201 lines
36 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

2026-04-23 00:01:34.751 INFO [tests.conftest] Fleet bootstrap inactive pass --fleet (or set FLEET_BOOTSTRAP=true) to connect local nodes to the waku.test fleet
2026-04-23 00:01:34.751 DEBUG [tests.conftest] Running fixture setup: test_id
2026-04-23 00:01:34.752 DEBUG [tests.conftest] Running test: test_relay_get_message_while_one_peer_is_paused with id: 2026-04-23_00-01-34__3ec56d7a-be5f-4d75-ab5a-8cb8f6c693f7
2026-04-23 00:01:34.752 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-04-23 00:01:34.752 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-04-23 00:01:34.752 DEBUG [src.steps.relay] Running fixture setup: setup_main_relay_nodes
2026-04-23 00:01:34.759 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-04-23 00:01:34.760 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-04-23_00-01-34__3ec56d7a-be5f-4d75-ab5a-8cb8f6c693f7__wakuorg_nwaku:latest.log
2026-04-23 00:01:34.760 DEBUG [src.node.waku_node] Starting Node...
2026-04-23 00:01:34.760 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-04-23 00:01:34.761 DEBUG [src.node.docker_mananger] Network waku already exists
2026-04-23 00:01:34.761 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.54.241
2026-04-23 00:01:34.762 DEBUG [src.node.docker_mananger] Generated ports ['11719', '11720', '11721', '11722', '11723']
2026-04-23 00:01:34.762 DEBUG [src.node.waku_node] RLN credentials were not set
2026-04-23 00:01:34.762 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-04-23 00:01:34.762 DEBUG [src.node.waku_node] Using volumes []
2026-04-23 00:01:34.762 DEBUG [src.node.docker_mananger] docker run -i -t -p 11719:11719 -p 11720:11720 -p 11721:11721 -p 11722:11722 -p 11723:11723 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=11721 --rest-port=11719 --tcp-port=11720 --discv5-udp-port=11722 --rest-address=0.0.0.0 --nat=extip:172.18.54.241 --peer-exchange=true --discv5-discovery=true --cluster-id=198 --nodekey=2ae0d53fcfe1d3aadb6f22af3dab80f25f6b0dbf98cc3952ddedeb51edada570 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=11723 --metrics-logging=true --relay=true
2026-04-23 00:01:34.955 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.54.241 waku 9c6cdc8d962848b6e521bb0857a6ca61dd5d1bd1ec6c2f739cf4790ef9f9b8e6
2026-04-23 00:01:34.989 DEBUG [src.node.docker_mananger] Container started with ID 9c6cdc8d9628. Setting up logs at ./log/docker/node1_2026-04-23_00-01-34__3ec56d7a-be5f-4d75-ab5a-8cb8f6c693f7__wakuorg_nwaku:latest.log
2026-04-23 00:01:34.990 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 11719
2026-04-23 00:01:34.990 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-04-23 00:01:35.081 ERROR [src.node.docker_mananger] Max retries reached for container c6ad32da2ca1. Exiting log stream.
2026-04-23 00:01:35.535 ERROR [src.node.docker_mananger] Max retries reached for container d7291079aa91. Exiting log stream.
2026-04-23 00:01:35.992 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:11719/health" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:35.995 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-04-23 00:01:35.995 INFO [src.node.waku_node] Node protocols are initialized !!
2026-04-23 00:01:35.996 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:11719/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:35.998 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.54.241/tcp/11720/p2p/16Uiu2HAmLPzqko6PTSHLfhWeERu56Xi6ueTvtRwmtoZqHGViXpVB","/ip4/172.18.54.241/tcp/11721/ws/p2p/16Uiu2HAmLPzqko6PTSHLfhWeERu56Xi6ueTvtRwmtoZqHGViXpVB"],"enrUri":"enr:-L24QPy49o0SoID5G4Y66f8YFm5E7T__K-8bDUHpa4aADXqZc5IiclG00oyVyk3wKi9fldRPFlWoSmN0-KUFEPGuKwECgmlkgnY0gmlwhKwSNvGKbXVsdGlhZGRyc5YACASsEjbxBi3IAAoErBI28QYtyd0DgnJzhQDGAQAAiXNlY3AyNTZrMaEDcwZWlV2uFgUZmAZhQXxg0rpwuuTbK3p31vM1XcanG36DdGNwgi3Ig3VkcIItyoV3YWt1MgE"}'
2026-04-23 00:01:35.998 INFO [src.node.waku_node] REST service is ready !!
2026-04-23 00:01:36.006 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-04-23 00:01:36.006 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-04-23_00-01-34__3ec56d7a-be5f-4d75-ab5a-8cb8f6c693f7__wakuorg_nwaku:latest.log
2026-04-23 00:01:36.006 DEBUG [src.node.waku_node] Starting Node...
2026-04-23 00:01:36.006 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-04-23 00:01:36.008 DEBUG [src.node.docker_mananger] Network waku already exists
2026-04-23 00:01:36.008 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.254.196
2026-04-23 00:01:36.008 DEBUG [src.node.docker_mananger] Generated ports ['38248', '38249', '38250', '38251', '38252']
2026-04-23 00:01:36.008 DEBUG [src.node.waku_node] RLN credentials were not set
2026-04-23 00:01:36.008 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-04-23 00:01:36.009 DEBUG [src.node.waku_node] Using volumes []
2026-04-23 00:01:36.009 DEBUG [src.node.docker_mananger] docker run -i -t -p 38248:38248 -p 38249:38249 -p 38250:38250 -p 38251:38251 -p 38252:38252 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=38250 --rest-port=38248 --tcp-port=38249 --discv5-udp-port=38251 --rest-address=0.0.0.0 --nat=extip:172.18.254.196 --peer-exchange=true --discv5-discovery=true --cluster-id=198 --nodekey=443ece1e8273ecca0dacdd244cbda98eefa2b52a0abc9e0183f9cca887c8ab16 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=38252 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QPy49o0SoID5G4Y66f8YFm5E7T__K-8bDUHpa4aADXqZc5IiclG00oyVyk3wKi9fldRPFlWoSmN0-KUFEPGuKwECgmlkgnY0gmlwhKwSNvGKbXVsdGlhZGRyc5YACASsEjbxBi3IAAoErBI28QYtyd0DgnJzhQDGAQAAiXNlY3AyNTZrMaEDcwZWlV2uFgUZmAZhQXxg0rpwuuTbK3p31vM1XcanG36DdGNwgi3Ig3VkcIItyoV3YWt1MgE
2026-04-23 00:01:36.193 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.254.196 waku b04c2f8774051fe84af9934972f6ef4fc21a91aa76c4bd05e484400d8ec51c31
2026-04-23 00:01:36.228 DEBUG [src.node.docker_mananger] Container started with ID b04c2f877405. Setting up logs at ./log/docker/node2_2026-04-23_00-01-34__3ec56d7a-be5f-4d75-ab5a-8cb8f6c693f7__wakuorg_nwaku:latest.log
2026-04-23 00:01:36.228 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 38248
2026-04-23 00:01:36.229 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-04-23 00:01:37.229 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:38248/health" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:37.235 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-04-23 00:01:37.236 INFO [src.node.waku_node] Node protocols are initialized !!
2026-04-23 00:01:37.237 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:38248/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:37.245 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.254.196/tcp/38249/p2p/16Uiu2HAkxC4aem3hyzUfsga6uo54eH3Bzgu9pCpcAg5oCHkTozJ7","/ip4/172.18.254.196/tcp/38250/ws/p2p/16Uiu2HAkxC4aem3hyzUfsga6uo54eH3Bzgu9pCpcAg5oCHkTozJ7"],"enrUri":"enr:-L24QA57hV5p1apEyirpYCUwDenvNeFk6E9MVMcloLbqjkHvSfT8O25I_psF4R_1xxAfO5R89crk6hrLv0HcUuRCO1gCgmlkgnY0gmlwhKwS_sSKbXVsdGlhZGRyc5YACASsEv7EBpVpAAoErBL-xAaVat0DgnJzhQDGAQAAiXNlY3AyNTZrMaECKRZ8B5GlBfjItJjgLW6vNFWQr0B_gOy1OSjZ_ohwOxSDdGNwgpVpg3VkcIKVa4V3YWt1MgE"}'
2026-04-23 00:01:37.246 INFO [src.node.waku_node] REST service is ready !!
2026-04-23 00:01:37.246 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:38248/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.54.241/tcp/11720/p2p/16Uiu2HAmLPzqko6PTSHLfhWeERu56Xi6ueTvtRwmtoZqHGViXpVB"]'
2026-04-23 00:01:37.251 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-04-23 00:01:37.252 DEBUG [src.steps.relay] Running fixture setup: setup_optional_relay_nodes
2026-04-23 00:01:37.262 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-04-23 00:01:37.262 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-04-23_00-01-34__3ec56d7a-be5f-4d75-ab5a-8cb8f6c693f7__wakuorg_nwaku:latest.log
2026-04-23 00:01:37.262 DEBUG [src.node.waku_node] Starting Node...
2026-04-23 00:01:37.262 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-04-23 00:01:37.264 DEBUG [src.node.docker_mananger] Network waku already exists
2026-04-23 00:01:37.264 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.121.110
2026-04-23 00:01:37.264 DEBUG [src.node.docker_mananger] Generated ports ['37215', '37216', '37217', '37218', '37219']
2026-04-23 00:01:37.264 DEBUG [src.node.waku_node] RLN credentials were not set
2026-04-23 00:01:37.264 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-04-23 00:01:37.264 DEBUG [src.node.waku_node] Using volumes []
2026-04-23 00:01:37.265 DEBUG [src.node.docker_mananger] docker run -i -t -p 37215:37215 -p 37216:37216 -p 37217:37217 -p 37218:37218 -p 37219:37219 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=37217 --rest-port=37215 --tcp-port=37216 --discv5-udp-port=37218 --rest-address=0.0.0.0 --nat=extip:172.18.121.110 --peer-exchange=true --discv5-discovery=true --cluster-id=198 --nodekey=79ede2bfcdcc7410f6b43cb1addafbca8cb3d2bfd011c0ea65de506274afa1c5 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=37219 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QPy49o0SoID5G4Y66f8YFm5E7T__K-8bDUHpa4aADXqZc5IiclG00oyVyk3wKi9fldRPFlWoSmN0-KUFEPGuKwECgmlkgnY0gmlwhKwSNvGKbXVsdGlhZGRyc5YACASsEjbxBi3IAAoErBI28QYtyd0DgnJzhQDGAQAAiXNlY3AyNTZrMaEDcwZWlV2uFgUZmAZhQXxg0rpwuuTbK3p31vM1XcanG36DdGNwgi3Ig3VkcIItyoV3YWt1MgE
2026-04-23 00:01:37.458 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.121.110 waku 92563117f0c79e6783f43783dd80691b6bf293bda51b3a16a686ab84306e8f94
2026-04-23 00:01:37.496 DEBUG [src.node.docker_mananger] Container started with ID 92563117f0c7. Setting up logs at ./log/docker/node3_2026-04-23_00-01-34__3ec56d7a-be5f-4d75-ab5a-8cb8f6c693f7__wakuorg_nwaku:latest.log
2026-04-23 00:01:37.496 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 37215
2026-04-23 00:01:37.496 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-04-23 00:01:38.498 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:37215/health" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:38.504 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-04-23 00:01:38.505 INFO [src.node.waku_node] Node protocols are initialized !!
2026-04-23 00:01:38.506 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:37215/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:38.509 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.121.110/tcp/37216/p2p/16Uiu2HAm7JEJQVjAWFwjZwW2kgYeAPxX6XZSEyDT4uBsxLo2448z","/ip4/172.18.121.110/tcp/37217/ws/p2p/16Uiu2HAm7JEJQVjAWFwjZwW2kgYeAPxX6XZSEyDT4uBsxLo2448z"],"enrUri":"enr:-L24QKpnLLON-KYfin47pODDD7by7SH_t92DJDMyQ1MmnR8uPMU2H5aDo_n9KcKFOnmaVf2xl4H2eU4GfbNfMt7Qic0CgmlkgnY0gmlwhKwSeW6KbXVsdGlhZGRyc5YACASsEnluBpFgAAoErBJ5bgaRYd0DgnJzhQDGAQAAiXNlY3AyNTZrMaECsGQkWnioqeuprEZjWBJ397fMND9Y2Xs0JMbUMezUfcODdGNwgpFgg3VkcIKRYoV3YWt1MgE"}'
2026-04-23 00:01:38.510 INFO [src.node.waku_node] REST service is ready !!
2026-04-23 00:01:38.510 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:37215/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.54.241/tcp/11720/p2p/16Uiu2HAmLPzqko6PTSHLfhWeERu56Xi6ueTvtRwmtoZqHGViXpVB"]'
2026-04-23 00:01:38.514 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-04-23 00:01:38.521 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-04-23 00:01:38.521 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-04-23_00-01-34__3ec56d7a-be5f-4d75-ab5a-8cb8f6c693f7__wakuorg_nwaku:latest.log
2026-04-23 00:01:38.521 DEBUG [src.node.waku_node] Starting Node...
2026-04-23 00:01:38.521 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-04-23 00:01:38.523 DEBUG [src.node.docker_mananger] Network waku already exists
2026-04-23 00:01:38.523 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.95.91
2026-04-23 00:01:38.523 DEBUG [src.node.docker_mananger] Generated ports ['11971', '11972', '11973', '11974', '11975']
2026-04-23 00:01:38.523 DEBUG [src.node.waku_node] RLN credentials were not set
2026-04-23 00:01:38.523 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-04-23 00:01:38.523 DEBUG [src.node.waku_node] Using volumes []
2026-04-23 00:01:38.523 DEBUG [src.node.docker_mananger] docker run -i -t -p 11971:11971 -p 11972:11972 -p 11973:11973 -p 11974:11974 -p 11975:11975 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=11973 --rest-port=11971 --tcp-port=11972 --discv5-udp-port=11974 --rest-address=0.0.0.0 --nat=extip:172.18.95.91 --peer-exchange=true --discv5-discovery=true --cluster-id=198 --nodekey=b7d6bc4dbb0cd95bdd7cc26aa92a0c68ee78b7cf138aa6e8effdae728ef2da56 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=11975 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QPy49o0SoID5G4Y66f8YFm5E7T__K-8bDUHpa4aADXqZc5IiclG00oyVyk3wKi9fldRPFlWoSmN0-KUFEPGuKwECgmlkgnY0gmlwhKwSNvGKbXVsdGlhZGRyc5YACASsEjbxBi3IAAoErBI28QYtyd0DgnJzhQDGAQAAiXNlY3AyNTZrMaEDcwZWlV2uFgUZmAZhQXxg0rpwuuTbK3p31vM1XcanG36DdGNwgi3Ig3VkcIItyoV3YWt1MgE
2026-04-23 00:01:38.720 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.95.91 waku 6ddec4a55a17bec9b32c5441f8483468e4872cab4558f1518d5868250e7cf542
2026-04-23 00:01:38.757 DEBUG [src.node.docker_mananger] Container started with ID 6ddec4a55a17. Setting up logs at ./log/docker/node4_2026-04-23_00-01-34__3ec56d7a-be5f-4d75-ab5a-8cb8f6c693f7__wakuorg_nwaku:latest.log
2026-04-23 00:01:38.758 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 11971
2026-04-23 00:01:38.758 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-04-23 00:01:39.760 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:11971/health" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:39.769 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-04-23 00:01:39.770 INFO [src.node.waku_node] Node protocols are initialized !!
2026-04-23 00:01:39.770 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:11971/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:39.774 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.95.91/tcp/11972/p2p/16Uiu2HAkwZLy7Ys2msn3vD2wdcxXWDNRpgN9fHDst2iB4HmrhncK","/ip4/172.18.95.91/tcp/11973/ws/p2p/16Uiu2HAkwZLy7Ys2msn3vD2wdcxXWDNRpgN9fHDst2iB4HmrhncK"],"enrUri":"enr:-L24QOFegbJqLywrN5skz1iZ9mu9SYjaMZJ5LDJTGR6SeCRJP_c-NoKnXe-jF6u1YPr-18vTzOzWez3MKZLu9-0CBTUCgmlkgnY0gmlwhKwSX1uKbXVsdGlhZGRyc5YACASsEl9bBi7EAAoErBJfWwYuxd0DgnJzhQDGAQAAiXNlY3AyNTZrMaECH66GEYCbPcwqXIWS1kGCH4MsOqrAXjGgUDGJ3pN0OKSDdGNwgi7Eg3VkcIIuxoV3YWt1MgE"}'
2026-04-23 00:01:39.774 INFO [src.node.waku_node] REST service is ready !!
2026-04-23 00:01:39.775 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:11971/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.54.241/tcp/11720/p2p/16Uiu2HAmLPzqko6PTSHLfhWeERu56Xi6ueTvtRwmtoZqHGViXpVB"]'
2026-04-23 00:01:39.778 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-04-23 00:01:39.785 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-04-23 00:01:39.785 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-04-23_00-01-34__3ec56d7a-be5f-4d75-ab5a-8cb8f6c693f7__wakuorg_nwaku:latest.log
2026-04-23 00:01:39.785 DEBUG [src.node.waku_node] Starting Node...
2026-04-23 00:01:39.785 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-04-23 00:01:39.786 DEBUG [src.node.docker_mananger] Network waku already exists
2026-04-23 00:01:39.787 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.141.155
2026-04-23 00:01:39.787 DEBUG [src.node.docker_mananger] Generated ports ['28874', '28875', '28876', '28877', '28878']
2026-04-23 00:01:39.787 DEBUG [src.node.waku_node] RLN credentials were not set
2026-04-23 00:01:39.787 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-04-23 00:01:39.787 DEBUG [src.node.waku_node] Using volumes []
2026-04-23 00:01:39.787 DEBUG [src.node.docker_mananger] docker run -i -t -p 28874:28874 -p 28875:28875 -p 28876:28876 -p 28877:28877 -p 28878:28878 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=28876 --rest-port=28874 --tcp-port=28875 --discv5-udp-port=28877 --rest-address=0.0.0.0 --nat=extip:172.18.141.155 --peer-exchange=true --discv5-discovery=true --cluster-id=198 --nodekey=cc3deacce8b0f4659836d9f4bde5a1b99ecde65a4dbfe54c9719f54cfae21ea8 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=28878 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QPy49o0SoID5G4Y66f8YFm5E7T__K-8bDUHpa4aADXqZc5IiclG00oyVyk3wKi9fldRPFlWoSmN0-KUFEPGuKwECgmlkgnY0gmlwhKwSNvGKbXVsdGlhZGRyc5YACASsEjbxBi3IAAoErBI28QYtyd0DgnJzhQDGAQAAiXNlY3AyNTZrMaEDcwZWlV2uFgUZmAZhQXxg0rpwuuTbK3p31vM1XcanG36DdGNwgi3Ig3VkcIItyoV3YWt1MgE
2026-04-23 00:01:39.974 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.141.155 waku 5e4963ff45037adc077d15c5756bf9413d29ba2624c777021d21f8fedff4ca4a
2026-04-23 00:01:40.006 DEBUG [src.node.docker_mananger] Container started with ID 5e4963ff4503. Setting up logs at ./log/docker/node5_2026-04-23_00-01-34__3ec56d7a-be5f-4d75-ab5a-8cb8f6c693f7__wakuorg_nwaku:latest.log
2026-04-23 00:01:40.007 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 28874
2026-04-23 00:01:40.007 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-04-23 00:01:41.008 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28874/health" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:41.017 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-04-23 00:01:41.018 INFO [src.node.waku_node] Node protocols are initialized !!
2026-04-23 00:01:41.018 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28874/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:41.021 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.141.155/tcp/28875/p2p/16Uiu2HAmHRrup7on4j1D4tvXMMkNNMkHVR6zyMEqSYs2eNjNo8Ho","/ip4/172.18.141.155/tcp/28876/ws/p2p/16Uiu2HAmHRrup7on4j1D4tvXMMkNNMkHVR6zyMEqSYs2eNjNo8Ho"],"enrUri":"enr:-L24QG0TNRVHRqEWEtEjxaZ4rTyUEPtNyJwmoF5NTl4Q0H0YCzsheSQS57x4s2fILC25qNsyGiNFb_z0RMLSK2npKS8CgmlkgnY0gmlwhKwSjZuKbXVsdGlhZGRyc5YACASsEo2bBnDLAAoErBKNmwZwzN0DgnJzhQDGAQAAiXNlY3AyNTZrMaEDRu1490jUhEjQ9_mzajJkcCkx4Ht_0vlg-NgB9xnWckqDdGNwgnDLg3VkcIJwzYV3YWt1MgE"}'
2026-04-23 00:01:41.022 INFO [src.node.waku_node] REST service is ready !!
2026-04-23 00:01:41.022 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28874/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.54.241/tcp/11720/p2p/16Uiu2HAmLPzqko6PTSHLfhWeERu56Xi6ueTvtRwmtoZqHGViXpVB"]'
2026-04-23 00:01:41.026 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-04-23 00:01:41.026 DEBUG [src.steps.relay] Running fixture setup: subscribe_main_relay_nodes
2026-04-23 00:01:41.027 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:11719/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/198/1"]'
2026-04-23 00:01:41.042 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-04-23 00:01:41.042 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:38248/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/198/1"]'
2026-04-23 00:01:41.051 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-04-23 00:01:41.051 DEBUG [src.steps.relay] Running fixture setup: subscribe_optional_relay_nodes
2026-04-23 00:01:41.052 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:37215/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/198/1"]'
2026-04-23 00:01:41.058 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-04-23 00:01:41.058 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:11971/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/198/1"]'
2026-04-23 00:01:41.064 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-04-23 00:01:41.066 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28874/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/198/1"]'
2026-04-23 00:01:41.070 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-04-23 00:01:41.072 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:11719/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-04-23 00:01:41.085 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-04-23 00:01:41.085 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-04-23 00:01:41.185 DEBUG [src.steps.relay] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-04-23 00:01:41.185 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:11719/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:41.188 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1776902501071878912,"ephemeral":false,"proof":""}]'
2026-04-23 00:01:41.190 DEBUG [src.steps.relay] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-04-23 00:01:41.190 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:38248/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:41.192 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1776902501071878912,"ephemeral":false,"proof":""}]'
2026-04-23 00:01:41.194 DEBUG [src.steps.relay] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-04-23 00:01:41.194 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:37215/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:41.196 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1776902501071878912,"ephemeral":false,"proof":""}]'
2026-04-23 00:01:41.198 DEBUG [src.steps.relay] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-04-23 00:01:41.198 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:11971/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:41.200 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1776902501071878912,"ephemeral":false,"proof":""}]'
2026-04-23 00:01:41.201 DEBUG [src.steps.relay] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-04-23 00:01:41.201 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28874/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:41.204 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1776902501071878912,"ephemeral":false,"proof":""}]'
2026-04-23 00:01:41.205 INFO [src.steps.relay] WARM UP successful!!
2026-04-23 00:01:41.206 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:11719/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-04-23 00:01:41.217 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-04-23 00:01:41.218 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-04-23 00:01:41.318 DEBUG [src.steps.relay] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-04-23 00:01:41.318 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:11719/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:41.321 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1776902501206698605,"ephemeral":false,"proof":""}]'
2026-04-23 00:01:41.322 DEBUG [src.steps.relay] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-04-23 00:01:41.323 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:38248/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:41.325 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1776902501206698605,"ephemeral":false,"proof":""}]'
2026-04-23 00:01:41.327 DEBUG [src.steps.relay] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-04-23 00:01:41.327 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:37215/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:41.329 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1776902501206698605,"ephemeral":false,"proof":""}]'
2026-04-23 00:01:41.331 DEBUG [src.steps.relay] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-04-23 00:01:41.331 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:11971/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:41.333 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1776902501206698605,"ephemeral":false,"proof":""}]'
2026-04-23 00:01:41.335 DEBUG [src.steps.relay] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-04-23 00:01:41.335 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28874/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:41.338 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1776902501206698605,"ephemeral":false,"proof":""}]'
2026-04-23 00:01:41.339 DEBUG [src.node.waku_node] Pausing container with id b04c2f877405
2026-04-23 00:01:41.350 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:11719/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-04-23 00:01:41.360 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-04-23 00:01:41.362 DEBUG [src.node.waku_node] Unpause container with id b04c2f877405
2026-04-23 00:01:41.374 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:11719/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-04-23 00:01:41.387 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-04-23 00:01:41.388 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:38248/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-04-23 00:01:41.392 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1776902501339674043,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1776902501339751398,"ephemeral":false,"proof":""}]'
2026-04-23 00:01:41.395 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-04-23 00:01:41.397 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-04-23 00:01:41.397 DEBUG [src.node.waku_node] Stopping container with id 9c6cdc8d9628
2026-04-23 00:01:41.907 DEBUG [src.node.waku_node] Container stopped.
2026-04-23 00:01:41.907 DEBUG [src.node.waku_node] Stopping container with id b04c2f877405
2026-04-23 00:01:42.375 DEBUG [src.node.waku_node] Container stopped.
2026-04-23 00:01:42.376 DEBUG [src.node.waku_node] Stopping container with id 92563117f0c7
2026-04-23 00:01:42.721 ERROR [src.node.docker_mananger] Max retries reached for container 9c6cdc8d9628. Exiting log stream.
2026-04-23 00:01:42.854 DEBUG [src.node.waku_node] Container stopped.
2026-04-23 00:01:42.855 DEBUG [src.node.waku_node] Stopping container with id 6ddec4a55a17
2026-04-23 00:01:43.216 ERROR [src.node.docker_mananger] Max retries reached for container b04c2f877405. Exiting log stream.
2026-04-23 00:01:43.307 DEBUG [src.node.waku_node] Container stopped.
2026-04-23 00:01:43.308 DEBUG [src.node.waku_node] Stopping container with id 5e4963ff4503
2026-04-23 00:01:43.669 ERROR [src.node.docker_mananger] Max retries reached for container 92563117f0c7. Exiting log stream.
2026-04-23 00:01:43.731 DEBUG [src.node.waku_node] Container stopped.
2026-04-23 00:01:43.734 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-04-23 00:01:43.756 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-04-23 00:01:43.773 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-04-23 00:01:43.783 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-04-23 00:01:43.791 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-04-23 00:01:43.800 DEBUG [src.node.docker_mananger] No errors found in the waku logs.