162 lines
30 KiB
Plaintext

2026-03-12 04:32:44.992 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-12 04:32:44.993 DEBUG [tests.conftest] Running test: test_store_no_peer_selected with id: 2026-03-12_04-32-44__29ce621c-3287-47ac-95ad-6dbe5a7d1792
2026-03-12 04:32:44.993 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-12 04:32:44.993 DEBUG [src.steps.filter] Running fixture setup: filter_setup
2026-03-12 04:32:44.994 DEBUG [src.steps.light_push] Running fixture setup: light_push_setup
2026-03-12 04:32:45.004 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-12 04:32:45.004 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-12_04-32-44__29ce621c-3287-47ac-95ad-6dbe5a7d1792__wakuorg_nwaku:latest.log
2026-03-12 04:32:45.011 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-12 04:32:45.011 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-12_04-32-44__29ce621c-3287-47ac-95ad-6dbe5a7d1792__wakuorg_nwaku:latest.log
2026-03-12 04:32:45.018 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-12 04:32:45.018 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-12_04-32-44__29ce621c-3287-47ac-95ad-6dbe5a7d1792__wakuorg_nwaku:latest.log
2026-03-12 04:32:45.018 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-12 04:32:45.019 DEBUG [src.steps.store] Running fixture setup: store_setup
2026-03-12 04:32:45.019 DEBUG [tests.e2e.test_e2e] Start 5 nodes
2026-03-12 04:32:45.026 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-12 04:32:45.026 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-12_04-32-44__29ce621c-3287-47ac-95ad-6dbe5a7d1792__wakuorg_nwaku:latest.log
2026-03-12 04:32:45.033 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-12 04:32:45.033 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-12_04-32-44__29ce621c-3287-47ac-95ad-6dbe5a7d1792__wakuorg_nwaku:latest.log
2026-03-12 04:32:45.039 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-12 04:32:45.039 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-12_04-32-44__29ce621c-3287-47ac-95ad-6dbe5a7d1792__wakuorg_nwaku:latest.log
2026-03-12 04:32:45.039 DEBUG [src.node.waku_node] Starting Node...
2026-03-12 04:32:45.040 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-12 04:32:45.086 DEBUG [src.node.docker_mananger] Network waku created
2026-03-12 04:32:45.086 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.176.70
2026-03-12 04:32:45.086 DEBUG [src.node.docker_mananger] Generated ports ['60058', '60059', '60060', '60061', '60062']
2026-03-12 04:32:45.086 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-12 04:32:45.087 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-12 04:32:45.087 DEBUG [src.node.waku_node] Using volumes []
2026-03-12 04:32:45.087 DEBUG [src.node.docker_mananger] docker run -i -t -p 60058:60058 -p 60059:60059 -p 60060:60060 -p 60061:60061 -p 60062:60062 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=60060 --rest-port=60058 --tcp-port=60059 --discv5-udp-port=60061 --rest-address=0.0.0.0 --nat=extip:172.18.176.70 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=52f0ea3df2c2ee00da7e0fbe165ddb40b8121c5004b6f87abcbeafc8c4b6f9ed --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=60062 --metrics-logging=true --relay=true --store=true
2026-03-12 04:32:48.658 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.176.70 waku ddccaeef881f54f06f95df755edea26c04eaf549e9fd7d3c45f50bfd0400054a
2026-03-12 04:32:48.694 DEBUG [src.node.docker_mananger] Container started with ID ddccaeef881f. Setting up logs at ./log/docker/node1_2026-03-12_04-32-44__29ce621c-3287-47ac-95ad-6dbe5a7d1792__wakuorg_nwaku:latest.log
2026-03-12 04:32:48.694 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 60058
2026-03-12 04:32:48.695 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-12 04:32:49.695 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:60058/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:32:49.698 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-12 04:32:49.699 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-12 04:32:49.699 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:60058/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:32:49.702 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.176.70/tcp/60059/p2p/16Uiu2HAkybbmDwH5RQeHenZRdpKcUtg7cextzaBZRJi2GUN1Zqub","/ip4/172.18.176.70/tcp/60060/ws/p2p/16Uiu2HAkybbmDwH5RQeHenZRdpKcUtg7cextzaBZRJi2GUN1Zqub"],"enrUri":"enr:-L24QAtWLmrXZVjhWV3F9XLjdD0DymCs7_BR8xC6RiWUbO3ZWzhxMfcSqIHYwnkxFoS6GmSigtVcEYTX0Rkq2FoCoL8CgmlkgnY0gmlwhKwSsEaKbXVsdGlhZGRyc5YACASsErBGBuqbAAoErBKwRgbqnN0DgnJzhQADAQAAiXNlY3AyNTZrMaECPfnJtkRVxEgYhSNoaBv7OQHZ2VWlzimh6NZN3DThAwqDdGNwguqbg3VkcILqnYV3YWt1MgM"}'
2026-03-12 04:32:49.702 INFO [src.node.waku_node] REST service is ready !!
2026-03-12 04:32:49.702 DEBUG [src.node.waku_node] Starting Node...
2026-03-12 04:32:49.702 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-12 04:32:49.704 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-12 04:32:49.704 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.176.40
2026-03-12 04:32:49.704 DEBUG [src.node.docker_mananger] Generated ports ['23642', '23643', '23644', '23645', '23646']
2026-03-12 04:32:49.704 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-12 04:32:49.704 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-12 04:32:49.704 DEBUG [src.node.waku_node] Using volumes []
2026-03-12 04:32:49.705 DEBUG [src.node.docker_mananger] docker run -i -t -p 23642:23642 -p 23643:23643 -p 23644:23644 -p 23645:23645 -p 23646:23646 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=23644 --rest-port=23642 --tcp-port=23643 --discv5-udp-port=23645 --rest-address=0.0.0.0 --nat=extip:172.18.176.40 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=5dfe1dbfa75c1bd04c3debefeecbeaaeccfdae54e6fbff3b2fe9f6fb8adaaed1 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=23646 --metrics-logging=true --store=false --relay=false --discv5-bootstrap-node=enr:-L24QAtWLmrXZVjhWV3F9XLjdD0DymCs7_BR8xC6RiWUbO3ZWzhxMfcSqIHYwnkxFoS6GmSigtVcEYTX0Rkq2FoCoL8CgmlkgnY0gmlwhKwSsEaKbXVsdGlhZGRyc5YACASsErBGBuqbAAoErBKwRgbqnN0DgnJzhQADAQAAiXNlY3AyNTZrMaECPfnJtkRVxEgYhSNoaBv7OQHZ2VWlzimh6NZN3DThAwqDdGNwguqbg3VkcILqnYV3YWt1MgM
2026-03-12 04:32:49.915 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.176.40 waku 268abb5e7e4f55495c4127193937896ce27e83cceaf8d859e1807898890725c8
2026-03-12 04:32:49.951 DEBUG [src.node.docker_mananger] Container started with ID 268abb5e7e4f. Setting up logs at ./log/docker/node2_2026-03-12_04-32-44__29ce621c-3287-47ac-95ad-6dbe5a7d1792__wakuorg_nwaku:latest.log
2026-03-12 04:32:49.951 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 23642
2026-03-12 04:32:49.952 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-12 04:32:50.953 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23642/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:32:50.956 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-12 04:32:50.956 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-12 04:32:50.956 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23642/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:32:50.959 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.176.40/tcp/23643/p2p/16Uiu2HAmLiDWuEAdDACQJGpEg5WQEc3RMGWMAmEeED7WkUFCczrp","/ip4/172.18.176.40/tcp/23644/ws/p2p/16Uiu2HAmLiDWuEAdDACQJGpEg5WQEc3RMGWMAmEeED7WkUFCczrp"],"enrUri":"enr:-L24QGBeAsn0swcs33GS0LfjfI5VwY9zZsdu_gY2Z7XvY2CURlQBwI4VfeuW3XwiwV31imNGmd5HpHKTyCcOLeCR-2ICgmlkgnY0gmlwhKwSsCiKbXVsdGlhZGRyc5YACASsErAoBlxbAAoErBKwKAZcXN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDd7Ef1fBvnrq1FdOP9TdiTrcW-vjuZdyaOz1pJxU793WDdGNwglxbg3VkcIJcXYV3YWt1MgA"}'
2026-03-12 04:32:50.959 INFO [src.node.waku_node] REST service is ready !!
2026-03-12 04:32:50.960 DEBUG [src.node.waku_node] Starting Node...
2026-03-12 04:32:50.960 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-12 04:32:50.961 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-12 04:32:50.961 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.219.5
2026-03-12 04:32:50.961 DEBUG [src.node.docker_mananger] Generated ports ['2840', '2841', '2842', '2843', '2844']
2026-03-12 04:32:50.962 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-12 04:32:50.962 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-12 04:32:50.962 DEBUG [src.node.waku_node] Using volumes []
2026-03-12 04:32:50.962 DEBUG [src.node.docker_mananger] docker run -i -t -p 2840:2840 -p 2841:2841 -p 2842:2842 -p 2843:2843 -p 2844:2844 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=2842 --rest-port=2840 --tcp-port=2841 --discv5-udp-port=2843 --rest-address=0.0.0.0 --nat=extip:172.18.219.5 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=72c9c45fbf64c9ec4ebba858bafbafc4fd9a335250ca0aa5e5dd9b53ab22cc6d --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=2844 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QGBeAsn0swcs33GS0LfjfI5VwY9zZsdu_gY2Z7XvY2CURlQBwI4VfeuW3XwiwV31imNGmd5HpHKTyCcOLeCR-2ICgmlkgnY0gmlwhKwSsCiKbXVsdGlhZGRyc5YACASsErAoBlxbAAoErBKwKAZcXN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDd7Ef1fBvnrq1FdOP9TdiTrcW-vjuZdyaOz1pJxU793WDdGNwglxbg3VkcIJcXYV3YWt1MgA
2026-03-12 04:32:51.169 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.219.5 waku bee725d1899f7d63d097f6cb3ae967546fdcc9c91b28d20a7819eb955371505d
2026-03-12 04:32:51.203 DEBUG [src.node.docker_mananger] Container started with ID bee725d1899f. Setting up logs at ./log/docker/node3_2026-03-12_04-32-44__29ce621c-3287-47ac-95ad-6dbe5a7d1792__wakuorg_nwaku:latest.log
2026-03-12 04:32:51.203 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 2840
2026-03-12 04:32:51.204 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-12 04:32:52.205 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2840/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:32:52.208 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-12 04:32:52.208 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-12 04:32:52.208 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2840/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:32:52.211 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.219.5/tcp/2841/p2p/16Uiu2HAkxXCeENQ1hPJtrj9PT3Yy163V373BRrrLvHqNHVceumL8","/ip4/172.18.219.5/tcp/2842/ws/p2p/16Uiu2HAkxXCeENQ1hPJtrj9PT3Yy163V373BRrrLvHqNHVceumL8"],"enrUri":"enr:-L24QOumfFYh-CVU2D8ynsNa7jK0knuqsMsK1kgrDSl2ep4LTZLvc55GWXkuGFNzuklmlvapU_xIr118BY1Fl8T-618CgmlkgnY0gmlwhKwS2wWKbXVsdGlhZGRyc5YACASsEtsFBgsZAAoErBLbBQYLGt0DgnJzhQADAQAAiXNlY3AyNTZrMaECLf2ioUPGRnixdPrLX_2E1rdn_v1e7lbZywfPzE5y8tWDdGNwggsZg3VkcIILG4V3YWt1MgA"}'
2026-03-12 04:32:52.211 INFO [src.node.waku_node] REST service is ready !!
2026-03-12 04:32:52.211 DEBUG [src.node.waku_node] Starting Node...
2026-03-12 04:32:52.211 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-12 04:32:52.213 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-12 04:32:52.213 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.31.213
2026-03-12 04:32:52.213 DEBUG [src.node.docker_mananger] Generated ports ['22817', '22818', '22819', '22820', '22821']
2026-03-12 04:32:52.213 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-12 04:32:52.213 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-12 04:32:52.213 DEBUG [src.node.waku_node] Using volumes []
2026-03-12 04:32:52.214 DEBUG [src.node.docker_mananger] docker run -i -t -p 22817:22817 -p 22818:22818 -p 22819:22819 -p 22820:22820 -p 22821:22821 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=22819 --rest-port=22817 --tcp-port=22818 --discv5-udp-port=22820 --rest-address=0.0.0.0 --nat=extip:172.18.31.213 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=65cb9e8f4c5f9e88b97abaccef46adff1f0eb50b37ffef78ca5ac50ea1fae3fe --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=22821 --metrics-logging=true --relay=true --store=false --discv5-bootstrap-node=enr:-L24QOumfFYh-CVU2D8ynsNa7jK0knuqsMsK1kgrDSl2ep4LTZLvc55GWXkuGFNzuklmlvapU_xIr118BY1Fl8T-618CgmlkgnY0gmlwhKwS2wWKbXVsdGlhZGRyc5YACASsEtsFBgsZAAoErBLbBQYLGt0DgnJzhQADAQAAiXNlY3AyNTZrMaECLf2ioUPGRnixdPrLX_2E1rdn_v1e7lbZywfPzE5y8tWDdGNwggsZg3VkcIILG4V3YWt1MgA
2026-03-12 04:32:52.419 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.31.213 waku ba4185fd6703cca49bd03fc6a582e230f8b6abab60539e1236f28b7b355b1400
2026-03-12 04:32:52.455 DEBUG [src.node.docker_mananger] Container started with ID ba4185fd6703. Setting up logs at ./log/docker/node3_2026-03-12_04-32-44__29ce621c-3287-47ac-95ad-6dbe5a7d1792__wakuorg_nwaku:latest.log
2026-03-12 04:32:52.455 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 22817
2026-03-12 04:32:52.455 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-12 04:32:53.456 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22817/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:32:53.477 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-12 04:32:53.479 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-12 04:32:53.479 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22817/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:32:53.484 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.31.213/tcp/22818/p2p/16Uiu2HAmPfWgi4MH2kUbEY27jUc6Vw7nZTEHWGL3DE3AmAdqPG3E","/ip4/172.18.31.213/tcp/22819/ws/p2p/16Uiu2HAmPfWgi4MH2kUbEY27jUc6Vw7nZTEHWGL3DE3AmAdqPG3E"],"enrUri":"enr:-L24QHolmbT3N5yCuIo2J2_GvRblMuuq0wSEl0nHVee4_RGLLxk_c2cVnlz9pH4leFMmU8ob3-i0jRx2bjkU3H6vk0ACgmlkgnY0gmlwhKwSH9WKbXVsdGlhZGRyc5YACASsEh_VBlkiAAoErBIf1QZZI90DgnJzhQADAQAAiXNlY3AyNTZrMaEDo5LbGgkSnP9vss7DYlEqWq2FJ2HHkKVfVIUeEKL9-c2DdGNwglkig3VkcIJZJIV3YWt1MgE"}'
2026-03-12 04:32:53.486 INFO [src.node.waku_node] REST service is ready !!
2026-03-12 04:32:53.486 DEBUG [src.node.waku_node] Starting Node...
2026-03-12 04:32:53.486 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-12 04:32:53.491 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-12 04:32:53.491 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.208.160
2026-03-12 04:32:53.491 DEBUG [src.node.docker_mananger] Generated ports ['26835', '26836', '26837', '26838', '26839']
2026-03-12 04:32:53.495 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-12 04:32:53.495 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-12 04:32:53.496 DEBUG [src.node.waku_node] Using volumes []
2026-03-12 04:32:53.496 DEBUG [src.node.docker_mananger] docker run -i -t -p 26835:26835 -p 26836:26836 -p 26837:26837 -p 26838:26838 -p 26839:26839 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=26837 --rest-port=26835 --tcp-port=26836 --discv5-udp-port=26838 --rest-address=0.0.0.0 --nat=extip:172.18.208.160 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=db8187d2664e1d75130dbbbbc8ebba183c317d7a1abd4f0a6cb6a4d8aca2fdcb --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=26839 --metrics-logging=true --relay=false --store=false --discv5-bootstrap-node=enr:-L24QHolmbT3N5yCuIo2J2_GvRblMuuq0wSEl0nHVee4_RGLLxk_c2cVnlz9pH4leFMmU8ob3-i0jRx2bjkU3H6vk0ACgmlkgnY0gmlwhKwSH9WKbXVsdGlhZGRyc5YACASsEh_VBlkiAAoErBIf1QZZI90DgnJzhQADAQAAiXNlY3AyNTZrMaEDo5LbGgkSnP9vss7DYlEqWq2FJ2HHkKVfVIUeEKL9-c2DdGNwglkig3VkcIJZJIV3YWt1MgE
2026-03-12 04:32:53.709 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.208.160 waku 88361fb9cdf331e55ef6752fb688e1351bcdc63cdf4f04403911c664213ca6ca
2026-03-12 04:32:53.748 DEBUG [src.node.docker_mananger] Container started with ID 88361fb9cdf3. Setting up logs at ./log/docker/node4_2026-03-12_04-32-44__29ce621c-3287-47ac-95ad-6dbe5a7d1792__wakuorg_nwaku:latest.log
2026-03-12 04:32:53.748 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 26835
2026-03-12 04:32:53.749 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-12 04:32:54.750 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26835/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:32:54.753 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-12 04:32:54.753 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-12 04:32:54.753 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26835/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:32:54.756 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.208.160/tcp/26836/p2p/16Uiu2HAmCzr8353zrNjTq8MCWimeWZubsTzZ8LoiSqXXWGMCAKPM","/ip4/172.18.208.160/tcp/26837/ws/p2p/16Uiu2HAmCzr8353zrNjTq8MCWimeWZubsTzZ8LoiSqXXWGMCAKPM"],"enrUri":"enr:-L24QP9G9R_Ru_JGjO2q9UouZuUC95UFCelvFkSmA4d52RoGPIg9232wQpfj4_iyl7k2WQT-tZZszcZ3fsJKIfnF7uYCgmlkgnY0gmlwhKwS0KCKbXVsdGlhZGRyc5YACASsEtCgBmjUAAoErBLQoAZo1d0DgnJzhQADAQAAiXNlY3AyNTZrMaEDBRZP6wENWBiGsTC4hatfEQ5wHuZ6DsGIz8LwPdzATXCDdGNwgmjUg3VkcIJo1oV3YWt1MgA"}'
2026-03-12 04:32:54.756 INFO [src.node.waku_node] REST service is ready !!
2026-03-12 04:32:54.756 DEBUG [tests.e2e.test_e2e] Add 3 peer nodes to node3
2026-03-12 04:32:54.756 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2840/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.176.70/tcp/60059/p2p/16Uiu2HAkybbmDwH5RQeHenZRdpKcUtg7cextzaBZRJi2GUN1Zqub"]'
2026-03-12 04:32:54.777 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-12 04:32:54.780 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2840/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.176.40/tcp/23643/p2p/16Uiu2HAmLiDWuEAdDACQJGpEg5WQEc3RMGWMAmEeED7WkUFCczrp"]'
2026-03-12 04:32:54.807 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-12 04:32:54.809 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2840/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.31.213/tcp/22818/p2p/16Uiu2HAmPfWgi4MH2kUbEY27jUc6Vw7nZTEHWGL3DE3AmAdqPG3E"]'
2026-03-12 04:32:54.822 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-12 04:32:54.826 DEBUG [tests.e2e.test_e2e] Subscribe nodes 1,2 to relay on pubsubtopic /waku/2/rs/3/1
2026-03-12 04:32:54.827 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22817/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-12 04:32:54.836 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-12 04:32:54.837 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:60058/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-12 04:32:54.840 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-12 04:32:54.841 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:60058/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:32:54.844 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.219.5/tcp/39234/p2p/16Uiu2HAkxXCeENQ1hPJtrj9PT3Yy163V373BRrrLvHqNHVceumL8","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"}]'
2026-03-12 04:32:54.844 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22817/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:32:54.846 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.219.5/tcp/2841/p2p/16Uiu2HAkxXCeENQ1hPJtrj9PT3Yy163V373BRrrLvHqNHVceumL8","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-12 04:32:54.846 DEBUG [src.libs.common] Sleeping for 30 seconds
2026-03-12 04:33:24.847 DEBUG [tests.e2e.test_e2e] Node1 publish message
2026-03-12 04:33:24.848 DEBUG [src.steps.store] Relaying message
2026-03-12 04:33:24.848 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22817/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": '$(date +%s%N)'}'
2026-03-12 04:33:24.853 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-12 04:33:24.854 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-12 04:33:25.055 DEBUG [tests.e2e.test_e2e] Check if node3 can inquiry stored message without stor peer specified
2026-03-12 04:33:25.055 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2840/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F1&contentTopics=%2Ftest%2F1%2Fwaku-filter%2Fproto&pageSize=5&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:33:25.064 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"ddfe7b7255b49c158044","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0xd02b28fc10bba41ec209db95c62abefd2a001c608f2e43ffad104f204fb9f4f3"}]}'
2026-03-12 04:33:25.067 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-12 04:33:25.067 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-12 04:33:25.068 DEBUG [src.node.waku_node] Stopping container with id ddccaeef881f
2026-03-12 04:33:25.636 DEBUG [src.node.waku_node] Container stopped.
2026-03-12 04:33:25.638 DEBUG [src.node.waku_node] Stopping container with id 268abb5e7e4f
2026-03-12 04:33:26.185 DEBUG [src.node.waku_node] Container stopped.
2026-03-12 04:33:26.186 DEBUG [src.node.waku_node] Stopping container with id bee725d1899f
2026-03-12 04:33:26.536 ERROR [src.node.docker_mananger] Max retries reached for container ddccaeef881f. Exiting log stream.
2026-03-12 04:33:26.761 DEBUG [src.node.waku_node] Container stopped.
2026-03-12 04:33:26.762 DEBUG [src.node.waku_node] Stopping container with id ba4185fd6703
2026-03-12 04:33:27.000 ERROR [src.node.docker_mananger] Max retries reached for container 268abb5e7e4f. Exiting log stream.
2026-03-12 04:33:27.334 DEBUG [src.node.waku_node] Container stopped.
2026-03-12 04:33:27.334 DEBUG [src.node.waku_node] Stopping container with id 88361fb9cdf3
2026-03-12 04:33:27.576 ERROR [src.node.docker_mananger] Max retries reached for container bee725d1899f. Exiting log stream.
2026-03-12 04:33:27.912 DEBUG [src.node.waku_node] Container stopped.
2026-03-12 04:33:27.914 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-12 04:33:27.956 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-12 04:33:27.967 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-12 04:33:27.990 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-12 04:33:28.013 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-12 04:33:28.015 DEBUG [src.node.docker_mananger] No errors found in the waku logs.