162 lines
30 KiB
Plaintext

2026-03-17 04:35:05.203 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-17 04:35:05.204 DEBUG [tests.conftest] Running test: test_store_no_peer_selected with id: 2026-03-17_04-35-05__27ab35e2-dd6d-42a1-a9bb-ffce34d478ce
2026-03-17 04:35:05.204 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-17 04:35:05.204 DEBUG [src.steps.filter] Running fixture setup: filter_setup
2026-03-17 04:35:05.204 DEBUG [src.steps.light_push] Running fixture setup: light_push_setup
2026-03-17 04:35:05.214 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-17 04:35:05.215 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-17_04-35-05__27ab35e2-dd6d-42a1-a9bb-ffce34d478ce__wakuorg_nwaku:latest.log
2026-03-17 04:35:05.221 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-17 04:35:05.221 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-17_04-35-05__27ab35e2-dd6d-42a1-a9bb-ffce34d478ce__wakuorg_nwaku:latest.log
2026-03-17 04:35:05.227 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-17 04:35:05.228 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-17_04-35-05__27ab35e2-dd6d-42a1-a9bb-ffce34d478ce__wakuorg_nwaku:latest.log
2026-03-17 04:35:05.228 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-17 04:35:05.228 DEBUG [src.steps.store] Running fixture setup: store_setup
2026-03-17 04:35:05.229 DEBUG [tests.e2e.test_e2e] Start 5 nodes
2026-03-17 04:35:05.235 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-17 04:35:05.235 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-17_04-35-05__27ab35e2-dd6d-42a1-a9bb-ffce34d478ce__wakuorg_nwaku:latest.log
2026-03-17 04:35:05.240 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-17 04:35:05.241 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-17_04-35-05__27ab35e2-dd6d-42a1-a9bb-ffce34d478ce__wakuorg_nwaku:latest.log
2026-03-17 04:35:05.246 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-17 04:35:05.246 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-17_04-35-05__27ab35e2-dd6d-42a1-a9bb-ffce34d478ce__wakuorg_nwaku:latest.log
2026-03-17 04:35:05.246 DEBUG [src.node.waku_node] Starting Node...
2026-03-17 04:35:05.246 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-17 04:35:05.290 DEBUG [src.node.docker_mananger] Network waku created
2026-03-17 04:35:05.290 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.94.9
2026-03-17 04:35:05.291 DEBUG [src.node.docker_mananger] Generated ports ['51692', '51693', '51694', '51695', '51696']
2026-03-17 04:35:05.291 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-17 04:35:05.291 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-17 04:35:05.291 DEBUG [src.node.waku_node] Using volumes []
2026-03-17 04:35:05.291 DEBUG [src.node.docker_mananger] docker run -i -t -p 51692:51692 -p 51693:51693 -p 51694:51694 -p 51695:51695 -p 51696:51696 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=51694 --rest-port=51692 --tcp-port=51693 --discv5-udp-port=51695 --rest-address=0.0.0.0 --nat=extip:172.18.94.9 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=ecffc6be6829d8e3a2ff7b5ec3daefdfc20e093934518e2bce16b4ccdcfbf04b --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=51696 --metrics-logging=true --relay=true --store=true
2026-03-17 04:35:08.102 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.94.9 waku dc88fce711e59c4fe5faf11dee5d667d333d9455b05d7520e094c6f91815a366
2026-03-17 04:35:08.136 DEBUG [src.node.docker_mananger] Container started with ID dc88fce711e5. Setting up logs at ./log/docker/node1_2026-03-17_04-35-05__27ab35e2-dd6d-42a1-a9bb-ffce34d478ce__wakuorg_nwaku:latest.log
2026-03-17 04:35:08.137 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 51692
2026-03-17 04:35:08.137 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-17 04:35:09.137 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51692/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:35:09.141 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-17 04:35:09.141 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-17 04:35:09.141 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51692/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:35:09.144 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.94.9/tcp/51693/p2p/16Uiu2HAmPG8w68yeqTNVHjFaym6MsZ2tHoSpEzZsJty9oeArFtPQ","/ip4/172.18.94.9/tcp/51694/ws/p2p/16Uiu2HAmPG8w68yeqTNVHjFaym6MsZ2tHoSpEzZsJty9oeArFtPQ"],"enrUri":"enr:-L24QHKKZm6Clxlz4P4sEuHEY1a885qj5FM7FUmHoApQkAl8c25nlK2NO9mQFDUbZXVsuN3ZnFqIppZMT0J6Lwj7-OgCgmlkgnY0gmlwhKwSXgmKbXVsdGlhZGRyc5YACASsEl4JBsntAAoErBJeCQbJ7t0DgnJzhQADAQAAiXNlY3AyNTZrMaEDnZXm8doy_gvzlGVtUDiiQaAauEKihoE8rjpbCOh6al-DdGNwgsntg3VkcILJ74V3YWt1MgM"}'
2026-03-17 04:35:09.144 INFO [src.node.waku_node] REST service is ready !!
2026-03-17 04:35:09.144 DEBUG [src.node.waku_node] Starting Node...
2026-03-17 04:35:09.144 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-17 04:35:09.146 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-17 04:35:09.146 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.174.219
2026-03-17 04:35:09.146 DEBUG [src.node.docker_mananger] Generated ports ['53920', '53921', '53922', '53923', '53924']
2026-03-17 04:35:09.146 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-17 04:35:09.146 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-17 04:35:09.146 DEBUG [src.node.waku_node] Using volumes []
2026-03-17 04:35:09.146 DEBUG [src.node.docker_mananger] docker run -i -t -p 53920:53920 -p 53921:53921 -p 53922:53922 -p 53923:53923 -p 53924:53924 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=53922 --rest-port=53920 --tcp-port=53921 --discv5-udp-port=53923 --rest-address=0.0.0.0 --nat=extip:172.18.174.219 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=171c3e9cefa4f9b22bb5faedb0cdce9c8a1b0fad2b808817bdd8fa3fe9a315cc --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=53924 --metrics-logging=true --store=false --relay=false --discv5-bootstrap-node=enr:-L24QHKKZm6Clxlz4P4sEuHEY1a885qj5FM7FUmHoApQkAl8c25nlK2NO9mQFDUbZXVsuN3ZnFqIppZMT0J6Lwj7-OgCgmlkgnY0gmlwhKwSXgmKbXVsdGlhZGRyc5YACASsEl4JBsntAAoErBJeCQbJ7t0DgnJzhQADAQAAiXNlY3AyNTZrMaEDnZXm8doy_gvzlGVtUDiiQaAauEKihoE8rjpbCOh6al-DdGNwgsntg3VkcILJ74V3YWt1MgM
2026-03-17 04:35:09.349 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.174.219 waku 44ee7754c03a43e448ddc18ec8f5098623db3dc03f143d5131eb5b1f0c1633d0
2026-03-17 04:35:09.382 DEBUG [src.node.docker_mananger] Container started with ID 44ee7754c03a. Setting up logs at ./log/docker/node2_2026-03-17_04-35-05__27ab35e2-dd6d-42a1-a9bb-ffce34d478ce__wakuorg_nwaku:latest.log
2026-03-17 04:35:09.382 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 53920
2026-03-17 04:35:09.383 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-17 04:35:10.383 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:53920/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:35:10.386 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-17 04:35:10.386 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-17 04:35:10.386 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:53920/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:35:10.389 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.174.219/tcp/53921/p2p/16Uiu2HAm1Q2ANUfzwFoKjwjZJBT4Mg6hNFyPXWkvbZ8AqAp1DkDZ","/ip4/172.18.174.219/tcp/53922/ws/p2p/16Uiu2HAm1Q2ANUfzwFoKjwjZJBT4Mg6hNFyPXWkvbZ8AqAp1DkDZ"],"enrUri":"enr:-L24QN6BB8YkQgTT4JlWYhE-l_fz7CEI5epAV4TthaYIYE6fSQVsLgtoLjMipRoCId1z2typqClXFVmjfy4jC4YZ2OwCgmlkgnY0gmlwhKwSrtuKbXVsdGlhZGRyc5YACASsEq7bBtKhAAoErBKu2wbSot0DgnJzhQADAQAAiXNlY3AyNTZrMaECWLnHvJHW-gTG1pPehmLjAUTcQstmGwgnoimNP4s9WWSDdGNwgtKhg3VkcILSo4V3YWt1MgA"}'
2026-03-17 04:35:10.389 INFO [src.node.waku_node] REST service is ready !!
2026-03-17 04:35:10.389 DEBUG [src.node.waku_node] Starting Node...
2026-03-17 04:35:10.389 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-17 04:35:10.391 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-17 04:35:10.391 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.131.18
2026-03-17 04:35:10.391 DEBUG [src.node.docker_mananger] Generated ports ['13352', '13353', '13354', '13355', '13356']
2026-03-17 04:35:10.391 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-17 04:35:10.391 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-17 04:35:10.391 DEBUG [src.node.waku_node] Using volumes []
2026-03-17 04:35:10.391 DEBUG [src.node.docker_mananger] docker run -i -t -p 13352:13352 -p 13353:13353 -p 13354:13354 -p 13355:13355 -p 13356:13356 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=13354 --rest-port=13352 --tcp-port=13353 --discv5-udp-port=13355 --rest-address=0.0.0.0 --nat=extip:172.18.131.18 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=084a75bc1d4de7aa447fbd2d8f65f8237ea8c6afbdaa6b5aed9ba5a6e9fd0bfe --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=13356 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QN6BB8YkQgTT4JlWYhE-l_fz7CEI5epAV4TthaYIYE6fSQVsLgtoLjMipRoCId1z2typqClXFVmjfy4jC4YZ2OwCgmlkgnY0gmlwhKwSrtuKbXVsdGlhZGRyc5YACASsEq7bBtKhAAoErBKu2wbSot0DgnJzhQADAQAAiXNlY3AyNTZrMaECWLnHvJHW-gTG1pPehmLjAUTcQstmGwgnoimNP4s9WWSDdGNwgtKhg3VkcILSo4V3YWt1MgA
2026-03-17 04:35:10.587 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.131.18 waku 64fef7e28a9089ee06ee5bedf3cda3e09eea59c78fd3fbb986c1710cef77ba5a
2026-03-17 04:35:10.625 DEBUG [src.node.docker_mananger] Container started with ID 64fef7e28a90. Setting up logs at ./log/docker/node3_2026-03-17_04-35-05__27ab35e2-dd6d-42a1-a9bb-ffce34d478ce__wakuorg_nwaku:latest.log
2026-03-17 04:35:10.625 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 13352
2026-03-17 04:35:10.625 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-17 04:35:11.626 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:13352/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:35:11.628 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-17 04:35:11.629 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-17 04:35:11.629 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:13352/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:35:11.631 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.131.18/tcp/13353/p2p/16Uiu2HAmUSMQU97t2AhUQCAErj4KhMuS1fPHisp5cT7vwyfTrGcp","/ip4/172.18.131.18/tcp/13354/ws/p2p/16Uiu2HAmUSMQU97t2AhUQCAErj4KhMuS1fPHisp5cT7vwyfTrGcp"],"enrUri":"enr:-L24QBVhnwSwhqW9wGRKDLBoZdNWFFdp2jnbKH6XLIzQb1YkfIWdCP2GuHMc5nC44M-LRLlnYI7QhFb2dQ_p-SZFNj0CgmlkgnY0gmlwhKwSgxKKbXVsdGlhZGRyc5YACASsEoMSBjQpAAoErBKDEgY0Kt0DgnJzhQADAQAAiXNlY3AyNTZrMaED6n4_Pr2wq0kNm0YSnvQsnB1Jx3tR-Ni3VqFYmI7lfkGDdGNwgjQpg3VkcII0K4V3YWt1MgA"}'
2026-03-17 04:35:11.631 INFO [src.node.waku_node] REST service is ready !!
2026-03-17 04:35:11.632 DEBUG [src.node.waku_node] Starting Node...
2026-03-17 04:35:11.632 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-17 04:35:11.633 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-17 04:35:11.633 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.21.27
2026-03-17 04:35:11.633 DEBUG [src.node.docker_mananger] Generated ports ['52926', '52927', '52928', '52929', '52930']
2026-03-17 04:35:11.634 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-17 04:35:11.634 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-17 04:35:11.634 DEBUG [src.node.waku_node] Using volumes []
2026-03-17 04:35:11.634 DEBUG [src.node.docker_mananger] docker run -i -t -p 52926:52926 -p 52927:52927 -p 52928:52928 -p 52929:52929 -p 52930:52930 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=52928 --rest-port=52926 --tcp-port=52927 --discv5-udp-port=52929 --rest-address=0.0.0.0 --nat=extip:172.18.21.27 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=ab58de2b5c30fdaa8ebd8c1bcbe6f6ccf14ecfa1cec12d967f3afed1244cbfcc --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=52930 --metrics-logging=true --relay=true --store=false --discv5-bootstrap-node=enr:-L24QBVhnwSwhqW9wGRKDLBoZdNWFFdp2jnbKH6XLIzQb1YkfIWdCP2GuHMc5nC44M-LRLlnYI7QhFb2dQ_p-SZFNj0CgmlkgnY0gmlwhKwSgxKKbXVsdGlhZGRyc5YACASsEoMSBjQpAAoErBKDEgY0Kt0DgnJzhQADAQAAiXNlY3AyNTZrMaED6n4_Pr2wq0kNm0YSnvQsnB1Jx3tR-Ni3VqFYmI7lfkGDdGNwgjQpg3VkcII0K4V3YWt1MgA
2026-03-17 04:35:11.830 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.21.27 waku eb33ca6617ab70dbefb5e07b1fd229bfabe938cf963706afc168a92d1c92bfeb
2026-03-17 04:35:11.865 DEBUG [src.node.docker_mananger] Container started with ID eb33ca6617ab. Setting up logs at ./log/docker/node3_2026-03-17_04-35-05__27ab35e2-dd6d-42a1-a9bb-ffce34d478ce__wakuorg_nwaku:latest.log
2026-03-17 04:35:11.865 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 52926
2026-03-17 04:35:11.866 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-17 04:35:12.867 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:52926/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:35:12.879 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-17 04:35:12.879 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-17 04:35:12.880 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:52926/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:35:12.886 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.21.27/tcp/52927/p2p/16Uiu2HAkydTMK36kc8tU7tqJ8C1X91c3vuKkMAkmGNgbBrSh5MJP","/ip4/172.18.21.27/tcp/52928/ws/p2p/16Uiu2HAkydTMK36kc8tU7tqJ8C1X91c3vuKkMAkmGNgbBrSh5MJP"],"enrUri":"enr:-L24QKzcVtYTwNj2DWKE3xS3-15xuu_J3eTmoGGAk5W3u-59WV2wUvLrt8LxnRU6nEnz7MKjAxh4my3bkCTc5U242BQCgmlkgnY0gmlwhKwSFRuKbXVsdGlhZGRyc5YACASsEhUbBs6_AAoErBIVGwbOwN0DgnJzhQADAQAAiXNlY3AyNTZrMaECPnNwEUhk2J5EsXfJWDbIZxWqqK8BQ22xsi44QWhB2cCDdGNwgs6_g3VkcILOwYV3YWt1MgE"}'
2026-03-17 04:35:12.888 INFO [src.node.waku_node] REST service is ready !!
2026-03-17 04:35:12.889 DEBUG [src.node.waku_node] Starting Node...
2026-03-17 04:35:12.889 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-17 04:35:12.891 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-17 04:35:12.892 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.160.253
2026-03-17 04:35:12.893 DEBUG [src.node.docker_mananger] Generated ports ['55644', '55645', '55646', '55647', '55648']
2026-03-17 04:35:12.894 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-17 04:35:12.895 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-17 04:35:12.896 DEBUG [src.node.waku_node] Using volumes []
2026-03-17 04:35:12.896 DEBUG [src.node.docker_mananger] docker run -i -t -p 55644:55644 -p 55645:55645 -p 55646:55646 -p 55647:55647 -p 55648:55648 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=55646 --rest-port=55644 --tcp-port=55645 --discv5-udp-port=55647 --rest-address=0.0.0.0 --nat=extip:172.18.160.253 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=e0500c992b9a7656cc8c5fc73922a0e2a00b407bdcd0ed1febbfd0eec0ade218 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=55648 --metrics-logging=true --relay=false --store=false --discv5-bootstrap-node=enr:-L24QKzcVtYTwNj2DWKE3xS3-15xuu_J3eTmoGGAk5W3u-59WV2wUvLrt8LxnRU6nEnz7MKjAxh4my3bkCTc5U242BQCgmlkgnY0gmlwhKwSFRuKbXVsdGlhZGRyc5YACASsEhUbBs6_AAoErBIVGwbOwN0DgnJzhQADAQAAiXNlY3AyNTZrMaECPnNwEUhk2J5EsXfJWDbIZxWqqK8BQ22xsi44QWhB2cCDdGNwgs6_g3VkcILOwYV3YWt1MgE
2026-03-17 04:35:13.093 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.160.253 waku c566d41f9b7d69ff5f66eb236e73ba8e2ddcddf722ce2ffb9ebb4142b8359f74
2026-03-17 04:35:13.127 DEBUG [src.node.docker_mananger] Container started with ID c566d41f9b7d. Setting up logs at ./log/docker/node4_2026-03-17_04-35-05__27ab35e2-dd6d-42a1-a9bb-ffce34d478ce__wakuorg_nwaku:latest.log
2026-03-17 04:35:13.127 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 55644
2026-03-17 04:35:13.127 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-17 04:35:14.129 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:55644/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:35:14.131 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-17 04:35:14.132 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-17 04:35:14.132 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:55644/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:35:14.134 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.160.253/tcp/55645/p2p/16Uiu2HAmRYVWhQiYq3K6Mrzv6UDGdsYSvZoHM2cuyFDggKMUZRL5","/ip4/172.18.160.253/tcp/55646/ws/p2p/16Uiu2HAmRYVWhQiYq3K6Mrzv6UDGdsYSvZoHM2cuyFDggKMUZRL5"],"enrUri":"enr:-L24QK3ten3KycLMFY8x8cgTCQofoFprTV6L6UuicrqOYv4fTiyVXRMF_Tx20S-EGvFYj8N1LuPI4_kbpBi-4fUeAdcCgmlkgnY0gmlwhKwSoP2KbXVsdGlhZGRyc5YACASsEqD9BtldAAoErBKg_QbZXt0DgnJzhQADAQAAiXNlY3AyNTZrMaEDv33Wk0QwRRD_O_M3ls18uQclPWlcm_YBPpuZmlBT9uKDdGNwgtldg3VkcILZX4V3YWt1MgA"}'
2026-03-17 04:35:14.135 INFO [src.node.waku_node] REST service is ready !!
2026-03-17 04:35:14.135 DEBUG [tests.e2e.test_e2e] Add 3 peer nodes to node3
2026-03-17 04:35:14.135 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:13352/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.94.9/tcp/51693/p2p/16Uiu2HAmPG8w68yeqTNVHjFaym6MsZ2tHoSpEzZsJty9oeArFtPQ"]'
2026-03-17 04:35:14.159 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-17 04:35:14.159 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:13352/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.174.219/tcp/53921/p2p/16Uiu2HAm1Q2ANUfzwFoKjwjZJBT4Mg6hNFyPXWkvbZ8AqAp1DkDZ"]'
2026-03-17 04:35:14.192 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-17 04:35:14.194 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:13352/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.21.27/tcp/52927/p2p/16Uiu2HAkydTMK36kc8tU7tqJ8C1X91c3vuKkMAkmGNgbBrSh5MJP"]'
2026-03-17 04:35:14.202 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-17 04:35:14.202 DEBUG [tests.e2e.test_e2e] Subscribe nodes 1,2 to relay on pubsubtopic /waku/2/rs/3/1
2026-03-17 04:35:14.203 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:52926/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-17 04:35:14.210 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-17 04:35:14.211 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:51692/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-17 04:35:14.215 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-17 04:35:14.215 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51692/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:35:14.218 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.131.18/tcp/54942/p2p/16Uiu2HAmUSMQU97t2AhUQCAErj4KhMuS1fPHisp5cT7vwyfTrGcp","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0","origin":"UnknownOrigin"}]'
2026-03-17 04:35:14.218 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:52926/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:35:14.220 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.131.18/tcp/13353/p2p/16Uiu2HAmUSMQU97t2AhUQCAErj4KhMuS1fPHisp5cT7vwyfTrGcp","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0","origin":"Discv5"}]'
2026-03-17 04:35:14.220 DEBUG [src.libs.common] Sleeping for 30 seconds
2026-03-17 04:35:44.221 DEBUG [tests.e2e.test_e2e] Node1 publish message
2026-03-17 04:35:44.221 DEBUG [src.steps.store] Relaying message
2026-03-17 04:35:44.222 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:52926/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": '$(date +%s%N)'}'
2026-03-17 04:35:44.227 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-17 04:35:44.228 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-17 04:35:44.429 DEBUG [tests.e2e.test_e2e] Check if node3 can inquiry stored message without stor peer specified
2026-03-17 04:35:44.429 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:13352/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F1&contentTopics=%2Ftest%2F1%2Fwaku-filter%2Fproto&pageSize=5&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:35:44.439 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"9ea079caebb59ffe63af","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0x674c6c29a608d85c58f4b40625a130fda96e45b412163bd8518d1fb1d408d424"}]}'
2026-03-17 04:35:44.442 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-17 04:35:44.442 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-17 04:35:44.443 DEBUG [src.node.waku_node] Stopping container with id dc88fce711e5
2026-03-17 04:35:44.983 DEBUG [src.node.waku_node] Container stopped.
2026-03-17 04:35:44.985 DEBUG [src.node.waku_node] Stopping container with id 44ee7754c03a
2026-03-17 04:35:45.500 DEBUG [src.node.waku_node] Container stopped.
2026-03-17 04:35:45.503 DEBUG [src.node.waku_node] Stopping container with id 64fef7e28a90
2026-03-17 04:35:45.882 ERROR [src.node.docker_mananger] Max retries reached for container dc88fce711e5. Exiting log stream.
2026-03-17 04:35:46.045 DEBUG [src.node.waku_node] Container stopped.
2026-03-17 04:35:46.045 DEBUG [src.node.waku_node] Stopping container with id eb33ca6617ab
2026-03-17 04:35:46.358 ERROR [src.node.docker_mananger] Max retries reached for container 44ee7754c03a. Exiting log stream.
2026-03-17 04:35:46.605 DEBUG [src.node.waku_node] Container stopped.
2026-03-17 04:35:46.608 DEBUG [src.node.waku_node] Stopping container with id c566d41f9b7d
2026-03-17 04:35:46.859 ERROR [src.node.docker_mananger] Max retries reached for container 64fef7e28a90. Exiting log stream.
2026-03-17 04:35:47.150 DEBUG [src.node.waku_node] Container stopped.
2026-03-17 04:35:47.153 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-17 04:35:47.193 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-17 04:35:47.211 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-17 04:35:47.235 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-17 04:35:47.258 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-17 04:35:47.260 DEBUG [src.node.docker_mananger] No errors found in the waku logs.