2026-02-25 20:22:16 +00:00

179 lines
34 KiB
Plaintext

2026-02-25 20:02:40.874 DEBUG [tests.conftest] Running fixture setup: test_id
2026-02-25 20:02:40.875 DEBUG [tests.conftest] Running test: test_multiple_edge_service_nodes_communication with id: 2026-02-25_20-02-40__ac6a36ce-e279-486f-81f4-3b62a6390832
2026-02-25 20:02:40.875 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-02-25 20:02:40.875 DEBUG [src.steps.filter] Running fixture setup: filter_setup
2026-02-25 20:02:40.876 DEBUG [src.steps.light_push] Running fixture setup: light_push_setup
2026-02-25 20:02:40.886 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-25 20:02:40.886 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-02-25_20-02-40__ac6a36ce-e279-486f-81f4-3b62a6390832__wakuorg_nwaku:latest.log
2026-02-25 20:02:40.894 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-25 20:02:40.894 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-02-25_20-02-40__ac6a36ce-e279-486f-81f4-3b62a6390832__wakuorg_nwaku:latest.log
2026-02-25 20:02:40.901 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-25 20:02:40.901 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-02-25_20-02-40__ac6a36ce-e279-486f-81f4-3b62a6390832__wakuorg_nwaku:latest.log
2026-02-25 20:02:40.901 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-02-25 20:02:40.902 DEBUG [src.steps.store] Running fixture setup: store_setup
2026-02-25 20:02:40.909 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-25 20:02:40.910 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-02-25_20-02-40__ac6a36ce-e279-486f-81f4-3b62a6390832__wakuorg_nwaku:latest.log
2026-02-25 20:02:40.916 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-25 20:02:40.917 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-02-25_20-02-40__ac6a36ce-e279-486f-81f4-3b62a6390832__wakuorg_nwaku:latest.log
2026-02-25 20:02:40.923 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-25 20:02:40.923 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node6_2026-02-25_20-02-40__ac6a36ce-e279-486f-81f4-3b62a6390832__wakuorg_nwaku:latest.log
2026-02-25 20:02:40.930 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-25 20:02:40.930 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node7_2026-02-25_20-02-40__ac6a36ce-e279-486f-81f4-3b62a6390832__wakuorg_nwaku:latest.log
2026-02-25 20:02:40.936 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-25 20:02:40.937 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node8_2026-02-25_20-02-40__ac6a36ce-e279-486f-81f4-3b62a6390832__wakuorg_nwaku:latest.log
2026-02-25 20:02:40.937 DEBUG [tests.e2e.test_e2e] Start 2 edges nodes and 3 service nodes
2026-02-25 20:02:40.937 DEBUG [src.node.waku_node] Starting Node...
2026-02-25 20:02:40.937 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-25 20:02:40.989 DEBUG [src.node.docker_mananger] Network waku created
2026-02-25 20:02:40.989 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.250.186
2026-02-25 20:02:40.989 DEBUG [src.node.docker_mananger] Generated ports ['7708', '7709', '7710', '7711', '7712']
2026-02-25 20:02:40.989 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-25 20:02:40.989 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-25 20:02:40.990 DEBUG [src.node.waku_node] Using volumes []
2026-02-25 20:02:40.990 DEBUG [src.node.docker_mananger] docker run -i -t -p 7708:7708 -p 7709:7709 -p 7710:7710 -p 7711:7711 -p 7712:7712 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=7710 --rest-port=7708 --tcp-port=7709 --discv5-udp-port=7711 --rest-address=0.0.0.0 --nat=extip:172.18.250.186 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=0f1d7c11c33a4ae1ec0d8852a4cd0eebf4ed2adfc87f7ac6b7e24d981d57970d --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=7712 --metrics-logging=true --relay=true --store=true --lightpush=true
2026-02-25 20:02:43.902 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.250.186 waku 041731b8895858fc48f51d2126025de2e223e22f5adf66ee8a39830b2e9953b5
2026-02-25 20:02:43.943 DEBUG [src.node.docker_mananger] Container started with ID 041731b88958. Setting up logs at ./log/docker/node6_2026-02-25_20-02-40__ac6a36ce-e279-486f-81f4-3b62a6390832__wakuorg_nwaku:latest.log
2026-02-25 20:02:43.944 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 7708
2026-02-25 20:02:43.946 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-25 20:02:44.948 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:7708/health" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:02:44.952 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Legacy Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-25 20:02:44.952 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-25 20:02:44.952 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:7708/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:02:44.955 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.250.186/tcp/7709/p2p/16Uiu2HAm1syd8WVQEmnZPozY7x9wofR1vGZK3Cm9F2oCTbp9VGPM","/ip4/172.18.250.186/tcp/7710/ws/p2p/16Uiu2HAm1syd8WVQEmnZPozY7x9wofR1vGZK3Cm9F2oCTbp9VGPM"],"enrUri":"enr:-L24QKgsAVavoQuY69X32abUyDnLqXW_GpHBkOAxy02I4_q4EsVprqIfvJrwOG7VWGEI_-rHIHBkrSejNds6vENf1JUCgmlkgnY0gmlwhKwS-rqKbXVsdGlhZGRyc5YACASsEvq6Bh4dAAoErBL6ugYeHt0DgnJzhQADAQAAiXNlY3AyNTZrMaECX-MsJHS3QjTAoZ8qczKGzM323RSa8h5n7Po_oaj9N-yDdGNwgh4dg3VkcIIeH4V3YWt1Mgs"}'
2026-02-25 20:02:44.956 INFO [src.node.waku_node] REST service is ready !!
2026-02-25 20:02:44.956 DEBUG [src.node.waku_node] Starting Node...
2026-02-25 20:02:44.956 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-25 20:02:44.958 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-25 20:02:44.958 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.52.143
2026-02-25 20:02:44.958 DEBUG [src.node.docker_mananger] Generated ports ['58581', '58582', '58583', '58584', '58585']
2026-02-25 20:02:44.959 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-25 20:02:44.959 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-25 20:02:44.959 DEBUG [src.node.waku_node] Using volumes []
2026-02-25 20:02:44.959 DEBUG [src.node.docker_mananger] docker run -i -t -p 58581:58581 -p 58582:58582 -p 58583:58583 -p 58584:58584 -p 58585:58585 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=58583 --rest-port=58581 --tcp-port=58582 --discv5-udp-port=58584 --rest-address=0.0.0.0 --nat=extip:172.18.52.143 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=3d287ec34a71d07ea26dcaadf39db48bdba5bceaa87faea7ef0438ed50ea30b2 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=58585 --metrics-logging=true --relay=false --lightpushnode=/ip4/172.18.250.186/tcp/7709/p2p/16Uiu2HAm1syd8WVQEmnZPozY7x9wofR1vGZK3Cm9F2oCTbp9VGPM --discv5-bootstrap-node=enr:-L24QKgsAVavoQuY69X32abUyDnLqXW_GpHBkOAxy02I4_q4EsVprqIfvJrwOG7VWGEI_-rHIHBkrSejNds6vENf1JUCgmlkgnY0gmlwhKwS-rqKbXVsdGlhZGRyc5YACASsEvq6Bh4dAAoErBL6ugYeHt0DgnJzhQADAQAAiXNlY3AyNTZrMaECX-MsJHS3QjTAoZ8qczKGzM323RSa8h5n7Po_oaj9N-yDdGNwgh4dg3VkcIIeH4V3YWt1Mgs
2026-02-25 20:02:45.171 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.52.143 waku 80b108164ad873ffac099d151c35230524fcfd1bee05d5f2a525e2aefbee577e
2026-02-25 20:02:45.211 DEBUG [src.node.docker_mananger] Container started with ID 80b108164ad8. Setting up logs at ./log/docker/node4_2026-02-25_20-02-40__ac6a36ce-e279-486f-81f4-3b62a6390832__wakuorg_nwaku:latest.log
2026-02-25 20:02:45.212 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 58581
2026-02-25 20:02:45.213 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-25 20:02:46.214 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58581/health" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:02:46.218 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-25 20:02:46.218 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-25 20:02:46.218 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58581/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:02:46.221 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.52.143/tcp/58582/p2p/16Uiu2HAm6gtWnsB4fsLtCG9rcdTiV5R7uaek1V96y2Q5GmpBJzmp","/ip4/172.18.52.143/tcp/58583/ws/p2p/16Uiu2HAm6gtWnsB4fsLtCG9rcdTiV5R7uaek1V96y2Q5GmpBJzmp"],"enrUri":"enr:-L24QJlubY6zz2voAOXkpvUX1QxOKikgB8NJ7HyjtMZ4VqQSIfnPU83YUCJakW0_Z_tfa_mCj3sEgVi9pDmTb1zuH9cCgmlkgnY0gmlwhKwSNI-KbXVsdGlhZGRyc5YACASsEjSPBuTWAAoErBI0jwbk190DgnJzhQADAQAAiXNlY3AyNTZrMaECp1Zw6i6oogKqaO98ujkqw89BPaUDXiadRX4dTWTd7VODdGNwguTWg3VkcILk2IV3YWt1MgA"}'
2026-02-25 20:02:46.222 INFO [src.node.waku_node] REST service is ready !!
2026-02-25 20:02:46.222 DEBUG [src.node.waku_node] Starting Node...
2026-02-25 20:02:46.222 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-25 20:02:46.224 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-25 20:02:46.224 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.163.30
2026-02-25 20:02:46.224 DEBUG [src.node.docker_mananger] Generated ports ['26672', '26673', '26674', '26675', '26676']
2026-02-25 20:02:46.224 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-25 20:02:46.225 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-25 20:02:46.225 DEBUG [src.node.waku_node] Using volumes []
2026-02-25 20:02:46.225 DEBUG [src.node.docker_mananger] docker run -i -t -p 26672:26672 -p 26673:26673 -p 26674:26674 -p 26675:26675 -p 26676:26676 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=26674 --rest-port=26672 --tcp-port=26673 --discv5-udp-port=26675 --rest-address=0.0.0.0 --nat=extip:172.18.163.30 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=fd47f2f7aab1c1aaa4cc5aa3e8a09c39aefaccecce8dedbb08f5de2faecbf793 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=26676 --metrics-logging=true --relay=true --store=true --discv5-bootstrap-node=enr:-L24QKgsAVavoQuY69X32abUyDnLqXW_GpHBkOAxy02I4_q4EsVprqIfvJrwOG7VWGEI_-rHIHBkrSejNds6vENf1JUCgmlkgnY0gmlwhKwS-rqKbXVsdGlhZGRyc5YACASsEvq6Bh4dAAoErBL6ugYeHt0DgnJzhQADAQAAiXNlY3AyNTZrMaECX-MsJHS3QjTAoZ8qczKGzM323RSa8h5n7Po_oaj9N-yDdGNwgh4dg3VkcIIeH4V3YWt1Mgs
2026-02-25 20:02:46.442 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.163.30 waku 1627f23989de19f218932cf090074d0a076367841e653abc035a59cc1760a0d6
2026-02-25 20:02:46.481 DEBUG [src.node.docker_mananger] Container started with ID 1627f23989de. Setting up logs at ./log/docker/node7_2026-02-25_20-02-40__ac6a36ce-e279-486f-81f4-3b62a6390832__wakuorg_nwaku:latest.log
2026-02-25 20:02:46.482 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 26672
2026-02-25 20:02:46.483 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-25 20:02:47.483 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26672/health" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:02:47.487 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-25 20:02:47.487 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-25 20:02:47.487 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26672/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:02:47.490 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.163.30/tcp/26673/p2p/16Uiu2HAm6pcJgjgngtkqzsj8xZwFVh2j3XBqffqDvMohrrkzve72","/ip4/172.18.163.30/tcp/26674/ws/p2p/16Uiu2HAm6pcJgjgngtkqzsj8xZwFVh2j3XBqffqDvMohrrkzve72"],"enrUri":"enr:-L24QIBbQELi5RAvPfBOKcebhh3EDIlpKC7ilLT-1kO8jLXIPGgu4TPCczXE8wxTEYRXLtvXbdLR7N6Qhq_qegUJJdoCgmlkgnY0gmlwhKwSox6KbXVsdGlhZGRyc5YACASsEqMeBmgxAAoErBKjHgZoMt0DgnJzhQADAQAAiXNlY3AyNTZrMaECqVDColTZ3bCATCpuHS9GEHg5GOpDZ3K8KR1ShKuAnQmDdGNwgmgxg3VkcIJoM4V3YWt1MgM"}'
2026-02-25 20:02:47.490 INFO [src.node.waku_node] REST service is ready !!
2026-02-25 20:02:47.490 DEBUG [src.node.waku_node] Starting Node...
2026-02-25 20:02:47.490 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-25 20:02:47.492 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-25 20:02:47.492 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.24.52
2026-02-25 20:02:47.493 DEBUG [src.node.docker_mananger] Generated ports ['34459', '34460', '34461', '34462', '34463']
2026-02-25 20:02:47.493 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-25 20:02:47.493 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-25 20:02:47.493 DEBUG [src.node.waku_node] Using volumes []
2026-02-25 20:02:47.493 DEBUG [src.node.docker_mananger] docker run -i -t -p 34459:34459 -p 34460:34460 -p 34461:34461 -p 34462:34462 -p 34463:34463 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=34461 --rest-port=34459 --tcp-port=34460 --discv5-udp-port=34462 --rest-address=0.0.0.0 --nat=extip:172.18.24.52 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=bb1932879fbdb256f6bd71abfbedab207cc46d755b674dddca3c9bce1ab9a4db --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=34463 --metrics-logging=true --relay=true --filter=true --storenode=/ip4/172.18.163.30/tcp/26673/p2p/16Uiu2HAm6pcJgjgngtkqzsj8xZwFVh2j3XBqffqDvMohrrkzve72 --discv5-bootstrap-node=enr:-L24QIBbQELi5RAvPfBOKcebhh3EDIlpKC7ilLT-1kO8jLXIPGgu4TPCczXE8wxTEYRXLtvXbdLR7N6Qhq_qegUJJdoCgmlkgnY0gmlwhKwSox6KbXVsdGlhZGRyc5YACASsEqMeBmgxAAoErBKjHgZoMt0DgnJzhQADAQAAiXNlY3AyNTZrMaECqVDColTZ3bCATCpuHS9GEHg5GOpDZ3K8KR1ShKuAnQmDdGNwgmgxg3VkcIJoM4V3YWt1MgM
2026-02-25 20:02:47.741 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.24.52 waku 9cefdec862839d4db5b825024e14416607e77515fab59d01f723afc0bf9ff82c
2026-02-25 20:02:47.784 DEBUG [src.node.docker_mananger] Container started with ID 9cefdec86283. Setting up logs at ./log/docker/node8_2026-02-25_20-02-40__ac6a36ce-e279-486f-81f4-3b62a6390832__wakuorg_nwaku:latest.log
2026-02-25 20:02:47.785 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 34459
2026-02-25 20:02:47.786 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-25 20:02:48.787 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:34459/health" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:02:48.799 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-25 20:02:48.800 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-25 20:02:48.800 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:34459/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:02:48.808 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.24.52/tcp/34460/p2p/16Uiu2HAm15bCQgfh6MzxP3DBLPVx4QRKP7Ykq5J2C3QFbKoMsj7W","/ip4/172.18.24.52/tcp/34461/ws/p2p/16Uiu2HAm15bCQgfh6MzxP3DBLPVx4QRKP7Ykq5J2C3QFbKoMsj7W"],"enrUri":"enr:-L24QJHy7R9yUMx7zNgOvXqvccavXbtB5GNPcFfuLtXfIdrCdo2gc2eMg8f09LHE7nhYK5vrtiB4tKxNpGbQojzDKQQCgmlkgnY0gmlwhKwSGDSKbXVsdGlhZGRyc5YACASsEhg0BoacAAoErBIYNAaGnd0DgnJzhQADAQAAiXNlY3AyNTZrMaECVAEZCV2HoRUUEzPHnxlxRhOZLLz_69hXfU1bl9K79rGDdGNwgoacg3VkcIKGnoV3YWt1MgU"}'
2026-02-25 20:02:48.809 INFO [src.node.waku_node] REST service is ready !!
2026-02-25 20:02:48.810 DEBUG [src.node.waku_node] Starting Node...
2026-02-25 20:02:48.810 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-25 20:02:48.812 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-25 20:02:48.812 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.167.131
2026-02-25 20:02:48.812 DEBUG [src.node.docker_mananger] Generated ports ['12770', '12771', '12772', '12773', '12774']
2026-02-25 20:02:48.812 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-25 20:02:48.813 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-25 20:02:48.813 DEBUG [src.node.waku_node] Using volumes []
2026-02-25 20:02:48.813 DEBUG [src.node.docker_mananger] docker run -i -t -p 12770:12770 -p 12771:12771 -p 12772:12772 -p 12773:12773 -p 12774:12774 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=12772 --rest-port=12770 --tcp-port=12771 --discv5-udp-port=12773 --rest-address=0.0.0.0 --nat=extip:172.18.167.131 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=3febfc4f7e5a2ee2fff686cceafec037793d9af1984eaa11b24fcf14dcad3632 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=12774 --metrics-logging=true --relay=false --filternode=/ip4/172.18.24.52/tcp/34460/p2p/16Uiu2HAm15bCQgfh6MzxP3DBLPVx4QRKP7Ykq5J2C3QFbKoMsj7W --storenode=/ip4/172.18.163.30/tcp/26673/p2p/16Uiu2HAm6pcJgjgngtkqzsj8xZwFVh2j3XBqffqDvMohrrkzve72 --discv5-bootstrap-node=enr:-L24QIBbQELi5RAvPfBOKcebhh3EDIlpKC7ilLT-1kO8jLXIPGgu4TPCczXE8wxTEYRXLtvXbdLR7N6Qhq_qegUJJdoCgmlkgnY0gmlwhKwSox6KbXVsdGlhZGRyc5YACASsEqMeBmgxAAoErBKjHgZoMt0DgnJzhQADAQAAiXNlY3AyNTZrMaECqVDColTZ3bCATCpuHS9GEHg5GOpDZ3K8KR1ShKuAnQmDdGNwgmgxg3VkcIJoM4V3YWt1MgM
2026-02-25 20:02:49.027 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.167.131 waku 679f0a452f73976ea86a8262f6acb830546a934a4d210307903927d5630ebd97
2026-02-25 20:02:49.067 DEBUG [src.node.docker_mananger] Container started with ID 679f0a452f73. Setting up logs at ./log/docker/node5_2026-02-25_20-02-40__ac6a36ce-e279-486f-81f4-3b62a6390832__wakuorg_nwaku:latest.log
2026-02-25 20:02:49.067 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 12770
2026-02-25 20:02:49.068 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-25 20:02:50.069 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:12770/health" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:02:50.073 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-25 20:02:50.073 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-25 20:02:50.073 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:12770/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:02:50.077 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.167.131/tcp/12771/p2p/16Uiu2HAmBuNo4oGsiXVvPih3Bv4WEe6dBRrrRrYfg84yZ9pFtaHr","/ip4/172.18.167.131/tcp/12772/ws/p2p/16Uiu2HAmBuNo4oGsiXVvPih3Bv4WEe6dBRrrRrYfg84yZ9pFtaHr"],"enrUri":"enr:-L24QGf5pDK63aB5-beXgfVou-bcXxgidcb7qXw8I6RvUQHgTtIIiEpJqhrB1BNqsDSM98Xcj74ZN6h3rmb9ehfMNYcCgmlkgnY0gmlwhKwSp4OKbXVsdGlhZGRyc5YACASsEqeDBjHjAAoErBKngwYx5N0DgnJzhQADAQAAiXNlY3AyNTZrMaEC9NPSiAz8PFgOwUNYAZwbagnM9Bra2Y6o2cmXIcnFlq2DdGNwgjHjg3VkcIIx5YV3YWt1MgA"}'
2026-02-25 20:02:50.077 INFO [src.node.waku_node] REST service is ready !!
2026-02-25 20:02:50.077 DEBUG [tests.e2e.test_e2e] Connect 3 service nodes to relay subscriptions
2026-02-25 20:02:50.077 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:7708/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-02-25 20:02:50.082 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 20:02:50.083 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:26672/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-02-25 20:02:50.089 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 20:02:50.089 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:34459/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-02-25 20:02:50.097 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 20:02:50.098 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:7708/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:02:50.102 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.163.30/tcp/39856/p2p/16Uiu2HAm6pcJgjgngtkqzsj8xZwFVh2j3XBqffqDvMohrrkzve72","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.36.0-156-gc7e0cc","origin":"Discv5"}]'
2026-02-25 20:02:50.103 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26672/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:02:50.109 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.17.0.5/tcp/55094/p2p/16Uiu2HAm15bCQgfh6MzxP3DBLPVx4QRKP7Ykq5J2C3QFbKoMsj7W","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-subscribe/2.0.0-beta1","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.36.0-156-gc7e0cc","origin":"UnknownOrigin"},{"multiaddr":"/ip4/172.18.250.186/tcp/7709/p2p/16Uiu2HAm1syd8WVQEmnZPozY7x9wofR1vGZK3Cm9F2oCTbp9VGPM","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/lightpush/3.0.0","/vac/waku/lightpush/2.0.0-beta1","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.36.0-156-gc7e0cc","origin":"Discv5"}]'
2026-02-25 20:02:50.111 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:34459/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:02:50.114 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.163.30/tcp/26673/p2p/16Uiu2HAm6pcJgjgngtkqzsj8xZwFVh2j3XBqffqDvMohrrkzve72","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.36.0-156-gc7e0cc","origin":"Discv5"}]'
2026-02-25 20:02:50.114 DEBUG [src.libs.common] Sleeping for 30 seconds
2026-02-25 20:03:20.115 DEBUG [tests.e2e.test_e2e] Edge node2 makes filter subscription to pubsubtopic /waku/2/rs/3/1 and content topic /test/1/waku-filter/proto
2026-02-25 20:03:20.115 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:12770/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "1", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-02-25 20:03:20.164 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"1","statusDesc":"OK"}'
2026-02-25 20:03:20.168 DEBUG [tests.e2e.test_e2e] Check if service node1 receives message sent by edge node1
2026-02-25 20:03:20.170 DEBUG [src.steps.light_push] Lightpushing message
2026-02-25 20:03:20.171 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:58581/lightpush/v1/message" -H "Content-Type: application/json" -d '{"pubsubTopic": "/waku/2/rs/3/1", "message": {"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": 1772049800169341599}}'
2026-02-25 20:03:20.231 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 20:03:20.234 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-02-25 20:03:20.336 DEBUG [src.steps.light_push] Checking that peer NODE_1:wakuorg/nwaku:latest can find the lightpushed message
2026-02-25 20:03:20.336 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:7708/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:03:20.340 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1772049800169341599,"ephemeral":false,"proof":""}]'
2026-02-25 20:03:20.342 DEBUG [tests.e2e.test_e2e] Check if edge node2 can query stored message
2026-02-25 20:03:20.342 DEBUG [src.steps.store] Checking that peer wakuorg/nwaku:latest can find the stored messages
2026-02-25 20:03:20.343 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:12770/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F1&pageSize=50&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:03:20.390 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"767eb0cae42e86c226c9","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0xf0dfde6d1d7bd029fc406a49e9b9024f6a14ced661e22e6e3510d09b62dfdaf4"}]}'
2026-02-25 20:03:20.391 DEBUG [src.steps.store] messages length is 1
2026-02-25 20:03:20.392 DEBUG [tests.e2e.test_e2e] Check if service node3 can query stored message
2026-02-25 20:03:20.394 DEBUG [src.steps.store] Checking that peer wakuorg/nwaku:latest can find the stored messages
2026-02-25 20:03:20.395 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:34459/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F1&pageSize=50&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:03:20.407 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"28f5ae4bff872c5b98b6","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0xf0dfde6d1d7bd029fc406a49e9b9024f6a14ced661e22e6e3510d09b62dfdaf4"}]}'
2026-02-25 20:03:20.408 DEBUG [src.steps.store] messages length is 1
2026-02-25 20:03:20.408 DEBUG [tests.e2e.test_e2e] Check if edge node2 can get sent message using filter get request
2026-02-25 20:03:20.409 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:12770/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-02-25 20:03:20.412 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1772049800169341599,"ephemeral":false}]'
2026-02-25 20:03:20.414 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-02-25 20:03:20.415 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-02-25 20:03:20.416 DEBUG [src.node.waku_node] Stopping container with id 041731b88958
2026-02-25 20:03:21.082 DEBUG [src.node.waku_node] Container stopped.
2026-02-25 20:03:21.084 DEBUG [src.node.waku_node] Stopping container with id 80b108164ad8
2026-02-25 20:03:21.687 DEBUG [src.node.waku_node] Container stopped.
2026-02-25 20:03:21.688 DEBUG [src.node.waku_node] Stopping container with id 1627f23989de
2026-02-25 20:03:21.898 ERROR [src.node.docker_mananger] Max retries reached for container 041731b88958. Exiting log stream.
2026-02-25 20:03:22.322 DEBUG [src.node.waku_node] Container stopped.
2026-02-25 20:03:22.322 DEBUG [src.node.waku_node] Stopping container with id 9cefdec86283
2026-02-25 20:03:22.505 ERROR [src.node.docker_mananger] Max retries reached for container 80b108164ad8. Exiting log stream.
2026-02-25 20:03:22.943 DEBUG [src.node.waku_node] Container stopped.
2026-02-25 20:03:22.943 DEBUG [src.node.waku_node] Stopping container with id 679f0a452f73
2026-02-25 20:03:23.143 ERROR [src.node.docker_mananger] Max retries reached for container 1627f23989de. Exiting log stream.
2026-02-25 20:03:23.525 DEBUG [src.node.waku_node] Container stopped.
2026-02-25 20:03:23.529 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-02-25 20:03:23.560 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-25 20:03:23.564 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-25 20:03:23.592 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-25 20:03:23.646 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-25 20:03:23.654 DEBUG [src.node.docker_mananger] No errors found in the waku logs.