179 lines
34 KiB
Plaintext

2026-03-08 04:31:31.260 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-08 04:31:31.261 DEBUG [tests.conftest] Running test: test_multiple_edge_service_nodes_communication with id: 2026-03-08_04-31-31__ab4f18e3-dd83-4e52-b6ee-733ab93118d3
2026-03-08 04:31:31.261 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-08 04:31:31.261 DEBUG [src.steps.filter] Running fixture setup: filter_setup
2026-03-08 04:31:31.262 DEBUG [src.steps.light_push] Running fixture setup: light_push_setup
2026-03-08 04:31:31.271 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:31:31.272 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-08_04-31-31__ab4f18e3-dd83-4e52-b6ee-733ab93118d3__wakuorg_nwaku:latest.log
2026-03-08 04:31:31.279 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:31:31.279 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-08_04-31-31__ab4f18e3-dd83-4e52-b6ee-733ab93118d3__wakuorg_nwaku:latest.log
2026-03-08 04:31:31.285 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:31:31.285 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-08_04-31-31__ab4f18e3-dd83-4e52-b6ee-733ab93118d3__wakuorg_nwaku:latest.log
2026-03-08 04:31:31.285 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-08 04:31:31.286 DEBUG [src.steps.store] Running fixture setup: store_setup
2026-03-08 04:31:31.292 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:31:31.292 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-08_04-31-31__ab4f18e3-dd83-4e52-b6ee-733ab93118d3__wakuorg_nwaku:latest.log
2026-03-08 04:31:31.298 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:31:31.298 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-08_04-31-31__ab4f18e3-dd83-4e52-b6ee-733ab93118d3__wakuorg_nwaku:latest.log
2026-03-08 04:31:31.304 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:31:31.304 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node6_2026-03-08_04-31-31__ab4f18e3-dd83-4e52-b6ee-733ab93118d3__wakuorg_nwaku:latest.log
2026-03-08 04:31:31.310 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:31:31.310 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node7_2026-03-08_04-31-31__ab4f18e3-dd83-4e52-b6ee-733ab93118d3__wakuorg_nwaku:latest.log
2026-03-08 04:31:31.316 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:31:31.316 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node8_2026-03-08_04-31-31__ab4f18e3-dd83-4e52-b6ee-733ab93118d3__wakuorg_nwaku:latest.log
2026-03-08 04:31:31.316 DEBUG [tests.e2e.test_e2e] Start 2 edges nodes and 3 service nodes
2026-03-08 04:31:31.316 DEBUG [src.node.waku_node] Starting Node...
2026-03-08 04:31:31.316 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-08 04:31:31.362 DEBUG [src.node.docker_mananger] Network waku created
2026-03-08 04:31:31.362 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.203.250
2026-03-08 04:31:31.362 DEBUG [src.node.docker_mananger] Generated ports ['29634', '29635', '29636', '29637', '29638']
2026-03-08 04:31:31.363 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-08 04:31:31.363 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-08 04:31:31.363 DEBUG [src.node.waku_node] Using volumes []
2026-03-08 04:31:31.363 DEBUG [src.node.docker_mananger] docker run -i -t -p 29634:29634 -p 29635:29635 -p 29636:29636 -p 29637:29637 -p 29638:29638 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=29636 --rest-port=29634 --tcp-port=29635 --discv5-udp-port=29637 --rest-address=0.0.0.0 --nat=extip:172.18.203.250 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=36d60021fabf0a3d0b407dac1bfb3d6a897aea45ffaecee2da097da9ee1dc4bd --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=29638 --metrics-logging=true --relay=true --store=true --lightpush=true
2026-03-08 04:31:34.718 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.203.250 waku d89262310b2d6c7401ca6210115ee156ac25c147db3b03d214b5fcf84f3cdde7
2026-03-08 04:31:34.752 DEBUG [src.node.docker_mananger] Container started with ID d89262310b2d. Setting up logs at ./log/docker/node6_2026-03-08_04-31-31__ab4f18e3-dd83-4e52-b6ee-733ab93118d3__wakuorg_nwaku:latest.log
2026-03-08 04:31:34.752 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 29634
2026-03-08 04:31:34.752 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-08 04:31:35.753 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:29634/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:31:35.756 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Legacy Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-08 04:31:35.756 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-08 04:31:35.757 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:29634/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:31:35.759 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.203.250/tcp/29635/p2p/16Uiu2HAkw4r9w2n2B7FKbTCKgY6QBYNL2d6SWRFtbXz2FvFB7ZAg","/ip4/172.18.203.250/tcp/29636/ws/p2p/16Uiu2HAkw4r9w2n2B7FKbTCKgY6QBYNL2d6SWRFtbXz2FvFB7ZAg"],"enrUri":"enr:-L24QEKLYV3UGa7bOyyt2P4kh8tvtrs3ajFmCq1dhYlpc87Mf2IAuJ3SjbCeA-OmoUSLTzP9DNoG1jRPuAfl2BYTBfUCgmlkgnY0gmlwhKwSy_qKbXVsdGlhZGRyc5YACASsEsv6BnPDAAoErBLL-gZzxN0DgnJzhQADAQAAiXNlY3AyNTZrMaECGGGuelWIFs7e5_cpj8tOBYv7XKXm7igqvwQBLbPlNkGDdGNwgnPDg3VkcIJzxYV3YWt1Mgs"}'
2026-03-08 04:31:35.759 INFO [src.node.waku_node] REST service is ready !!
2026-03-08 04:31:35.759 DEBUG [src.node.waku_node] Starting Node...
2026-03-08 04:31:35.760 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-08 04:31:35.761 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-08 04:31:35.761 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.41.168
2026-03-08 04:31:35.761 DEBUG [src.node.docker_mananger] Generated ports ['38439', '38440', '38441', '38442', '38443']
2026-03-08 04:31:35.762 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-08 04:31:35.762 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-08 04:31:35.762 DEBUG [src.node.waku_node] Using volumes []
2026-03-08 04:31:35.762 DEBUG [src.node.docker_mananger] docker run -i -t -p 38439:38439 -p 38440:38440 -p 38441:38441 -p 38442:38442 -p 38443:38443 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=38441 --rest-port=38439 --tcp-port=38440 --discv5-udp-port=38442 --rest-address=0.0.0.0 --nat=extip:172.18.41.168 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=41c6be2adc30fc0cadd45da9689ef44d9db5ce4975f8b9feb2ce5ec8ad6f4f9d --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=38443 --metrics-logging=true --relay=false --lightpushnode=/ip4/172.18.203.250/tcp/29635/p2p/16Uiu2HAkw4r9w2n2B7FKbTCKgY6QBYNL2d6SWRFtbXz2FvFB7ZAg --discv5-bootstrap-node=enr:-L24QEKLYV3UGa7bOyyt2P4kh8tvtrs3ajFmCq1dhYlpc87Mf2IAuJ3SjbCeA-OmoUSLTzP9DNoG1jRPuAfl2BYTBfUCgmlkgnY0gmlwhKwSy_qKbXVsdGlhZGRyc5YACASsEsv6BnPDAAoErBLL-gZzxN0DgnJzhQADAQAAiXNlY3AyNTZrMaECGGGuelWIFs7e5_cpj8tOBYv7XKXm7igqvwQBLbPlNkGDdGNwgnPDg3VkcIJzxYV3YWt1Mgs
2026-03-08 04:31:35.958 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.41.168 waku 3209e31b20e7c47eb041a98bec422f26bdc164a65628b29bcebbe956f461565e
2026-03-08 04:31:35.992 DEBUG [src.node.docker_mananger] Container started with ID 3209e31b20e7. Setting up logs at ./log/docker/node4_2026-03-08_04-31-31__ab4f18e3-dd83-4e52-b6ee-733ab93118d3__wakuorg_nwaku:latest.log
2026-03-08 04:31:35.992 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 38439
2026-03-08 04:31:35.992 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-08 04:31:36.993 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:38439/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:31:36.996 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-08 04:31:36.996 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-08 04:31:36.997 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:38439/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:31:36.999 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.41.168/tcp/38440/p2p/16Uiu2HAmP6CAyqwEY2B4d19UE1ajawSX6M974zrx9KQjwDL5bADf","/ip4/172.18.41.168/tcp/38441/ws/p2p/16Uiu2HAmP6CAyqwEY2B4d19UE1ajawSX6M974zrx9KQjwDL5bADf"],"enrUri":"enr:-L24QGWQ3TsWqisW-6bIPD_llRa96G9QWGjAhZ1NH1vcM1bTIp7126bDKDbG97ndEKe4AL_1r2R5P-opKFbW7e35fg0CgmlkgnY0gmlwhKwSKaiKbXVsdGlhZGRyc5YACASsEimoBpYoAAoErBIpqAaWKd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDmwnByDDkQTcOTv_8Qzylpi9Mws8qwgm3PN-CikwhG5KDdGNwgpYog3VkcIKWKoV3YWt1MgA"}'
2026-03-08 04:31:36.999 INFO [src.node.waku_node] REST service is ready !!
2026-03-08 04:31:36.999 DEBUG [src.node.waku_node] Starting Node...
2026-03-08 04:31:37.000 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-08 04:31:37.001 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-08 04:31:37.001 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.176.154
2026-03-08 04:31:37.001 DEBUG [src.node.docker_mananger] Generated ports ['4487', '4488', '4489', '4490', '4491']
2026-03-08 04:31:37.001 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-08 04:31:37.002 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-08 04:31:37.002 DEBUG [src.node.waku_node] Using volumes []
2026-03-08 04:31:37.002 DEBUG [src.node.docker_mananger] docker run -i -t -p 4487:4487 -p 4488:4488 -p 4489:4489 -p 4490:4490 -p 4491:4491 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=4489 --rest-port=4487 --tcp-port=4488 --discv5-udp-port=4490 --rest-address=0.0.0.0 --nat=extip:172.18.176.154 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=609e4493c1cebc07ea1431c24afcbb8dcd22edf6a2dadef9d55e466200fa5925 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=4491 --metrics-logging=true --relay=true --store=true --discv5-bootstrap-node=enr:-L24QEKLYV3UGa7bOyyt2P4kh8tvtrs3ajFmCq1dhYlpc87Mf2IAuJ3SjbCeA-OmoUSLTzP9DNoG1jRPuAfl2BYTBfUCgmlkgnY0gmlwhKwSy_qKbXVsdGlhZGRyc5YACASsEsv6BnPDAAoErBLL-gZzxN0DgnJzhQADAQAAiXNlY3AyNTZrMaECGGGuelWIFs7e5_cpj8tOBYv7XKXm7igqvwQBLbPlNkGDdGNwgnPDg3VkcIJzxYV3YWt1Mgs
2026-03-08 04:31:37.192 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.176.154 waku 16b885d356523374b77050b42debcf077d4728c18afc14959f85bb1433fa6271
2026-03-08 04:31:37.230 DEBUG [src.node.docker_mananger] Container started with ID 16b885d35652. Setting up logs at ./log/docker/node7_2026-03-08_04-31-31__ab4f18e3-dd83-4e52-b6ee-733ab93118d3__wakuorg_nwaku:latest.log
2026-03-08 04:31:37.230 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 4487
2026-03-08 04:31:37.231 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-08 04:31:38.231 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4487/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:31:38.234 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-08 04:31:38.234 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-08 04:31:38.235 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4487/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:31:38.237 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.176.154/tcp/4488/p2p/16Uiu2HAkyCu6gML5dfofu61aWVhuLnqi794eE5sJz7nSdu8Kd5mD","/ip4/172.18.176.154/tcp/4489/ws/p2p/16Uiu2HAkyCu6gML5dfofu61aWVhuLnqi794eE5sJz7nSdu8Kd5mD"],"enrUri":"enr:-L24QGjuiyFZieLyXzsyPxfveqrJoNsa1fLvmBa581j9SRTmfplEf-n7-U6VjmR-9jlV1oztKlpNJJD_OGEXONRbhMACgmlkgnY0gmlwhKwSsJqKbXVsdGlhZGRyc5YACASsErCaBhGIAAoErBKwmgYRid0DgnJzhQADAQAAiXNlY3AyNTZrMaECOCkH04Y4S-mcFPZZbfFuDQofzwY1ZrtfElrh7VcsiLSDdGNwghGIg3VkcIIRioV3YWt1MgM"}'
2026-03-08 04:31:38.237 INFO [src.node.waku_node] REST service is ready !!
2026-03-08 04:31:38.237 DEBUG [src.node.waku_node] Starting Node...
2026-03-08 04:31:38.238 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-08 04:31:38.239 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-08 04:31:38.239 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.189.6
2026-03-08 04:31:38.239 DEBUG [src.node.docker_mananger] Generated ports ['42529', '42530', '42531', '42532', '42533']
2026-03-08 04:31:38.240 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-08 04:31:38.240 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-08 04:31:38.240 DEBUG [src.node.waku_node] Using volumes []
2026-03-08 04:31:38.240 DEBUG [src.node.docker_mananger] docker run -i -t -p 42529:42529 -p 42530:42530 -p 42531:42531 -p 42532:42532 -p 42533:42533 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=42531 --rest-port=42529 --tcp-port=42530 --discv5-udp-port=42532 --rest-address=0.0.0.0 --nat=extip:172.18.189.6 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=684bbbf4adeb2cd110f44f4114acfcdc584da8ff5e2399ea496fff6efafcbfaa --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=42533 --metrics-logging=true --relay=true --filter=true --storenode=/ip4/172.18.176.154/tcp/4488/p2p/16Uiu2HAkyCu6gML5dfofu61aWVhuLnqi794eE5sJz7nSdu8Kd5mD --discv5-bootstrap-node=enr:-L24QGjuiyFZieLyXzsyPxfveqrJoNsa1fLvmBa581j9SRTmfplEf-n7-U6VjmR-9jlV1oztKlpNJJD_OGEXONRbhMACgmlkgnY0gmlwhKwSsJqKbXVsdGlhZGRyc5YACASsErCaBhGIAAoErBKwmgYRid0DgnJzhQADAQAAiXNlY3AyNTZrMaECOCkH04Y4S-mcFPZZbfFuDQofzwY1ZrtfElrh7VcsiLSDdGNwghGIg3VkcIIRioV3YWt1MgM
2026-03-08 04:31:38.472 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.189.6 waku aab928f2f0e9178e97f14395709206200bcea6c451a3fa57e60a9caf19dfcba4
2026-03-08 04:31:38.507 DEBUG [src.node.docker_mananger] Container started with ID aab928f2f0e9. Setting up logs at ./log/docker/node8_2026-03-08_04-31-31__ab4f18e3-dd83-4e52-b6ee-733ab93118d3__wakuorg_nwaku:latest.log
2026-03-08 04:31:38.507 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 42529
2026-03-08 04:31:38.508 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-08 04:31:39.510 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:42529/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:31:39.519 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-08 04:31:39.523 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-08 04:31:39.523 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:42529/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:31:39.539 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.189.6/tcp/42530/p2p/16Uiu2HAkyS77EHUcMeuRc8hjGPVSfJ2243qREz9Rw1yFQL48SrSt","/ip4/172.18.189.6/tcp/42531/ws/p2p/16Uiu2HAkyS77EHUcMeuRc8hjGPVSfJ2243qREz9Rw1yFQL48SrSt"],"enrUri":"enr:-L24QMwMFI2aUaCliR025ywZ8AJQvIpcgjsUj84hEK5_Qqy1SW_t3a-1tkGngz2so8xN9H8jb49-fQYkBZJC0kMh7o8CgmlkgnY0gmlwhKwSvQaKbXVsdGlhZGRyc5YACASsEr0GBqYiAAoErBK9BgamI90DgnJzhQADAQAAiXNlY3AyNTZrMaECO4spBzJOIX1GuyR21wrIFyu3R4pfLEEjBlkFxKTEqjmDdGNwgqYig3VkcIKmJIV3YWt1MgU"}'
2026-03-08 04:31:39.540 INFO [src.node.waku_node] REST service is ready !!
2026-03-08 04:31:39.542 DEBUG [src.node.waku_node] Starting Node...
2026-03-08 04:31:39.542 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-08 04:31:39.547 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-08 04:31:39.547 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.151.48
2026-03-08 04:31:39.548 DEBUG [src.node.docker_mananger] Generated ports ['47097', '47098', '47099', '47100', '47101']
2026-03-08 04:31:39.548 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-08 04:31:39.549 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-08 04:31:39.549 DEBUG [src.node.waku_node] Using volumes []
2026-03-08 04:31:39.550 DEBUG [src.node.docker_mananger] docker run -i -t -p 47097:47097 -p 47098:47098 -p 47099:47099 -p 47100:47100 -p 47101:47101 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=47099 --rest-port=47097 --tcp-port=47098 --discv5-udp-port=47100 --rest-address=0.0.0.0 --nat=extip:172.18.151.48 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=2ebd579a14a1dd9f9d3e3a76b9c989cd9e2def62cd770facc0d0bdb6a37acceb --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=47101 --metrics-logging=true --relay=false --filternode=/ip4/172.18.189.6/tcp/42530/p2p/16Uiu2HAkyS77EHUcMeuRc8hjGPVSfJ2243qREz9Rw1yFQL48SrSt --storenode=/ip4/172.18.176.154/tcp/4488/p2p/16Uiu2HAkyCu6gML5dfofu61aWVhuLnqi794eE5sJz7nSdu8Kd5mD --discv5-bootstrap-node=enr:-L24QGjuiyFZieLyXzsyPxfveqrJoNsa1fLvmBa581j9SRTmfplEf-n7-U6VjmR-9jlV1oztKlpNJJD_OGEXONRbhMACgmlkgnY0gmlwhKwSsJqKbXVsdGlhZGRyc5YACASsErCaBhGIAAoErBKwmgYRid0DgnJzhQADAQAAiXNlY3AyNTZrMaECOCkH04Y4S-mcFPZZbfFuDQofzwY1ZrtfElrh7VcsiLSDdGNwghGIg3VkcIIRioV3YWt1MgM
2026-03-08 04:31:39.748 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.151.48 waku f9c6f759de22ba69a968859c6e8878d7e4830bf59eaa107fbae399a673b6604e
2026-03-08 04:31:39.786 DEBUG [src.node.docker_mananger] Container started with ID f9c6f759de22. Setting up logs at ./log/docker/node5_2026-03-08_04-31-31__ab4f18e3-dd83-4e52-b6ee-733ab93118d3__wakuorg_nwaku:latest.log
2026-03-08 04:31:39.787 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 47097
2026-03-08 04:31:39.787 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-08 04:31:40.787 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:47097/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:31:40.790 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-08 04:31:40.790 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-08 04:31:40.791 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:47097/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:31:40.793 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.151.48/tcp/47098/p2p/16Uiu2HAm9PtGDV6aQa5hCLwnh37aou7YbfCSpjxfTXq6A7Uugmyv","/ip4/172.18.151.48/tcp/47099/ws/p2p/16Uiu2HAm9PtGDV6aQa5hCLwnh37aou7YbfCSpjxfTXq6A7Uugmyv"],"enrUri":"enr:-L24QG1b9Ap_uTdrQNhko16z-do8cODSpRlux7MSY-OQ9CHNWYYEymvwRjq5nFkcJRIslXKbI655YpWhspZrJx4N-E8CgmlkgnY0gmlwhKwSlzCKbXVsdGlhZGRyc5YACASsEpcwBrf6AAoErBKXMAa3-90DgnJzhQADAQAAiXNlY3AyNTZrMaECz45YDRgbRl4awuy3K1GKtF18ZqU0A4MMjoycMCizK02DdGNwgrf6g3VkcIK3_IV3YWt1MgA"}'
2026-03-08 04:31:40.793 INFO [src.node.waku_node] REST service is ready !!
2026-03-08 04:31:40.793 DEBUG [tests.e2e.test_e2e] Connect 3 service nodes to relay subscriptions
2026-03-08 04:31:40.794 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:29634/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-08 04:31:40.797 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:31:40.798 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:4487/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-08 04:31:40.806 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:31:40.806 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:42529/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-08 04:31:40.813 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:31:40.814 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:29634/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:31:40.818 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.176.154/tcp/46520/p2p/16Uiu2HAkyCu6gML5dfofu61aWVhuLnqi794eE5sJz7nSdu8Kd5mD","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-08 04:31:40.818 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4487/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:31:40.820 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.203.250/tcp/29635/p2p/16Uiu2HAkw4r9w2n2B7FKbTCKgY6QBYNL2d6SWRFtbXz2FvFB7ZAg","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/lightpush/3.0.0","/vac/waku/lightpush/2.0.0-beta1","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"},{"multiaddr":"/ip4/172.17.0.5/tcp/60396/p2p/16Uiu2HAkyS77EHUcMeuRc8hjGPVSfJ2243qREz9Rw1yFQL48SrSt","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-subscribe/2.0.0-beta1","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"}]'
2026-03-08 04:31:40.820 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:42529/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:31:40.823 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.176.154/tcp/4488/p2p/16Uiu2HAkyCu6gML5dfofu61aWVhuLnqi794eE5sJz7nSdu8Kd5mD","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-08 04:31:40.823 DEBUG [src.libs.common] Sleeping for 30 seconds
2026-03-08 04:32:10.823 DEBUG [tests.e2e.test_e2e] Edge node2 makes filter subscription to pubsubtopic /waku/2/rs/3/1 and content topic /test/1/waku-filter/proto
2026-03-08 04:32:10.824 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:47097/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "1", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-08 04:32:10.867 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"1","statusDesc":"OK"}'
2026-03-08 04:32:10.868 DEBUG [tests.e2e.test_e2e] Check if service node1 receives message sent by edge node1
2026-03-08 04:32:10.869 DEBUG [src.steps.light_push] Lightpushing message
2026-03-08 04:32:10.871 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:38439/lightpush/v1/message" -H "Content-Type: application/json" -d '{"pubsubTopic": "/waku/2/rs/3/1", "message": {"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": 1772944330868983298}}'
2026-03-08 04:32:10.932 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:32:10.934 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-08 04:32:11.036 DEBUG [src.steps.light_push] Checking that peer NODE_1:wakuorg/nwaku:latest can find the lightpushed message
2026-03-08 04:32:11.036 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:29634/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:32:11.039 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1772944330868983298,"ephemeral":false,"proof":""}]'
2026-03-08 04:32:11.041 DEBUG [tests.e2e.test_e2e] Check if edge node2 can query stored message
2026-03-08 04:32:11.041 DEBUG [src.steps.store] Checking that peer wakuorg/nwaku:latest can find the stored messages
2026-03-08 04:32:11.042 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:47097/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F1&pageSize=50&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:32:11.076 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"3ec0d03e0299825d7613","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0x28770ffee298a9a182cf5641a298e9f48ecac2ab6fd29fe7c2cf27bf15c954f1"}]}'
2026-03-08 04:32:11.077 DEBUG [src.steps.store] messages length is 1
2026-03-08 04:32:11.080 DEBUG [tests.e2e.test_e2e] Check if service node3 can query stored message
2026-03-08 04:32:11.081 DEBUG [src.steps.store] Checking that peer wakuorg/nwaku:latest can find the stored messages
2026-03-08 04:32:11.081 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:42529/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F1&pageSize=50&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:32:11.094 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"f413d6448545b1e859bd","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0x28770ffee298a9a182cf5641a298e9f48ecac2ab6fd29fe7c2cf27bf15c954f1"}]}'
2026-03-08 04:32:11.094 DEBUG [src.steps.store] messages length is 1
2026-03-08 04:32:11.094 DEBUG [tests.e2e.test_e2e] Check if edge node2 can get sent message using filter get request
2026-03-08 04:32:11.095 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:47097/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:32:11.097 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1772944330868983298,"ephemeral":false}]'
2026-03-08 04:32:11.099 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-08 04:32:11.100 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-08 04:32:11.101 DEBUG [src.node.waku_node] Stopping container with id d89262310b2d
2026-03-08 04:32:11.679 DEBUG [src.node.waku_node] Container stopped.
2026-03-08 04:32:11.682 DEBUG [src.node.waku_node] Stopping container with id 3209e31b20e7
2026-03-08 04:32:12.235 DEBUG [src.node.waku_node] Container stopped.
2026-03-08 04:32:12.237 DEBUG [src.node.waku_node] Stopping container with id 16b885d35652
2026-03-08 04:32:12.491 ERROR [src.node.docker_mananger] Max retries reached for container d89262310b2d. Exiting log stream.
2026-03-08 04:32:12.780 DEBUG [src.node.waku_node] Container stopped.
2026-03-08 04:32:12.780 DEBUG [src.node.waku_node] Stopping container with id aab928f2f0e9
2026-03-08 04:32:13.049 ERROR [src.node.docker_mananger] Max retries reached for container 3209e31b20e7. Exiting log stream.
2026-03-08 04:32:13.379 DEBUG [src.node.waku_node] Container stopped.
2026-03-08 04:32:13.381 DEBUG [src.node.waku_node] Stopping container with id f9c6f759de22
2026-03-08 04:32:13.597 ERROR [src.node.docker_mananger] Max retries reached for container 16b885d35652. Exiting log stream.
2026-03-08 04:32:13.929 DEBUG [src.node.waku_node] Container stopped.
2026-03-08 04:32:13.931 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-08 04:32:13.959 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-08 04:32:13.963 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-08 04:32:13.991 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-08 04:32:14.040 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-08 04:32:14.048 DEBUG [src.node.docker_mananger] No errors found in the waku logs.