179 lines
34 KiB
Plaintext

2026-03-05 04:32:22.958 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-05 04:32:22.959 DEBUG [tests.conftest] Running test: test_multiple_edge_service_nodes_communication with id: 2026-03-05_04-32-22__e8104c2b-a654-4de7-9e41-ab567e4f18b2
2026-03-05 04:32:22.959 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-05 04:32:22.959 DEBUG [src.steps.filter] Running fixture setup: filter_setup
2026-03-05 04:32:22.960 DEBUG [src.steps.light_push] Running fixture setup: light_push_setup
2026-03-05 04:32:22.969 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-05 04:32:22.969 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-05_04-32-22__e8104c2b-a654-4de7-9e41-ab567e4f18b2__wakuorg_nwaku:latest.log
2026-03-05 04:32:22.975 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-05 04:32:22.975 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-05_04-32-22__e8104c2b-a654-4de7-9e41-ab567e4f18b2__wakuorg_nwaku:latest.log
2026-03-05 04:32:22.981 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-05 04:32:22.981 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-05_04-32-22__e8104c2b-a654-4de7-9e41-ab567e4f18b2__wakuorg_nwaku:latest.log
2026-03-05 04:32:22.982 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-05 04:32:22.982 DEBUG [src.steps.store] Running fixture setup: store_setup
2026-03-05 04:32:22.988 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-05 04:32:22.988 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-05_04-32-22__e8104c2b-a654-4de7-9e41-ab567e4f18b2__wakuorg_nwaku:latest.log
2026-03-05 04:32:22.994 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-05 04:32:22.994 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-05_04-32-22__e8104c2b-a654-4de7-9e41-ab567e4f18b2__wakuorg_nwaku:latest.log
2026-03-05 04:32:23.000 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-05 04:32:23.000 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node6_2026-03-05_04-32-22__e8104c2b-a654-4de7-9e41-ab567e4f18b2__wakuorg_nwaku:latest.log
2026-03-05 04:32:23.005 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-05 04:32:23.006 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node7_2026-03-05_04-32-22__e8104c2b-a654-4de7-9e41-ab567e4f18b2__wakuorg_nwaku:latest.log
2026-03-05 04:32:23.011 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-05 04:32:23.011 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node8_2026-03-05_04-32-22__e8104c2b-a654-4de7-9e41-ab567e4f18b2__wakuorg_nwaku:latest.log
2026-03-05 04:32:23.011 DEBUG [tests.e2e.test_e2e] Start 2 edges nodes and 3 service nodes
2026-03-05 04:32:23.011 DEBUG [src.node.waku_node] Starting Node...
2026-03-05 04:32:23.012 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-05 04:32:23.056 DEBUG [src.node.docker_mananger] Network waku created
2026-03-05 04:32:23.056 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.243.166
2026-03-05 04:32:23.056 DEBUG [src.node.docker_mananger] Generated ports ['61429', '61430', '61431', '61432', '61433']
2026-03-05 04:32:23.057 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-05 04:32:23.057 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-05 04:32:23.057 DEBUG [src.node.waku_node] Using volumes []
2026-03-05 04:32:23.057 DEBUG [src.node.docker_mananger] docker run -i -t -p 61429:61429 -p 61430:61430 -p 61431:61431 -p 61432:61432 -p 61433:61433 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=61431 --rest-port=61429 --tcp-port=61430 --discv5-udp-port=61432 --rest-address=0.0.0.0 --nat=extip:172.18.243.166 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=acdd8fe7c6cb1a4c3cedab1ece9302e9e263a95aff5f5b20ada62ad6c24dc644 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=61433 --metrics-logging=true --relay=true --store=true --lightpush=true
2026-03-05 04:32:26.428 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.243.166 waku 422a4d8bba4e08e333dde4e87da69467cd04ecf978325eaea90ca693385578c6
2026-03-05 04:32:26.462 DEBUG [src.node.docker_mananger] Container started with ID 422a4d8bba4e. Setting up logs at ./log/docker/node6_2026-03-05_04-32-22__e8104c2b-a654-4de7-9e41-ab567e4f18b2__wakuorg_nwaku:latest.log
2026-03-05 04:32:26.463 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 61429
2026-03-05 04:32:26.463 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-05 04:32:27.463 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:61429/health" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:32:27.467 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Legacy Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-05 04:32:27.467 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-05 04:32:27.467 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:61429/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:32:27.470 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.243.166/tcp/61430/p2p/16Uiu2HAmJ31Yw9jmFC9bqpxxjj4Fp4Py4h3UCK9tFf5J4CvBmun8","/ip4/172.18.243.166/tcp/61431/ws/p2p/16Uiu2HAmJ31Yw9jmFC9bqpxxjj4Fp4Py4h3UCK9tFf5J4CvBmun8"],"enrUri":"enr:-L24QOSxJPWAgXJKby0B3iYTTEVRqXI5Mssb5q0-ShM4YDmTFC4vLUS9BS4p5DlKi5zAFUfR60EmmI0UwaCharsUvHECgmlkgnY0gmlwhKwS86aKbXVsdGlhZGRyc5YACASsEvOmBu_2AAoErBLzpgbv990DgnJzhQADAQAAiXNlY3AyNTZrMaEDT-6R9QA8BkD7hW-25eVdZ9JKa3x8Cl5gyplDIvk5hmmDdGNwgu_2g3VkcILv-IV3YWt1Mgs"}'
2026-03-05 04:32:27.470 INFO [src.node.waku_node] REST service is ready !!
2026-03-05 04:32:27.470 DEBUG [src.node.waku_node] Starting Node...
2026-03-05 04:32:27.470 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-05 04:32:27.472 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-05 04:32:27.472 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.154.10
2026-03-05 04:32:27.472 DEBUG [src.node.docker_mananger] Generated ports ['13216', '13217', '13218', '13219', '13220']
2026-03-05 04:32:27.472 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-05 04:32:27.472 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-05 04:32:27.473 DEBUG [src.node.waku_node] Using volumes []
2026-03-05 04:32:27.473 DEBUG [src.node.docker_mananger] docker run -i -t -p 13216:13216 -p 13217:13217 -p 13218:13218 -p 13219:13219 -p 13220:13220 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=13218 --rest-port=13216 --tcp-port=13217 --discv5-udp-port=13219 --rest-address=0.0.0.0 --nat=extip:172.18.154.10 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=b9a0e95414c96008143ed201ffbbb6ce0e2de7b840964dcf1b026aebffccc19f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=13220 --metrics-logging=true --relay=false --lightpushnode=/ip4/172.18.243.166/tcp/61430/p2p/16Uiu2HAmJ31Yw9jmFC9bqpxxjj4Fp4Py4h3UCK9tFf5J4CvBmun8 --discv5-bootstrap-node=enr:-L24QOSxJPWAgXJKby0B3iYTTEVRqXI5Mssb5q0-ShM4YDmTFC4vLUS9BS4p5DlKi5zAFUfR60EmmI0UwaCharsUvHECgmlkgnY0gmlwhKwS86aKbXVsdGlhZGRyc5YACASsEvOmBu_2AAoErBLzpgbv990DgnJzhQADAQAAiXNlY3AyNTZrMaEDT-6R9QA8BkD7hW-25eVdZ9JKa3x8Cl5gyplDIvk5hmmDdGNwgu_2g3VkcILv-IV3YWt1Mgs
2026-03-05 04:32:27.666 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.154.10 waku afb8d9255ae3aad3281f9cb8c898f9f68bc5fdcc97925e438e76d107b5c1217b
2026-03-05 04:32:27.699 DEBUG [src.node.docker_mananger] Container started with ID afb8d9255ae3. Setting up logs at ./log/docker/node4_2026-03-05_04-32-22__e8104c2b-a654-4de7-9e41-ab567e4f18b2__wakuorg_nwaku:latest.log
2026-03-05 04:32:27.699 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 13216
2026-03-05 04:32:27.699 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-05 04:32:28.701 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:13216/health" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:32:28.704 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-05 04:32:28.704 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-05 04:32:28.704 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:13216/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:32:28.706 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.154.10/tcp/13217/p2p/16Uiu2HAmSJ3h6vn8AZCAJKNxPVtxB5Rp2MePh8QqP5Wfy6KbpoXo","/ip4/172.18.154.10/tcp/13218/ws/p2p/16Uiu2HAmSJ3h6vn8AZCAJKNxPVtxB5Rp2MePh8QqP5Wfy6KbpoXo"],"enrUri":"enr:-L24QCajJKQCri6WZR3bjrDMewKLZgszqxmb9kbM_4CX3UVTcPjQPKezyYO-uP7bao9w7kqI0agcMHgl-ikr8E9x5HsCgmlkgnY0gmlwhKwSmgqKbXVsdGlhZGRyc5YACASsEpoKBjOhAAoErBKaCgYzot0DgnJzhQADAQAAiXNlY3AyNTZrMaEDyqYyxqyRZFKqNnuy4zz1uN0yAbNC3WFKtCiMBdn09WqDdGNwgjOhg3VkcIIzo4V3YWt1MgA"}'
2026-03-05 04:32:28.707 INFO [src.node.waku_node] REST service is ready !!
2026-03-05 04:32:28.707 DEBUG [src.node.waku_node] Starting Node...
2026-03-05 04:32:28.707 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-05 04:32:28.708 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-05 04:32:28.708 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.18.146
2026-03-05 04:32:28.709 DEBUG [src.node.docker_mananger] Generated ports ['9349', '9350', '9351', '9352', '9353']
2026-03-05 04:32:28.709 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-05 04:32:28.709 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-05 04:32:28.709 DEBUG [src.node.waku_node] Using volumes []
2026-03-05 04:32:28.709 DEBUG [src.node.docker_mananger] docker run -i -t -p 9349:9349 -p 9350:9350 -p 9351:9351 -p 9352:9352 -p 9353:9353 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=9351 --rest-port=9349 --tcp-port=9350 --discv5-udp-port=9352 --rest-address=0.0.0.0 --nat=extip:172.18.18.146 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=341d81ac0bb2bc4ab8cec75db5399fb8b2d4cdc2e2b8a4d6fc39ccfdbdd4626e --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=9353 --metrics-logging=true --relay=true --store=true --discv5-bootstrap-node=enr:-L24QOSxJPWAgXJKby0B3iYTTEVRqXI5Mssb5q0-ShM4YDmTFC4vLUS9BS4p5DlKi5zAFUfR60EmmI0UwaCharsUvHECgmlkgnY0gmlwhKwS86aKbXVsdGlhZGRyc5YACASsEvOmBu_2AAoErBLzpgbv990DgnJzhQADAQAAiXNlY3AyNTZrMaEDT-6R9QA8BkD7hW-25eVdZ9JKa3x8Cl5gyplDIvk5hmmDdGNwgu_2g3VkcILv-IV3YWt1Mgs
2026-03-05 04:32:28.902 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.18.146 waku 1f4eafd4f35d8d5feaf36606d63655a9f41a70147550603e4d8d62d6d3d5d082
2026-03-05 04:32:28.936 DEBUG [src.node.docker_mananger] Container started with ID 1f4eafd4f35d. Setting up logs at ./log/docker/node7_2026-03-05_04-32-22__e8104c2b-a654-4de7-9e41-ab567e4f18b2__wakuorg_nwaku:latest.log
2026-03-05 04:32:28.936 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 9349
2026-03-05 04:32:28.936 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-05 04:32:29.937 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:9349/health" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:32:29.940 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-05 04:32:29.940 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-05 04:32:29.940 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:9349/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:32:29.943 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.18.146/tcp/9350/p2p/16Uiu2HAm4VyCEsJC49Xc3MqwfRujX8pwiyyGYS7mvDMfzyAC3KBb","/ip4/172.18.18.146/tcp/9351/ws/p2p/16Uiu2HAm4VyCEsJC49Xc3MqwfRujX8pwiyyGYS7mvDMfzyAC3KBb"],"enrUri":"enr:-L24QEexL1bqL3SgiL2bL4enoNomIXC2EU37jLgkPCCKqc6mYhkdgq9QEshkQOXdnX9O0d6SBsVl9-hO968fjEXTZeYCgmlkgnY0gmlwhKwSEpKKbXVsdGlhZGRyc5YACASsEhKSBiSGAAoErBISkgYkh90DgnJzhQADAQAAiXNlY3AyNTZrMaEChtL4XW-6PmBUySf8KWDsDv7gJjlIBP1dDYnw4wB_MA6DdGNwgiSGg3VkcIIkiIV3YWt1MgM"}'
2026-03-05 04:32:29.943 INFO [src.node.waku_node] REST service is ready !!
2026-03-05 04:32:29.943 DEBUG [src.node.waku_node] Starting Node...
2026-03-05 04:32:29.943 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-05 04:32:29.945 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-05 04:32:29.945 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.57.195
2026-03-05 04:32:29.945 DEBUG [src.node.docker_mananger] Generated ports ['51213', '51214', '51215', '51216', '51217']
2026-03-05 04:32:29.945 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-05 04:32:29.945 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-05 04:32:29.945 DEBUG [src.node.waku_node] Using volumes []
2026-03-05 04:32:29.946 DEBUG [src.node.docker_mananger] docker run -i -t -p 51213:51213 -p 51214:51214 -p 51215:51215 -p 51216:51216 -p 51217:51217 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=51215 --rest-port=51213 --tcp-port=51214 --discv5-udp-port=51216 --rest-address=0.0.0.0 --nat=extip:172.18.57.195 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=cad3cd870a2b2ce18ecac999bac88dfafed7a9da4024b9b86e75fefadaed1bb0 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=51217 --metrics-logging=true --relay=true --filter=true --storenode=/ip4/172.18.18.146/tcp/9350/p2p/16Uiu2HAm4VyCEsJC49Xc3MqwfRujX8pwiyyGYS7mvDMfzyAC3KBb --discv5-bootstrap-node=enr:-L24QEexL1bqL3SgiL2bL4enoNomIXC2EU37jLgkPCCKqc6mYhkdgq9QEshkQOXdnX9O0d6SBsVl9-hO968fjEXTZeYCgmlkgnY0gmlwhKwSEpKKbXVsdGlhZGRyc5YACASsEhKSBiSGAAoErBISkgYkh90DgnJzhQADAQAAiXNlY3AyNTZrMaEChtL4XW-6PmBUySf8KWDsDv7gJjlIBP1dDYnw4wB_MA6DdGNwgiSGg3VkcIIkiIV3YWt1MgM
2026-03-05 04:32:30.175 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.57.195 waku 745eb4bd9b8f721d20b6d9ec4dd610bd0d0d2c60a0a2da9f1230a7452e3df710
2026-03-05 04:32:30.210 DEBUG [src.node.docker_mananger] Container started with ID 745eb4bd9b8f. Setting up logs at ./log/docker/node8_2026-03-05_04-32-22__e8104c2b-a654-4de7-9e41-ab567e4f18b2__wakuorg_nwaku:latest.log
2026-03-05 04:32:30.211 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 51213
2026-03-05 04:32:30.211 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-05 04:32:31.212 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51213/health" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:32:31.215 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-05 04:32:31.215 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-05 04:32:31.215 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51213/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:32:31.220 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.57.195/tcp/51214/p2p/16Uiu2HAmLKn2SZZiUrbkAP6PHKu4JV4c9vWBwf8Vvb1o1vzHLnWk","/ip4/172.18.57.195/tcp/51215/ws/p2p/16Uiu2HAmLKn2SZZiUrbkAP6PHKu4JV4c9vWBwf8Vvb1o1vzHLnWk"],"enrUri":"enr:-L24QFnUvjlc4yw_K4PWCqJZIprhCtn0O0jmSIW4pYKjinTpZdquTkUKUR5qsiJDMQR1cQDj3XEiWNzM8Cq1A8OiTCECgmlkgnY0gmlwhKwSOcOKbXVsdGlhZGRyc5YACASsEjnDBsgOAAoErBI5wwbID90DgnJzhQADAQAAiXNlY3AyNTZrMaEDcfGGLpxwkYw_lJ_N5yI1CJgIBSug-_J_bsq6jGg8MemDdGNwgsgOg3VkcILIEIV3YWt1MgU"}'
2026-03-05 04:32:31.220 INFO [src.node.waku_node] REST service is ready !!
2026-03-05 04:32:31.221 DEBUG [src.node.waku_node] Starting Node...
2026-03-05 04:32:31.221 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-05 04:32:31.227 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-05 04:32:31.228 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.90.243
2026-03-05 04:32:31.228 DEBUG [src.node.docker_mananger] Generated ports ['26374', '26375', '26376', '26377', '26378']
2026-03-05 04:32:31.228 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-05 04:32:31.228 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-05 04:32:31.228 DEBUG [src.node.waku_node] Using volumes []
2026-03-05 04:32:31.229 DEBUG [src.node.docker_mananger] docker run -i -t -p 26374:26374 -p 26375:26375 -p 26376:26376 -p 26377:26377 -p 26378:26378 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=26376 --rest-port=26374 --tcp-port=26375 --discv5-udp-port=26377 --rest-address=0.0.0.0 --nat=extip:172.18.90.243 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=4a1cf7b7cddbf9fadda2ddf3d395cb7eaf4b44a1daf819e5cc24b563ebc5a807 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=26378 --metrics-logging=true --relay=false --filternode=/ip4/172.18.57.195/tcp/51214/p2p/16Uiu2HAmLKn2SZZiUrbkAP6PHKu4JV4c9vWBwf8Vvb1o1vzHLnWk --storenode=/ip4/172.18.18.146/tcp/9350/p2p/16Uiu2HAm4VyCEsJC49Xc3MqwfRujX8pwiyyGYS7mvDMfzyAC3KBb --discv5-bootstrap-node=enr:-L24QEexL1bqL3SgiL2bL4enoNomIXC2EU37jLgkPCCKqc6mYhkdgq9QEshkQOXdnX9O0d6SBsVl9-hO968fjEXTZeYCgmlkgnY0gmlwhKwSEpKKbXVsdGlhZGRyc5YACASsEhKSBiSGAAoErBISkgYkh90DgnJzhQADAQAAiXNlY3AyNTZrMaEChtL4XW-6PmBUySf8KWDsDv7gJjlIBP1dDYnw4wB_MA6DdGNwgiSGg3VkcIIkiIV3YWt1MgM
2026-03-05 04:32:31.444 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.90.243 waku 781669c3105a5787b04e5341de9f28dc8de7337a246f934f0b563460230bd06c
2026-03-05 04:32:31.476 DEBUG [src.node.docker_mananger] Container started with ID 781669c3105a. Setting up logs at ./log/docker/node5_2026-03-05_04-32-22__e8104c2b-a654-4de7-9e41-ab567e4f18b2__wakuorg_nwaku:latest.log
2026-03-05 04:32:31.477 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 26374
2026-03-05 04:32:31.477 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-05 04:32:32.477 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26374/health" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:32:32.481 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-05 04:32:32.481 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-05 04:32:32.481 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26374/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:32:32.484 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.90.243/tcp/26375/p2p/16Uiu2HAmDbSZiAWfQ2YWuFZc3D14w6KYpWoCPPL7MHhswGXegTK6","/ip4/172.18.90.243/tcp/26376/ws/p2p/16Uiu2HAmDbSZiAWfQ2YWuFZc3D14w6KYpWoCPPL7MHhswGXegTK6"],"enrUri":"enr:-L24QFLXjbDeGjMI7atfJV3BVyQkxE7jk4lHsM39HhUfdigmW6QUTE_NU9G1vavj-O9dymv4HH0MiDp-5yW6p_OkoLMCgmlkgnY0gmlwhKwSWvOKbXVsdGlhZGRyc5YACASsElrzBmcHAAoErBJa8wZnCN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDDfMBDjjFqfD2pfUlIAVl1-a6dj_lIzubaAr3pT0bmKmDdGNwgmcHg3VkcIJnCYV3YWt1MgA"}'
2026-03-05 04:32:32.484 INFO [src.node.waku_node] REST service is ready !!
2026-03-05 04:32:32.484 DEBUG [tests.e2e.test_e2e] Connect 3 service nodes to relay subscriptions
2026-03-05 04:32:32.485 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:61429/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-05 04:32:32.488 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 04:32:32.488 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:9349/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-05 04:32:32.494 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 04:32:32.497 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:51213/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-05 04:32:32.505 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 04:32:32.506 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:61429/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:32:32.508 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.18.146/tcp/46654/p2p/16Uiu2HAm4VyCEsJC49Xc3MqwfRujX8pwiyyGYS7mvDMfzyAC3KBb","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-05 04:32:32.508 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:9349/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:32:32.511 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.243.166/tcp/61430/p2p/16Uiu2HAmJ31Yw9jmFC9bqpxxjj4Fp4Py4h3UCK9tFf5J4CvBmun8","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/lightpush/3.0.0","/vac/waku/lightpush/2.0.0-beta1","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"},{"multiaddr":"/ip4/172.17.0.5/tcp/37714/p2p/16Uiu2HAmLKn2SZZiUrbkAP6PHKu4JV4c9vWBwf8Vvb1o1vzHLnWk","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-subscribe/2.0.0-beta1","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"}]'
2026-03-05 04:32:32.511 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51213/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:32:32.514 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.18.146/tcp/9350/p2p/16Uiu2HAm4VyCEsJC49Xc3MqwfRujX8pwiyyGYS7mvDMfzyAC3KBb","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-05 04:32:32.514 DEBUG [src.libs.common] Sleeping for 30 seconds
2026-03-05 04:33:02.514 DEBUG [tests.e2e.test_e2e] Edge node2 makes filter subscription to pubsubtopic /waku/2/rs/3/1 and content topic /test/1/waku-filter/proto
2026-03-05 04:33:02.514 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:26374/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "1", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-05 04:33:02.562 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"1","statusDesc":"OK"}'
2026-03-05 04:33:02.563 DEBUG [tests.e2e.test_e2e] Check if service node1 receives message sent by edge node1
2026-03-05 04:33:02.564 DEBUG [src.steps.light_push] Lightpushing message
2026-03-05 04:33:02.565 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:13216/lightpush/v1/message" -H "Content-Type: application/json" -d '{"pubsubTopic": "/waku/2/rs/3/1", "message": {"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": 1772685182564394059}}'
2026-03-05 04:33:02.621 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 04:33:02.624 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-05 04:33:02.726 DEBUG [src.steps.light_push] Checking that peer NODE_1:wakuorg/nwaku:latest can find the lightpushed message
2026-03-05 04:33:02.726 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:61429/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:33:02.729 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1772685182564394059,"ephemeral":false,"proof":""}]'
2026-03-05 04:33:02.731 DEBUG [tests.e2e.test_e2e] Check if edge node2 can query stored message
2026-03-05 04:33:02.731 DEBUG [src.steps.store] Checking that peer wakuorg/nwaku:latest can find the stored messages
2026-03-05 04:33:02.731 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26374/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F1&pageSize=50&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:33:02.765 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"df6175293ada5082349e","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0x7d4219e320ef6076e91cebf9a44fa075f2fe04708f28235aaf0042aad79ef1ca"}]}'
2026-03-05 04:33:02.766 DEBUG [src.steps.store] messages length is 1
2026-03-05 04:33:02.767 DEBUG [tests.e2e.test_e2e] Check if service node3 can query stored message
2026-03-05 04:33:02.769 DEBUG [src.steps.store] Checking that peer wakuorg/nwaku:latest can find the stored messages
2026-03-05 04:33:02.769 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51213/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F1&pageSize=50&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:33:02.780 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"1272dfbfad7c6e646eb0","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0x7d4219e320ef6076e91cebf9a44fa075f2fe04708f28235aaf0042aad79ef1ca"}]}'
2026-03-05 04:33:02.781 DEBUG [src.steps.store] messages length is 1
2026-03-05 04:33:02.781 DEBUG [tests.e2e.test_e2e] Check if edge node2 can get sent message using filter get request
2026-03-05 04:33:02.781 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26374/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-05 04:33:02.783 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1772685182564394059,"ephemeral":false}]'
2026-03-05 04:33:02.786 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-05 04:33:02.787 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-05 04:33:02.787 DEBUG [src.node.waku_node] Stopping container with id 422a4d8bba4e
2026-03-05 04:33:03.351 DEBUG [src.node.waku_node] Container stopped.
2026-03-05 04:33:03.352 DEBUG [src.node.waku_node] Stopping container with id afb8d9255ae3
2026-03-05 04:33:03.916 DEBUG [src.node.waku_node] Container stopped.
2026-03-05 04:33:03.916 DEBUG [src.node.waku_node] Stopping container with id 1f4eafd4f35d
2026-03-05 04:33:04.167 ERROR [src.node.docker_mananger] Max retries reached for container 422a4d8bba4e. Exiting log stream.
2026-03-05 04:33:04.463 DEBUG [src.node.waku_node] Container stopped.
2026-03-05 04:33:04.463 DEBUG [src.node.waku_node] Stopping container with id 745eb4bd9b8f
2026-03-05 04:33:04.729 ERROR [src.node.docker_mananger] Max retries reached for container afb8d9255ae3. Exiting log stream.
2026-03-05 04:33:04.999 DEBUG [src.node.waku_node] Container stopped.
2026-03-05 04:33:05.000 DEBUG [src.node.waku_node] Stopping container with id 781669c3105a
2026-03-05 04:33:05.275 ERROR [src.node.docker_mananger] Max retries reached for container 1f4eafd4f35d. Exiting log stream.
2026-03-05 04:33:05.534 DEBUG [src.node.waku_node] Container stopped.
2026-03-05 04:33:05.536 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-05 04:33:05.566 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-05 04:33:05.571 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-05 04:33:05.599 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-05 04:33:05.624 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-05 04:33:05.634 DEBUG [src.node.docker_mananger] No errors found in the waku logs.