180 lines
34 KiB
Plaintext

2026-03-21 04:38:59.546 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-21 04:38:59.547 DEBUG [tests.conftest] Running test: test_filter_get_message_after_one_peer_was_stopped with id: 2026-03-21_04-38-59__de517c56-5511-424c-9b74-3b8013800c03
2026-03-21 04:38:59.547 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-21 04:38:59.547 DEBUG [src.steps.filter] Running fixture setup: filter_setup
2026-03-21 04:38:59.547 DEBUG [src.steps.filter] Running fixture setup: setup_main_relay_node
2026-03-21 04:38:59.554 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-21 04:38:59.554 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-21_04-38-59__de517c56-5511-424c-9b74-3b8013800c03__wakuorg_nwaku:latest.log
2026-03-21 04:38:59.555 DEBUG [src.node.waku_node] Starting Node...
2026-03-21 04:38:59.555 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-21 04:38:59.556 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-21 04:38:59.556 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.150.11
2026-03-21 04:38:59.556 DEBUG [src.node.docker_mananger] Generated ports ['33449', '33450', '33451', '33452', '33453']
2026-03-21 04:38:59.557 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-21 04:38:59.557 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-21 04:38:59.557 DEBUG [src.node.waku_node] Using volumes []
2026-03-21 04:38:59.557 DEBUG [src.node.docker_mananger] docker run -i -t -p 33449:33449 -p 33450:33450 -p 33451:33451 -p 33452:33452 -p 33453:33453 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=33451 --rest-port=33449 --tcp-port=33450 --discv5-udp-port=33452 --rest-address=0.0.0.0 --nat=extip:172.18.150.11 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=ed4c68a7f4c1863d0c2124240c1d0e83ca0e9ee63adeeabfd5d0fd6904eead36 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=33453 --metrics-logging=true --relay=true --filter=true
2026-03-21 04:38:59.750 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.150.11 waku 07fba83ee5e8c76a16a6998fff6edaec7a7ee1cb5882bbe7fe8df34f2ae18309
2026-03-21 04:38:59.786 DEBUG [src.node.docker_mananger] Container started with ID 07fba83ee5e8. Setting up logs at ./log/docker/node1_2026-03-21_04-38-59__de517c56-5511-424c-9b74-3b8013800c03__wakuorg_nwaku:latest.log
2026-03-21 04:38:59.786 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 33449
2026-03-21 04:38:59.786 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-21 04:38:59.794 ERROR [src.node.docker_mananger] Max retries reached for container 41d356d93d04. Exiting log stream.
2026-03-21 04:39:00.354 ERROR [src.node.docker_mananger] Max retries reached for container ad72d91a1152. Exiting log stream.
2026-03-21 04:39:00.787 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33449/health" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:00.789 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-21 04:39:00.790 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-21 04:39:00.790 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33449/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:00.792 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.150.11/tcp/33450/p2p/16Uiu2HAmQHVwf6z9HodmYBP7MckhZZoLznHSszDuH6HpPgaYUDjo","/ip4/172.18.150.11/tcp/33451/ws/p2p/16Uiu2HAmQHVwf6z9HodmYBP7MckhZZoLznHSszDuH6HpPgaYUDjo"],"enrUri":"enr:-L24QOhOVN6QX1R7KEvrn5NomS59Wt91TTvVIkh8a8ik_s78JjsPfJQdNI3mgK9qtAJRZ1WpEaf1XC658Rva2ueD2h0CgmlkgnY0gmlwhKwSlguKbXVsdGlhZGRyc5YACASsEpYLBoKqAAoErBKWCwaCq90DgnJzhQADAQAAiXNlY3AyNTZrMaEDrMrs-_H8mkfJPx-4Arf07TPOhR5VPhapvYSdFZj_ReqDdGNwgoKqg3VkcIKCrIV3YWt1MgU"}'
2026-03-21 04:39:00.792 INFO [src.node.waku_node] REST service is ready !!
2026-03-21 04:39:00.793 DEBUG [src.steps.filter] Running fixture setup: setup_main_filter_node
2026-03-21 04:39:00.800 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-21 04:39:00.800 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-21_04-38-59__de517c56-5511-424c-9b74-3b8013800c03__wakuorg_nwaku:latest.log
2026-03-21 04:39:00.800 DEBUG [src.node.waku_node] Starting Node...
2026-03-21 04:39:00.800 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-21 04:39:00.801 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-21 04:39:00.802 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.193.11
2026-03-21 04:39:00.802 DEBUG [src.node.docker_mananger] Generated ports ['20003', '20004', '20005', '20006', '20007']
2026-03-21 04:39:00.802 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-21 04:39:00.802 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-21 04:39:00.802 DEBUG [src.node.waku_node] Using volumes []
2026-03-21 04:39:00.802 DEBUG [src.node.docker_mananger] docker run -i -t -p 20003:20003 -p 20004:20004 -p 20005:20005 -p 20006:20006 -p 20007:20007 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=20005 --rest-port=20003 --tcp-port=20004 --discv5-udp-port=20006 --rest-address=0.0.0.0 --nat=extip:172.18.193.11 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=06fca7cacb0e8aa8adcedaea8d3afaba5dcc18fcd6b06cadec2ae4605ca22c5f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=20007 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QOhOVN6QX1R7KEvrn5NomS59Wt91TTvVIkh8a8ik_s78JjsPfJQdNI3mgK9qtAJRZ1WpEaf1XC658Rva2ueD2h0CgmlkgnY0gmlwhKwSlguKbXVsdGlhZGRyc5YACASsEpYLBoKqAAoErBKWCwaCq90DgnJzhQADAQAAiXNlY3AyNTZrMaEDrMrs-_H8mkfJPx-4Arf07TPOhR5VPhapvYSdFZj_ReqDdGNwgoKqg3VkcIKCrIV3YWt1MgU --filternode=/ip4/172.18.150.11/tcp/33450/p2p/16Uiu2HAmQHVwf6z9HodmYBP7MckhZZoLznHSszDuH6HpPgaYUDjo
2026-03-21 04:39:00.999 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.193.11 waku 29f931932fcd79392c1d5c201f56e796665c58428fa9d246491f7267215a3c95
2026-03-21 04:39:01.038 DEBUG [src.node.docker_mananger] Container started with ID 29f931932fcd. Setting up logs at ./log/docker/node2_2026-03-21_04-38-59__de517c56-5511-424c-9b74-3b8013800c03__wakuorg_nwaku:latest.log
2026-03-21 04:39:01.039 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 20003
2026-03-21 04:39:01.039 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-21 04:39:02.039 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:20003/health" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:02.042 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-21 04:39:02.043 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-21 04:39:02.043 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:20003/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:02.045 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.193.11/tcp/20004/p2p/16Uiu2HAm9DPThJsTLxSNJBDApBK3ZonDFaq7TT4njspnTX36Ta44","/ip4/172.18.193.11/tcp/20005/ws/p2p/16Uiu2HAm9DPThJsTLxSNJBDApBK3ZonDFaq7TT4njspnTX36Ta44"],"enrUri":"enr:-L24QByDRY8Aai8566D5j5x5DOwWm2DhAQ4i8RFuL1NczKH2VF0B4OJ_g6tnJvrVdGddfTAOY9MAkb_VChJHVdfRWh4CgmlkgnY0gmlwhKwSwQuKbXVsdGlhZGRyc5YACASsEsELBk4kAAoErBLBCwZOJd0DgnJzhQADAQAAiXNlY3AyNTZrMaECzN34IMq2E0n48K999T3ovb-5b44FX4csZXtwseLiLnWDdGNwgk4kg3VkcIJOJoV3YWt1MgA"}'
2026-03-21 04:39:02.045 INFO [src.node.waku_node] REST service is ready !!
2026-03-21 04:39:02.045 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:20003/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.150.11/tcp/33450/p2p/16Uiu2HAmQHVwf6z9HodmYBP7MckhZZoLznHSszDuH6HpPgaYUDjo"]'
2026-03-21 04:39:02.079 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-21 04:39:02.091 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-21 04:39:02.091 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-21_04-38-59__de517c56-5511-424c-9b74-3b8013800c03__wakuorg_nwaku:latest.log
2026-03-21 04:39:02.091 DEBUG [src.node.waku_node] Starting Node...
2026-03-21 04:39:02.092 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-21 04:39:02.093 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-21 04:39:02.093 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.61.104
2026-03-21 04:39:02.093 DEBUG [src.node.docker_mananger] Generated ports ['65341', '65342', '65343', '65344', '65345']
2026-03-21 04:39:02.094 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-21 04:39:02.094 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-21 04:39:02.094 DEBUG [src.node.waku_node] Using volumes []
2026-03-21 04:39:02.094 DEBUG [src.node.docker_mananger] docker run -i -t -p 65341:65341 -p 65342:65342 -p 65343:65343 -p 65344:65344 -p 65345:65345 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=65343 --rest-port=65341 --tcp-port=65342 --discv5-udp-port=65344 --rest-address=0.0.0.0 --nat=extip:172.18.61.104 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=5ffc7abbbcedff4fb76b6abc80ffdbbed7ed44ad319b7cda24f39b931f1bcedc --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=65345 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QOhOVN6QX1R7KEvrn5NomS59Wt91TTvVIkh8a8ik_s78JjsPfJQdNI3mgK9qtAJRZ1WpEaf1XC658Rva2ueD2h0CgmlkgnY0gmlwhKwSlguKbXVsdGlhZGRyc5YACASsEpYLBoKqAAoErBKWCwaCq90DgnJzhQADAQAAiXNlY3AyNTZrMaEDrMrs-_H8mkfJPx-4Arf07TPOhR5VPhapvYSdFZj_ReqDdGNwgoKqg3VkcIKCrIV3YWt1MgU --filternode=/ip4/172.18.150.11/tcp/33450/p2p/16Uiu2HAmQHVwf6z9HodmYBP7MckhZZoLznHSszDuH6HpPgaYUDjo
2026-03-21 04:39:02.301 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.61.104 waku 948373fe20b0b5f99c259653e07b71bce24e21529993f088cfb6c32ac148d14e
2026-03-21 04:39:02.338 DEBUG [src.node.docker_mananger] Container started with ID 948373fe20b0. Setting up logs at ./log/docker/node3_2026-03-21_04-38-59__de517c56-5511-424c-9b74-3b8013800c03__wakuorg_nwaku:latest.log
2026-03-21 04:39:02.338 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 65341
2026-03-21 04:39:02.339 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-21 04:39:03.339 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65341/health" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:03.343 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-21 04:39:03.343 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-21 04:39:03.343 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65341/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:03.345 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.61.104/tcp/65342/p2p/16Uiu2HAm7hDFEYgtAre7qibGBRNSBCVjPvATzdNtFHWJCaAZZbVa","/ip4/172.18.61.104/tcp/65343/ws/p2p/16Uiu2HAm7hDFEYgtAre7qibGBRNSBCVjPvATzdNtFHWJCaAZZbVa"],"enrUri":"enr:-L24QLwOPZUXTxWn2x-YF1VV0bAx7hRGV9qRCm-IVb1FZdMmTMmwzhc8yBZ2vhRVbLRD4cSX-Y0e03LrGPiKEcvv_ogCgmlkgnY0gmlwhKwSPWiKbXVsdGlhZGRyc5YACASsEj1oBv8-AAoErBI9aAb_P90DgnJzhQADAQAAiXNlY3AyNTZrMaECtkdO3nSoAyILfG4Tx4FP9Anga_g9_Li2QKVKgJBQk6GDdGNwgv8-g3VkcIL_QIV3YWt1MgA"}'
2026-03-21 04:39:03.346 INFO [src.node.waku_node] REST service is ready !!
2026-03-21 04:39:03.346 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:65341/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.150.11/tcp/33450/p2p/16Uiu2HAmQHVwf6z9HodmYBP7MckhZZoLznHSszDuH6HpPgaYUDjo"]'
2026-03-21 04:39:03.381 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-21 04:39:03.395 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-21 04:39:03.396 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-21_04-38-59__de517c56-5511-424c-9b74-3b8013800c03__wakuorg_nwaku:latest.log
2026-03-21 04:39:03.396 DEBUG [src.node.waku_node] Starting Node...
2026-03-21 04:39:03.396 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-21 04:39:03.397 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-21 04:39:03.397 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.171.252
2026-03-21 04:39:03.397 DEBUG [src.node.docker_mananger] Generated ports ['62166', '62167', '62168', '62169', '62170']
2026-03-21 04:39:03.398 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-21 04:39:03.398 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-21 04:39:03.398 DEBUG [src.node.waku_node] Using volumes []
2026-03-21 04:39:03.398 DEBUG [src.node.docker_mananger] docker run -i -t -p 62166:62166 -p 62167:62167 -p 62168:62168 -p 62169:62169 -p 62170:62170 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=62168 --rest-port=62166 --tcp-port=62167 --discv5-udp-port=62169 --rest-address=0.0.0.0 --nat=extip:172.18.171.252 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=106a8eeb426b826d7f5f0daab099b11d784ae504355b16f26a257a843aceec53 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=62170 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QOhOVN6QX1R7KEvrn5NomS59Wt91TTvVIkh8a8ik_s78JjsPfJQdNI3mgK9qtAJRZ1WpEaf1XC658Rva2ueD2h0CgmlkgnY0gmlwhKwSlguKbXVsdGlhZGRyc5YACASsEpYLBoKqAAoErBKWCwaCq90DgnJzhQADAQAAiXNlY3AyNTZrMaEDrMrs-_H8mkfJPx-4Arf07TPOhR5VPhapvYSdFZj_ReqDdGNwgoKqg3VkcIKCrIV3YWt1MgU --filternode=/ip4/172.18.150.11/tcp/33450/p2p/16Uiu2HAmQHVwf6z9HodmYBP7MckhZZoLznHSszDuH6HpPgaYUDjo
2026-03-21 04:39:03.599 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.171.252 waku 8f660740ef634cb667115aeb8883a4ba7ea2ace35b715e5ba0a7ccf801d1548c
2026-03-21 04:39:03.635 DEBUG [src.node.docker_mananger] Container started with ID 8f660740ef63. Setting up logs at ./log/docker/node4_2026-03-21_04-38-59__de517c56-5511-424c-9b74-3b8013800c03__wakuorg_nwaku:latest.log
2026-03-21 04:39:03.635 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 62166
2026-03-21 04:39:03.636 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-21 04:39:04.636 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62166/health" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:04.639 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-21 04:39:04.639 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-21 04:39:04.640 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62166/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:04.642 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.171.252/tcp/62167/p2p/16Uiu2HAmRjERQY78mhT5g2uvHYQQZTWXAef2xXi2VswgS4Bx7dQE","/ip4/172.18.171.252/tcp/62168/ws/p2p/16Uiu2HAmRjERQY78mhT5g2uvHYQQZTWXAef2xXi2VswgS4Bx7dQE"],"enrUri":"enr:-L24QEHc94SSRgkDFW08hBTTbXGiWU16839-F3Ifj3rTdathfRsdsIVT54u8s3InXYW7BvVy7AaPL2qh-j-XAaolJ0YCgmlkgnY0gmlwhKwSq_yKbXVsdGlhZGRyc5YACASsEqv8BvLXAAoErBKr_Aby2N0DgnJzhQADAQAAiXNlY3AyNTZrMaEDwj4p05Sov_ZYRamXBTMjAn-vu4z72rEv5r6SfLbSdjODdGNwgvLXg3VkcILy2YV3YWt1MgA"}'
2026-03-21 04:39:04.642 INFO [src.node.waku_node] REST service is ready !!
2026-03-21 04:39:04.643 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62166/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.150.11/tcp/33450/p2p/16Uiu2HAmQHVwf6z9HodmYBP7MckhZZoLznHSszDuH6HpPgaYUDjo"]'
2026-03-21 04:39:04.681 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-21 04:39:04.695 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-21 04:39:04.695 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-21_04-38-59__de517c56-5511-424c-9b74-3b8013800c03__wakuorg_nwaku:latest.log
2026-03-21 04:39:04.695 DEBUG [src.node.waku_node] Starting Node...
2026-03-21 04:39:04.696 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-21 04:39:04.700 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-21 04:39:04.700 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.200.152
2026-03-21 04:39:04.700 DEBUG [src.node.docker_mananger] Generated ports ['62960', '62961', '62962', '62963', '62964']
2026-03-21 04:39:04.701 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-21 04:39:04.701 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-21 04:39:04.701 DEBUG [src.node.waku_node] Using volumes []
2026-03-21 04:39:04.701 DEBUG [src.node.docker_mananger] docker run -i -t -p 62960:62960 -p 62961:62961 -p 62962:62962 -p 62963:62963 -p 62964:62964 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=62962 --rest-port=62960 --tcp-port=62961 --discv5-udp-port=62963 --rest-address=0.0.0.0 --nat=extip:172.18.200.152 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=aa619a7507019efa60a563cba0de9be80c32ceaf1ca50d0e413acf60ddc3cf5c --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=62964 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QOhOVN6QX1R7KEvrn5NomS59Wt91TTvVIkh8a8ik_s78JjsPfJQdNI3mgK9qtAJRZ1WpEaf1XC658Rva2ueD2h0CgmlkgnY0gmlwhKwSlguKbXVsdGlhZGRyc5YACASsEpYLBoKqAAoErBKWCwaCq90DgnJzhQADAQAAiXNlY3AyNTZrMaEDrMrs-_H8mkfJPx-4Arf07TPOhR5VPhapvYSdFZj_ReqDdGNwgoKqg3VkcIKCrIV3YWt1MgU --filternode=/ip4/172.18.150.11/tcp/33450/p2p/16Uiu2HAmQHVwf6z9HodmYBP7MckhZZoLznHSszDuH6HpPgaYUDjo
2026-03-21 04:39:04.907 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.200.152 waku f3a3d1eff4bca866e046579c8321faff311d9a0f9e59a93c9e721b8e68fd11b3
2026-03-21 04:39:04.946 DEBUG [src.node.docker_mananger] Container started with ID f3a3d1eff4bc. Setting up logs at ./log/docker/node5_2026-03-21_04-38-59__de517c56-5511-424c-9b74-3b8013800c03__wakuorg_nwaku:latest.log
2026-03-21 04:39:04.947 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 62960
2026-03-21 04:39:04.947 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-21 04:39:05.948 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62960/health" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:05.951 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-21 04:39:05.952 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-21 04:39:05.952 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62960/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:05.954 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.200.152/tcp/62961/p2p/16Uiu2HAmS6QP6eWsehbpqhWnLPNvopwrJx9NL1bXg9V5zvtxHeX7","/ip4/172.18.200.152/tcp/62962/ws/p2p/16Uiu2HAmS6QP6eWsehbpqhWnLPNvopwrJx9NL1bXg9V5zvtxHeX7"],"enrUri":"enr:-L24QBjhxuUrqSoPC6ch2CRmgQ1sf5kCEwWc4ro_rkQbvkFvP5Vfb9WOU0-fpcdmenKg-pDFwZaJQUDR8VhlnDsg3dICgmlkgnY0gmlwhKwSyJiKbXVsdGlhZGRyc5YACASsEsiYBvXxAAoErBLImAb18t0DgnJzhQADAQAAiXNlY3AyNTZrMaEDx6qfVirqAJ4LnNrl27r6Io-ekj8jRHPtJGq-4dZSApaDdGNwgvXxg3VkcIL184V3YWt1MgA"}'
2026-03-21 04:39:05.954 INFO [src.node.waku_node] REST service is ready !!
2026-03-21 04:39:05.955 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62960/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.150.11/tcp/33450/p2p/16Uiu2HAmQHVwf6z9HodmYBP7MckhZZoLznHSszDuH6HpPgaYUDjo"]'
2026-03-21 04:39:05.995 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-21 04:39:05.995 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:33449/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-21 04:39:06.002 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-21 04:39:06.003 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:20003/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "05dfc2da-c7c9-4e93-ab91-31ff1a620463", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-21 04:39:06.015 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"05dfc2da-c7c9-4e93-ab91-31ff1a620463","statusDesc":"OK"}'
2026-03-21 04:39:06.017 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:65341/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "e14b38bc-4589-4ecb-9879-24358033163b", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-21 04:39:06.033 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"e14b38bc-4589-4ecb-9879-24358033163b","statusDesc":"OK"}'
2026-03-21 04:39:06.035 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62166/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "17a620bc-b542-4a6a-a037-a7986e1dc292", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-21 04:39:06.046 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"17a620bc-b542-4a6a-a037-a7986e1dc292","statusDesc":"OK"}'
2026-03-21 04:39:06.047 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62960/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "b53b0032-8a47-4bea-93da-f97267774297", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-21 04:39:06.058 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"b53b0032-8a47-4bea-93da-f97267774297","statusDesc":"OK"}'
2026-03-21 04:39:06.059 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:33449/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": '$(date +%s%N)'}'
2026-03-21 04:39:06.076 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-21 04:39:06.077 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-21 04:39:06.177 DEBUG [src.steps.filter] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-21 04:39:06.178 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:20003/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:06.180 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774067946059194711,"ephemeral":false}]'
2026-03-21 04:39:06.182 DEBUG [src.steps.filter] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-03-21 04:39:06.183 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65341/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:06.185 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774067946059194711,"ephemeral":false}]'
2026-03-21 04:39:06.186 DEBUG [src.steps.filter] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-03-21 04:39:06.187 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62166/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:06.189 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774067946059194711,"ephemeral":false}]'
2026-03-21 04:39:06.191 DEBUG [src.steps.filter] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-03-21 04:39:06.191 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62960/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:06.193 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774067946059194711,"ephemeral":false}]'
2026-03-21 04:39:06.195 DEBUG [src.node.waku_node] Stopping container with id 29f931932fcd
2026-03-21 04:39:06.772 DEBUG [src.node.waku_node] Container stopped.
2026-03-21 04:39:06.776 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:33449/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": '$(date +%s%N)'}'
2026-03-21 04:39:06.785 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-21 04:39:06.786 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-21 04:39:06.887 DEBUG [src.steps.filter] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-21 04:39:06.888 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65341/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:06.891 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774067946776209053,"ephemeral":false}]'
2026-03-21 04:39:06.892 DEBUG [src.steps.filter] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-03-21 04:39:06.892 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62166/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:06.895 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774067946776209053,"ephemeral":false}]'
2026-03-21 04:39:06.896 DEBUG [src.steps.filter] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-03-21 04:39:06.896 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62960/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:39:06.899 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774067946776209053,"ephemeral":false}]'
2026-03-21 04:39:06.902 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-21 04:39:06.903 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-21 04:39:06.903 DEBUG [src.node.waku_node] Stopping container with id 07fba83ee5e8
2026-03-21 04:39:07.494 DEBUG [src.node.waku_node] Container stopped.
2026-03-21 04:39:07.494 DEBUG [src.node.waku_node] Stopping container with id 948373fe20b0
2026-03-21 04:39:07.610 ERROR [src.node.docker_mananger] Max retries reached for container 29f931932fcd. Exiting log stream.
2026-03-21 04:39:08.060 DEBUG [src.node.waku_node] Container stopped.
2026-03-21 04:39:08.062 DEBUG [src.node.waku_node] Stopping container with id 8f660740ef63
2026-03-21 04:39:08.306 ERROR [src.node.docker_mananger] Max retries reached for container 07fba83ee5e8. Exiting log stream.
2026-03-21 04:39:08.632 DEBUG [src.node.waku_node] Container stopped.
2026-03-21 04:39:08.633 DEBUG [src.node.waku_node] Stopping container with id f3a3d1eff4bc
2026-03-21 04:39:08.875 ERROR [src.node.docker_mananger] Max retries reached for container 948373fe20b0. Exiting log stream.
2026-03-21 04:39:09.197 DEBUG [src.node.waku_node] Container stopped.
2026-03-21 04:39:09.200 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-21 04:39:09.215 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-21 04:39:09.223 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-21 04:39:09.227 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-21 04:39:09.231 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-21 04:39:09.236 DEBUG [src.node.docker_mananger] No errors found in the waku logs.