177 lines
34 KiB
Plaintext

2026-03-22 04:44:59.281 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-22 04:44:59.283 DEBUG [tests.conftest] Running test: test_filter_get_message_while_one_peer_is_paused with id: 2026-03-22_04-44-59__9007fb3e-de8f-4653-abab-badc39eb5415
2026-03-22 04:44:59.283 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-22 04:44:59.283 DEBUG [src.steps.filter] Running fixture setup: filter_setup
2026-03-22 04:44:59.285 DEBUG [src.steps.filter] Running fixture setup: setup_main_relay_node
2026-03-22 04:44:59.293 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-22 04:44:59.293 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-22_04-44-59__9007fb3e-de8f-4653-abab-badc39eb5415__wakuorg_nwaku:latest.log
2026-03-22 04:44:59.294 DEBUG [src.node.waku_node] Starting Node...
2026-03-22 04:44:59.294 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-22 04:44:59.296 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-22 04:44:59.296 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.32.54
2026-03-22 04:44:59.296 DEBUG [src.node.docker_mananger] Generated ports ['45215', '45216', '45217', '45218', '45219']
2026-03-22 04:44:59.297 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-22 04:44:59.297 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-22 04:44:59.297 DEBUG [src.node.waku_node] Using volumes []
2026-03-22 04:44:59.297 DEBUG [src.node.docker_mananger] docker run -i -t -p 45215:45215 -p 45216:45216 -p 45217:45217 -p 45218:45218 -p 45219:45219 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=45217 --rest-port=45215 --tcp-port=45216 --discv5-udp-port=45218 --rest-address=0.0.0.0 --nat=extip:172.18.32.54 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=b4ce9cfce31be57ced0706ceaa55c4aa9facbe3a20bfec8c2c5b61a25bfb66ca --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=45219 --metrics-logging=true --relay=true --filter=true
2026-03-22 04:44:59.475 ERROR [src.node.docker_mananger] Max retries reached for container 3615ce820a2d. Exiting log stream.
2026-03-22 04:44:59.540 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.32.54 waku de787dd2bcb3f376bec46ae5be4880e9e33a92f73d5e4d63b9505a96f6356ec8
2026-03-22 04:44:59.588 DEBUG [src.node.docker_mananger] Container started with ID de787dd2bcb3. Setting up logs at ./log/docker/node1_2026-03-22_04-44-59__9007fb3e-de8f-4653-abab-badc39eb5415__wakuorg_nwaku:latest.log
2026-03-22 04:44:59.589 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 45215
2026-03-22 04:44:59.590 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-22 04:45:00.462 ERROR [src.node.docker_mananger] Max retries reached for container a99edd0c53ce. Exiting log stream.
2026-03-22 04:45:00.592 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45215/health" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:45:00.595 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-22 04:45:00.595 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-22 04:45:00.595 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45215/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:45:00.597 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.32.54/tcp/45216/p2p/16Uiu2HAmLoWEYRdEkH8tHCo9CKRvGzvfVrrXuZ8fMPnjCUYTAaqa","/ip4/172.18.32.54/tcp/45217/ws/p2p/16Uiu2HAmLoWEYRdEkH8tHCo9CKRvGzvfVrrXuZ8fMPnjCUYTAaqa"],"enrUri":"enr:-L24QNajojIwDVK2RZ0Nw5V-lgQ9UOxeogl-8J1HCI1LBYQfZeoQHF4YIvC1zk32AtB1GOPDyONvyx68q69kx0iW8RYCgmlkgnY0gmlwhKwSIDaKbXVsdGlhZGRyc5YACASsEiA2BrCgAAoErBIgNgawod0DgnJzhQADAQAAiXNlY3AyNTZrMaEDeQvuc83qxBmA9AoD89bo_UUP68f8C4RIfggWP_kjyk2DdGNwgrCgg3VkcIKwooV3YWt1MgU"}'
2026-03-22 04:45:00.597 INFO [src.node.waku_node] REST service is ready !!
2026-03-22 04:45:00.598 DEBUG [src.steps.filter] Running fixture setup: setup_main_filter_node
2026-03-22 04:45:00.604 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-22 04:45:00.604 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-22_04-44-59__9007fb3e-de8f-4653-abab-badc39eb5415__wakuorg_nwaku:latest.log
2026-03-22 04:45:00.605 DEBUG [src.node.waku_node] Starting Node...
2026-03-22 04:45:00.605 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-22 04:45:00.606 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-22 04:45:00.606 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.68.33
2026-03-22 04:45:00.606 DEBUG [src.node.docker_mananger] Generated ports ['59850', '59851', '59852', '59853', '59854']
2026-03-22 04:45:00.606 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-22 04:45:00.607 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-22 04:45:00.607 DEBUG [src.node.waku_node] Using volumes []
2026-03-22 04:45:00.607 DEBUG [src.node.docker_mananger] docker run -i -t -p 59850:59850 -p 59851:59851 -p 59852:59852 -p 59853:59853 -p 59854:59854 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=59852 --rest-port=59850 --tcp-port=59851 --discv5-udp-port=59853 --rest-address=0.0.0.0 --nat=extip:172.18.68.33 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=12a70cadd7f16da42edac3fc26618fbd0b4f1dab29baaccdac5450ac0decce0d --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=59854 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QNajojIwDVK2RZ0Nw5V-lgQ9UOxeogl-8J1HCI1LBYQfZeoQHF4YIvC1zk32AtB1GOPDyONvyx68q69kx0iW8RYCgmlkgnY0gmlwhKwSIDaKbXVsdGlhZGRyc5YACASsEiA2BrCgAAoErBIgNgawod0DgnJzhQADAQAAiXNlY3AyNTZrMaEDeQvuc83qxBmA9AoD89bo_UUP68f8C4RIfggWP_kjyk2DdGNwgrCgg3VkcIKwooV3YWt1MgU --filternode=/ip4/172.18.32.54/tcp/45216/p2p/16Uiu2HAmLoWEYRdEkH8tHCo9CKRvGzvfVrrXuZ8fMPnjCUYTAaqa
2026-03-22 04:45:00.801 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.68.33 waku 23130e4af6f7f4dc425b43644ee540ac30a2700ec8d564327d9326309b71d216
2026-03-22 04:45:00.834 DEBUG [src.node.docker_mananger] Container started with ID 23130e4af6f7. Setting up logs at ./log/docker/node2_2026-03-22_04-44-59__9007fb3e-de8f-4653-abab-badc39eb5415__wakuorg_nwaku:latest.log
2026-03-22 04:45:00.834 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 59850
2026-03-22 04:45:00.835 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-22 04:45:01.836 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59850/health" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:45:01.839 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-22 04:45:01.839 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-22 04:45:01.839 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59850/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:45:01.841 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.68.33/tcp/59851/p2p/16Uiu2HAm3zyzmvkUoucsRRHmFB9jDCNBC8g6EEnirWzYE3LniMj5","/ip4/172.18.68.33/tcp/59852/ws/p2p/16Uiu2HAm3zyzmvkUoucsRRHmFB9jDCNBC8g6EEnirWzYE3LniMj5"],"enrUri":"enr:-L24QEoU4W4V1Huud70W3NsJSuPxMGZuQchC39tlg_XaraVbD6VrxuHuiBV7KhW5r7Hp6fuXZ6j05jsdjwtnZRneA2wCgmlkgnY0gmlwhKwSRCGKbXVsdGlhZGRyc5YACASsEkQhBunLAAoErBJEIQbpzN0DgnJzhQADAQAAiXNlY3AyNTZrMaECf2YIrsu9gRg1_piPTfi1xea4R7Esnrjt9P8EcXcv0XCDdGNwgunLg3VkcILpzYV3YWt1MgA"}'
2026-03-22 04:45:01.841 INFO [src.node.waku_node] REST service is ready !!
2026-03-22 04:45:01.842 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59850/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.32.54/tcp/45216/p2p/16Uiu2HAmLoWEYRdEkH8tHCo9CKRvGzvfVrrXuZ8fMPnjCUYTAaqa"]'
2026-03-22 04:45:01.874 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:45:01.885 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-22 04:45:01.886 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-22_04-44-59__9007fb3e-de8f-4653-abab-badc39eb5415__wakuorg_nwaku:latest.log
2026-03-22 04:45:01.886 DEBUG [src.node.waku_node] Starting Node...
2026-03-22 04:45:01.886 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-22 04:45:01.887 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-22 04:45:01.887 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.10.128
2026-03-22 04:45:01.888 DEBUG [src.node.docker_mananger] Generated ports ['64454', '64455', '64456', '64457', '64458']
2026-03-22 04:45:01.888 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-22 04:45:01.888 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-22 04:45:01.888 DEBUG [src.node.waku_node] Using volumes []
2026-03-22 04:45:01.888 DEBUG [src.node.docker_mananger] docker run -i -t -p 64454:64454 -p 64455:64455 -p 64456:64456 -p 64457:64457 -p 64458:64458 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=64456 --rest-port=64454 --tcp-port=64455 --discv5-udp-port=64457 --rest-address=0.0.0.0 --nat=extip:172.18.10.128 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=cb5dba050a9bc579dce8fcdbba82ada0b8627f8da74c76cbecab45a53180fc36 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=64458 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QNajojIwDVK2RZ0Nw5V-lgQ9UOxeogl-8J1HCI1LBYQfZeoQHF4YIvC1zk32AtB1GOPDyONvyx68q69kx0iW8RYCgmlkgnY0gmlwhKwSIDaKbXVsdGlhZGRyc5YACASsEiA2BrCgAAoErBIgNgawod0DgnJzhQADAQAAiXNlY3AyNTZrMaEDeQvuc83qxBmA9AoD89bo_UUP68f8C4RIfggWP_kjyk2DdGNwgrCgg3VkcIKwooV3YWt1MgU --filternode=/ip4/172.18.32.54/tcp/45216/p2p/16Uiu2HAmLoWEYRdEkH8tHCo9CKRvGzvfVrrXuZ8fMPnjCUYTAaqa
2026-03-22 04:45:02.096 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.10.128 waku 964b50ebb81892f955b1867451a5461e19002650bfef172357f5f5f374786817
2026-03-22 04:45:02.130 DEBUG [src.node.docker_mananger] Container started with ID 964b50ebb818. Setting up logs at ./log/docker/node3_2026-03-22_04-44-59__9007fb3e-de8f-4653-abab-badc39eb5415__wakuorg_nwaku:latest.log
2026-03-22 04:45:02.131 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 64454
2026-03-22 04:45:02.131 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-22 04:45:03.131 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:64454/health" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:45:03.134 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-22 04:45:03.134 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-22 04:45:03.134 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:64454/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:45:03.137 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.10.128/tcp/64455/p2p/16Uiu2HAm641Es6ms3HQ62X4MyF6oTQgLjQnpThynTRHyCPG4AwXa","/ip4/172.18.10.128/tcp/64456/ws/p2p/16Uiu2HAm641Es6ms3HQ62X4MyF6oTQgLjQnpThynTRHyCPG4AwXa"],"enrUri":"enr:-L24QIjfRySoLW34_qh5dzdceb2h78vqM1DzKzPZkxwjNUP0Qk4WTHDiIa1raF7NigdhqAzgOsp8apC_7ts-2PxBMTUCgmlkgnY0gmlwhKwSCoCKbXVsdGlhZGRyc5YACASsEgqABvvHAAoErBIKgAb7yN0DgnJzhQADAQAAiXNlY3AyNTZrMaECneOOObHI3oL_HYMP4VhS-OSmoAwx1nCiuLOs24EXeH2DdGNwgvvHg3VkcIL7yYV3YWt1MgA"}'
2026-03-22 04:45:03.137 INFO [src.node.waku_node] REST service is ready !!
2026-03-22 04:45:03.137 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:64454/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.32.54/tcp/45216/p2p/16Uiu2HAmLoWEYRdEkH8tHCo9CKRvGzvfVrrXuZ8fMPnjCUYTAaqa"]'
2026-03-22 04:45:03.173 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:45:03.184 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-22 04:45:03.185 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-22_04-44-59__9007fb3e-de8f-4653-abab-badc39eb5415__wakuorg_nwaku:latest.log
2026-03-22 04:45:03.185 DEBUG [src.node.waku_node] Starting Node...
2026-03-22 04:45:03.186 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-22 04:45:03.188 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-22 04:45:03.189 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.245.147
2026-03-22 04:45:03.189 DEBUG [src.node.docker_mananger] Generated ports ['46160', '46161', '46162', '46163', '46164']
2026-03-22 04:45:03.189 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-22 04:45:03.189 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-22 04:45:03.189 DEBUG [src.node.waku_node] Using volumes []
2026-03-22 04:45:03.190 DEBUG [src.node.docker_mananger] docker run -i -t -p 46160:46160 -p 46161:46161 -p 46162:46162 -p 46163:46163 -p 46164:46164 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=46162 --rest-port=46160 --tcp-port=46161 --discv5-udp-port=46163 --rest-address=0.0.0.0 --nat=extip:172.18.245.147 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=fd0d739bdb3de9e61b9c5767c62bf501647982372da08677bd548af172dfa18d --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=46164 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QNajojIwDVK2RZ0Nw5V-lgQ9UOxeogl-8J1HCI1LBYQfZeoQHF4YIvC1zk32AtB1GOPDyONvyx68q69kx0iW8RYCgmlkgnY0gmlwhKwSIDaKbXVsdGlhZGRyc5YACASsEiA2BrCgAAoErBIgNgawod0DgnJzhQADAQAAiXNlY3AyNTZrMaEDeQvuc83qxBmA9AoD89bo_UUP68f8C4RIfggWP_kjyk2DdGNwgrCgg3VkcIKwooV3YWt1MgU --filternode=/ip4/172.18.32.54/tcp/45216/p2p/16Uiu2HAmLoWEYRdEkH8tHCo9CKRvGzvfVrrXuZ8fMPnjCUYTAaqa
2026-03-22 04:45:03.384 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.245.147 waku 639e1d410a2c5d766dce94ead810db1990f45b6ca4fd627fbc65d666d0d49f13
2026-03-22 04:45:03.419 DEBUG [src.node.docker_mananger] Container started with ID 639e1d410a2c. Setting up logs at ./log/docker/node4_2026-03-22_04-44-59__9007fb3e-de8f-4653-abab-badc39eb5415__wakuorg_nwaku:latest.log
2026-03-22 04:45:03.419 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 46160
2026-03-22 04:45:03.419 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-22 04:45:04.421 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:46160/health" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:45:04.423 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-22 04:45:04.423 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-22 04:45:04.424 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:46160/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:45:04.426 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.245.147/tcp/46161/p2p/16Uiu2HAmKxAiy7f5wbGNKfXFCPvmf1cEYtYnwPJoJD23MxBRrxZW","/ip4/172.18.245.147/tcp/46162/ws/p2p/16Uiu2HAmKxAiy7f5wbGNKfXFCPvmf1cEYtYnwPJoJD23MxBRrxZW"],"enrUri":"enr:-L24QMXhGbb9BzL0WVr7uYHzwfmT1sRA81o1C4z-fRsMxWDwaAXA_GJj33BOYwrdw4lsjHnCGuOdvOVuXq6j_acF_80CgmlkgnY0gmlwhKwS9ZOKbXVsdGlhZGRyc5YACASsEvWTBrRRAAoErBL1kwa0Ut0DgnJzhQADAQAAiXNlY3AyNTZrMaEDbGhpcdqADcjDDDHwkaOeCMIOO0FFACJSFyfpy7b_q0GDdGNwgrRRg3VkcIK0U4V3YWt1MgA"}'
2026-03-22 04:45:04.426 INFO [src.node.waku_node] REST service is ready !!
2026-03-22 04:45:04.426 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:46160/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.32.54/tcp/45216/p2p/16Uiu2HAmLoWEYRdEkH8tHCo9CKRvGzvfVrrXuZ8fMPnjCUYTAaqa"]'
2026-03-22 04:45:04.458 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:45:04.479 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-22 04:45:04.479 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-22_04-44-59__9007fb3e-de8f-4653-abab-badc39eb5415__wakuorg_nwaku:latest.log
2026-03-22 04:45:04.479 DEBUG [src.node.waku_node] Starting Node...
2026-03-22 04:45:04.481 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-22 04:45:04.482 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-22 04:45:04.482 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.167.113
2026-03-22 04:45:04.482 DEBUG [src.node.docker_mananger] Generated ports ['18658', '18659', '18660', '18661', '18662']
2026-03-22 04:45:04.483 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-22 04:45:04.483 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-22 04:45:04.483 DEBUG [src.node.waku_node] Using volumes []
2026-03-22 04:45:04.483 DEBUG [src.node.docker_mananger] docker run -i -t -p 18658:18658 -p 18659:18659 -p 18660:18660 -p 18661:18661 -p 18662:18662 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=18660 --rest-port=18658 --tcp-port=18659 --discv5-udp-port=18661 --rest-address=0.0.0.0 --nat=extip:172.18.167.113 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=ffbcb4b838d28fccdfed8e0e76fafe44dd9ccccfe6575cdf63f3da1afcde94ca --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=18662 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QNajojIwDVK2RZ0Nw5V-lgQ9UOxeogl-8J1HCI1LBYQfZeoQHF4YIvC1zk32AtB1GOPDyONvyx68q69kx0iW8RYCgmlkgnY0gmlwhKwSIDaKbXVsdGlhZGRyc5YACASsEiA2BrCgAAoErBIgNgawod0DgnJzhQADAQAAiXNlY3AyNTZrMaEDeQvuc83qxBmA9AoD89bo_UUP68f8C4RIfggWP_kjyk2DdGNwgrCgg3VkcIKwooV3YWt1MgU --filternode=/ip4/172.18.32.54/tcp/45216/p2p/16Uiu2HAmLoWEYRdEkH8tHCo9CKRvGzvfVrrXuZ8fMPnjCUYTAaqa
2026-03-22 04:45:04.690 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.167.113 waku ce6224f36ae3b766d4f7d071f257033bac7ef868efeb90ebe081818660702833
2026-03-22 04:45:04.724 DEBUG [src.node.docker_mananger] Container started with ID ce6224f36ae3. Setting up logs at ./log/docker/node5_2026-03-22_04-44-59__9007fb3e-de8f-4653-abab-badc39eb5415__wakuorg_nwaku:latest.log
2026-03-22 04:45:04.724 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 18658
2026-03-22 04:45:04.724 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-22 04:45:05.725 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18658/health" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:45:05.728 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-22 04:45:05.728 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-22 04:45:05.728 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18658/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:45:05.730 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.167.113/tcp/18659/p2p/16Uiu2HAmJVp6CYCaQ35HpCPhW24ueadj5F37QzvjCHUgtqJgYAPc","/ip4/172.18.167.113/tcp/18660/ws/p2p/16Uiu2HAmJVp6CYCaQ35HpCPhW24ueadj5F37QzvjCHUgtqJgYAPc"],"enrUri":"enr:-L24QDQ0N_30r5WkprgEPLpMAM5Rq_ilBpKxJ0i1v7VFLJUJf0Hm7ufZqfKI-qKjcTd14lQEaVW64D3nRxjwgmP4FxUCgmlkgnY0gmlwhKwSp3GKbXVsdGlhZGRyc5YACASsEqdxBkjjAAoErBKncQZI5N0DgnJzhQADAQAAiXNlY3AyNTZrMaEDVsxLBfp16T8GskwGbXAQDcMkTHR0tvxDxJqfSCs1s8uDdGNwgkjjg3VkcIJI5YV3YWt1MgA"}'
2026-03-22 04:45:05.731 INFO [src.node.waku_node] REST service is ready !!
2026-03-22 04:45:05.731 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:18658/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.32.54/tcp/45216/p2p/16Uiu2HAmLoWEYRdEkH8tHCo9CKRvGzvfVrrXuZ8fMPnjCUYTAaqa"]'
2026-03-22 04:45:05.767 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:45:05.769 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:45215/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-22 04:45:05.774 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:45:05.779 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59850/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "d46e33a1-e651-4146-b1ff-33b0ff61b13b", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-22 04:45:05.792 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"d46e33a1-e651-4146-b1ff-33b0ff61b13b","statusDesc":"OK"}'
2026-03-22 04:45:05.794 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:64454/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "06a914cb-698e-4a0c-9d04-04534431067e", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-22 04:45:05.804 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"06a914cb-698e-4a0c-9d04-04534431067e","statusDesc":"OK"}'
2026-03-22 04:45:05.804 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:46160/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "6e3be466-6ed3-4b5c-bbff-127c93a98642", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-22 04:45:05.814 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"6e3be466-6ed3-4b5c-bbff-127c93a98642","statusDesc":"OK"}'
2026-03-22 04:45:05.814 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:18658/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "0368f374-a575-4728-b4fb-8e9bd4e2093d", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-22 04:45:05.823 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"0368f374-a575-4728-b4fb-8e9bd4e2093d","statusDesc":"OK"}'
2026-03-22 04:45:05.824 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:45215/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": '$(date +%s%N)'}'
2026-03-22 04:45:05.841 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:45:05.841 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-22 04:45:05.942 DEBUG [src.steps.filter] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-22 04:45:05.942 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59850/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:45:05.944 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774154705824036508,"ephemeral":false}]'
2026-03-22 04:45:05.946 DEBUG [src.steps.filter] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-03-22 04:45:05.946 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:64454/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:45:05.948 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774154705824036508,"ephemeral":false}]'
2026-03-22 04:45:05.950 DEBUG [src.steps.filter] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-03-22 04:45:05.950 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:46160/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:45:05.952 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774154705824036508,"ephemeral":false}]'
2026-03-22 04:45:05.953 DEBUG [src.steps.filter] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-03-22 04:45:05.954 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18658/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:45:05.956 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774154705824036508,"ephemeral":false}]'
2026-03-22 04:45:05.957 DEBUG [src.node.waku_node] Pausing container with id 23130e4af6f7
2026-03-22 04:45:05.966 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:45215/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": '$(date +%s%N)'}'
2026-03-22 04:45:05.972 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:45:05.972 DEBUG [src.node.waku_node] Unpause container with id 23130e4af6f7
2026-03-22 04:45:05.981 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:45215/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": '$(date +%s%N)'}'
2026-03-22 04:45:05.988 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:45:05.988 DEBUG [src.libs.common] Sleeping for 0.5 seconds
2026-03-22 04:45:06.489 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59850/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:45:06.491 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774154705957412562,"ephemeral":false},{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774154705957484987,"ephemeral":false}]'
2026-03-22 04:45:06.494 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-22 04:45:06.495 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-22 04:45:06.495 DEBUG [src.node.waku_node] Stopping container with id de787dd2bcb3
2026-03-22 04:45:07.083 DEBUG [src.node.waku_node] Container stopped.
2026-03-22 04:45:07.084 DEBUG [src.node.waku_node] Stopping container with id 23130e4af6f7
2026-03-22 04:45:07.670 DEBUG [src.node.waku_node] Container stopped.
2026-03-22 04:45:07.672 DEBUG [src.node.waku_node] Stopping container with id 964b50ebb818
2026-03-22 04:45:07.952 ERROR [src.node.docker_mananger] Max retries reached for container de787dd2bcb3. Exiting log stream.
2026-03-22 04:45:08.266 DEBUG [src.node.waku_node] Container stopped.
2026-03-22 04:45:08.269 DEBUG [src.node.waku_node] Stopping container with id 639e1d410a2c
2026-03-22 04:45:08.505 ERROR [src.node.docker_mananger] Max retries reached for container 23130e4af6f7. Exiting log stream.
2026-03-22 04:45:08.843 DEBUG [src.node.waku_node] Container stopped.
2026-03-22 04:45:08.843 DEBUG [src.node.waku_node] Stopping container with id ce6224f36ae3
2026-03-22 04:45:09.080 ERROR [src.node.docker_mananger] Max retries reached for container 964b50ebb818. Exiting log stream.
2026-03-22 04:45:09.394 DEBUG [src.node.waku_node] Container stopped.
2026-03-22 04:45:09.395 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-22 04:45:09.426 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-22 04:45:09.435 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-22 04:45:09.439 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-22 04:45:09.444 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-22 04:45:09.448 DEBUG [src.node.docker_mananger] No errors found in the waku logs.