177 lines
34 KiB
Plaintext

2026-03-25 04:45:35.923 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-25 04:45:35.924 DEBUG [tests.conftest] Running test: test_filter_get_message_while_one_peer_is_paused with id: 2026-03-25_04-45-35__bfb2a47e-014b-4404-a181-82c1535e57d8
2026-03-25 04:45:35.925 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-25 04:45:35.925 DEBUG [src.steps.filter] Running fixture setup: filter_setup
2026-03-25 04:45:35.927 DEBUG [src.steps.filter] Running fixture setup: setup_main_relay_node
2026-03-25 04:45:35.937 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-25 04:45:35.938 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-25_04-45-35__bfb2a47e-014b-4404-a181-82c1535e57d8__wakuorg_nwaku:latest.log
2026-03-25 04:45:35.938 DEBUG [src.node.waku_node] Starting Node...
2026-03-25 04:45:35.938 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-25 04:45:35.943 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-25 04:45:35.944 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.202.15
2026-03-25 04:45:35.945 DEBUG [src.node.docker_mananger] Generated ports ['22534', '22535', '22536', '22537', '22538']
2026-03-25 04:45:35.945 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-25 04:45:35.945 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-25 04:45:35.946 DEBUG [src.node.waku_node] Using volumes []
2026-03-25 04:45:35.946 DEBUG [src.node.docker_mananger] docker run -i -t -p 22534:22534 -p 22535:22535 -p 22536:22536 -p 22537:22537 -p 22538:22538 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=22536 --rest-port=22534 --tcp-port=22535 --discv5-udp-port=22537 --rest-address=0.0.0.0 --nat=extip:172.18.202.15 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=aed37b2bfd5c505c7a1940f11adbbabab15d621ab1050be81e417efc0c0b0def --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=22538 --metrics-logging=true --relay=true --filter=true
2026-03-25 04:45:36.115 ERROR [src.node.docker_mananger] Max retries reached for container d66a3ac8e9ff. Exiting log stream.
2026-03-25 04:45:36.192 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.202.15 waku 527adb6a2370dfa42901236ab1369355081ee56992ac8925c951bfdba4f2ac18
2026-03-25 04:45:36.240 DEBUG [src.node.docker_mananger] Container started with ID 527adb6a2370. Setting up logs at ./log/docker/node1_2026-03-25_04-45-35__bfb2a47e-014b-4404-a181-82c1535e57d8__wakuorg_nwaku:latest.log
2026-03-25 04:45:36.241 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 22534
2026-03-25 04:45:36.242 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-25 04:45:37.114 ERROR [src.node.docker_mananger] Max retries reached for container 87c2e4687376. Exiting log stream.
2026-03-25 04:45:37.242 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22534/health" -H "Content-Type: application/json" -d 'None'
2026-03-25 04:45:37.245 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-25 04:45:37.246 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-25 04:45:37.246 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22534/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-25 04:45:37.248 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.202.15/tcp/22535/p2p/16Uiu2HAmQ94AsTTW8xSD8C5A5hj6frgyb4TNwPKMU3LpKmXomEjj","/ip4/172.18.202.15/tcp/22536/ws/p2p/16Uiu2HAmQ94AsTTW8xSD8C5A5hj6frgyb4TNwPKMU3LpKmXomEjj"],"enrUri":"enr:-L24QJiDpUD2Ev0lWAMsm_gzUq1cQXtoMkZaSnmca05D09Coa5_-X4RNZareceDsf2WmRntro-l1GQXzkXJagLZMYVACgmlkgnY0gmlwhKwSyg-KbXVsdGlhZGRyc5YACASsEsoPBlgHAAoErBLKDwZYCN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDqqEjd0VwABNG2KOkDH-HupGwSVKliuo4Oql0cJd6wgKDdGNwglgHg3VkcIJYCYV3YWt1MgU"}'
2026-03-25 04:45:37.248 INFO [src.node.waku_node] REST service is ready !!
2026-03-25 04:45:37.249 DEBUG [src.steps.filter] Running fixture setup: setup_main_filter_node
2026-03-25 04:45:37.256 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-25 04:45:37.256 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-25_04-45-35__bfb2a47e-014b-4404-a181-82c1535e57d8__wakuorg_nwaku:latest.log
2026-03-25 04:45:37.256 DEBUG [src.node.waku_node] Starting Node...
2026-03-25 04:45:37.256 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-25 04:45:37.258 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-25 04:45:37.258 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.246.71
2026-03-25 04:45:37.258 DEBUG [src.node.docker_mananger] Generated ports ['6348', '6349', '6350', '6351', '6352']
2026-03-25 04:45:37.258 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-25 04:45:37.258 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-25 04:45:37.258 DEBUG [src.node.waku_node] Using volumes []
2026-03-25 04:45:37.259 DEBUG [src.node.docker_mananger] docker run -i -t -p 6348:6348 -p 6349:6349 -p 6350:6350 -p 6351:6351 -p 6352:6352 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=6350 --rest-port=6348 --tcp-port=6349 --discv5-udp-port=6351 --rest-address=0.0.0.0 --nat=extip:172.18.246.71 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=d5b59ce98a47f5bb65db12a944abdca25e85c80417ab90ccdafaa2fa02a8beeb --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=6352 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QJiDpUD2Ev0lWAMsm_gzUq1cQXtoMkZaSnmca05D09Coa5_-X4RNZareceDsf2WmRntro-l1GQXzkXJagLZMYVACgmlkgnY0gmlwhKwSyg-KbXVsdGlhZGRyc5YACASsEsoPBlgHAAoErBLKDwZYCN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDqqEjd0VwABNG2KOkDH-HupGwSVKliuo4Oql0cJd6wgKDdGNwglgHg3VkcIJYCYV3YWt1MgU --filternode=/ip4/172.18.202.15/tcp/22535/p2p/16Uiu2HAmQ94AsTTW8xSD8C5A5hj6frgyb4TNwPKMU3LpKmXomEjj
2026-03-25 04:45:37.461 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.246.71 waku 83add069ffbb9234381fc499e34c12d950da3b2d1904b4574f2996566f9cc7ba
2026-03-25 04:45:37.494 DEBUG [src.node.docker_mananger] Container started with ID 83add069ffbb. Setting up logs at ./log/docker/node2_2026-03-25_04-45-35__bfb2a47e-014b-4404-a181-82c1535e57d8__wakuorg_nwaku:latest.log
2026-03-25 04:45:37.495 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 6348
2026-03-25 04:45:37.496 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-25 04:45:38.496 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6348/health" -H "Content-Type: application/json" -d 'None'
2026-03-25 04:45:38.499 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-25 04:45:38.500 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-25 04:45:38.500 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6348/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-25 04:45:38.502 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.246.71/tcp/6349/p2p/16Uiu2HAm1vL4WN1zwKVQMRSoS2J42iy3r3MfRFcopTCtq9endjmq","/ip4/172.18.246.71/tcp/6350/ws/p2p/16Uiu2HAm1vL4WN1zwKVQMRSoS2J42iy3r3MfRFcopTCtq9endjmq"],"enrUri":"enr:-L24QJ8pWfc3Ac3oxaKT4To9QP8_UTo8st4rpbZb6VBb78B1aWk9czlbjqM00tcrg0mkX0-NBJFwhZVuy_Y9-ZmOCEYCgmlkgnY0gmlwhKwS9keKbXVsdGlhZGRyc5YACASsEvZHBhjNAAoErBL2RwYYzt0DgnJzhQADAQAAiXNlY3AyNTZrMaECYH1xXeovAG8rh-8EGJfqtC8YRpcAzShdf3ifwWr1RqCDdGNwghjNg3VkcIIYz4V3YWt1MgA"}'
2026-03-25 04:45:38.502 INFO [src.node.waku_node] REST service is ready !!
2026-03-25 04:45:38.503 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6348/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.202.15/tcp/22535/p2p/16Uiu2HAmQ94AsTTW8xSD8C5A5hj6frgyb4TNwPKMU3LpKmXomEjj"]'
2026-03-25 04:45:38.538 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-25 04:45:38.549 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-25 04:45:38.550 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-25_04-45-35__bfb2a47e-014b-4404-a181-82c1535e57d8__wakuorg_nwaku:latest.log
2026-03-25 04:45:38.550 DEBUG [src.node.waku_node] Starting Node...
2026-03-25 04:45:38.550 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-25 04:45:38.551 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-25 04:45:38.552 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.244.0
2026-03-25 04:45:38.552 DEBUG [src.node.docker_mananger] Generated ports ['44280', '44281', '44282', '44283', '44284']
2026-03-25 04:45:38.552 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-25 04:45:38.552 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-25 04:45:38.552 DEBUG [src.node.waku_node] Using volumes []
2026-03-25 04:45:38.552 DEBUG [src.node.docker_mananger] docker run -i -t -p 44280:44280 -p 44281:44281 -p 44282:44282 -p 44283:44283 -p 44284:44284 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=44282 --rest-port=44280 --tcp-port=44281 --discv5-udp-port=44283 --rest-address=0.0.0.0 --nat=extip:172.18.244.0 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=73f106f3dadfae23baaf06324beebafe1e7fd48eccd89c4c6fee5bc78c933de7 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=44284 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QJiDpUD2Ev0lWAMsm_gzUq1cQXtoMkZaSnmca05D09Coa5_-X4RNZareceDsf2WmRntro-l1GQXzkXJagLZMYVACgmlkgnY0gmlwhKwSyg-KbXVsdGlhZGRyc5YACASsEsoPBlgHAAoErBLKDwZYCN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDqqEjd0VwABNG2KOkDH-HupGwSVKliuo4Oql0cJd6wgKDdGNwglgHg3VkcIJYCYV3YWt1MgU --filternode=/ip4/172.18.202.15/tcp/22535/p2p/16Uiu2HAmQ94AsTTW8xSD8C5A5hj6frgyb4TNwPKMU3LpKmXomEjj
2026-03-25 04:45:38.762 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.244.0 waku 17d20c51c2a2ee40573b207de2feec2b9bee3443a9d74503c973820b13a07e5e
2026-03-25 04:45:38.797 DEBUG [src.node.docker_mananger] Container started with ID 17d20c51c2a2. Setting up logs at ./log/docker/node3_2026-03-25_04-45-35__bfb2a47e-014b-4404-a181-82c1535e57d8__wakuorg_nwaku:latest.log
2026-03-25 04:45:38.797 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 44280
2026-03-25 04:45:38.798 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-25 04:45:39.798 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:44280/health" -H "Content-Type: application/json" -d 'None'
2026-03-25 04:45:39.801 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-25 04:45:39.802 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-25 04:45:39.802 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:44280/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-25 04:45:39.804 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.244.0/tcp/44281/p2p/16Uiu2HAkxrsHzuYPY8ZkLPEofEAeVYzjdRsneUsRqBnNsBwwoPBD","/ip4/172.18.244.0/tcp/44282/ws/p2p/16Uiu2HAkxrsHzuYPY8ZkLPEofEAeVYzjdRsneUsRqBnNsBwwoPBD"],"enrUri":"enr:-L24QEHc7TA_LYj4IQQO5L6g74u-Ssp_t2A0zuZwok7Qd_iGa9EZk-rpJRCQpRN2cfSRAK8evZAMJwsoWVSyTrfn1bUCgmlkgnY0gmlwhKwS9ACKbXVsdGlhZGRyc5YACASsEvQABqz5AAoErBL0AAas-t0DgnJzhQADAQAAiXNlY3AyNTZrMaECMwdfqGT2Biw9RSf7bUjzLhGZXyv7s2gOafHQEDbQQjiDdGNwgqz5g3VkcIKs-4V3YWt1MgA"}'
2026-03-25 04:45:39.805 INFO [src.node.waku_node] REST service is ready !!
2026-03-25 04:45:39.805 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:44280/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.202.15/tcp/22535/p2p/16Uiu2HAmQ94AsTTW8xSD8C5A5hj6frgyb4TNwPKMU3LpKmXomEjj"]'
2026-03-25 04:45:39.846 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-25 04:45:39.860 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-25 04:45:39.861 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-25_04-45-35__bfb2a47e-014b-4404-a181-82c1535e57d8__wakuorg_nwaku:latest.log
2026-03-25 04:45:39.863 DEBUG [src.node.waku_node] Starting Node...
2026-03-25 04:45:39.863 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-25 04:45:39.866 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-25 04:45:39.867 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.67.50
2026-03-25 04:45:39.867 DEBUG [src.node.docker_mananger] Generated ports ['29616', '29617', '29618', '29619', '29620']
2026-03-25 04:45:39.867 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-25 04:45:39.868 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-25 04:45:39.869 DEBUG [src.node.waku_node] Using volumes []
2026-03-25 04:45:39.869 DEBUG [src.node.docker_mananger] docker run -i -t -p 29616:29616 -p 29617:29617 -p 29618:29618 -p 29619:29619 -p 29620:29620 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=29618 --rest-port=29616 --tcp-port=29617 --discv5-udp-port=29619 --rest-address=0.0.0.0 --nat=extip:172.18.67.50 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=fe4feabbfe49cc2f318fa668b18e547c37cfa9e065fa1de7bc06b0c39f05da29 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=29620 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QJiDpUD2Ev0lWAMsm_gzUq1cQXtoMkZaSnmca05D09Coa5_-X4RNZareceDsf2WmRntro-l1GQXzkXJagLZMYVACgmlkgnY0gmlwhKwSyg-KbXVsdGlhZGRyc5YACASsEsoPBlgHAAoErBLKDwZYCN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDqqEjd0VwABNG2KOkDH-HupGwSVKliuo4Oql0cJd6wgKDdGNwglgHg3VkcIJYCYV3YWt1MgU --filternode=/ip4/172.18.202.15/tcp/22535/p2p/16Uiu2HAmQ94AsTTW8xSD8C5A5hj6frgyb4TNwPKMU3LpKmXomEjj
2026-03-25 04:45:40.066 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.67.50 waku 98ea835165bfd585e00078ca7d2210811837cb5f23f6f1d6e6d8048303f2145f
2026-03-25 04:45:40.103 DEBUG [src.node.docker_mananger] Container started with ID 98ea835165bf. Setting up logs at ./log/docker/node4_2026-03-25_04-45-35__bfb2a47e-014b-4404-a181-82c1535e57d8__wakuorg_nwaku:latest.log
2026-03-25 04:45:40.103 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 29616
2026-03-25 04:45:40.104 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-25 04:45:41.104 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:29616/health" -H "Content-Type: application/json" -d 'None'
2026-03-25 04:45:41.107 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-25 04:45:41.108 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-25 04:45:41.108 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:29616/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-25 04:45:41.110 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.67.50/tcp/29617/p2p/16Uiu2HAkxs3KWhgZFvzEthUxdoCJjLjiagiu119RvRMteHoXGtDi","/ip4/172.18.67.50/tcp/29618/ws/p2p/16Uiu2HAkxs3KWhgZFvzEthUxdoCJjLjiagiu119RvRMteHoXGtDi"],"enrUri":"enr:-L24QJaJ7fg1x_b0mXAOiq25v-iFjHTC98e0ZyUj3FabD60ib0RUKSzcspWYqfpFvuUcn12osiBvFI_hb7DF69Ww-JwCgmlkgnY0gmlwhKwSQzKKbXVsdGlhZGRyc5YACASsEkMyBnOxAAoErBJDMgZzst0DgnJzhQADAQAAiXNlY3AyNTZrMaECMxK10EIfkm6QWmthLpnDBjs_oF8snDyJqLJ2eKa4_4WDdGNwgnOxg3VkcIJzs4V3YWt1MgA"}'
2026-03-25 04:45:41.110 INFO [src.node.waku_node] REST service is ready !!
2026-03-25 04:45:41.111 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:29616/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.202.15/tcp/22535/p2p/16Uiu2HAmQ94AsTTW8xSD8C5A5hj6frgyb4TNwPKMU3LpKmXomEjj"]'
2026-03-25 04:45:41.150 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-25 04:45:41.162 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-25 04:45:41.163 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-25_04-45-35__bfb2a47e-014b-4404-a181-82c1535e57d8__wakuorg_nwaku:latest.log
2026-03-25 04:45:41.163 DEBUG [src.node.waku_node] Starting Node...
2026-03-25 04:45:41.163 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-25 04:45:41.166 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-25 04:45:41.166 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.1.160
2026-03-25 04:45:41.167 DEBUG [src.node.docker_mananger] Generated ports ['33109', '33110', '33111', '33112', '33113']
2026-03-25 04:45:41.168 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-25 04:45:41.168 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-25 04:45:41.169 DEBUG [src.node.waku_node] Using volumes []
2026-03-25 04:45:41.169 DEBUG [src.node.docker_mananger] docker run -i -t -p 33109:33109 -p 33110:33110 -p 33111:33111 -p 33112:33112 -p 33113:33113 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=33111 --rest-port=33109 --tcp-port=33110 --discv5-udp-port=33112 --rest-address=0.0.0.0 --nat=extip:172.18.1.160 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=a3cdde6eef0ea9fc9dcad25ab90bfb369ac46a3aa9a0ea17dda5309ac2fce93f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=33113 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QJiDpUD2Ev0lWAMsm_gzUq1cQXtoMkZaSnmca05D09Coa5_-X4RNZareceDsf2WmRntro-l1GQXzkXJagLZMYVACgmlkgnY0gmlwhKwSyg-KbXVsdGlhZGRyc5YACASsEsoPBlgHAAoErBLKDwZYCN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDqqEjd0VwABNG2KOkDH-HupGwSVKliuo4Oql0cJd6wgKDdGNwglgHg3VkcIJYCYV3YWt1MgU --filternode=/ip4/172.18.202.15/tcp/22535/p2p/16Uiu2HAmQ94AsTTW8xSD8C5A5hj6frgyb4TNwPKMU3LpKmXomEjj
2026-03-25 04:45:41.370 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.1.160 waku 659df17e7af2396654e88cd43e57fcf274e8a08f5e2ccd01f328c04fca203676
2026-03-25 04:45:41.404 DEBUG [src.node.docker_mananger] Container started with ID 659df17e7af2. Setting up logs at ./log/docker/node5_2026-03-25_04-45-35__bfb2a47e-014b-4404-a181-82c1535e57d8__wakuorg_nwaku:latest.log
2026-03-25 04:45:41.404 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 33109
2026-03-25 04:45:41.405 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-25 04:45:42.406 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33109/health" -H "Content-Type: application/json" -d 'None'
2026-03-25 04:45:42.409 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-25 04:45:42.409 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-25 04:45:42.409 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33109/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-25 04:45:42.412 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.1.160/tcp/33110/p2p/16Uiu2HAm2whCMtBYfgQKzoQGo9kH953DiLXHHK1cGbXA87PFCU8B","/ip4/172.18.1.160/tcp/33111/ws/p2p/16Uiu2HAm2whCMtBYfgQKzoQGo9kH953DiLXHHK1cGbXA87PFCU8B"],"enrUri":"enr:-L24QNOFGUzQzk5kZgphosdCx0clPMu7LDVecYI2DKXFsUa0cFgTm0nr3XrdcSIp1C8aKqT5B68pRKlkdh7O9AjbRbECgmlkgnY0gmlwhKwSAaCKbXVsdGlhZGRyc5YACASsEgGgBoFWAAoErBIBoAaBV90DgnJzhQADAQAAiXNlY3AyNTZrMaECb7Kbw_xu8BlHBszxtz8lQec77musg5E2CYDhQw3UisSDdGNwgoFWg3VkcIKBWIV3YWt1MgA"}'
2026-03-25 04:45:42.412 INFO [src.node.waku_node] REST service is ready !!
2026-03-25 04:45:42.412 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:33109/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.202.15/tcp/22535/p2p/16Uiu2HAmQ94AsTTW8xSD8C5A5hj6frgyb4TNwPKMU3LpKmXomEjj"]'
2026-03-25 04:45:42.450 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-25 04:45:42.451 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22534/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-25 04:45:42.457 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-25 04:45:42.458 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6348/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "66590e3e-4d63-46d1-817c-8d458525ed4d", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-25 04:45:42.471 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"66590e3e-4d63-46d1-817c-8d458525ed4d","statusDesc":"OK"}'
2026-03-25 04:45:42.473 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:44280/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "c2820e51-814a-429b-b763-c2d8222dc785", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-25 04:45:42.483 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"c2820e51-814a-429b-b763-c2d8222dc785","statusDesc":"OK"}'
2026-03-25 04:45:42.484 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:29616/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "78670c23-9ac9-4426-9ca8-21fcbd8ffc9a", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-25 04:45:42.493 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"78670c23-9ac9-4426-9ca8-21fcbd8ffc9a","statusDesc":"OK"}'
2026-03-25 04:45:42.494 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:33109/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "f464890b-d074-4e1f-85a3-d9e4e01960fa", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-25 04:45:42.503 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"f464890b-d074-4e1f-85a3-d9e4e01960fa","statusDesc":"OK"}'
2026-03-25 04:45:42.503 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22534/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": '$(date +%s%N)'}'
2026-03-25 04:45:42.520 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-25 04:45:42.521 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-25 04:45:42.622 DEBUG [src.steps.filter] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-25 04:45:42.622 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6348/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-25 04:45:42.626 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774413942503850442,"ephemeral":false}]'
2026-03-25 04:45:42.628 DEBUG [src.steps.filter] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-03-25 04:45:42.628 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:44280/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-25 04:45:42.631 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774413942503850442,"ephemeral":false}]'
2026-03-25 04:45:42.633 DEBUG [src.steps.filter] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-03-25 04:45:42.633 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:29616/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-25 04:45:42.636 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774413942503850442,"ephemeral":false}]'
2026-03-25 04:45:42.637 DEBUG [src.steps.filter] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-03-25 04:45:42.638 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33109/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-25 04:45:42.640 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774413942503850442,"ephemeral":false}]'
2026-03-25 04:45:42.642 DEBUG [src.node.waku_node] Pausing container with id 83add069ffbb
2026-03-25 04:45:42.651 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22534/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": '$(date +%s%N)'}'
2026-03-25 04:45:42.658 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-25 04:45:42.658 DEBUG [src.node.waku_node] Unpause container with id 83add069ffbb
2026-03-25 04:45:42.667 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22534/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": '$(date +%s%N)'}'
2026-03-25 04:45:42.673 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-25 04:45:42.675 DEBUG [src.libs.common] Sleeping for 0.5 seconds
2026-03-25 04:45:43.176 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6348/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-25 04:45:43.179 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774413942641961796,"ephemeral":false},{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1774413942642039040,"ephemeral":false}]'
2026-03-25 04:45:43.181 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-25 04:45:43.181 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-25 04:45:43.182 DEBUG [src.node.waku_node] Stopping container with id 527adb6a2370
2026-03-25 04:45:43.771 DEBUG [src.node.waku_node] Container stopped.
2026-03-25 04:45:43.771 DEBUG [src.node.waku_node] Stopping container with id 83add069ffbb
2026-03-25 04:45:44.340 DEBUG [src.node.waku_node] Container stopped.
2026-03-25 04:45:44.342 DEBUG [src.node.waku_node] Stopping container with id 17d20c51c2a2
2026-03-25 04:45:44.585 ERROR [src.node.docker_mananger] Max retries reached for container 527adb6a2370. Exiting log stream.
2026-03-25 04:45:44.940 DEBUG [src.node.waku_node] Container stopped.
2026-03-25 04:45:44.942 DEBUG [src.node.waku_node] Stopping container with id 98ea835165bf
2026-03-25 04:45:45.177 ERROR [src.node.docker_mananger] Max retries reached for container 83add069ffbb. Exiting log stream.
2026-03-25 04:45:45.483 DEBUG [src.node.waku_node] Container stopped.
2026-03-25 04:45:45.483 DEBUG [src.node.waku_node] Stopping container with id 659df17e7af2
2026-03-25 04:45:45.754 ERROR [src.node.docker_mananger] Max retries reached for container 17d20c51c2a2. Exiting log stream.
2026-03-25 04:45:46.038 DEBUG [src.node.waku_node] Container stopped.
2026-03-25 04:45:46.040 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-25 04:45:46.057 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-25 04:45:46.066 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-25 04:45:46.072 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-25 04:45:46.077 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-25 04:45:46.083 DEBUG [src.node.docker_mananger] No errors found in the waku logs.