177 lines
34 KiB
Plaintext

2026-03-04 04:40:34.265 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-04 04:40:34.265 DEBUG [tests.conftest] Running test: test_filter_get_message_while_one_peer_is_paused with id: 2026-03-04_04-40-34__1590048c-7f4d-440f-9913-ac4dc6e31392
2026-03-04 04:40:34.265 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-04 04:40:34.266 DEBUG [src.steps.filter] Running fixture setup: filter_setup
2026-03-04 04:40:34.266 DEBUG [src.steps.filter] Running fixture setup: setup_main_relay_node
2026-03-04 04:40:34.274 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-04 04:40:34.274 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-04_04-40-34__1590048c-7f4d-440f-9913-ac4dc6e31392__wakuorg_nwaku:latest.log
2026-03-04 04:40:34.275 DEBUG [src.node.waku_node] Starting Node...
2026-03-04 04:40:34.275 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-04 04:40:34.276 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-04 04:40:34.277 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.79.31
2026-03-04 04:40:34.277 DEBUG [src.node.docker_mananger] Generated ports ['2617', '2618', '2619', '2620', '2621']
2026-03-04 04:40:34.277 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-04 04:40:34.277 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-04 04:40:34.277 DEBUG [src.node.waku_node] Using volumes []
2026-03-04 04:40:34.277 DEBUG [src.node.docker_mananger] docker run -i -t -p 2617:2617 -p 2618:2618 -p 2619:2619 -p 2620:2620 -p 2621:2621 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=2619 --rest-port=2617 --tcp-port=2618 --discv5-udp-port=2620 --rest-address=0.0.0.0 --nat=extip:172.18.79.31 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=9a78740c05dca28feede4ee5ff1c16acae2af9ebd8bbc5f9bd9a527b469a341b --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=2621 --metrics-logging=true --relay=true --filter=true
2026-03-04 04:40:34.486 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.79.31 waku cbdb80763d97e9a30660d9ece9d65f64a846be7f4b0bd89c0d8096009b9a93fb
2026-03-04 04:40:34.521 DEBUG [src.node.docker_mananger] Container started with ID cbdb80763d97. Setting up logs at ./log/docker/node1_2026-03-04_04-40-34__1590048c-7f4d-440f-9913-ac4dc6e31392__wakuorg_nwaku:latest.log
2026-03-04 04:40:34.521 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 2617
2026-03-04 04:40:34.521 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-04 04:40:34.576 ERROR [src.node.docker_mananger] Max retries reached for container db2ad31ffd76. Exiting log stream.
2026-03-04 04:40:34.924 ERROR [src.node.docker_mananger] Max retries reached for container 12b566e2c8a6. Exiting log stream.
2026-03-04 04:40:35.522 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2617/health" -H "Content-Type: application/json" -d 'None'
2026-03-04 04:40:35.525 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-04 04:40:35.525 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-04 04:40:35.525 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2617/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-04 04:40:35.528 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.79.31/tcp/2618/p2p/16Uiu2HAkwxoLVqKb8J24SK8MbinFAMvRpP8bzApff9s1jNqjhYUT","/ip4/172.18.79.31/tcp/2619/ws/p2p/16Uiu2HAkwxoLVqKb8J24SK8MbinFAMvRpP8bzApff9s1jNqjhYUT"],"enrUri":"enr:-L24QHvOElRqEG6LCBDptj_vQz8kO-6ZuwaX8cpWNVGLxZ9sQjBi-U68gJ2vzcXMcGppMn44fb-P_Lv9217MQ8-GJlECgmlkgnY0gmlwhKwSTx-KbXVsdGlhZGRyc5YACASsEk8fBgo6AAoErBJPHwYKO90DgnJzhQADAQAAiXNlY3AyNTZrMaECJbCyjq8nP2EH-aw7XqIVoVNov68-dnBMOvre2vl0prSDdGNwggo6g3VkcIIKPIV3YWt1MgU"}'
2026-03-04 04:40:35.528 INFO [src.node.waku_node] REST service is ready !!
2026-03-04 04:40:35.528 DEBUG [src.steps.filter] Running fixture setup: setup_main_filter_node
2026-03-04 04:40:35.535 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-04 04:40:35.535 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-04_04-40-34__1590048c-7f4d-440f-9913-ac4dc6e31392__wakuorg_nwaku:latest.log
2026-03-04 04:40:35.536 DEBUG [src.node.waku_node] Starting Node...
2026-03-04 04:40:35.536 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-04 04:40:35.537 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-04 04:40:35.537 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.226.119
2026-03-04 04:40:35.538 DEBUG [src.node.docker_mananger] Generated ports ['3311', '3312', '3313', '3314', '3315']
2026-03-04 04:40:35.538 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-04 04:40:35.538 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-04 04:40:35.538 DEBUG [src.node.waku_node] Using volumes []
2026-03-04 04:40:35.538 DEBUG [src.node.docker_mananger] docker run -i -t -p 3311:3311 -p 3312:3312 -p 3313:3313 -p 3314:3314 -p 3315:3315 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=3313 --rest-port=3311 --tcp-port=3312 --discv5-udp-port=3314 --rest-address=0.0.0.0 --nat=extip:172.18.226.119 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=93b0cfed0f06d23eaacf48d27dbd6cf14f1becc39d9be3679bf9dfdb59b86bec --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=3315 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QHvOElRqEG6LCBDptj_vQz8kO-6ZuwaX8cpWNVGLxZ9sQjBi-U68gJ2vzcXMcGppMn44fb-P_Lv9217MQ8-GJlECgmlkgnY0gmlwhKwSTx-KbXVsdGlhZGRyc5YACASsEk8fBgo6AAoErBJPHwYKO90DgnJzhQADAQAAiXNlY3AyNTZrMaECJbCyjq8nP2EH-aw7XqIVoVNov68-dnBMOvre2vl0prSDdGNwggo6g3VkcIIKPIV3YWt1MgU --filternode=/ip4/172.18.79.31/tcp/2618/p2p/16Uiu2HAkwxoLVqKb8J24SK8MbinFAMvRpP8bzApff9s1jNqjhYUT
2026-03-04 04:40:35.750 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.226.119 waku 9667be8926c9eaca8800f1d74b830c8a343a757d204bb02e290f2b549e96880e
2026-03-04 04:40:35.783 DEBUG [src.node.docker_mananger] Container started with ID 9667be8926c9. Setting up logs at ./log/docker/node2_2026-03-04_04-40-34__1590048c-7f4d-440f-9913-ac4dc6e31392__wakuorg_nwaku:latest.log
2026-03-04 04:40:35.783 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 3311
2026-03-04 04:40:35.784 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-04 04:40:36.785 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3311/health" -H "Content-Type: application/json" -d 'None'
2026-03-04 04:40:36.788 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-04 04:40:36.788 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-04 04:40:36.788 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3311/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-04 04:40:36.791 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.226.119/tcp/3312/p2p/16Uiu2HAmTLo7aYcvBY1wntd9nLGaqnzrWhD34FYDq4LRdqQJJHYq","/ip4/172.18.226.119/tcp/3313/ws/p2p/16Uiu2HAmTLo7aYcvBY1wntd9nLGaqnzrWhD34FYDq4LRdqQJJHYq"],"enrUri":"enr:-L24QPuM_o0mnv-JQupl6gSq9_qdq0mAaRo1D80LBaggvL1edE7eSxf02TPhaSGryrjhQfp3f6PKa3gQB2nCM9SMW3ECgmlkgnY0gmlwhKwS4neKbXVsdGlhZGRyc5YACASsEuJ3BgzwAAoErBLidwYM8d0DgnJzhQADAQAAiXNlY3AyNTZrMaED2jYk8RgBXnaI4Y4Q-wuB-uyusmwDPAgObmF0lbmCnY6DdGNwggzwg3VkcIIM8oV3YWt1MgA"}'
2026-03-04 04:40:36.791 INFO [src.node.waku_node] REST service is ready !!
2026-03-04 04:40:36.791 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:3311/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.79.31/tcp/2618/p2p/16Uiu2HAkwxoLVqKb8J24SK8MbinFAMvRpP8bzApff9s1jNqjhYUT"]'
2026-03-04 04:40:36.831 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-04 04:40:36.843 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-04 04:40:36.843 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-04_04-40-34__1590048c-7f4d-440f-9913-ac4dc6e31392__wakuorg_nwaku:latest.log
2026-03-04 04:40:36.843 DEBUG [src.node.waku_node] Starting Node...
2026-03-04 04:40:36.843 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-04 04:40:36.845 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-04 04:40:36.845 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.188.254
2026-03-04 04:40:36.845 DEBUG [src.node.docker_mananger] Generated ports ['25787', '25788', '25789', '25790', '25791']
2026-03-04 04:40:36.845 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-04 04:40:36.846 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-04 04:40:36.846 DEBUG [src.node.waku_node] Using volumes []
2026-03-04 04:40:36.846 DEBUG [src.node.docker_mananger] docker run -i -t -p 25787:25787 -p 25788:25788 -p 25789:25789 -p 25790:25790 -p 25791:25791 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=25789 --rest-port=25787 --tcp-port=25788 --discv5-udp-port=25790 --rest-address=0.0.0.0 --nat=extip:172.18.188.254 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=1c773a17b9ce8e7fea0f0ab91dacc4e9acc1badec3e71ab0a8fec6a8c6fce5fb --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=25791 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QHvOElRqEG6LCBDptj_vQz8kO-6ZuwaX8cpWNVGLxZ9sQjBi-U68gJ2vzcXMcGppMn44fb-P_Lv9217MQ8-GJlECgmlkgnY0gmlwhKwSTx-KbXVsdGlhZGRyc5YACASsEk8fBgo6AAoErBJPHwYKO90DgnJzhQADAQAAiXNlY3AyNTZrMaECJbCyjq8nP2EH-aw7XqIVoVNov68-dnBMOvre2vl0prSDdGNwggo6g3VkcIIKPIV3YWt1MgU --filternode=/ip4/172.18.79.31/tcp/2618/p2p/16Uiu2HAkwxoLVqKb8J24SK8MbinFAMvRpP8bzApff9s1jNqjhYUT
2026-03-04 04:40:37.057 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.188.254 waku 7b1c6ad3cedf9e82a48a399cb885ba07cef18f96267795ec6859b0be0f42fe85
2026-03-04 04:40:37.089 DEBUG [src.node.docker_mananger] Container started with ID 7b1c6ad3cedf. Setting up logs at ./log/docker/node3_2026-03-04_04-40-34__1590048c-7f4d-440f-9913-ac4dc6e31392__wakuorg_nwaku:latest.log
2026-03-04 04:40:37.089 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 25787
2026-03-04 04:40:37.089 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-04 04:40:38.090 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25787/health" -H "Content-Type: application/json" -d 'None'
2026-03-04 04:40:38.093 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-04 04:40:38.093 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-04 04:40:38.093 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25787/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-04 04:40:38.095 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.188.254/tcp/25788/p2p/16Uiu2HAmHnRmRhjoyRU27iLcqSNoR5Q3NnQFGtfmWisHG2S8hwPQ","/ip4/172.18.188.254/tcp/25789/ws/p2p/16Uiu2HAmHnRmRhjoyRU27iLcqSNoR5Q3NnQFGtfmWisHG2S8hwPQ"],"enrUri":"enr:-L24QGP2Bz3f7VXHl97VWytJCYr3FY80I-njIpl9nwnwfg7yfjufvk8kDBxJhRsCJZY0pEewzbyptVfjD9GB5i8Em9YCgmlkgnY0gmlwhKwSvP6KbXVsdGlhZGRyc5YACASsErz-BmS8AAoErBK8_gZkvd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDTDI9CcgiEe04ydXQ5GvMENg5m-MA2g3XqW_KtaeybLuDdGNwgmS8g3VkcIJkvoV3YWt1MgA"}'
2026-03-04 04:40:38.096 INFO [src.node.waku_node] REST service is ready !!
2026-03-04 04:40:38.096 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:25787/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.79.31/tcp/2618/p2p/16Uiu2HAkwxoLVqKb8J24SK8MbinFAMvRpP8bzApff9s1jNqjhYUT"]'
2026-03-04 04:40:38.130 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-04 04:40:38.144 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-04 04:40:38.145 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-04_04-40-34__1590048c-7f4d-440f-9913-ac4dc6e31392__wakuorg_nwaku:latest.log
2026-03-04 04:40:38.145 DEBUG [src.node.waku_node] Starting Node...
2026-03-04 04:40:38.145 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-04 04:40:38.149 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-04 04:40:38.150 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.174.222
2026-03-04 04:40:38.150 DEBUG [src.node.docker_mananger] Generated ports ['18613', '18614', '18615', '18616', '18617']
2026-03-04 04:40:38.151 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-04 04:40:38.152 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-04 04:40:38.152 DEBUG [src.node.waku_node] Using volumes []
2026-03-04 04:40:38.152 DEBUG [src.node.docker_mananger] docker run -i -t -p 18613:18613 -p 18614:18614 -p 18615:18615 -p 18616:18616 -p 18617:18617 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=18615 --rest-port=18613 --tcp-port=18614 --discv5-udp-port=18616 --rest-address=0.0.0.0 --nat=extip:172.18.174.222 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=4cb7aeeaf76b3d809c4cafbb23d085cc888bb49be54fdbb6be24eb4ffad0a3a4 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=18617 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QHvOElRqEG6LCBDptj_vQz8kO-6ZuwaX8cpWNVGLxZ9sQjBi-U68gJ2vzcXMcGppMn44fb-P_Lv9217MQ8-GJlECgmlkgnY0gmlwhKwSTx-KbXVsdGlhZGRyc5YACASsEk8fBgo6AAoErBJPHwYKO90DgnJzhQADAQAAiXNlY3AyNTZrMaECJbCyjq8nP2EH-aw7XqIVoVNov68-dnBMOvre2vl0prSDdGNwggo6g3VkcIIKPIV3YWt1MgU --filternode=/ip4/172.18.79.31/tcp/2618/p2p/16Uiu2HAkwxoLVqKb8J24SK8MbinFAMvRpP8bzApff9s1jNqjhYUT
2026-03-04 04:40:38.372 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.174.222 waku 292a4b7c34de487d93b52c454cc6f2ab3ff3b9e718fe933c640069e459f32162
2026-03-04 04:40:38.410 DEBUG [src.node.docker_mananger] Container started with ID 292a4b7c34de. Setting up logs at ./log/docker/node4_2026-03-04_04-40-34__1590048c-7f4d-440f-9913-ac4dc6e31392__wakuorg_nwaku:latest.log
2026-03-04 04:40:38.410 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 18613
2026-03-04 04:40:38.410 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-04 04:40:39.411 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18613/health" -H "Content-Type: application/json" -d 'None'
2026-03-04 04:40:39.414 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-04 04:40:39.414 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-04 04:40:39.414 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18613/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-04 04:40:39.417 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.174.222/tcp/18614/p2p/16Uiu2HAmAFWkBri8E4eKJ7mX1CWYoxPLXrqNseR2wqpzhrC1EyBP","/ip4/172.18.174.222/tcp/18615/ws/p2p/16Uiu2HAmAFWkBri8E4eKJ7mX1CWYoxPLXrqNseR2wqpzhrC1EyBP"],"enrUri":"enr:-L24QBGqZkaxUSMw4sN9w-3EbwZ8O5M5NPv6XlzQ1jhXEkp1TQZwCOP-caBGI0Kr2sRac8eK3eWcA28_8kmgfVZOmg8CgmlkgnY0gmlwhKwSrt6KbXVsdGlhZGRyc5YACASsEq7eBki2AAoErBKu3gZIt90DgnJzhQADAQAAiXNlY3AyNTZrMaEC3EUN4kpzhkh994vzzJkjpS4UaBJfRkikhE1wJq15mWKDdGNwgki2g3VkcIJIuIV3YWt1MgA"}'
2026-03-04 04:40:39.417 INFO [src.node.waku_node] REST service is ready !!
2026-03-04 04:40:39.417 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:18613/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.79.31/tcp/2618/p2p/16Uiu2HAkwxoLVqKb8J24SK8MbinFAMvRpP8bzApff9s1jNqjhYUT"]'
2026-03-04 04:40:39.453 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-04 04:40:39.468 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-04 04:40:39.468 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-04_04-40-34__1590048c-7f4d-440f-9913-ac4dc6e31392__wakuorg_nwaku:latest.log
2026-03-04 04:40:39.469 DEBUG [src.node.waku_node] Starting Node...
2026-03-04 04:40:39.469 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-04 04:40:39.474 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-04 04:40:39.474 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.150.241
2026-03-04 04:40:39.474 DEBUG [src.node.docker_mananger] Generated ports ['58809', '58810', '58811', '58812', '58813']
2026-03-04 04:40:39.474 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-04 04:40:39.474 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-04 04:40:39.475 DEBUG [src.node.waku_node] Using volumes []
2026-03-04 04:40:39.475 DEBUG [src.node.docker_mananger] docker run -i -t -p 58809:58809 -p 58810:58810 -p 58811:58811 -p 58812:58812 -p 58813:58813 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=58811 --rest-port=58809 --tcp-port=58810 --discv5-udp-port=58812 --rest-address=0.0.0.0 --nat=extip:172.18.150.241 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=f4e829e79efa33c04c4aaee89fb1472b4bde6dbfda2f6f14aeacaedbbbaa310b --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=58813 --metrics-logging=true --relay=false --discv5-bootstrap-node=enr:-L24QHvOElRqEG6LCBDptj_vQz8kO-6ZuwaX8cpWNVGLxZ9sQjBi-U68gJ2vzcXMcGppMn44fb-P_Lv9217MQ8-GJlECgmlkgnY0gmlwhKwSTx-KbXVsdGlhZGRyc5YACASsEk8fBgo6AAoErBJPHwYKO90DgnJzhQADAQAAiXNlY3AyNTZrMaECJbCyjq8nP2EH-aw7XqIVoVNov68-dnBMOvre2vl0prSDdGNwggo6g3VkcIIKPIV3YWt1MgU --filternode=/ip4/172.18.79.31/tcp/2618/p2p/16Uiu2HAkwxoLVqKb8J24SK8MbinFAMvRpP8bzApff9s1jNqjhYUT
2026-03-04 04:40:39.681 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.150.241 waku 0970e288ad92dda744bd5a0e8d3aa519a14747784b0236173c33052e4b2eadf3
2026-03-04 04:40:39.718 DEBUG [src.node.docker_mananger] Container started with ID 0970e288ad92. Setting up logs at ./log/docker/node5_2026-03-04_04-40-34__1590048c-7f4d-440f-9913-ac4dc6e31392__wakuorg_nwaku:latest.log
2026-03-04 04:40:39.719 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 58809
2026-03-04 04:40:39.720 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-04 04:40:40.721 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58809/health" -H "Content-Type: application/json" -d 'None'
2026-03-04 04:40:40.724 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-04 04:40:40.724 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-04 04:40:40.724 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58809/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-04 04:40:40.726 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.150.241/tcp/58810/p2p/16Uiu2HAmH2q1Msg3WiujM6eaMdj3wLySEaq5mG2cpsRXzuUsQ68s","/ip4/172.18.150.241/tcp/58811/ws/p2p/16Uiu2HAmH2q1Msg3WiujM6eaMdj3wLySEaq5mG2cpsRXzuUsQ68s"],"enrUri":"enr:-L24QNtxVa9go5fBzORo1EsCu4XvgSWMjnoBmS0Su-f-tTUYHAtMVU0N0wQksouaMx1uGd_NTSMr0sANG4HUzxnvWVICgmlkgnY0gmlwhKwSlvGKbXVsdGlhZGRyc5YACASsEpbxBuW6AAoErBKW8Qblu90DgnJzhQADAQAAiXNlY3AyNTZrMaEDQQb17Ye0F4bMWFzLGJZpu8X05uHE6FyanhcGei48K5SDdGNwguW6g3VkcILlvIV3YWt1MgA"}'
2026-03-04 04:40:40.727 INFO [src.node.waku_node] REST service is ready !!
2026-03-04 04:40:40.727 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:58809/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.79.31/tcp/2618/p2p/16Uiu2HAkwxoLVqKb8J24SK8MbinFAMvRpP8bzApff9s1jNqjhYUT"]'
2026-03-04 04:40:40.761 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-04 04:40:40.762 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2617/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-04 04:40:40.772 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-04 04:40:40.775 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:3311/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "ab668828-a2c8-4a93-8b6d-36c93730512c", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-04 04:40:40.786 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"ab668828-a2c8-4a93-8b6d-36c93730512c","statusDesc":"OK"}'
2026-03-04 04:40:40.790 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:25787/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "add92a9a-4b54-4ad2-a5e8-959cb1b1cb44", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-04 04:40:40.801 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"add92a9a-4b54-4ad2-a5e8-959cb1b1cb44","statusDesc":"OK"}'
2026-03-04 04:40:40.802 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:18613/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "dcb6b78a-ccd3-4d89-9c8f-6cf27db6f728", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-04 04:40:40.812 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"dcb6b78a-ccd3-4d89-9c8f-6cf27db6f728","statusDesc":"OK"}'
2026-03-04 04:40:40.812 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:58809/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "47b076ca-b053-485c-8bfb-6051d171e7b7", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-04 04:40:40.821 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"47b076ca-b053-485c-8bfb-6051d171e7b7","statusDesc":"OK"}'
2026-03-04 04:40:40.822 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2617/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": '$(date +%s%N)'}'
2026-03-04 04:40:40.838 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-04 04:40:40.840 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-04 04:40:40.942 DEBUG [src.steps.filter] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-04 04:40:40.942 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3311/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-04 04:40:40.945 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1772599240822430591,"ephemeral":false}]'
2026-03-04 04:40:40.947 DEBUG [src.steps.filter] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-03-04 04:40:40.947 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25787/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-04 04:40:40.950 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1772599240822430591,"ephemeral":false}]'
2026-03-04 04:40:40.951 DEBUG [src.steps.filter] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-03-04 04:40:40.952 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18613/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-04 04:40:40.954 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1772599240822430591,"ephemeral":false}]'
2026-03-04 04:40:40.955 DEBUG [src.steps.filter] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-03-04 04:40:40.956 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58809/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-04 04:40:40.958 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1772599240822430591,"ephemeral":false}]'
2026-03-04 04:40:40.960 DEBUG [src.node.waku_node] Pausing container with id 9667be8926c9
2026-03-04 04:40:40.970 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2617/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": '$(date +%s%N)'}'
2026-03-04 04:40:40.976 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-04 04:40:40.977 DEBUG [src.node.waku_node] Unpause container with id 9667be8926c9
2026-03-04 04:40:40.985 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2617/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": '$(date +%s%N)'}'
2026-03-04 04:40:40.991 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-04 04:40:40.992 DEBUG [src.libs.common] Sleeping for 0.5 seconds
2026-03-04 04:40:41.493 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3311/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-04 04:40:41.496 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1772599240960145531,"ephemeral":false},{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1772599240960256149,"ephemeral":false}]'
2026-03-04 04:40:41.498 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-04 04:40:41.500 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-04 04:40:41.500 DEBUG [src.node.waku_node] Stopping container with id cbdb80763d97
2026-03-04 04:40:42.094 DEBUG [src.node.waku_node] Container stopped.
2026-03-04 04:40:42.094 DEBUG [src.node.waku_node] Stopping container with id 9667be8926c9
2026-03-04 04:40:42.685 DEBUG [src.node.waku_node] Container stopped.
2026-03-04 04:40:42.685 DEBUG [src.node.waku_node] Stopping container with id 7b1c6ad3cedf
2026-03-04 04:40:42.908 ERROR [src.node.docker_mananger] Max retries reached for container cbdb80763d97. Exiting log stream.
2026-03-04 04:40:43.297 DEBUG [src.node.waku_node] Container stopped.
2026-03-04 04:40:43.298 DEBUG [src.node.waku_node] Stopping container with id 292a4b7c34de
2026-03-04 04:40:43.501 ERROR [src.node.docker_mananger] Max retries reached for container 9667be8926c9. Exiting log stream.
2026-03-04 04:40:43.857 DEBUG [src.node.waku_node] Container stopped.
2026-03-04 04:40:43.857 DEBUG [src.node.waku_node] Stopping container with id 0970e288ad92
2026-03-04 04:40:44.112 ERROR [src.node.docker_mananger] Max retries reached for container 7b1c6ad3cedf. Exiting log stream.
2026-03-04 04:40:44.425 DEBUG [src.node.waku_node] Container stopped.
2026-03-04 04:40:44.428 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-04 04:40:44.442 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-04 04:40:44.448 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-04 04:40:44.454 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-04 04:40:44.461 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-04 04:40:44.469 DEBUG [src.node.docker_mananger] No errors found in the waku logs.