180 lines
32 KiB
Plaintext

2026-02-02 04:48:57.153 DEBUG [tests.conftest] Running fixture setup: test_id
2026-02-02 04:48:57.154 DEBUG [tests.conftest] Running test: test_static_shard_relay with id: 2026-02-02_04-48-57__1d5f6742-59da-439d-823f-0beb52b8cc89
2026-02-02 04:48:57.154 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-02-02 04:48:57.156 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-02-02 04:48:57.157 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-02-02 04:48:57.168 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-02 04:48:57.168 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-02-02_04-48-57__1d5f6742-59da-439d-823f-0beb52b8cc89__wakuorg_nwaku:latest.log
2026-02-02 04:48:57.168 DEBUG [src.node.waku_node] Starting Node...
2026-02-02 04:48:57.168 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-02 04:48:57.170 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-02 04:48:57.170 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.3.11
2026-02-02 04:48:57.171 DEBUG [src.node.docker_mananger] Generated ports ['19435', '19436', '19437', '19438', '19439']
2026-02-02 04:48:57.172 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-02-02 04:48:57.172 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-02-02 04:48:57.172 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-02 04:48:57.174 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-02 04:48:57.175 DEBUG [src.node.waku_node] Using volumes []
2026-02-02 04:48:57.175 DEBUG [src.node.docker_mananger] docker run -i -t -p 19435:19435 -p 19436:19436 -p 19437:19437 -p 19438:19438 -p 19439:19439 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=19437 --rest-port=19435 --tcp-port=19436 --discv5-udp-port=19438 --rest-address=0.0.0.0 --nat=extip:172.18.3.11 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=6dffe4f5f0dac2f919fdad0ad07de8ffcbfaa9ddfcccdadce3557ecdd865ea35 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=19439 --metrics-logging=true --relay=true --filter=true
2026-02-02 04:48:57.357 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.3.11 waku 0b79a68652e0aa0ba36075534b34d9bb3efa3d13e17ed18acc6d3d4da99520c7
2026-02-02 04:48:57.388 DEBUG [src.node.docker_mananger] Container started with ID 0b79a68652e0. Setting up logs at ./log/docker/node1_2026-02-02_04-48-57__1d5f6742-59da-439d-823f-0beb52b8cc89__wakuorg_nwaku:latest.log
2026-02-02 04:48:57.388 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 19435
2026-02-02 04:48:57.389 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-02 04:48:57.399 ERROR [src.node.docker_mananger] Max retries reached for container ff6f9c1d9531. Exiting log stream.
2026-02-02 04:48:57.988 ERROR [src.node.docker_mananger] Max retries reached for container 952873855f28. Exiting log stream.
2026-02-02 04:48:58.390 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:19435/health" -H "Content-Type: application/json" -d 'None'
2026-02-02 04:48:58.393 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"}]}'
2026-02-02 04:48:58.394 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-02 04:48:58.394 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:19435/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-02 04:48:58.396 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.3.11/tcp/19436/p2p/16Uiu2HAmCR1K7ezmRyz8eBEfi1mcNNjas5AkTHv1BgxzWA5621he","/ip4/172.18.3.11/tcp/19437/ws/p2p/16Uiu2HAmCR1K7ezmRyz8eBEfi1mcNNjas5AkTHv1BgxzWA5621he"],"enrUri":"enr:-L24QOtag931U4DG-TSgdO6BRbJg9-0LzuE85VOYSSisIB6kGCopoIcjHLmTgn5N9T8k01dXibTjRMxyt7rQTPmrWkACgmlkgnY0gmlwhKwSAwuKbXVsdGlhZGRyc5YACASsEgMLBkvsAAoErBIDCwZL7d0DgnJzhQACAQAAiXNlY3AyNTZrMaEC_Gr0_TKsQN4-DmGc_kfCDYy50ZoGu6xvVemJXEw7nu2DdGNwgkvsg3VkcIJL7oV3YWt1MgU"}'
2026-02-02 04:48:58.396 INFO [src.node.waku_node] REST service is ready !!
2026-02-02 04:48:58.403 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-02 04:48:58.403 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-02-02_04-48-57__1d5f6742-59da-439d-823f-0beb52b8cc89__wakuorg_nwaku:latest.log
2026-02-02 04:48:58.403 DEBUG [src.node.waku_node] Starting Node...
2026-02-02 04:48:58.403 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-02 04:48:58.405 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-02 04:48:58.405 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.99.191
2026-02-02 04:48:58.405 DEBUG [src.node.docker_mananger] Generated ports ['17330', '17331', '17332', '17333', '17334']
2026-02-02 04:48:58.405 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-02-02 04:48:58.405 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-02-02 04:48:58.405 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-02 04:48:58.405 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-02 04:48:58.406 DEBUG [src.node.waku_node] Using volumes []
2026-02-02 04:48:58.406 DEBUG [src.node.docker_mananger] docker run -i -t -p 17330:17330 -p 17331:17331 -p 17332:17332 -p 17333:17333 -p 17334:17334 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=17332 --rest-port=17330 --tcp-port=17331 --discv5-udp-port=17333 --rest-address=0.0.0.0 --nat=extip:172.18.99.191 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=b0ddf7d7bb09fb1dfcadb8deb7dc9390b0b4f4c32430bae964cddbb30745c92b --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=17334 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QOtag931U4DG-TSgdO6BRbJg9-0LzuE85VOYSSisIB6kGCopoIcjHLmTgn5N9T8k01dXibTjRMxyt7rQTPmrWkACgmlkgnY0gmlwhKwSAwuKbXVsdGlhZGRyc5YACASsEgMLBkvsAAoErBIDCwZL7d0DgnJzhQACAQAAiXNlY3AyNTZrMaEC_Gr0_TKsQN4-DmGc_kfCDYy50ZoGu6xvVemJXEw7nu2DdGNwgkvsg3VkcIJL7oV3YWt1MgU
2026-02-02 04:48:58.597 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.99.191 waku 97ba4ddcdc2d18ee567a3205f50ad3fd6f3d5db9c04a93898d50b91e3ed0e170
2026-02-02 04:48:58.628 DEBUG [src.node.docker_mananger] Container started with ID 97ba4ddcdc2d. Setting up logs at ./log/docker/node2_2026-02-02_04-48-57__1d5f6742-59da-439d-823f-0beb52b8cc89__wakuorg_nwaku:latest.log
2026-02-02 04:48:58.628 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 17330
2026-02-02 04:48:58.629 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-02 04:48:59.630 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17330/health" -H "Content-Type: application/json" -d 'None'
2026-02-02 04:48:59.641 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"}]}'
2026-02-02 04:48:59.642 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-02 04:48:59.643 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17330/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-02 04:48:59.647 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.99.191/tcp/17331/p2p/16Uiu2HAm1GdC8biiQ8hYPFCWKKtJDN5LcoJcFXhanmaNE6ZPuZMG","/ip4/172.18.99.191/tcp/17332/ws/p2p/16Uiu2HAm1GdC8biiQ8hYPFCWKKtJDN5LcoJcFXhanmaNE6ZPuZMG"],"enrUri":"enr:-L24QB7MiqHdduqBhYzMGF2GzjdhJQwbbRqji6SMGkpb-WTxYZNqu7Avt4ulTxTV07DOGpehPAHo735D2Me0RY8N6TQCgmlkgnY0gmlwhKwSY7-KbXVsdGlhZGRyc5YACASsEmO_BkOzAAoErBJjvwZDtN0DgnJzhQACAQAAiXNlY3AyNTZrMaECVtS-IY_a6Kv1Yjrt9J51EsBI3OOWdbbaA7wgtm-wK1eDdGNwgkOzg3VkcIJDtYV3YWt1MgE"}'
2026-02-02 04:48:59.647 INFO [src.node.waku_node] REST service is ready !!
2026-02-02 04:48:59.648 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:17330/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.3.11/tcp/19436/p2p/16Uiu2HAmCR1K7ezmRyz8eBEfi1mcNNjas5AkTHv1BgxzWA5621he"]'
2026-02-02 04:48:59.651 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-02 04:48:59.658 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-02 04:48:59.658 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-02-02_04-48-57__1d5f6742-59da-439d-823f-0beb52b8cc89__wakuorg_nwaku:latest.log
2026-02-02 04:48:59.658 DEBUG [src.node.waku_node] Starting Node...
2026-02-02 04:48:59.659 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-02 04:48:59.660 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-02 04:48:59.660 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.167.174
2026-02-02 04:48:59.660 DEBUG [src.node.docker_mananger] Generated ports ['16078', '16079', '16080', '16081', '16082']
2026-02-02 04:48:59.660 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-02-02 04:48:59.661 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-02-02 04:48:59.661 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-02 04:48:59.661 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-02 04:48:59.661 DEBUG [src.node.waku_node] Using volumes []
2026-02-02 04:48:59.661 DEBUG [src.node.docker_mananger] docker run -i -t -p 16078:16078 -p 16079:16079 -p 16080:16080 -p 16081:16081 -p 16082:16082 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=16080 --rest-port=16078 --tcp-port=16079 --discv5-udp-port=16081 --rest-address=0.0.0.0 --nat=extip:172.18.167.174 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=cf4d049c8cb5b9bc3bedd4387534a69ddfacacb6ffc36862bccff8e95de80aa8 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=16082 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QOtag931U4DG-TSgdO6BRbJg9-0LzuE85VOYSSisIB6kGCopoIcjHLmTgn5N9T8k01dXibTjRMxyt7rQTPmrWkACgmlkgnY0gmlwhKwSAwuKbXVsdGlhZGRyc5YACASsEgMLBkvsAAoErBIDCwZL7d0DgnJzhQACAQAAiXNlY3AyNTZrMaEC_Gr0_TKsQN4-DmGc_kfCDYy50ZoGu6xvVemJXEw7nu2DdGNwgkvsg3VkcIJL7oV3YWt1MgU
2026-02-02 04:48:59.855 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.167.174 waku 75a7ff3b93d25d8e542a44e4c9126d21e65f9cce8f72d9fd32cf5cfee49761ea
2026-02-02 04:48:59.885 DEBUG [src.node.docker_mananger] Container started with ID 75a7ff3b93d2. Setting up logs at ./log/docker/node3_2026-02-02_04-48-57__1d5f6742-59da-439d-823f-0beb52b8cc89__wakuorg_nwaku:latest.log
2026-02-02 04:48:59.886 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 16078
2026-02-02 04:48:59.886 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-02 04:49:00.887 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16078/health" -H "Content-Type: application/json" -d 'None'
2026-02-02 04:49:00.898 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"}]}'
2026-02-02 04:49:00.898 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-02 04:49:00.899 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16078/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-02 04:49:00.904 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.167.174/tcp/16079/p2p/16Uiu2HAm3fthy7kdh4h5S9SH9mik2SZ7jL4YBFheQie4joupwNJt","/ip4/172.18.167.174/tcp/16080/ws/p2p/16Uiu2HAm3fthy7kdh4h5S9SH9mik2SZ7jL4YBFheQie4joupwNJt"],"enrUri":"enr:-L24QNiCMuj7yxqtjREZl7ChdFCt_fm9VnA_H_0nmSlhr0dEDl6-Vh9AiVusuki2oA3OMAUnWnFtgDZbH72PVgHUvWECgmlkgnY0gmlwhKwSp66KbXVsdGlhZGRyc5YACASsEqeuBj7PAAoErBKnrgY-0N0DgnJzhQACAQAAiXNlY3AyNTZrMaECeoIEcqqJobXVL3ghMV9PBFdM43Q-seBtMagckawmkGGDdGNwgj7Pg3VkcII-0YV3YWt1MgE"}'
2026-02-02 04:49:00.904 INFO [src.node.waku_node] REST service is ready !!
2026-02-02 04:49:00.905 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:16078/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.3.11/tcp/19436/p2p/16Uiu2HAmCR1K7ezmRyz8eBEfi1mcNNjas5AkTHv1BgxzWA5621he"]'
2026-02-02 04:49:00.909 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-02 04:49:00.916 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-02 04:49:00.916 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-02-02_04-48-57__1d5f6742-59da-439d-823f-0beb52b8cc89__wakuorg_nwaku:latest.log
2026-02-02 04:49:00.916 DEBUG [src.node.waku_node] Starting Node...
2026-02-02 04:49:00.916 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-02 04:49:00.918 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-02 04:49:00.918 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.35.238
2026-02-02 04:49:00.918 DEBUG [src.node.docker_mananger] Generated ports ['34709', '34710', '34711', '34712', '34713']
2026-02-02 04:49:00.918 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-02-02 04:49:00.918 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-02-02 04:49:00.918 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-02 04:49:00.918 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-02 04:49:00.919 DEBUG [src.node.waku_node] Using volumes []
2026-02-02 04:49:00.919 DEBUG [src.node.docker_mananger] docker run -i -t -p 34709:34709 -p 34710:34710 -p 34711:34711 -p 34712:34712 -p 34713:34713 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=34711 --rest-port=34709 --tcp-port=34710 --discv5-udp-port=34712 --rest-address=0.0.0.0 --nat=extip:172.18.35.238 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=7a8284fb2f1ed53ba170af860f5e5ce8ff29ce2d59d265a4dcec9dc2a70beed1 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=34713 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QOtag931U4DG-TSgdO6BRbJg9-0LzuE85VOYSSisIB6kGCopoIcjHLmTgn5N9T8k01dXibTjRMxyt7rQTPmrWkACgmlkgnY0gmlwhKwSAwuKbXVsdGlhZGRyc5YACASsEgMLBkvsAAoErBIDCwZL7d0DgnJzhQACAQAAiXNlY3AyNTZrMaEC_Gr0_TKsQN4-DmGc_kfCDYy50ZoGu6xvVemJXEw7nu2DdGNwgkvsg3VkcIJL7oV3YWt1MgU
2026-02-02 04:49:01.113 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.35.238 waku a95b73bec0dd0de1feb81fe8c32e0289f82389ac369b0aeb454007515e15d90e
2026-02-02 04:49:01.143 DEBUG [src.node.docker_mananger] Container started with ID a95b73bec0dd. Setting up logs at ./log/docker/node4_2026-02-02_04-48-57__1d5f6742-59da-439d-823f-0beb52b8cc89__wakuorg_nwaku:latest.log
2026-02-02 04:49:01.144 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 34709
2026-02-02 04:49:01.144 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-02 04:49:02.146 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:34709/health" -H "Content-Type: application/json" -d 'None'
2026-02-02 04:49:02.157 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"}]}'
2026-02-02 04:49:02.157 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-02 04:49:02.158 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:34709/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-02 04:49:02.163 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.35.238/tcp/34710/p2p/16Uiu2HAmDDyWJdp8Zcc5kQN71RGoWGMZeznbxLX1yhicg87QgvLQ","/ip4/172.18.35.238/tcp/34711/ws/p2p/16Uiu2HAmDDyWJdp8Zcc5kQN71RGoWGMZeznbxLX1yhicg87QgvLQ"],"enrUri":"enr:-L24QKUD4NJIFadyAHcqouLyEEiHRi7Wkv-sJgFnvoazfegoCbGwtkjZfjVjXzEKlHfe9LrxHIYUuNcWD3WY6VOBljoCgmlkgnY0gmlwhKwSI-6KbXVsdGlhZGRyc5YACASsEiPuBoeWAAoErBIj7gaHl90DgnJzhQACAQAAiXNlY3AyNTZrMaEDCHM2NLoQna_1YuGw25j-iuIQfpwUsiJpmB1QhTxpmWGDdGNwgoeWg3VkcIKHmIV3YWt1MgE"}'
2026-02-02 04:49:02.164 INFO [src.node.waku_node] REST service is ready !!
2026-02-02 04:49:02.164 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:34709/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.3.11/tcp/19436/p2p/16Uiu2HAmCR1K7ezmRyz8eBEfi1mcNNjas5AkTHv1BgxzWA5621he"]'
2026-02-02 04:49:02.168 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-02 04:49:02.175 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-02 04:49:02.175 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-02-02_04-48-57__1d5f6742-59da-439d-823f-0beb52b8cc89__wakuorg_nwaku:latest.log
2026-02-02 04:49:02.175 DEBUG [src.node.waku_node] Starting Node...
2026-02-02 04:49:02.175 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-02 04:49:02.177 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-02 04:49:02.177 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.13.69
2026-02-02 04:49:02.177 DEBUG [src.node.docker_mananger] Generated ports ['19065', '19066', '19067', '19068', '19069']
2026-02-02 04:49:02.177 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-02-02 04:49:02.177 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-02-02 04:49:02.177 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-02 04:49:02.177 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-02 04:49:02.178 DEBUG [src.node.waku_node] Using volumes []
2026-02-02 04:49:02.178 DEBUG [src.node.docker_mananger] docker run -i -t -p 19065:19065 -p 19066:19066 -p 19067:19067 -p 19068:19068 -p 19069:19069 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=19067 --rest-port=19065 --tcp-port=19066 --discv5-udp-port=19068 --rest-address=0.0.0.0 --nat=extip:172.18.13.69 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=7faedb9fceb2c6cb9f22aa454acd36d060e4cce1b7aba47d98fe6ccbecece67c --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=19069 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QOtag931U4DG-TSgdO6BRbJg9-0LzuE85VOYSSisIB6kGCopoIcjHLmTgn5N9T8k01dXibTjRMxyt7rQTPmrWkACgmlkgnY0gmlwhKwSAwuKbXVsdGlhZGRyc5YACASsEgMLBkvsAAoErBIDCwZL7d0DgnJzhQACAQAAiXNlY3AyNTZrMaEC_Gr0_TKsQN4-DmGc_kfCDYy50ZoGu6xvVemJXEw7nu2DdGNwgkvsg3VkcIJL7oV3YWt1MgU
2026-02-02 04:49:02.367 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.13.69 waku f62a9789b22f7c6dbe826da811f8dca6967e38669c6203213c9fdf2fba06f340
2026-02-02 04:49:02.397 DEBUG [src.node.docker_mananger] Container started with ID f62a9789b22f. Setting up logs at ./log/docker/node5_2026-02-02_04-48-57__1d5f6742-59da-439d-823f-0beb52b8cc89__wakuorg_nwaku:latest.log
2026-02-02 04:49:02.398 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 19065
2026-02-02 04:49:02.398 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-02 04:49:03.399 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:19065/health" -H "Content-Type: application/json" -d 'None'
2026-02-02 04:49:03.408 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"}]}'
2026-02-02 04:49:03.408 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-02 04:49:03.408 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:19065/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-02 04:49:03.414 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.13.69/tcp/19066/p2p/16Uiu2HAm9xGLT8S6rMKv4iQHa2VaWz2t5ebdWv8q9UxTmpub2WVq","/ip4/172.18.13.69/tcp/19067/ws/p2p/16Uiu2HAm9xGLT8S6rMKv4iQHa2VaWz2t5ebdWv8q9UxTmpub2WVq"],"enrUri":"enr:-L24QNZqKPutrKLRcU4y8n4HHIU5XMZlQd9IguqtIRlBkSqMQk7ZCKryN-bGNUu3JnfaOxOvcQmGQicAYgTxI-JOexoCgmlkgnY0gmlwhKwSDUWKbXVsdGlhZGRyc5YACASsEg1FBkp6AAoErBINRQZKe90DgnJzhQACAQAAiXNlY3AyNTZrMaEC19njTG4dz2TgrF6x1cc8ASIMqNa0IQciaAW2YVXs7iSDdGNwgkp6g3VkcIJKfIV3YWt1MgE"}'
2026-02-02 04:49:03.414 INFO [src.node.waku_node] REST service is ready !!
2026-02-02 04:49:03.415 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:19065/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.3.11/tcp/19436/p2p/16Uiu2HAmCR1K7ezmRyz8eBEfi1mcNNjas5AkTHv1BgxzWA5621he"]'
2026-02-02 04:49:03.418 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-02 04:49:03.418 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:19435/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-02-02 04:49:03.420 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-02 04:49:03.421 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:17330/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-02-02 04:49:03.423 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-02 04:49:03.423 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:16078/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-02-02 04:49:03.426 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-02 04:49:03.426 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:34709/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-02-02 04:49:03.429 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-02 04:49:03.429 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:19065/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-02-02 04:49:03.431 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-02 04:49:03.432 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:19435/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-02-02 04:49:03.440 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-02 04:49:03.440 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-02-02 04:49:03.542 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-02-02 04:49:03.542 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:19435/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-02-02 04:49:03.545 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1770007743432218000,"ephemeral":false,"proof":""}]'
2026-02-02 04:49:03.547 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-02-02 04:49:03.547 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17330/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-02-02 04:49:03.549 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1770007743432218000,"ephemeral":false,"proof":""}]'
2026-02-02 04:49:03.551 DEBUG [src.steps.sharding] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-02-02 04:49:03.551 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16078/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-02-02 04:49:03.553 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1770007743432218000,"ephemeral":false,"proof":""}]'
2026-02-02 04:49:03.555 DEBUG [src.steps.sharding] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-02-02 04:49:03.555 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:34709/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-02-02 04:49:03.557 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1770007743432218000,"ephemeral":false,"proof":""}]'
2026-02-02 04:49:03.559 DEBUG [src.steps.sharding] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-02-02 04:49:03.559 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:19065/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-02-02 04:49:03.561 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1770007743432218000,"ephemeral":false,"proof":""}]'
2026-02-02 04:49:03.564 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-02-02 04:49:03.565 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-02-02 04:49:03.565 DEBUG [src.node.waku_node] Stopping container with id 0b79a68652e0
2026-02-02 04:49:04.138 DEBUG [src.node.waku_node] Container stopped.
2026-02-02 04:49:04.139 DEBUG [src.node.waku_node] Stopping container with id 97ba4ddcdc2d
2026-02-02 04:49:04.663 DEBUG [src.node.waku_node] Container stopped.
2026-02-02 04:49:04.664 DEBUG [src.node.waku_node] Stopping container with id 75a7ff3b93d2
2026-02-02 04:49:04.952 ERROR [src.node.docker_mananger] Max retries reached for container 0b79a68652e0. Exiting log stream.
2026-02-02 04:49:05.216 DEBUG [src.node.waku_node] Container stopped.
2026-02-02 04:49:05.216 DEBUG [src.node.waku_node] Stopping container with id a95b73bec0dd
2026-02-02 04:49:05.476 ERROR [src.node.docker_mananger] Max retries reached for container 97ba4ddcdc2d. Exiting log stream.
2026-02-02 04:49:05.758 DEBUG [src.node.waku_node] Container stopped.
2026-02-02 04:49:05.758 DEBUG [src.node.waku_node] Stopping container with id f62a9789b22f
2026-02-02 04:49:06.029 ERROR [src.node.docker_mananger] Max retries reached for container 75a7ff3b93d2. Exiting log stream.
2026-02-02 04:49:06.299 DEBUG [src.node.waku_node] Container stopped.
2026-02-02 04:49:06.300 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-02-02 04:49:06.314 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-02 04:49:06.320 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-02 04:49:06.324 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-02 04:49:06.329 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-02 04:49:06.334 DEBUG [src.node.docker_mananger] No errors found in the waku logs.