180 lines
32 KiB
Plaintext

2026-03-02 04:43:34.313 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-02 04:43:34.314 DEBUG [tests.conftest] Running test: test_static_shard_relay with id: 2026-03-02_04-43-34__ad14b983-4113-437b-86e8-e9503ae12df5
2026-03-02 04:43:34.314 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-02 04:43:34.314 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-02 04:43:34.314 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-02 04:43:34.322 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-02 04:43:34.322 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-02_04-43-34__ad14b983-4113-437b-86e8-e9503ae12df5__wakuorg_nwaku:latest.log
2026-03-02 04:43:34.322 DEBUG [src.node.waku_node] Starting Node...
2026-03-02 04:43:34.323 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-02 04:43:34.324 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-02 04:43:34.324 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.229.89
2026-03-02 04:43:34.324 DEBUG [src.node.docker_mananger] Generated ports ['1148', '1149', '1150', '1151', '1152']
2026-03-02 04:43:34.324 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-02 04:43:34.324 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-02 04:43:34.325 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-02 04:43:34.325 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-02 04:43:34.325 DEBUG [src.node.waku_node] Using volumes []
2026-03-02 04:43:34.325 DEBUG [src.node.docker_mananger] docker run -i -t -p 1148:1148 -p 1149:1149 -p 1150:1150 -p 1151:1151 -p 1152:1152 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=1150 --rest-port=1148 --tcp-port=1149 --discv5-udp-port=1151 --rest-address=0.0.0.0 --nat=extip:172.18.229.89 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=5b3baaccbad97dcbece1f3ed0fadcd0bd5e4a5e6a70effa2a0055dcb4a1cefa8 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=1152 --metrics-logging=true --relay=true --filter=true
2026-03-02 04:43:34.491 ERROR [src.node.docker_mananger] Max retries reached for container 8ac17eb8e879. Exiting log stream.
2026-03-02 04:43:34.530 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.229.89 waku 01693e07f260ce836c001542dd5ca857d2fe155705680561cff1c4c8b86681a6
2026-03-02 04:43:34.565 DEBUG [src.node.docker_mananger] Container started with ID 01693e07f260. Setting up logs at ./log/docker/node1_2026-03-02_04-43-34__ad14b983-4113-437b-86e8-e9503ae12df5__wakuorg_nwaku:latest.log
2026-03-02 04:43:34.565 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 1148
2026-03-02 04:43:34.565 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-02 04:43:35.120 ERROR [src.node.docker_mananger] Max retries reached for container 972be288b2a1. Exiting log stream.
2026-03-02 04:43:35.567 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:1148/health" -H "Content-Type: application/json" -d 'None'
2026-03-02 04:43:35.570 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-02 04:43:35.570 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-02 04:43:35.570 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:1148/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-02 04:43:35.573 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.229.89/tcp/1149/p2p/16Uiu2HAmDPwPRk9sHFBkz3yCpJ3F3dA6M7vBwWjEr21E5vitf7aQ","/ip4/172.18.229.89/tcp/1150/ws/p2p/16Uiu2HAmDPwPRk9sHFBkz3yCpJ3F3dA6M7vBwWjEr21E5vitf7aQ"],"enrUri":"enr:-L24QFanw-IxLn-y9EEJlXnToyTSKgh2sYnTktSdxNS62u55c2D3IMnBKoHz8xz-7yL1Tag9MKomLDZA5mA2hdjsUqMCgmlkgnY0gmlwhKwS5VmKbXVsdGlhZGRyc5YACASsEuVZBgR9AAoErBLlWQYEft0DgnJzhQACAQAAiXNlY3AyNTZrMaEDCwCf1XZ9Kxv52GeKVCgfzpiRCqd-YpkuPfpLaLm452mDdGNwggR9g3VkcIIEf4V3YWt1MgU"}'
2026-03-02 04:43:35.573 INFO [src.node.waku_node] REST service is ready !!
2026-03-02 04:43:35.580 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-02 04:43:35.580 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-02_04-43-34__ad14b983-4113-437b-86e8-e9503ae12df5__wakuorg_nwaku:latest.log
2026-03-02 04:43:35.580 DEBUG [src.node.waku_node] Starting Node...
2026-03-02 04:43:35.580 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-02 04:43:35.582 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-02 04:43:35.582 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.92.83
2026-03-02 04:43:35.582 DEBUG [src.node.docker_mananger] Generated ports ['49911', '49912', '49913', '49914', '49915']
2026-03-02 04:43:35.582 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-02 04:43:35.582 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-02 04:43:35.582 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-02 04:43:35.582 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-02 04:43:35.582 DEBUG [src.node.waku_node] Using volumes []
2026-03-02 04:43:35.582 DEBUG [src.node.docker_mananger] docker run -i -t -p 49911:49911 -p 49912:49912 -p 49913:49913 -p 49914:49914 -p 49915:49915 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=49913 --rest-port=49911 --tcp-port=49912 --discv5-udp-port=49914 --rest-address=0.0.0.0 --nat=extip:172.18.92.83 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=84b744a1fe2efaa5caebcfe88bbcb1b1f71064ff7aaeebdc2bfdfceb0be7f2a1 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=49915 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QFanw-IxLn-y9EEJlXnToyTSKgh2sYnTktSdxNS62u55c2D3IMnBKoHz8xz-7yL1Tag9MKomLDZA5mA2hdjsUqMCgmlkgnY0gmlwhKwS5VmKbXVsdGlhZGRyc5YACASsEuVZBgR9AAoErBLlWQYEft0DgnJzhQACAQAAiXNlY3AyNTZrMaEDCwCf1XZ9Kxv52GeKVCgfzpiRCqd-YpkuPfpLaLm452mDdGNwggR9g3VkcIIEf4V3YWt1MgU
2026-03-02 04:43:35.797 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.92.83 waku a47c5e376482c68d5d7ec50a50ac5e5b02e547d9833063f47aaf8c5e00ee1cce
2026-03-02 04:43:35.832 DEBUG [src.node.docker_mananger] Container started with ID a47c5e376482. Setting up logs at ./log/docker/node2_2026-03-02_04-43-34__ad14b983-4113-437b-86e8-e9503ae12df5__wakuorg_nwaku:latest.log
2026-03-02 04:43:35.832 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 49911
2026-03-02 04:43:35.832 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-02 04:43:36.833 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:49911/health" -H "Content-Type: application/json" -d 'None'
2026-03-02 04:43:36.862 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-02 04:43:36.863 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-02 04:43:36.864 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:49911/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-02 04:43:36.870 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.92.83/tcp/49912/p2p/16Uiu2HAkyScYwLN6wJuAz4rx4cHDmkU3kuyeW2ZV7FYfRdfjfCsw","/ip4/172.18.92.83/tcp/49913/ws/p2p/16Uiu2HAkyScYwLN6wJuAz4rx4cHDmkU3kuyeW2ZV7FYfRdfjfCsw"],"enrUri":"enr:-L24QN1yFddMQFplzusV-7accBWRSO1CVENjVf6wZuvryWWDNo34sTFgiXm34mISpdOShxcc0zIaAZ2xBIaw81ef7MYCgmlkgnY0gmlwhKwSXFOKbXVsdGlhZGRyc5YACASsElxTBsL4AAoErBJcUwbC-d0DgnJzhQACAQAAiXNlY3AyNTZrMaECO6xzq9DEtswSk5ZwCp7AzhAzU46XehXLBt4nFxnriWaDdGNwgsL4g3VkcILC-oV3YWt1MgE"}'
2026-03-02 04:43:36.870 INFO [src.node.waku_node] REST service is ready !!
2026-03-02 04:43:36.871 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:49911/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.229.89/tcp/1149/p2p/16Uiu2HAmDPwPRk9sHFBkz3yCpJ3F3dA6M7vBwWjEr21E5vitf7aQ"]'
2026-03-02 04:43:36.876 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-02 04:43:36.884 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-02 04:43:36.884 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-02_04-43-34__ad14b983-4113-437b-86e8-e9503ae12df5__wakuorg_nwaku:latest.log
2026-03-02 04:43:36.884 DEBUG [src.node.waku_node] Starting Node...
2026-03-02 04:43:36.885 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-02 04:43:36.886 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-02 04:43:36.886 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.74.169
2026-03-02 04:43:36.886 DEBUG [src.node.docker_mananger] Generated ports ['18915', '18916', '18917', '18918', '18919']
2026-03-02 04:43:36.887 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-02 04:43:36.887 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-02 04:43:36.887 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-02 04:43:36.887 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-02 04:43:36.887 DEBUG [src.node.waku_node] Using volumes []
2026-03-02 04:43:36.887 DEBUG [src.node.docker_mananger] docker run -i -t -p 18915:18915 -p 18916:18916 -p 18917:18917 -p 18918:18918 -p 18919:18919 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=18917 --rest-port=18915 --tcp-port=18916 --discv5-udp-port=18918 --rest-address=0.0.0.0 --nat=extip:172.18.74.169 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=a0fd3d5bcbc0744cc44ccbc9ac6b0feccb62e24a1feba0c9aac3cb8deec35fc4 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=18919 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QFanw-IxLn-y9EEJlXnToyTSKgh2sYnTktSdxNS62u55c2D3IMnBKoHz8xz-7yL1Tag9MKomLDZA5mA2hdjsUqMCgmlkgnY0gmlwhKwS5VmKbXVsdGlhZGRyc5YACASsEuVZBgR9AAoErBLlWQYEft0DgnJzhQACAQAAiXNlY3AyNTZrMaEDCwCf1XZ9Kxv52GeKVCgfzpiRCqd-YpkuPfpLaLm452mDdGNwggR9g3VkcIIEf4V3YWt1MgU
2026-03-02 04:43:37.091 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.74.169 waku c156e5fc822f9c3bf54f03780815155c6d95dfc33e37d895675b0f135026db02
2026-03-02 04:43:37.127 DEBUG [src.node.docker_mananger] Container started with ID c156e5fc822f. Setting up logs at ./log/docker/node3_2026-03-02_04-43-34__ad14b983-4113-437b-86e8-e9503ae12df5__wakuorg_nwaku:latest.log
2026-03-02 04:43:37.127 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 18915
2026-03-02 04:43:37.127 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-02 04:43:38.128 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18915/health" -H "Content-Type: application/json" -d 'None'
2026-03-02 04:43:38.140 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-02 04:43:38.141 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-02 04:43:38.141 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18915/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-02 04:43:38.148 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.74.169/tcp/18916/p2p/16Uiu2HAkv7Uax8gXaCD7D7QPfarBoCStQ8iwJ6x4FpNKiGQwXJ4Q","/ip4/172.18.74.169/tcp/18917/ws/p2p/16Uiu2HAkv7Uax8gXaCD7D7QPfarBoCStQ8iwJ6x4FpNKiGQwXJ4Q"],"enrUri":"enr:-L24QIlcRicvmtKtrzJidOI1oVrTL-aRuVt5Rt8TlWyRHnW9ZDclIDHnMR_vH5FwOCODy90Vii6RjGeVreBswy3mKRkCgmlkgnY0gmlwhKwSSqmKbXVsdGlhZGRyc5YACASsEkqpBknkAAoErBJKqQZJ5d0DgnJzhQACAQAAiXNlY3AyNTZrMaECCjJZbOz1UsYiz-veUgwyLtEMeAvH_XdxFmefBQmQ_9mDdGNwgknkg3VkcIJJ5oV3YWt1MgE"}'
2026-03-02 04:43:38.148 INFO [src.node.waku_node] REST service is ready !!
2026-03-02 04:43:38.149 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:18915/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.229.89/tcp/1149/p2p/16Uiu2HAmDPwPRk9sHFBkz3yCpJ3F3dA6M7vBwWjEr21E5vitf7aQ"]'
2026-03-02 04:43:38.152 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-02 04:43:38.159 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-02 04:43:38.159 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-02_04-43-34__ad14b983-4113-437b-86e8-e9503ae12df5__wakuorg_nwaku:latest.log
2026-03-02 04:43:38.159 DEBUG [src.node.waku_node] Starting Node...
2026-03-02 04:43:38.159 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-02 04:43:38.160 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-02 04:43:38.161 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.92.18
2026-03-02 04:43:38.161 DEBUG [src.node.docker_mananger] Generated ports ['22820', '22821', '22822', '22823', '22824']
2026-03-02 04:43:38.161 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-02 04:43:38.161 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-02 04:43:38.161 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-02 04:43:38.161 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-02 04:43:38.161 DEBUG [src.node.waku_node] Using volumes []
2026-03-02 04:43:38.161 DEBUG [src.node.docker_mananger] docker run -i -t -p 22820:22820 -p 22821:22821 -p 22822:22822 -p 22823:22823 -p 22824:22824 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=22822 --rest-port=22820 --tcp-port=22821 --discv5-udp-port=22823 --rest-address=0.0.0.0 --nat=extip:172.18.92.18 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=11f9dada5f9b1c7d1e2c0f7a2eee7a963e5d7f0b8209ad7580b82adbd8f63290 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=22824 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QFanw-IxLn-y9EEJlXnToyTSKgh2sYnTktSdxNS62u55c2D3IMnBKoHz8xz-7yL1Tag9MKomLDZA5mA2hdjsUqMCgmlkgnY0gmlwhKwS5VmKbXVsdGlhZGRyc5YACASsEuVZBgR9AAoErBLlWQYEft0DgnJzhQACAQAAiXNlY3AyNTZrMaEDCwCf1XZ9Kxv52GeKVCgfzpiRCqd-YpkuPfpLaLm452mDdGNwggR9g3VkcIIEf4V3YWt1MgU
2026-03-02 04:43:38.369 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.92.18 waku 41faa768ff7bd07362717702e5aa5df8eff92f3dcc9263d56771352c4aaf9745
2026-03-02 04:43:38.405 DEBUG [src.node.docker_mananger] Container started with ID 41faa768ff7b. Setting up logs at ./log/docker/node4_2026-03-02_04-43-34__ad14b983-4113-437b-86e8-e9503ae12df5__wakuorg_nwaku:latest.log
2026-03-02 04:43:38.406 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 22820
2026-03-02 04:43:38.406 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-02 04:43:39.407 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22820/health" -H "Content-Type: application/json" -d 'None'
2026-03-02 04:43:39.421 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-02 04:43:39.421 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-02 04:43:39.422 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22820/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-02 04:43:39.427 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.92.18/tcp/22821/p2p/16Uiu2HAm5X8ejJ3SS6Vn3FiS2CAyR4YKGZzfyJEKXXWBGHEf4v9H","/ip4/172.18.92.18/tcp/22822/ws/p2p/16Uiu2HAm5X8ejJ3SS6Vn3FiS2CAyR4YKGZzfyJEKXXWBGHEf4v9H"],"enrUri":"enr:-L24QFuMbKSDzWpDKV7FUkn6wINrMCwxnikBhWrDBKDBGo6NGncs7usKzVWuagmyEQdyCdr51ffuRi6RkDWShggf35gCgmlkgnY0gmlwhKwSXBKKbXVsdGlhZGRyc5YACASsElwSBlklAAoErBJcEgZZJt0DgnJzhQACAQAAiXNlY3AyNTZrMaEClfruQBd2jRRBtJKtK6EG1HDkt64z7jLgyI-9p79R98yDdGNwglklg3VkcIJZJ4V3YWt1MgE"}'
2026-03-02 04:43:39.428 INFO [src.node.waku_node] REST service is ready !!
2026-03-02 04:43:39.429 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22820/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.229.89/tcp/1149/p2p/16Uiu2HAmDPwPRk9sHFBkz3yCpJ3F3dA6M7vBwWjEr21E5vitf7aQ"]'
2026-03-02 04:43:39.433 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-02 04:43:39.440 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-02 04:43:39.440 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-02_04-43-34__ad14b983-4113-437b-86e8-e9503ae12df5__wakuorg_nwaku:latest.log
2026-03-02 04:43:39.440 DEBUG [src.node.waku_node] Starting Node...
2026-03-02 04:43:39.440 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-02 04:43:39.441 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-02 04:43:39.441 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.10.161
2026-03-02 04:43:39.442 DEBUG [src.node.docker_mananger] Generated ports ['21377', '21378', '21379', '21380', '21381']
2026-03-02 04:43:39.442 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-02 04:43:39.442 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-02 04:43:39.442 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-02 04:43:39.442 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-02 04:43:39.442 DEBUG [src.node.waku_node] Using volumes []
2026-03-02 04:43:39.442 DEBUG [src.node.docker_mananger] docker run -i -t -p 21377:21377 -p 21378:21378 -p 21379:21379 -p 21380:21380 -p 21381:21381 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=21379 --rest-port=21377 --tcp-port=21378 --discv5-udp-port=21380 --rest-address=0.0.0.0 --nat=extip:172.18.10.161 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=5dc9a6a34d466fbd663e5d49d89695ab99edae5ed478dbaec82d975b5fdaeea8 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=21381 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QFanw-IxLn-y9EEJlXnToyTSKgh2sYnTktSdxNS62u55c2D3IMnBKoHz8xz-7yL1Tag9MKomLDZA5mA2hdjsUqMCgmlkgnY0gmlwhKwS5VmKbXVsdGlhZGRyc5YACASsEuVZBgR9AAoErBLlWQYEft0DgnJzhQACAQAAiXNlY3AyNTZrMaEDCwCf1XZ9Kxv52GeKVCgfzpiRCqd-YpkuPfpLaLm452mDdGNwggR9g3VkcIIEf4V3YWt1MgU
2026-03-02 04:43:39.657 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.10.161 waku 9fc1d115a51a979446daa16333cc602929c45570d0343913334dfc225176a82d
2026-03-02 04:43:39.696 DEBUG [src.node.docker_mananger] Container started with ID 9fc1d115a51a. Setting up logs at ./log/docker/node5_2026-03-02_04-43-34__ad14b983-4113-437b-86e8-e9503ae12df5__wakuorg_nwaku:latest.log
2026-03-02 04:43:39.696 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 21377
2026-03-02 04:43:39.696 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-02 04:43:40.697 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:21377/health" -H "Content-Type: application/json" -d 'None'
2026-03-02 04:43:40.712 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-02 04:43:40.714 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-02 04:43:40.714 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:21377/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-02 04:43:40.720 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.10.161/tcp/21378/p2p/16Uiu2HAkzLhLWzpG2gwfMW5V75EbXVzLS5RjR2Y2xdN6FkNuJFYW","/ip4/172.18.10.161/tcp/21379/ws/p2p/16Uiu2HAkzLhLWzpG2gwfMW5V75EbXVzLS5RjR2Y2xdN6FkNuJFYW"],"enrUri":"enr:-L24QACkiQc_j7GE4iRroYl_HGEpZcY6ZnqyPp-yRwYVLWKmOwGqLzSPXfUUmG3b3kg8Cr5RjEsUQVcUmGN6huWYY4UCgmlkgnY0gmlwhKwSCqGKbXVsdGlhZGRyc5YACASsEgqhBlOCAAoErBIKoQZTg90DgnJzhQACAQAAiXNlY3AyNTZrMaECSQQQuEIQ5i3-b_qt66N425NZHHsTEoQiRKdZGwFVJ-ODdGNwglOCg3VkcIJThIV3YWt1MgE"}'
2026-03-02 04:43:40.720 INFO [src.node.waku_node] REST service is ready !!
2026-03-02 04:43:40.720 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:21377/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.229.89/tcp/1149/p2p/16Uiu2HAmDPwPRk9sHFBkz3yCpJ3F3dA6M7vBwWjEr21E5vitf7aQ"]'
2026-03-02 04:43:40.723 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-02 04:43:40.724 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:1148/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-02 04:43:40.726 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-02 04:43:40.726 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:49911/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-02 04:43:40.729 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-02 04:43:40.729 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:18915/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-02 04:43:40.732 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-02 04:43:40.732 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22820/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-02 04:43:40.735 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-02 04:43:40.735 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:21377/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-02 04:43:40.737 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-02 04:43:40.738 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:1148/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-02 04:43:40.747 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-02 04:43:40.748 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-02 04:43:40.848 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-02 04:43:40.849 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:1148/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-02 04:43:40.851 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772426620738207927,"ephemeral":false,"proof":""}]'
2026-03-02 04:43:40.853 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-02 04:43:40.853 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:49911/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-02 04:43:40.855 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772426620738207927,"ephemeral":false,"proof":""}]'
2026-03-02 04:43:40.857 DEBUG [src.steps.sharding] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-03-02 04:43:40.857 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18915/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-02 04:43:40.859 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772426620738207927,"ephemeral":false,"proof":""}]'
2026-03-02 04:43:40.861 DEBUG [src.steps.sharding] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-03-02 04:43:40.861 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22820/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-02 04:43:40.863 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772426620738207927,"ephemeral":false,"proof":""}]'
2026-03-02 04:43:40.864 DEBUG [src.steps.sharding] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-03-02 04:43:40.865 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:21377/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-02 04:43:40.867 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772426620738207927,"ephemeral":false,"proof":""}]'
2026-03-02 04:43:40.869 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-02 04:43:40.871 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-02 04:43:40.871 DEBUG [src.node.waku_node] Stopping container with id 01693e07f260
2026-03-02 04:43:41.477 DEBUG [src.node.waku_node] Container stopped.
2026-03-02 04:43:41.477 DEBUG [src.node.waku_node] Stopping container with id a47c5e376482
2026-03-02 04:43:42.084 DEBUG [src.node.waku_node] Container stopped.
2026-03-02 04:43:42.084 DEBUG [src.node.waku_node] Stopping container with id c156e5fc822f
2026-03-02 04:43:42.342 ERROR [src.node.docker_mananger] Max retries reached for container 01693e07f260. Exiting log stream.
2026-03-02 04:43:42.689 DEBUG [src.node.waku_node] Container stopped.
2026-03-02 04:43:42.689 DEBUG [src.node.waku_node] Stopping container with id 41faa768ff7b
2026-03-02 04:43:42.917 ERROR [src.node.docker_mananger] Max retries reached for container a47c5e376482. Exiting log stream.
2026-03-02 04:43:43.255 DEBUG [src.node.waku_node] Container stopped.
2026-03-02 04:43:43.255 DEBUG [src.node.waku_node] Stopping container with id 9fc1d115a51a
2026-03-02 04:43:43.503 ERROR [src.node.docker_mananger] Max retries reached for container c156e5fc822f. Exiting log stream.
2026-03-02 04:43:43.839 DEBUG [src.node.waku_node] Container stopped.
2026-03-02 04:43:43.842 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-02 04:43:43.870 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-02 04:43:43.880 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-02 04:43:43.885 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-02 04:43:43.896 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-02 04:43:43.902 DEBUG [src.node.docker_mananger] No errors found in the waku logs.