2026-03-18 09:27:31 +00:00

192 lines
36 KiB
Plaintext

2026-03-18 09:07:37.410 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-18 09:07:37.411 DEBUG [tests.conftest] Running test: test_last_node_to_start_publishes with id: 2026-03-18_09-07-37__409e45db-4569-4ef6-b160-84f20601bfa1
2026-03-18 09:07:37.411 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-18 09:07:37.411 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-18 09:07:37.411 DEBUG [src.steps.relay] Running fixture setup: setup_main_relay_nodes
2026-03-18 09:07:37.419 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0
2026-03-18 09:07:37.419 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-18_09-07-37__409e45db-4569-4ef6-b160-84f20601bfa1__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:07:37.419 DEBUG [src.node.waku_node] Starting Node...
2026-03-18 09:07:37.419 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-18 09:07:37.421 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-18 09:07:37.421 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.131.250
2026-03-18 09:07:37.421 DEBUG [src.node.docker_mananger] Generated ports ['40952', '40953', '40954', '40955', '40956']
2026-03-18 09:07:37.421 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-18 09:07:37.422 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-18 09:07:37.422 DEBUG [src.node.waku_node] Using volumes []
2026-03-18 09:07:37.422 DEBUG [src.node.docker_mananger] docker run -i -t -p 40952:40952 -p 40953:40953 -p 40954:40954 -p 40955:40955 -p 40956:40956 harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=40954 --rest-port=40952 --tcp-port=40953 --discv5-udp-port=40955 --rest-address=0.0.0.0 --nat=extip:172.18.131.250 --peer-exchange=true --discv5-discovery=true --cluster-id=198 --nodekey=79f9c110849d4c014daadb9fc1f5b6af4e0ec2cf9ac5a535b5f016c4c012d441 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=40956 --metrics-logging=true --relay=true
2026-03-18 09:07:37.639 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.131.250 waku a255091f6a16b208d10c78f36955c4fbec53c166db5c8054d7a7156e70bf4067
2026-03-18 09:07:37.640 ERROR [src.node.docker_mananger] Max retries reached for container afaf90a6d15e. Exiting log stream.
2026-03-18 09:07:37.686 DEBUG [src.node.docker_mananger] Container started with ID a255091f6a16. Setting up logs at ./log/docker/node1_2026-03-18_09-07-37__409e45db-4569-4ef6-b160-84f20601bfa1__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:07:37.687 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0. REST: 40952
2026-03-18 09:07:37.687 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-18 09:07:38.209 ERROR [src.node.docker_mananger] Max retries reached for container 285316626cce. Exiting log stream.
2026-03-18 09:07:38.687 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:40952/health" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:38.691 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-18 09:07:38.691 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-18 09:07:38.691 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:40952/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:38.693 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.131.250/tcp/40953/p2p/16Uiu2HAm4LfebsfjLLYSmSGLLLiAK6KzrLsSTXoXWFujdVG3oq6H","/ip4/172.18.131.250/tcp/40954/ws/p2p/16Uiu2HAm4LfebsfjLLYSmSGLLLiAK6KzrLsSTXoXWFujdVG3oq6H"],"enrUri":"enr:-L24QHXqxVOF7VeU3Ii4AgD7MCSbrOQRnyZTJ84odWz3_L4fRuQZAFE-5cKHexHgsm-Ih7AgTGOF9z9NZ8e22x67YnsCgmlkgnY0gmlwhKwSg_qKbXVsdGlhZGRyc5YACASsEoP6Bp_5AAoErBKD-gaf-t0DgnJzhQDGAQAAiXNlY3AyNTZrMaEChHDndtRhdHE-grVkmPQEhmMQiQxxyIhljMqS5Vl25KKDdGNwgp_5g3VkcIKf-4V3YWt1MgE"}'
2026-03-18 09:07:38.693 INFO [src.node.waku_node] REST service is ready !!
2026-03-18 09:07:38.701 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0
2026-03-18 09:07:38.701 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-18_09-07-37__409e45db-4569-4ef6-b160-84f20601bfa1__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:07:38.701 DEBUG [src.node.waku_node] Starting Node...
2026-03-18 09:07:38.701 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-18 09:07:38.703 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-18 09:07:38.703 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.122.74
2026-03-18 09:07:38.703 DEBUG [src.node.docker_mananger] Generated ports ['35652', '35653', '35654', '35655', '35656']
2026-03-18 09:07:38.703 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-18 09:07:38.703 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-18 09:07:38.703 DEBUG [src.node.waku_node] Using volumes []
2026-03-18 09:07:38.703 DEBUG [src.node.docker_mananger] docker run -i -t -p 35652:35652 -p 35653:35653 -p 35654:35654 -p 35655:35655 -p 35656:35656 harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=35654 --rest-port=35652 --tcp-port=35653 --discv5-udp-port=35655 --rest-address=0.0.0.0 --nat=extip:172.18.122.74 --peer-exchange=true --discv5-discovery=true --cluster-id=198 --nodekey=ad1a37f98c3fa6d0fccef8df3b60e3d24d56f468ea8a42c05d44b3ba157e1b5f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=35656 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QHXqxVOF7VeU3Ii4AgD7MCSbrOQRnyZTJ84odWz3_L4fRuQZAFE-5cKHexHgsm-Ih7AgTGOF9z9NZ8e22x67YnsCgmlkgnY0gmlwhKwSg_qKbXVsdGlhZGRyc5YACASsEoP6Bp_5AAoErBKD-gaf-t0DgnJzhQDGAQAAiXNlY3AyNTZrMaEChHDndtRhdHE-grVkmPQEhmMQiQxxyIhljMqS5Vl25KKDdGNwgp_5g3VkcIKf-4V3YWt1MgE
2026-03-18 09:07:38.906 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.122.74 waku e17d9b6473337f390368493cf8723361748325c6d3b16357434ab635b717c824
2026-03-18 09:07:38.946 DEBUG [src.node.docker_mananger] Container started with ID e17d9b647333. Setting up logs at ./log/docker/node2_2026-03-18_09-07-37__409e45db-4569-4ef6-b160-84f20601bfa1__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:07:38.946 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0. REST: 35652
2026-03-18 09:07:38.947 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-18 09:07:39.948 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:35652/health" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:39.967 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-18 09:07:39.968 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-18 09:07:39.969 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:35652/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:39.972 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.122.74/tcp/35653/p2p/16Uiu2HAm8AzX4borrsdUKw8f8UH6xnqn9Qz5vTbd9nefzMLvbRZA","/ip4/172.18.122.74/tcp/35654/ws/p2p/16Uiu2HAm8AzX4borrsdUKw8f8UH6xnqn9Qz5vTbd9nefzMLvbRZA"],"enrUri":"enr:-L24QKVVaPoYh9QK-D2EC2sNFve9nXN35d2FvTyxC-h0nkt6WgNo44bcGhQuK2VOErlOSWm6nrthppIzclGSevW-9akCgmlkgnY0gmlwhKwSekqKbXVsdGlhZGRyc5YACASsEnpKBotFAAoErBJ6SgaLRt0DgnJzhQDGAQAAiXNlY3AyNTZrMaECvWUuHz7yqBVDPGYbvewtxzgwLjR1vaKULrY5a7nVWqmDdGNwgotFg3VkcIKLR4V3YWt1MgE"}'
2026-03-18 09:07:39.972 INFO [src.node.waku_node] REST service is ready !!
2026-03-18 09:07:39.973 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:35652/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.131.250/tcp/40953/p2p/16Uiu2HAm4LfebsfjLLYSmSGLLLiAK6KzrLsSTXoXWFujdVG3oq6H"]'
2026-03-18 09:07:39.976 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:07:39.976 DEBUG [src.steps.relay] Running fixture setup: setup_optional_relay_nodes
2026-03-18 09:07:39.984 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0
2026-03-18 09:07:39.984 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-18_09-07-37__409e45db-4569-4ef6-b160-84f20601bfa1__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:07:39.984 DEBUG [src.node.waku_node] Starting Node...
2026-03-18 09:07:39.984 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-18 09:07:39.986 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-18 09:07:39.986 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.245.19
2026-03-18 09:07:39.986 DEBUG [src.node.docker_mananger] Generated ports ['60600', '60601', '60602', '60603', '60604']
2026-03-18 09:07:39.986 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-18 09:07:39.986 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-18 09:07:39.986 DEBUG [src.node.waku_node] Using volumes []
2026-03-18 09:07:39.986 DEBUG [src.node.docker_mananger] docker run -i -t -p 60600:60600 -p 60601:60601 -p 60602:60602 -p 60603:60603 -p 60604:60604 harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=60602 --rest-port=60600 --tcp-port=60601 --discv5-udp-port=60603 --rest-address=0.0.0.0 --nat=extip:172.18.245.19 --peer-exchange=true --discv5-discovery=true --cluster-id=198 --nodekey=e1cfcfe7a8e39197be0072833afbf7e1b1f214dc9fde6a345bac01d608ab1108 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=60604 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QHXqxVOF7VeU3Ii4AgD7MCSbrOQRnyZTJ84odWz3_L4fRuQZAFE-5cKHexHgsm-Ih7AgTGOF9z9NZ8e22x67YnsCgmlkgnY0gmlwhKwSg_qKbXVsdGlhZGRyc5YACASsEoP6Bp_5AAoErBKD-gaf-t0DgnJzhQDGAQAAiXNlY3AyNTZrMaEChHDndtRhdHE-grVkmPQEhmMQiQxxyIhljMqS5Vl25KKDdGNwgp_5g3VkcIKf-4V3YWt1MgE
2026-03-18 09:07:40.184 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.245.19 waku 4951d969c39ae7ac085b400e12299aaf918f0d121cec211b5fbe59d20ed819c1
2026-03-18 09:07:40.222 DEBUG [src.node.docker_mananger] Container started with ID 4951d969c39a. Setting up logs at ./log/docker/node3_2026-03-18_09-07-37__409e45db-4569-4ef6-b160-84f20601bfa1__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:07:40.222 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0. REST: 60600
2026-03-18 09:07:40.222 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-18 09:07:41.224 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:60600/health" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:41.239 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-18 09:07:41.240 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-18 09:07:41.242 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:60600/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:41.248 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.245.19/tcp/60601/p2p/16Uiu2HAkzzqtGTKB9RWUrJZnXYRrpPrr5dpDZLAM7Nzgq9RDBVMf","/ip4/172.18.245.19/tcp/60602/ws/p2p/16Uiu2HAkzzqtGTKB9RWUrJZnXYRrpPrr5dpDZLAM7Nzgq9RDBVMf"],"enrUri":"enr:-L24QFVT5HocenxTPqHCeL5_o_qFnw-_HbkVZ6h-ydKCyBI8A5S08cEBRefEr3wWKjz9cwLuw2dtQK7-9B0LZyCfKdICgmlkgnY0gmlwhKwS9ROKbXVsdGlhZGRyc5YACASsEvUTBuy5AAoErBL1Ewbsut0DgnJzhQDGAQAAiXNlY3AyNTZrMaECUsnM9lEfUVm4Z-5vKZE5CeJPXw-KiUtch8UJR2ZnPO6DdGNwguy5g3VkcILsu4V3YWt1MgE"}'
2026-03-18 09:07:41.248 INFO [src.node.waku_node] REST service is ready !!
2026-03-18 09:07:41.248 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:60600/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.131.250/tcp/40953/p2p/16Uiu2HAm4LfebsfjLLYSmSGLLLiAK6KzrLsSTXoXWFujdVG3oq6H"]'
2026-03-18 09:07:41.251 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:07:41.258 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0
2026-03-18 09:07:41.259 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-18_09-07-37__409e45db-4569-4ef6-b160-84f20601bfa1__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:07:41.259 DEBUG [src.node.waku_node] Starting Node...
2026-03-18 09:07:41.259 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-18 09:07:41.260 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-18 09:07:41.260 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.97.126
2026-03-18 09:07:41.261 DEBUG [src.node.docker_mananger] Generated ports ['59488', '59489', '59490', '59491', '59492']
2026-03-18 09:07:41.261 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-18 09:07:41.261 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-18 09:07:41.261 DEBUG [src.node.waku_node] Using volumes []
2026-03-18 09:07:41.261 DEBUG [src.node.docker_mananger] docker run -i -t -p 59488:59488 -p 59489:59489 -p 59490:59490 -p 59491:59491 -p 59492:59492 harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=59490 --rest-port=59488 --tcp-port=59489 --discv5-udp-port=59491 --rest-address=0.0.0.0 --nat=extip:172.18.97.126 --peer-exchange=true --discv5-discovery=true --cluster-id=198 --nodekey=ef15e56eb747e82491dd94df8bdea66a5ee5b86760ffbb3acdba5bfcd337cfd5 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=59492 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QHXqxVOF7VeU3Ii4AgD7MCSbrOQRnyZTJ84odWz3_L4fRuQZAFE-5cKHexHgsm-Ih7AgTGOF9z9NZ8e22x67YnsCgmlkgnY0gmlwhKwSg_qKbXVsdGlhZGRyc5YACASsEoP6Bp_5AAoErBKD-gaf-t0DgnJzhQDGAQAAiXNlY3AyNTZrMaEChHDndtRhdHE-grVkmPQEhmMQiQxxyIhljMqS5Vl25KKDdGNwgp_5g3VkcIKf-4V3YWt1MgE
2026-03-18 09:07:41.466 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.97.126 waku 03130213114abd07daf36027ddfd975b5630571dad431e8a8ec96b0d9f81786f
2026-03-18 09:07:41.508 DEBUG [src.node.docker_mananger] Container started with ID 03130213114a. Setting up logs at ./log/docker/node4_2026-03-18_09-07-37__409e45db-4569-4ef6-b160-84f20601bfa1__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:07:41.508 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0. REST: 59488
2026-03-18 09:07:41.509 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-18 09:07:42.509 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59488/health" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:42.525 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-18 09:07:42.526 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-18 09:07:42.526 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59488/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:42.529 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.97.126/tcp/59489/p2p/16Uiu2HAkvr5WUQFECYvXc8EVnQJKTXjeYFfKozBNEX4KutLXujE2","/ip4/172.18.97.126/tcp/59490/ws/p2p/16Uiu2HAkvr5WUQFECYvXc8EVnQJKTXjeYFfKozBNEX4KutLXujE2"],"enrUri":"enr:-L24QNIndd_Z3m8AV2ZtSIHNDDCPOo_GvmjddZ0PItDKeaWRHR67A4-G_2bZCtqUE0QdYGzAeurYOXFzf2Y_ELEsB8gCgmlkgnY0gmlwhKwSYX6KbXVsdGlhZGRyc5YACASsEmF-BuhhAAoErBJhfgboYt0DgnJzhQDGAQAAiXNlY3AyNTZrMaECFRw7CjybHda-84wdOn30WlVugNy4Cpu1cz3tZt47TnuDdGNwguhhg3VkcILoY4V3YWt1MgE"}'
2026-03-18 09:07:42.529 INFO [src.node.waku_node] REST service is ready !!
2026-03-18 09:07:42.530 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59488/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.131.250/tcp/40953/p2p/16Uiu2HAm4LfebsfjLLYSmSGLLLiAK6KzrLsSTXoXWFujdVG3oq6H"]'
2026-03-18 09:07:42.532 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:07:42.540 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0
2026-03-18 09:07:42.540 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-18_09-07-37__409e45db-4569-4ef6-b160-84f20601bfa1__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:07:42.540 DEBUG [src.node.waku_node] Starting Node...
2026-03-18 09:07:42.540 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-18 09:07:42.542 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-18 09:07:42.542 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.252.129
2026-03-18 09:07:42.542 DEBUG [src.node.docker_mananger] Generated ports ['23341', '23342', '23343', '23344', '23345']
2026-03-18 09:07:42.542 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-18 09:07:42.542 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-18 09:07:42.542 DEBUG [src.node.waku_node] Using volumes []
2026-03-18 09:07:42.542 DEBUG [src.node.docker_mananger] docker run -i -t -p 23341:23341 -p 23342:23342 -p 23343:23343 -p 23344:23344 -p 23345:23345 harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=23343 --rest-port=23341 --tcp-port=23342 --discv5-udp-port=23344 --rest-address=0.0.0.0 --nat=extip:172.18.252.129 --peer-exchange=true --discv5-discovery=true --cluster-id=198 --nodekey=0e1fa27bae4edef70b884bcdae99ce1f0fc5d5ebaa0948d6ac699d4bd22fdcde --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=23345 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QHXqxVOF7VeU3Ii4AgD7MCSbrOQRnyZTJ84odWz3_L4fRuQZAFE-5cKHexHgsm-Ih7AgTGOF9z9NZ8e22x67YnsCgmlkgnY0gmlwhKwSg_qKbXVsdGlhZGRyc5YACASsEoP6Bp_5AAoErBKD-gaf-t0DgnJzhQDGAQAAiXNlY3AyNTZrMaEChHDndtRhdHE-grVkmPQEhmMQiQxxyIhljMqS5Vl25KKDdGNwgp_5g3VkcIKf-4V3YWt1MgE
2026-03-18 09:07:42.744 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.252.129 waku e76b6b96445c48066849097e5c035043fd55164d9ee814ec7f8cb34d3b777836
2026-03-18 09:07:42.781 DEBUG [src.node.docker_mananger] Container started with ID e76b6b96445c. Setting up logs at ./log/docker/node5_2026-03-18_09-07-37__409e45db-4569-4ef6-b160-84f20601bfa1__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:07:42.782 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0. REST: 23341
2026-03-18 09:07:42.782 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-18 09:07:43.782 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23341/health" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:43.798 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-18 09:07:43.798 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-18 09:07:43.801 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23341/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:43.805 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.252.129/tcp/23342/p2p/16Uiu2HAmJZcoeseoX3UtogWtPonrc3FbdM1S5N685KufM5GnqQDy","/ip4/172.18.252.129/tcp/23343/ws/p2p/16Uiu2HAmJZcoeseoX3UtogWtPonrc3FbdM1S5N685KufM5GnqQDy"],"enrUri":"enr:-L24QCoyu8Vu315aPj-vVhGJAFavDC0KvKWzxOoB7Dl_buS3ADdCLPF-nLGl7mpREmaJyxF3x4nEVPwp28s--ehyiXECgmlkgnY0gmlwhKwS_IGKbXVsdGlhZGRyc5YACASsEvyBBlsuAAoErBL8gQZbL90DgnJzhQDGAQAAiXNlY3AyNTZrMaEDV8XbIoI95qaidF-KNd-AaGBoyNzdui71ECJG24BHOdyDdGNwglsug3VkcIJbMIV3YWt1MgE"}'
2026-03-18 09:07:43.806 INFO [src.node.waku_node] REST service is ready !!
2026-03-18 09:07:43.806 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:23341/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.131.250/tcp/40953/p2p/16Uiu2HAm4LfebsfjLLYSmSGLLLiAK6KzrLsSTXoXWFujdVG3oq6H"]'
2026-03-18 09:07:43.810 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:07:43.810 DEBUG [src.steps.relay] Running fixture setup: subscribe_main_relay_nodes
2026-03-18 09:07:43.811 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:40952/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/198/1"]'
2026-03-18 09:07:43.819 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:07:43.820 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:35652/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/198/1"]'
2026-03-18 09:07:43.826 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:07:43.826 DEBUG [src.steps.relay] Running fixture setup: subscribe_optional_relay_nodes
2026-03-18 09:07:43.827 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:60600/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/198/1"]'
2026-03-18 09:07:43.833 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:07:43.834 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59488/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/198/1"]'
2026-03-18 09:07:43.839 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:07:43.839 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:23341/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/198/1"]'
2026-03-18 09:07:43.846 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:07:43.848 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:40952/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-18 09:07:43.860 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:07:43.862 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-18 09:07:43.963 DEBUG [src.steps.relay] Checking that peer NODE_1:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:07:43.963 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:40952/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:43.966 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824863848226560,"ephemeral":false,"proof":""}]'
2026-03-18 09:07:43.967 DEBUG [src.steps.relay] Checking that peer NODE_2:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:07:43.968 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:35652/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:43.970 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824863848226560,"ephemeral":false,"proof":""}]'
2026-03-18 09:07:43.971 DEBUG [src.steps.relay] Checking that peer NODE_3:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:07:43.971 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:60600/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:43.974 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824863848226560,"ephemeral":false,"proof":""}]'
2026-03-18 09:07:43.975 DEBUG [src.steps.relay] Checking that peer NODE_4:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:07:43.975 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59488/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:43.978 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824863848226560,"ephemeral":false,"proof":""}]'
2026-03-18 09:07:43.979 DEBUG [src.steps.relay] Checking that peer NODE_5:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:07:43.979 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23341/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:43.982 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824863848226560,"ephemeral":false,"proof":""}]'
2026-03-18 09:07:43.983 INFO [src.steps.relay] WARM UP successful!!
2026-03-18 09:07:43.986 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:23341/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-18 09:07:43.992 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:07:43.993 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-18 09:07:44.094 DEBUG [src.steps.relay] Checking that peer NODE_1:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:07:44.094 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:40952/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:44.097 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824863986106879,"ephemeral":false,"proof":""}]'
2026-03-18 09:07:44.099 DEBUG [src.steps.relay] Checking that peer NODE_2:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:07:44.099 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:35652/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:44.101 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824863986106879,"ephemeral":false,"proof":""}]'
2026-03-18 09:07:44.103 DEBUG [src.steps.relay] Checking that peer NODE_3:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:07:44.103 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:60600/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:44.105 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824863986106879,"ephemeral":false,"proof":""}]'
2026-03-18 09:07:44.106 DEBUG [src.steps.relay] Checking that peer NODE_4:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:07:44.106 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59488/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:44.109 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824863986106879,"ephemeral":false,"proof":""}]'
2026-03-18 09:07:44.110 DEBUG [src.steps.relay] Checking that peer NODE_5:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:07:44.110 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23341/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:07:44.113 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824863986106879,"ephemeral":false,"proof":""}]'
2026-03-18 09:07:44.116 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-18 09:07:44.118 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-18 09:07:44.118 DEBUG [src.node.waku_node] Stopping container with id a255091f6a16
2026-03-18 09:07:44.725 DEBUG [src.node.waku_node] Container stopped.
2026-03-18 09:07:44.726 DEBUG [src.node.waku_node] Stopping container with id e17d9b647333
2026-03-18 09:07:45.311 DEBUG [src.node.waku_node] Container stopped.
2026-03-18 09:07:45.313 DEBUG [src.node.waku_node] Stopping container with id 4951d969c39a
2026-03-18 09:07:45.539 ERROR [src.node.docker_mananger] Max retries reached for container a255091f6a16. Exiting log stream.
2026-03-18 09:07:45.886 DEBUG [src.node.waku_node] Container stopped.
2026-03-18 09:07:45.886 DEBUG [src.node.waku_node] Stopping container with id 03130213114a
2026-03-18 09:07:46.127 ERROR [src.node.docker_mananger] Max retries reached for container e17d9b647333. Exiting log stream.
2026-03-18 09:07:46.454 DEBUG [src.node.waku_node] Container stopped.
2026-03-18 09:07:46.455 DEBUG [src.node.waku_node] Stopping container with id e76b6b96445c
2026-03-18 09:07:46.702 ERROR [src.node.docker_mananger] Max retries reached for container 4951d969c39a. Exiting log stream.
2026-03-18 09:07:47.000 DEBUG [src.node.waku_node] Container stopped.
2026-03-18 09:07:47.003 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-18 09:07:47.020 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-18 09:07:47.027 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-18 09:07:47.033 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-18 09:07:47.039 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-18 09:07:47.044 DEBUG [src.node.docker_mananger] No errors found in the waku logs.