2026-03-18 09:27:31 +00:00

192 lines
36 KiB
Plaintext

2026-03-18 09:06:41.288 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-18 09:06:41.288 DEBUG [tests.conftest] Running test: test_first_node_to_start_publishes with id: 2026-03-18_09-06-41__7f08f22d-a402-4de2-8d14-34645d5c3d1b
2026-03-18 09:06:41.288 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-18 09:06:41.289 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-18 09:06:41.289 DEBUG [src.steps.relay] Running fixture setup: setup_main_relay_nodes
2026-03-18 09:06:41.297 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0
2026-03-18 09:06:41.297 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-18_09-06-41__7f08f22d-a402-4de2-8d14-34645d5c3d1b__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:06:41.297 DEBUG [src.node.waku_node] Starting Node...
2026-03-18 09:06:41.297 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-18 09:06:41.299 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-18 09:06:41.299 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.95.16
2026-03-18 09:06:41.299 DEBUG [src.node.docker_mananger] Generated ports ['31433', '31434', '31435', '31436', '31437']
2026-03-18 09:06:41.299 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-18 09:06:41.299 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-18 09:06:41.299 DEBUG [src.node.waku_node] Using volumes []
2026-03-18 09:06:41.299 DEBUG [src.node.docker_mananger] docker run -i -t -p 31433:31433 -p 31434:31434 -p 31435:31435 -p 31436:31436 -p 31437:31437 harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=31435 --rest-port=31433 --tcp-port=31434 --discv5-udp-port=31436 --rest-address=0.0.0.0 --nat=extip:172.18.95.16 --peer-exchange=true --discv5-discovery=true --cluster-id=198 --nodekey=5c6adceaa8867e4fce1fadbd8917f03ce53cacbfdbae5bddcefc1bfacdba75cd --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=31437 --metrics-logging=true --relay=true
2026-03-18 09:06:41.492 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.95.16 waku 408bdf96c1504da02546b1aa496e0641ede167829bac9522664b374728fdd4d3
2026-03-18 09:06:41.526 DEBUG [src.node.docker_mananger] Container started with ID 408bdf96c150. Setting up logs at ./log/docker/node1_2026-03-18_09-06-41__7f08f22d-a402-4de2-8d14-34645d5c3d1b__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:06:41.527 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0. REST: 31433
2026-03-18 09:06:41.528 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-18 09:06:41.556 ERROR [src.node.docker_mananger] Max retries reached for container fdf9053e2c00. Exiting log stream.
2026-03-18 09:06:42.072 ERROR [src.node.docker_mananger] Max retries reached for container 8904b01e9942. Exiting log stream.
2026-03-18 09:06:42.528 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:31433/health" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:42.531 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-18 09:06:42.531 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-18 09:06:42.532 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:31433/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:42.534 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.95.16/tcp/31434/p2p/16Uiu2HAm9uyFPU6QJBrHsLcUqus1zjKWXhZJ1iyi5FXcKZcEjHBF","/ip4/172.18.95.16/tcp/31435/ws/p2p/16Uiu2HAm9uyFPU6QJBrHsLcUqus1zjKWXhZJ1iyi5FXcKZcEjHBF"],"enrUri":"enr:-L24QDkILmjqcKJy3eeh9-j2aEziwuIjy7fczFpX1PWxCUYIU-AWMYUVyd08BPOp1thvHefpw8hIYCKQh269dwyFvwUCgmlkgnY0gmlwhKwSXxCKbXVsdGlhZGRyc5YACASsEl8QBnrKAAoErBJfEAZ6y90DgnJzhQDGAQAAiXNlY3AyNTZrMaEC10Nn1XA7e1xqh7aWMu3Z3cmfg-zYRrgkRh7GH8BDedKDdGNwgnrKg3VkcIJ6zIV3YWt1MgE"}'
2026-03-18 09:06:42.534 INFO [src.node.waku_node] REST service is ready !!
2026-03-18 09:06:42.540 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0
2026-03-18 09:06:42.540 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-18_09-06-41__7f08f22d-a402-4de2-8d14-34645d5c3d1b__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:06:42.541 DEBUG [src.node.waku_node] Starting Node...
2026-03-18 09:06:42.541 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-18 09:06:42.542 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-18 09:06:42.542 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.226.57
2026-03-18 09:06:42.542 DEBUG [src.node.docker_mananger] Generated ports ['49870', '49871', '49872', '49873', '49874']
2026-03-18 09:06:42.543 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-18 09:06:42.543 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-18 09:06:42.543 DEBUG [src.node.waku_node] Using volumes []
2026-03-18 09:06:42.543 DEBUG [src.node.docker_mananger] docker run -i -t -p 49870:49870 -p 49871:49871 -p 49872:49872 -p 49873:49873 -p 49874:49874 harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=49872 --rest-port=49870 --tcp-port=49871 --discv5-udp-port=49873 --rest-address=0.0.0.0 --nat=extip:172.18.226.57 --peer-exchange=true --discv5-discovery=true --cluster-id=198 --nodekey=5adfd78b59cf4b63fb6f39dfeda849329dea40fe5ffc548c9d24dded3bb1dc64 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=49874 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QDkILmjqcKJy3eeh9-j2aEziwuIjy7fczFpX1PWxCUYIU-AWMYUVyd08BPOp1thvHefpw8hIYCKQh269dwyFvwUCgmlkgnY0gmlwhKwSXxCKbXVsdGlhZGRyc5YACASsEl8QBnrKAAoErBJfEAZ6y90DgnJzhQDGAQAAiXNlY3AyNTZrMaEC10Nn1XA7e1xqh7aWMu3Z3cmfg-zYRrgkRh7GH8BDedKDdGNwgnrKg3VkcIJ6zIV3YWt1MgE
2026-03-18 09:06:42.737 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.226.57 waku 0b329749b34db807dc3faed8b5ab75c50cd46b27546e4d4e3fd65271b6601964
2026-03-18 09:06:42.773 DEBUG [src.node.docker_mananger] Container started with ID 0b329749b34d. Setting up logs at ./log/docker/node2_2026-03-18_09-06-41__7f08f22d-a402-4de2-8d14-34645d5c3d1b__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:06:42.773 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0. REST: 49870
2026-03-18 09:06:42.774 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-18 09:06:43.774 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:49870/health" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:43.792 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-18 09:06:43.794 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-18 09:06:43.795 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:49870/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:43.799 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.226.57/tcp/49871/p2p/16Uiu2HAkxebzNsRyuKPfAfkaQsJLb9EahH97QmFDESR6KttkyGFp","/ip4/172.18.226.57/tcp/49872/ws/p2p/16Uiu2HAkxebzNsRyuKPfAfkaQsJLb9EahH97QmFDESR6KttkyGFp"],"enrUri":"enr:-L24QBQk6B8A2IqTlMerA8I3dMWOQmSL0ix644aoDtrCUwLQSvyA4gwqRdBZinA6-HdNlfmBhT9CUSwsubXfW3-XvbcCgmlkgnY0gmlwhKwS4jmKbXVsdGlhZGRyc5YACASsEuI5BsLPAAoErBLiOQbC0N0DgnJzhQDGAQAAiXNlY3AyNTZrMaECL-MZkWJplfRmNXGwxIglzc9tYejgIIxjQWeo_Gw4foeDdGNwgsLPg3VkcILC0YV3YWt1MgE"}'
2026-03-18 09:06:43.799 INFO [src.node.waku_node] REST service is ready !!
2026-03-18 09:06:43.800 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:49870/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.95.16/tcp/31434/p2p/16Uiu2HAm9uyFPU6QJBrHsLcUqus1zjKWXhZJ1iyi5FXcKZcEjHBF"]'
2026-03-18 09:06:43.803 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:06:43.803 DEBUG [src.steps.relay] Running fixture setup: setup_optional_relay_nodes
2026-03-18 09:06:43.810 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0
2026-03-18 09:06:43.810 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-18_09-06-41__7f08f22d-a402-4de2-8d14-34645d5c3d1b__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:06:43.810 DEBUG [src.node.waku_node] Starting Node...
2026-03-18 09:06:43.810 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-18 09:06:43.811 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-18 09:06:43.812 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.237.71
2026-03-18 09:06:43.812 DEBUG [src.node.docker_mananger] Generated ports ['38285', '38286', '38287', '38288', '38289']
2026-03-18 09:06:43.812 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-18 09:06:43.812 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-18 09:06:43.812 DEBUG [src.node.waku_node] Using volumes []
2026-03-18 09:06:43.812 DEBUG [src.node.docker_mananger] docker run -i -t -p 38285:38285 -p 38286:38286 -p 38287:38287 -p 38288:38288 -p 38289:38289 harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=38287 --rest-port=38285 --tcp-port=38286 --discv5-udp-port=38288 --rest-address=0.0.0.0 --nat=extip:172.18.237.71 --peer-exchange=true --discv5-discovery=true --cluster-id=198 --nodekey=2be44b5eabaab4c0d5dfd7acecbef3becc9af09f85e7cf1832c34116bdaeeddf --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=38289 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QDkILmjqcKJy3eeh9-j2aEziwuIjy7fczFpX1PWxCUYIU-AWMYUVyd08BPOp1thvHefpw8hIYCKQh269dwyFvwUCgmlkgnY0gmlwhKwSXxCKbXVsdGlhZGRyc5YACASsEl8QBnrKAAoErBJfEAZ6y90DgnJzhQDGAQAAiXNlY3AyNTZrMaEC10Nn1XA7e1xqh7aWMu3Z3cmfg-zYRrgkRh7GH8BDedKDdGNwgnrKg3VkcIJ6zIV3YWt1MgE
2026-03-18 09:06:44.007 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.237.71 waku e82a8d4fef026e68e1400fb30c2ba685b2b9b9bb0c17806be8e47b9ecc684709
2026-03-18 09:06:44.040 DEBUG [src.node.docker_mananger] Container started with ID e82a8d4fef02. Setting up logs at ./log/docker/node3_2026-03-18_09-06-41__7f08f22d-a402-4de2-8d14-34645d5c3d1b__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:06:44.040 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0. REST: 38285
2026-03-18 09:06:44.041 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-18 09:06:45.041 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:38285/health" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:45.061 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-18 09:06:45.061 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-18 09:06:45.063 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:38285/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:45.070 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.237.71/tcp/38286/p2p/16Uiu2HAkwqrZVELqY5QNRpZwZHyjB9QjvGBYqK8bUXXdHCQcXdZ1","/ip4/172.18.237.71/tcp/38287/ws/p2p/16Uiu2HAkwqrZVELqY5QNRpZwZHyjB9QjvGBYqK8bUXXdHCQcXdZ1"],"enrUri":"enr:-L24QNinnM2Dy1FBgkXI-M1F2o1IPDWZtrqKJpSmejocT_0KCjFOGS7IJWrl9EuAdPTnwcbLB6Q73Xfe6f4kdGroggYCgmlkgnY0gmlwhKwS7UeKbXVsdGlhZGRyc5YACASsEu1HBpWOAAoErBLtRwaVj90DgnJzhQDGAQAAiXNlY3AyNTZrMaECI-lG9HivhHkuNkujbuxSTLfp5w8ZsvdfHc7YBw_JWFCDdGNwgpWOg3VkcIKVkIV3YWt1MgE"}'
2026-03-18 09:06:45.070 INFO [src.node.waku_node] REST service is ready !!
2026-03-18 09:06:45.071 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:38285/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.95.16/tcp/31434/p2p/16Uiu2HAm9uyFPU6QJBrHsLcUqus1zjKWXhZJ1iyi5FXcKZcEjHBF"]'
2026-03-18 09:06:45.073 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:06:45.080 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0
2026-03-18 09:06:45.080 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-18_09-06-41__7f08f22d-a402-4de2-8d14-34645d5c3d1b__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:06:45.080 DEBUG [src.node.waku_node] Starting Node...
2026-03-18 09:06:45.080 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-18 09:06:45.082 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-18 09:06:45.082 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.126.107
2026-03-18 09:06:45.082 DEBUG [src.node.docker_mananger] Generated ports ['3019', '3020', '3021', '3022', '3023']
2026-03-18 09:06:45.082 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-18 09:06:45.082 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-18 09:06:45.082 DEBUG [src.node.waku_node] Using volumes []
2026-03-18 09:06:45.083 DEBUG [src.node.docker_mananger] docker run -i -t -p 3019:3019 -p 3020:3020 -p 3021:3021 -p 3022:3022 -p 3023:3023 harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=3021 --rest-port=3019 --tcp-port=3020 --discv5-udp-port=3022 --rest-address=0.0.0.0 --nat=extip:172.18.126.107 --peer-exchange=true --discv5-discovery=true --cluster-id=198 --nodekey=9d7c85bfbcfcd1cdda157fdded630ad3532dae9ab2bde9faae9c1818330e7c10 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=3023 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QDkILmjqcKJy3eeh9-j2aEziwuIjy7fczFpX1PWxCUYIU-AWMYUVyd08BPOp1thvHefpw8hIYCKQh269dwyFvwUCgmlkgnY0gmlwhKwSXxCKbXVsdGlhZGRyc5YACASsEl8QBnrKAAoErBJfEAZ6y90DgnJzhQDGAQAAiXNlY3AyNTZrMaEC10Nn1XA7e1xqh7aWMu3Z3cmfg-zYRrgkRh7GH8BDedKDdGNwgnrKg3VkcIJ6zIV3YWt1MgE
2026-03-18 09:06:45.278 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.126.107 waku 6986a7c279d71c157e73c2a4ec5356a5321369aaade30572ce1e0272ff27d6d5
2026-03-18 09:06:45.313 DEBUG [src.node.docker_mananger] Container started with ID 6986a7c279d7. Setting up logs at ./log/docker/node4_2026-03-18_09-06-41__7f08f22d-a402-4de2-8d14-34645d5c3d1b__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:06:45.314 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0. REST: 3019
2026-03-18 09:06:45.314 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-18 09:06:46.316 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3019/health" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:46.332 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-18 09:06:46.333 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-18 09:06:46.333 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3019/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:46.339 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.126.107/tcp/3020/p2p/16Uiu2HAmMWwgc3QDyfo2SUooQGfw5Bv8hKkrS5x1imAhtNNvCM3F","/ip4/172.18.126.107/tcp/3021/ws/p2p/16Uiu2HAmMWwgc3QDyfo2SUooQGfw5Bv8hKkrS5x1imAhtNNvCM3F"],"enrUri":"enr:-L24QN34T5q0JKgbwX4cQ9yIucqaudUyfFPJUMvE1PHfVXdyHzq8ZMgGLXJlxMVTrS25EQmWbiJV9mJWQkMm_sLODKoCgmlkgnY0gmlwhKwSfmuKbXVsdGlhZGRyc5YACASsEn5rBgvMAAoErBJ-awYLzd0DgnJzhQDGAQAAiXNlY3AyNTZrMaEDg6mFMlnXBMfC7accgl-UNIgT47ymMl44hLUWljj7BjqDdGNwggvMg3VkcIILzoV3YWt1MgE"}'
2026-03-18 09:06:46.340 INFO [src.node.waku_node] REST service is ready !!
2026-03-18 09:06:46.340 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:3019/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.95.16/tcp/31434/p2p/16Uiu2HAm9uyFPU6QJBrHsLcUqus1zjKWXhZJ1iyi5FXcKZcEjHBF"]'
2026-03-18 09:06:46.344 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:06:46.351 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0
2026-03-18 09:06:46.351 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-18_09-06-41__7f08f22d-a402-4de2-8d14-34645d5c3d1b__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:06:46.351 DEBUG [src.node.waku_node] Starting Node...
2026-03-18 09:06:46.351 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-18 09:06:46.353 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-18 09:06:46.353 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.130.66
2026-03-18 09:06:46.353 DEBUG [src.node.docker_mananger] Generated ports ['52729', '52730', '52731', '52732', '52733']
2026-03-18 09:06:46.353 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-18 09:06:46.353 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-18 09:06:46.353 DEBUG [src.node.waku_node] Using volumes []
2026-03-18 09:06:46.354 DEBUG [src.node.docker_mananger] docker run -i -t -p 52729:52729 -p 52730:52730 -p 52731:52731 -p 52732:52732 -p 52733:52733 harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=52731 --rest-port=52729 --tcp-port=52730 --discv5-udp-port=52732 --rest-address=0.0.0.0 --nat=extip:172.18.130.66 --peer-exchange=true --discv5-discovery=true --cluster-id=198 --nodekey=b1cb7d9b53b608c4fdf7fb7ce8dbe1d0c8bdc1226f6febc576f32015bd17e5d0 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=52733 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QDkILmjqcKJy3eeh9-j2aEziwuIjy7fczFpX1PWxCUYIU-AWMYUVyd08BPOp1thvHefpw8hIYCKQh269dwyFvwUCgmlkgnY0gmlwhKwSXxCKbXVsdGlhZGRyc5YACASsEl8QBnrKAAoErBJfEAZ6y90DgnJzhQDGAQAAiXNlY3AyNTZrMaEC10Nn1XA7e1xqh7aWMu3Z3cmfg-zYRrgkRh7GH8BDedKDdGNwgnrKg3VkcIJ6zIV3YWt1MgE
2026-03-18 09:06:46.561 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.130.66 waku a62169786f26d52cff631a1bc6e666e675321612c46b36d2974bb4ccb3d4a3da
2026-03-18 09:06:46.596 DEBUG [src.node.docker_mananger] Container started with ID a62169786f26. Setting up logs at ./log/docker/node5_2026-03-18_09-06-41__7f08f22d-a402-4de2-8d14-34645d5c3d1b__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:06:46.596 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0. REST: 52729
2026-03-18 09:06:46.596 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-18 09:06:47.597 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:52729/health" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:47.612 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-18 09:06:47.615 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-18 09:06:47.617 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:52729/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:47.621 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.130.66/tcp/52730/p2p/16Uiu2HAmDhNT8MasmiPTXwh1e1VaCtca8xDRqaUfeaaVwfdGbGNt","/ip4/172.18.130.66/tcp/52731/ws/p2p/16Uiu2HAmDhNT8MasmiPTXwh1e1VaCtca8xDRqaUfeaaVwfdGbGNt"],"enrUri":"enr:-L24QLEwzPtBYLZP1rqtawGTI6bO5A9YOmDyHSUflCGHcW_9d2jNBVBuETl6Cr73vk00g-7PlBTm9fyUZ1qcocsgQvUCgmlkgnY0gmlwhKwSgkKKbXVsdGlhZGRyc5YACASsEoJCBs36AAoErBKCQgbN-90DgnJzhQDGAQAAiXNlY3AyNTZrMaEDD3fWfAgzShNVZsVYUdDUbGQUioRkppyMGj0mbn9GGFGDdGNwgs36g3VkcILN_IV3YWt1MgE"}'
2026-03-18 09:06:47.622 INFO [src.node.waku_node] REST service is ready !!
2026-03-18 09:06:47.623 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:52729/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.95.16/tcp/31434/p2p/16Uiu2HAm9uyFPU6QJBrHsLcUqus1zjKWXhZJ1iyi5FXcKZcEjHBF"]'
2026-03-18 09:06:47.626 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:06:47.626 DEBUG [src.steps.relay] Running fixture setup: subscribe_main_relay_nodes
2026-03-18 09:06:47.627 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:31433/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/198/1"]'
2026-03-18 09:06:47.634 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:06:47.634 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:49870/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/198/1"]'
2026-03-18 09:06:47.643 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:06:47.645 DEBUG [src.steps.relay] Running fixture setup: subscribe_optional_relay_nodes
2026-03-18 09:06:47.645 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:38285/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/198/1"]'
2026-03-18 09:06:47.652 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:06:47.652 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:3019/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/198/1"]'
2026-03-18 09:06:47.659 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:06:47.659 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:52729/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/198/1"]'
2026-03-18 09:06:47.665 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:06:47.667 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:31433/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-18 09:06:47.679 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:06:47.679 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-18 09:06:47.779 DEBUG [src.steps.relay] Checking that peer NODE_1:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:06:47.780 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:31433/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:47.782 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824807667506432,"ephemeral":false,"proof":""}]'
2026-03-18 09:06:47.784 DEBUG [src.steps.relay] Checking that peer NODE_2:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:06:47.784 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:49870/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:47.786 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824807667506432,"ephemeral":false,"proof":""}]'
2026-03-18 09:06:47.787 DEBUG [src.steps.relay] Checking that peer NODE_3:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:06:47.788 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:38285/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:47.790 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824807667506432,"ephemeral":false,"proof":""}]'
2026-03-18 09:06:47.791 DEBUG [src.steps.relay] Checking that peer NODE_4:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:06:47.791 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3019/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:47.794 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824807667506432,"ephemeral":false,"proof":""}]'
2026-03-18 09:06:47.795 DEBUG [src.steps.relay] Checking that peer NODE_5:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:06:47.796 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:52729/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:47.798 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824807667506432,"ephemeral":false,"proof":""}]'
2026-03-18 09:06:47.800 INFO [src.steps.relay] WARM UP successful!!
2026-03-18 09:06:47.801 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:31433/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-18 09:06:47.812 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:06:47.813 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-18 09:06:47.913 DEBUG [src.steps.relay] Checking that peer NODE_1:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:06:47.913 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:31433/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:47.915 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824807801341438,"ephemeral":false,"proof":""}]'
2026-03-18 09:06:47.917 DEBUG [src.steps.relay] Checking that peer NODE_2:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:06:47.917 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:49870/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:47.919 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824807801341438,"ephemeral":false,"proof":""}]'
2026-03-18 09:06:47.921 DEBUG [src.steps.relay] Checking that peer NODE_3:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:06:47.921 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:38285/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:47.923 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824807801341438,"ephemeral":false,"proof":""}]'
2026-03-18 09:06:47.924 DEBUG [src.steps.relay] Checking that peer NODE_4:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:06:47.924 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3019/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:47.927 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824807801341438,"ephemeral":false,"proof":""}]'
2026-03-18 09:06:47.928 DEBUG [src.steps.relay] Checking that peer NODE_5:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:06:47.928 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:52729/relay/v1/messages/%2Fwaku%2F2%2Frs%2F198%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:06:47.930 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773824807801341438,"ephemeral":false,"proof":""}]'
2026-03-18 09:06:47.933 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-18 09:06:47.935 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-18 09:06:47.935 DEBUG [src.node.waku_node] Stopping container with id 408bdf96c150
2026-03-18 09:06:48.507 DEBUG [src.node.waku_node] Container stopped.
2026-03-18 09:06:48.507 DEBUG [src.node.waku_node] Stopping container with id 0b329749b34d
2026-03-18 09:06:49.047 DEBUG [src.node.waku_node] Container stopped.
2026-03-18 09:06:49.049 DEBUG [src.node.waku_node] Stopping container with id e82a8d4fef02
2026-03-18 09:06:49.320 ERROR [src.node.docker_mananger] Max retries reached for container 408bdf96c150. Exiting log stream.
2026-03-18 09:06:49.588 DEBUG [src.node.waku_node] Container stopped.
2026-03-18 09:06:49.588 DEBUG [src.node.waku_node] Stopping container with id 6986a7c279d7
2026-03-18 09:06:49.861 ERROR [src.node.docker_mananger] Max retries reached for container 0b329749b34d. Exiting log stream.
2026-03-18 09:06:50.163 DEBUG [src.node.waku_node] Container stopped.
2026-03-18 09:06:50.164 DEBUG [src.node.waku_node] Stopping container with id a62169786f26
2026-03-18 09:06:50.406 ERROR [src.node.docker_mananger] Max retries reached for container e82a8d4fef02. Exiting log stream.
2026-03-18 09:06:50.718 DEBUG [src.node.waku_node] Container stopped.
2026-03-18 09:06:50.720 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-18 09:06:50.736 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-18 09:06:50.744 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-18 09:06:50.751 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-18 09:06:50.763 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-18 09:06:50.770 DEBUG [src.node.docker_mananger] No errors found in the waku logs.