2026-03-05 15:25:13 +00:00

117 lines
18 KiB
Plaintext

2026-03-05 15:06:35.142 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-05 15:06:35.143 DEBUG [tests.conftest] Running test: test_metrics_after_relay_publish with id: 2026-03-05_15-06-35__3ac4d877-d749-4e27-a618-df674e0ad5f9
2026-03-05 15:06:35.143 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-05 15:06:35.143 DEBUG [src.steps.filter] Running fixture setup: filter_setup
2026-03-05 15:06:35.144 DEBUG [src.steps.light_push] Running fixture setup: light_push_setup
2026-03-05 15:06:35.144 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-05 15:06:35.144 DEBUG [src.steps.store] Running fixture setup: store_setup
2026-03-05 15:06:35.144 DEBUG [src.steps.relay] Running fixture setup: setup_main_relay_nodes
2026-03-05 15:06:35.152 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-beta
2026-03-05 15:06:35.152 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-05_15-06-35__3ac4d877-d749-4e27-a618-df674e0ad5f9__harbor.status.im_wakuorg_nwaku:v0.38.0-beta.log
2026-03-05 15:06:35.152 DEBUG [src.node.waku_node] Starting Node...
2026-03-05 15:06:35.152 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-05 15:06:35.153 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-05 15:06:35.154 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.31.65
2026-03-05 15:06:35.154 DEBUG [src.node.docker_mananger] Generated ports ['22259', '22260', '22261', '22262', '22263']
2026-03-05 15:06:35.154 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-05 15:06:35.154 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-05 15:06:35.154 DEBUG [src.node.waku_node] Using volumes []
2026-03-05 15:06:35.154 DEBUG [src.node.docker_mananger] docker run -i -t -p 22259:22259 -p 22260:22260 -p 22261:22261 -p 22262:22262 -p 22263:22263 harbor.status.im/wakuorg/nwaku:v0.38.0-beta --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=22261 --rest-port=22259 --tcp-port=22260 --discv5-udp-port=22262 --rest-address=0.0.0.0 --nat=extip:172.18.31.65 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=5d8de6b07e79fcb2ebfccdeb3a5eaafa1769fcffb4cd6abd036eff4e69dad0bd --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=22263 --metrics-logging=true --relay=true
2026-03-05 15:06:35.341 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.31.65 waku a28f1e8c965c07d931b1bdd7568a45bdc7b669b2754d780deed199483505ec63
2026-03-05 15:06:35.377 DEBUG [src.node.docker_mananger] Container started with ID a28f1e8c965c. Setting up logs at ./log/docker/node1_2026-03-05_15-06-35__3ac4d877-d749-4e27-a618-df674e0ad5f9__harbor.status.im_wakuorg_nwaku:v0.38.0-beta.log
2026-03-05 15:06:35.378 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-beta. REST: 22259
2026-03-05 15:06:35.378 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-05 15:06:35.385 ERROR [src.node.docker_mananger] Max retries reached for container afd904691c86. Exiting log stream.
2026-03-05 15:06:35.930 ERROR [src.node.docker_mananger] Max retries reached for container fee720f61c3e. Exiting log stream.
2026-03-05 15:06:36.379 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22259/health" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:06:36.382 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-05 15:06:36.382 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-05 15:06:36.382 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22259/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:06:36.385 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.31.65/tcp/22260/p2p/16Uiu2HAkyqeiqguZSikhCu95xivgFMd414sX6hQ2HxToACExeEy5","/ip4/172.18.31.65/tcp/22261/ws/p2p/16Uiu2HAkyqeiqguZSikhCu95xivgFMd414sX6hQ2HxToACExeEy5"],"enrUri":"enr:-L24QG6M5GJUBByqy9alKlwOnYycsjCdvAysMmLKzRiOx6MJPe09EFQP22YMAhYdT8nkRD-Dnci2dekdN6lWEbg8IqoCgmlkgnY0gmlwhKwSH0GKbXVsdGlhZGRyc5YACASsEh9BBlb0AAoErBIfQQZW9d0DgnJzhQADAQAAiXNlY3AyNTZrMaECQZND0Pw82l1DGS2ixrU3B8AHp0QGwo6Jvb15Pp3oaCCDdGNwglb0g3VkcIJW9oV3YWt1MgE"}'
2026-03-05 15:06:36.385 INFO [src.node.waku_node] REST service is ready !!
2026-03-05 15:06:36.392 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-beta
2026-03-05 15:06:36.392 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-05_15-06-35__3ac4d877-d749-4e27-a618-df674e0ad5f9__harbor.status.im_wakuorg_nwaku:v0.38.0-beta.log
2026-03-05 15:06:36.392 DEBUG [src.node.waku_node] Starting Node...
2026-03-05 15:06:36.392 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-05 15:06:36.393 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-05 15:06:36.394 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.218.49
2026-03-05 15:06:36.394 DEBUG [src.node.docker_mananger] Generated ports ['40934', '40935', '40936', '40937', '40938']
2026-03-05 15:06:36.394 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-05 15:06:36.394 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-05 15:06:36.394 DEBUG [src.node.waku_node] Using volumes []
2026-03-05 15:06:36.394 DEBUG [src.node.docker_mananger] docker run -i -t -p 40934:40934 -p 40935:40935 -p 40936:40936 -p 40937:40937 -p 40938:40938 harbor.status.im/wakuorg/nwaku:v0.38.0-beta --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=40936 --rest-port=40934 --tcp-port=40935 --discv5-udp-port=40937 --rest-address=0.0.0.0 --nat=extip:172.18.218.49 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=ee817d59b2d9e6c8ccb0befefaa6feca19ab2ee7d854c8cfdada4f3b73a0fc22 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=40938 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QG6M5GJUBByqy9alKlwOnYycsjCdvAysMmLKzRiOx6MJPe09EFQP22YMAhYdT8nkRD-Dnci2dekdN6lWEbg8IqoCgmlkgnY0gmlwhKwSH0GKbXVsdGlhZGRyc5YACASsEh9BBlb0AAoErBIfQQZW9d0DgnJzhQADAQAAiXNlY3AyNTZrMaECQZND0Pw82l1DGS2ixrU3B8AHp0QGwo6Jvb15Pp3oaCCDdGNwglb0g3VkcIJW9oV3YWt1MgE
2026-03-05 15:06:36.590 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.218.49 waku fe045532aff43fca94de26f3c62a03f4582310e36a09290f113a83b40156325b
2026-03-05 15:06:36.622 DEBUG [src.node.docker_mananger] Container started with ID fe045532aff4. Setting up logs at ./log/docker/node2_2026-03-05_15-06-35__3ac4d877-d749-4e27-a618-df674e0ad5f9__harbor.status.im_wakuorg_nwaku:v0.38.0-beta.log
2026-03-05 15:06:36.622 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-beta. REST: 40934
2026-03-05 15:06:36.623 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-05 15:06:37.625 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:40934/health" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:06:37.649 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-05 15:06:37.652 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-05 15:06:37.654 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:40934/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:06:37.660 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.218.49/tcp/40935/p2p/16Uiu2HAmVPeeMo9Uf671iFBfwD9fLff1qH3GzqGRzByqsW4Frk2e","/ip4/172.18.218.49/tcp/40936/ws/p2p/16Uiu2HAmVPeeMo9Uf671iFBfwD9fLff1qH3GzqGRzByqsW4Frk2e"],"enrUri":"enr:-L24QIQQH49JGtAC4yl8HanObYoSajo-h0sviu3MxHSTry4sI1SJj4ldjzcP8qWWjcbr56F5U7mOtLsBuXQwgKjFHTUCgmlkgnY0gmlwhKwS2jGKbXVsdGlhZGRyc5YACASsEtoxBp_nAAoErBLaMQaf6N0DgnJzhQADAQAAiXNlY3AyNTZrMaED-KivLsblB_w9ooOO-S48RF-J3ILmFYpk8LDwlG2rYRODdGNwgp_ng3VkcIKf6YV3YWt1MgE"}'
2026-03-05 15:06:37.661 INFO [src.node.waku_node] REST service is ready !!
2026-03-05 15:06:37.662 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:40934/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.31.65/tcp/22260/p2p/16Uiu2HAkyqeiqguZSikhCu95xivgFMd414sX6hQ2HxToACExeEy5"]'
2026-03-05 15:06:37.666 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 15:06:37.667 DEBUG [src.steps.relay] Running fixture setup: subscribe_main_relay_nodes
2026-03-05 15:06:37.667 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22259/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-05 15:06:37.672 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 15:06:37.672 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:40934/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-05 15:06:37.677 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 15:06:37.678 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22259/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-05 15:06:37.684 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 15:06:37.684 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-05 15:06:37.785 DEBUG [src.steps.relay] Checking that peer NODE_1:harbor.status.im/wakuorg/nwaku:v0.38.0-beta can find the published message
2026-03-05 15:06:37.786 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22259/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:06:37.788 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1772723197678389248,"ephemeral":false,"proof":""}]'
2026-03-05 15:06:37.790 DEBUG [src.steps.relay] Checking that peer NODE_2:harbor.status.im/wakuorg/nwaku:v0.38.0-beta can find the published message
2026-03-05 15:06:37.790 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:40934/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:06:37.793 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1772723197678389248,"ephemeral":false,"proof":""}]'
2026-03-05 15:06:37.794 INFO [src.steps.relay] WARM UP successful!!
2026-03-05 15:06:37.795 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22259/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-05 15:06:37.800 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 15:06:37.801 DEBUG [src.libs.common] Sleeping for 0.5 seconds
2026-03-05 15:06:38.301 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:40934/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:06:38.304 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1772723197795591537,"ephemeral":false,"proof":""}]'
2026-03-05 15:06:38.304 DEBUG [src.libs.common] Sleeping for 5 seconds
2026-03-05 15:06:43.305 DEBUG [src.steps.metrics] Checking metric: libp2p_peers has 1
2026-03-05 15:06:43.310 DEBUG [src.steps.metrics] Found metric: libp2p_peers with value 1.0
2026-03-05 15:06:43.310 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_peers has 1
2026-03-05 15:06:43.313 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_peers with value 1.0
2026-03-05 15:06:43.314 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_topics has 1
2026-03-05 15:06:43.317 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_topics with value 2.0
2026-03-05 15:06:43.317 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_subscriptions_total has 1
2026-03-05 15:06:43.320 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_subscriptions_total with value 2.0
2026-03-05 15:06:43.320 DEBUG [src.steps.metrics] Checking metric: libp2p_gossipsub_peers_per_topic_mesh{topic="other"} has 1
2026-03-05 15:06:43.323 DEBUG [src.steps.metrics] Found metric: libp2p_gossipsub_peers_per_topic_mesh{topic="other"} with value 2.0
2026-03-05 15:06:43.324 DEBUG [src.steps.metrics] Checking metric: waku_peer_store_size has 1
2026-03-05 15:06:43.327 DEBUG [src.steps.metrics] Found metric: waku_peer_store_size with value 1.0
2026-03-05 15:06:43.327 DEBUG [src.steps.metrics] Checking metric: waku_histogram_message_size_count has 1
2026-03-05 15:06:43.330 DEBUG [src.steps.metrics] Found metric: waku_histogram_message_size_count with value 2.0
2026-03-05 15:06:43.330 DEBUG [src.steps.metrics] Checking metric: waku_node_messages_total{type="relay"} has 1
2026-03-05 15:06:43.334 DEBUG [src.steps.metrics] Found metric: waku_node_messages_total{type="relay"} with value 2.0
2026-03-05 15:06:43.334 DEBUG [src.steps.metrics] Checking metric: libp2p_peers has 1
2026-03-05 15:06:43.337 DEBUG [src.steps.metrics] Found metric: libp2p_peers with value 1.0
2026-03-05 15:06:43.338 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_peers has 1
2026-03-05 15:06:43.341 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_peers with value 1.0
2026-03-05 15:06:43.341 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_topics has 1
2026-03-05 15:06:43.344 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_topics with value 2.0
2026-03-05 15:06:43.345 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_subscriptions_total has 1
2026-03-05 15:06:43.348 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_subscriptions_total with value 2.0
2026-03-05 15:06:43.348 DEBUG [src.steps.metrics] Checking metric: libp2p_gossipsub_peers_per_topic_mesh{topic="other"} has 1
2026-03-05 15:06:43.351 DEBUG [src.steps.metrics] Found metric: libp2p_gossipsub_peers_per_topic_mesh{topic="other"} with value 2.0
2026-03-05 15:06:43.351 DEBUG [src.steps.metrics] Checking metric: waku_peer_store_size has 1
2026-03-05 15:06:43.354 DEBUG [src.steps.metrics] Found metric: waku_peer_store_size with value 1.0
2026-03-05 15:06:43.355 DEBUG [src.steps.metrics] Checking metric: waku_histogram_message_size_count has 1
2026-03-05 15:06:43.358 DEBUG [src.steps.metrics] Found metric: waku_histogram_message_size_count with value 2.0
2026-03-05 15:06:43.358 DEBUG [src.steps.metrics] Checking metric: waku_node_messages_total{type="relay"} has 1
2026-03-05 15:06:43.361 DEBUG [src.steps.metrics] Found metric: waku_node_messages_total{type="relay"} with value 2.0
2026-03-05 15:06:43.364 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-05 15:06:43.365 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-05 15:06:43.365 DEBUG [src.node.waku_node] Stopping container with id a28f1e8c965c
2026-03-05 15:06:43.955 DEBUG [src.node.waku_node] Container stopped.
2026-03-05 15:06:43.955 DEBUG [src.node.waku_node] Stopping container with id fe045532aff4
2026-03-05 15:06:44.507 DEBUG [src.node.waku_node] Container stopped.
2026-03-05 15:06:44.508 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-05 15:06:44.517 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-05 15:06:44.524 DEBUG [src.node.docker_mananger] No errors found in the waku logs.