117 lines
18 KiB
Plaintext

2026-03-26 04:48:02.773 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-26 04:48:02.773 DEBUG [tests.conftest] Running test: test_metrics_after_relay_publish with id: 2026-03-26_04-48-02__c5f6578d-9027-4272-b0cd-e83d90b25125
2026-03-26 04:48:02.773 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-26 04:48:02.773 DEBUG [src.steps.filter] Running fixture setup: filter_setup
2026-03-26 04:48:02.774 DEBUG [src.steps.light_push] Running fixture setup: light_push_setup
2026-03-26 04:48:02.774 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-26 04:48:02.774 DEBUG [src.steps.store] Running fixture setup: store_setup
2026-03-26 04:48:02.775 DEBUG [src.steps.relay] Running fixture setup: setup_main_relay_nodes
2026-03-26 04:48:02.781 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:48:02.781 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-26_04-48-02__c5f6578d-9027-4272-b0cd-e83d90b25125__wakuorg_nwaku:latest.log
2026-03-26 04:48:02.781 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:48:02.782 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:48:02.783 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:48:02.783 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.69.45
2026-03-26 04:48:02.783 DEBUG [src.node.docker_mananger] Generated ports ['2724', '2725', '2726', '2727', '2728']
2026-03-26 04:48:02.783 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:48:02.784 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:48:02.784 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:48:02.784 DEBUG [src.node.docker_mananger] docker run -i -t -p 2724:2724 -p 2725:2725 -p 2726:2726 -p 2727:2727 -p 2728:2728 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=2726 --rest-port=2724 --tcp-port=2725 --discv5-udp-port=2727 --rest-address=0.0.0.0 --nat=extip:172.18.69.45 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=e10dfa7af9befea5d268166c66da1e7199d0194e83201d9eee2796698df0419d --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=2728 --metrics-logging=true --relay=true
2026-03-26 04:48:02.977 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.69.45 waku fb74c743e3dedfb341ab34eafb7404892a7ba69f85fa7652eeff539c4efc6d28
2026-03-26 04:48:03.016 DEBUG [src.node.docker_mananger] Container started with ID fb74c743e3de. Setting up logs at ./log/docker/node1_2026-03-26_04-48-02__c5f6578d-9027-4272-b0cd-e83d90b25125__wakuorg_nwaku:latest.log
2026-03-26 04:48:03.016 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 2724
2026-03-26 04:48:03.017 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:48:03.040 ERROR [src.node.docker_mananger] Max retries reached for container b69b867361b2. Exiting log stream.
2026-03-26 04:48:03.578 ERROR [src.node.docker_mananger] Max retries reached for container c2211e05c92e. Exiting log stream.
2026-03-26 04:48:04.017 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2724/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:48:04.020 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:48:04.020 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:48:04.021 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2724/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:48:04.023 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.69.45/tcp/2725/p2p/16Uiu2HAmGKVtQRuYvQpvkkV2oVpeoVpEBC3AzQ9ur8KuUW5MRkJG","/ip4/172.18.69.45/tcp/2726/ws/p2p/16Uiu2HAmGKVtQRuYvQpvkkV2oVpeoVpEBC3AzQ9ur8KuUW5MRkJG"],"enrUri":"enr:-L24QLttSHwX2-653gD84O_EA2VNF95sqICUfVNekf0827L3ZlPvrJWCxBHp9XDgG79k_TXcoqA-EuwQJEWs3WkXkcECgmlkgnY0gmlwhKwSRS2KbXVsdGlhZGRyc5YACASsEkUtBgqlAAoErBJFLQYKpt0DgnJzhQADAQAAiXNlY3AyNTZrMaEDNnCHTGzlLI_PUcZVp2UATFWAvFWKP4XD8_K4X9DJRTWDdGNwggqlg3VkcIIKp4V3YWt1MgE"}'
2026-03-26 04:48:04.023 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:48:04.030 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:48:04.030 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-26_04-48-02__c5f6578d-9027-4272-b0cd-e83d90b25125__wakuorg_nwaku:latest.log
2026-03-26 04:48:04.030 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:48:04.030 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:48:04.032 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:48:04.032 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.132.13
2026-03-26 04:48:04.032 DEBUG [src.node.docker_mananger] Generated ports ['63879', '63880', '63881', '63882', '63883']
2026-03-26 04:48:04.032 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:48:04.033 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:48:04.033 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:48:04.033 DEBUG [src.node.docker_mananger] docker run -i -t -p 63879:63879 -p 63880:63880 -p 63881:63881 -p 63882:63882 -p 63883:63883 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=63881 --rest-port=63879 --tcp-port=63880 --discv5-udp-port=63882 --rest-address=0.0.0.0 --nat=extip:172.18.132.13 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=45f4ccf8b9931eaf50dc479ada969f880ecfacdac9c4d61eedc6a4e4b0e0faa2 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=63883 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QLttSHwX2-653gD84O_EA2VNF95sqICUfVNekf0827L3ZlPvrJWCxBHp9XDgG79k_TXcoqA-EuwQJEWs3WkXkcECgmlkgnY0gmlwhKwSRS2KbXVsdGlhZGRyc5YACASsEkUtBgqlAAoErBJFLQYKpt0DgnJzhQADAQAAiXNlY3AyNTZrMaEDNnCHTGzlLI_PUcZVp2UATFWAvFWKP4XD8_K4X9DJRTWDdGNwggqlg3VkcIIKp4V3YWt1MgE
2026-03-26 04:48:04.220 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.132.13 waku a0dc1494ef2e4b79c182cee59fe947404ab1ed00578c588f9aae7d6cb0afa8d1
2026-03-26 04:48:04.257 DEBUG [src.node.docker_mananger] Container started with ID a0dc1494ef2e. Setting up logs at ./log/docker/node2_2026-03-26_04-48-02__c5f6578d-9027-4272-b0cd-e83d90b25125__wakuorg_nwaku:latest.log
2026-03-26 04:48:04.258 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 63879
2026-03-26 04:48:04.258 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:48:05.258 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:63879/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:48:05.280 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:48:05.280 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:48:05.281 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:63879/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:48:05.287 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.132.13/tcp/63880/p2p/16Uiu2HAm1WeYuuBQubGnNyVe3iVqtheBmkBYn8Yz7CKYt3qjp61M","/ip4/172.18.132.13/tcp/63881/ws/p2p/16Uiu2HAm1WeYuuBQubGnNyVe3iVqtheBmkBYn8Yz7CKYt3qjp61M"],"enrUri":"enr:-L24QK_uDtDoud3SZBUW36H9VQRniEybE4OOkoFPFV2kGkz6duK64VFWnSO1-SpqmivLbt3Npm9clhuMugc1BGkyHg4CgmlkgnY0gmlwhKwShA2KbXVsdGlhZGRyc5YACASsEoQNBvmIAAoErBKEDQb5id0DgnJzhQADAQAAiXNlY3AyNTZrMaECWmxo7DsSfJx5h30pvjw6YeGgPZDSzcT8h73-zHUuJMCDdGNwgvmIg3VkcIL5ioV3YWt1MgE"}'
2026-03-26 04:48:05.287 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:48:05.288 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:63879/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.69.45/tcp/2725/p2p/16Uiu2HAmGKVtQRuYvQpvkkV2oVpeoVpEBC3AzQ9ur8KuUW5MRkJG"]'
2026-03-26 04:48:05.292 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:48:05.292 DEBUG [src.steps.relay] Running fixture setup: subscribe_main_relay_nodes
2026-03-26 04:48:05.293 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2724/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-26 04:48:05.298 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:48:05.298 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:63879/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-26 04:48:05.303 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:48:05.304 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2724/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-26 04:48:05.311 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:48:05.312 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-26 04:48:05.413 DEBUG [src.steps.relay] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-26 04:48:05.413 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2724/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:48:05.415 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774500485304568832,"ephemeral":false,"proof":""}]'
2026-03-26 04:48:05.417 DEBUG [src.steps.relay] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-26 04:48:05.417 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:63879/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:48:05.419 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774500485304568832,"ephemeral":false,"proof":""}]'
2026-03-26 04:48:05.421 INFO [src.steps.relay] WARM UP successful!!
2026-03-26 04:48:05.421 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2724/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-26 04:48:05.426 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:48:05.427 DEBUG [src.libs.common] Sleeping for 0.5 seconds
2026-03-26 04:48:05.927 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:63879/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:48:05.930 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774500485421869382,"ephemeral":false,"proof":""}]'
2026-03-26 04:48:05.930 DEBUG [src.libs.common] Sleeping for 5 seconds
2026-03-26 04:48:10.931 DEBUG [src.steps.metrics] Checking metric: libp2p_peers has 1
2026-03-26 04:48:10.935 DEBUG [src.steps.metrics] Found metric: libp2p_peers with value 1.0
2026-03-26 04:48:10.936 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_peers has 1
2026-03-26 04:48:10.939 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_peers with value 1.0
2026-03-26 04:48:10.939 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_topics has 1
2026-03-26 04:48:10.942 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_topics with value 2.0
2026-03-26 04:48:10.942 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_subscriptions_total has 1
2026-03-26 04:48:10.946 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_subscriptions_total with value 2.0
2026-03-26 04:48:10.946 DEBUG [src.steps.metrics] Checking metric: libp2p_gossipsub_peers_per_topic_mesh{topic="other"} has 1
2026-03-26 04:48:10.949 DEBUG [src.steps.metrics] Found metric: libp2p_gossipsub_peers_per_topic_mesh{topic="other"} with value 2.0
2026-03-26 04:48:10.949 DEBUG [src.steps.metrics] Checking metric: waku_peer_store_size has 1
2026-03-26 04:48:10.952 DEBUG [src.steps.metrics] Found metric: waku_peer_store_size with value 1.0
2026-03-26 04:48:10.952 DEBUG [src.steps.metrics] Checking metric: waku_histogram_message_size_count has 1
2026-03-26 04:48:10.956 DEBUG [src.steps.metrics] Found metric: waku_histogram_message_size_count with value 2.0
2026-03-26 04:48:10.956 DEBUG [src.steps.metrics] Checking metric: waku_node_messages_total{type="relay"} has 1
2026-03-26 04:48:10.959 DEBUG [src.steps.metrics] Found metric: waku_node_messages_total{type="relay"} with value 2.0
2026-03-26 04:48:10.959 DEBUG [src.steps.metrics] Checking metric: libp2p_peers has 1
2026-03-26 04:48:10.963 DEBUG [src.steps.metrics] Found metric: libp2p_peers with value 1.0
2026-03-26 04:48:10.963 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_peers has 1
2026-03-26 04:48:10.966 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_peers with value 1.0
2026-03-26 04:48:10.967 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_topics has 1
2026-03-26 04:48:10.970 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_topics with value 2.0
2026-03-26 04:48:10.970 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_subscriptions_total has 1
2026-03-26 04:48:10.974 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_subscriptions_total with value 2.0
2026-03-26 04:48:10.974 DEBUG [src.steps.metrics] Checking metric: libp2p_gossipsub_peers_per_topic_mesh{topic="other"} has 1
2026-03-26 04:48:10.977 DEBUG [src.steps.metrics] Found metric: libp2p_gossipsub_peers_per_topic_mesh{topic="other"} with value 2.0
2026-03-26 04:48:10.978 DEBUG [src.steps.metrics] Checking metric: waku_peer_store_size has 1
2026-03-26 04:48:10.981 DEBUG [src.steps.metrics] Found metric: waku_peer_store_size with value 1.0
2026-03-26 04:48:10.981 DEBUG [src.steps.metrics] Checking metric: waku_histogram_message_size_count has 1
2026-03-26 04:48:10.984 DEBUG [src.steps.metrics] Found metric: waku_histogram_message_size_count with value 2.0
2026-03-26 04:48:10.985 DEBUG [src.steps.metrics] Checking metric: waku_node_messages_total{type="relay"} has 1
2026-03-26 04:48:10.988 DEBUG [src.steps.metrics] Found metric: waku_node_messages_total{type="relay"} with value 2.0
2026-03-26 04:48:10.991 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-26 04:48:10.992 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-26 04:48:10.992 DEBUG [src.node.waku_node] Stopping container with id fb74c743e3de
2026-03-26 04:48:11.547 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:48:11.547 DEBUG [src.node.waku_node] Stopping container with id a0dc1494ef2e
2026-03-26 04:48:12.104 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:48:12.105 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-26 04:48:12.118 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-26 04:48:12.124 DEBUG [src.node.docker_mananger] No errors found in the waku logs.