2026-03-05 15:25:13 +00:00

113 lines
17 KiB
Plaintext

2026-03-05 15:13:47.839 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-05 15:13:47.839 DEBUG [tests.conftest] Running test: test_metrics_after_store_get with id: 2026-03-05_15-13-47__cc28a81d-d5b5-4360-895a-1c227671a422
2026-03-05 15:13:47.840 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-05 15:13:47.840 DEBUG [src.steps.filter] Running fixture setup: filter_setup
2026-03-05 15:13:47.840 DEBUG [src.steps.light_push] Running fixture setup: light_push_setup
2026-03-05 15:13:47.841 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-05 15:13:47.841 DEBUG [src.steps.store] Running fixture setup: store_setup
2026-03-05 15:13:47.841 DEBUG [src.steps.store] Running fixture setup: node_setup
2026-03-05 15:13:47.848 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-beta
2026-03-05 15:13:47.848 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/publishing_node1_2026-03-05_15-13-47__cc28a81d-d5b5-4360-895a-1c227671a422__harbor.status.im_wakuorg_nwaku:v0.38.0-beta.log
2026-03-05 15:13:47.848 DEBUG [src.node.waku_node] Starting Node...
2026-03-05 15:13:47.849 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-05 15:13:47.850 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-05 15:13:47.850 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.60.38
2026-03-05 15:13:47.850 DEBUG [src.node.docker_mananger] Generated ports ['46276', '46277', '46278', '46279', '46280']
2026-03-05 15:13:47.850 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-05 15:13:47.850 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-05 15:13:47.851 DEBUG [src.node.waku_node] Using volumes []
2026-03-05 15:13:47.851 DEBUG [src.node.docker_mananger] docker run -i -t -p 46276:46276 -p 46277:46277 -p 46278:46278 -p 46279:46279 -p 46280:46280 harbor.status.im/wakuorg/nwaku:v0.38.0-beta --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=46278 --rest-port=46276 --tcp-port=46277 --discv5-udp-port=46279 --rest-address=0.0.0.0 --nat=extip:172.18.60.38 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=b90ccccbbe7aabdae6c93bbaed9f02a44ab1bf3a0201cbac23bbd9f52bddaea3 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=46280 --metrics-logging=true --store=true --relay=true
2026-03-05 15:13:48.036 ERROR [src.node.docker_mananger] Max retries reached for container 30fee297b9d5. Exiting log stream.
2026-03-05 15:13:48.050 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.60.38 waku fa991f132516998e839652619fdc8b2b00d5a03f9469ab544bf22d8091007252
2026-03-05 15:13:48.086 DEBUG [src.node.docker_mananger] Container started with ID fa991f132516. Setting up logs at ./log/docker/publishing_node1_2026-03-05_15-13-47__cc28a81d-d5b5-4360-895a-1c227671a422__harbor.status.im_wakuorg_nwaku:v0.38.0-beta.log
2026-03-05 15:13:48.086 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-beta. REST: 46276
2026-03-05 15:13:48.087 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-05 15:13:48.626 ERROR [src.node.docker_mananger] Max retries reached for container f2255d01b9ff. Exiting log stream.
2026-03-05 15:13:49.087 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:46276/health" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:13:49.090 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-05 15:13:49.090 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-05 15:13:49.090 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:46276/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:13:49.093 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.60.38/tcp/46277/p2p/16Uiu2HAmK2WWWTfcYjhjAKTUniaKxQuoqNWhQ6BkR61kj43nBF6s","/ip4/172.18.60.38/tcp/46278/ws/p2p/16Uiu2HAmK2WWWTfcYjhjAKTUniaKxQuoqNWhQ6BkR61kj43nBF6s"],"enrUri":"enr:-L24QI_Fvk_InQs2ubWRQmvVsooOSBKxa7mSHQpZ23APnn5GKCgfbrgsGwTEab8itlkZC8mifvrZ_KxxWVULy_OvxIICgmlkgnY0gmlwhKwSPCaKbXVsdGlhZGRyc5YACASsEjwmBrTFAAoErBI8Jga0xt0DgnJzhQADAQAAiXNlY3AyNTZrMaEDXqlrXFUZ_7zfcZoT6APHSRf1WGL4TUm3Il9Dz-jzcKyDdGNwgrTFg3VkcIK0x4V3YWt1MgM"}'
2026-03-05 15:13:49.093 INFO [src.node.waku_node] REST service is ready !!
2026-03-05 15:13:49.100 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-beta
2026-03-05 15:13:49.100 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/store_node1_2026-03-05_15-13-47__cc28a81d-d5b5-4360-895a-1c227671a422__harbor.status.im_wakuorg_nwaku:v0.38.0-beta.log
2026-03-05 15:13:49.100 DEBUG [src.node.waku_node] Starting Node...
2026-03-05 15:13:49.100 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-05 15:13:49.102 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-05 15:13:49.102 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.1.58
2026-03-05 15:13:49.102 DEBUG [src.node.docker_mananger] Generated ports ['7745', '7746', '7747', '7748', '7749']
2026-03-05 15:13:49.102 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-05 15:13:49.102 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-05 15:13:49.103 DEBUG [src.node.waku_node] Using volumes []
2026-03-05 15:13:49.103 DEBUG [src.node.docker_mananger] docker run -i -t -p 7745:7745 -p 7746:7746 -p 7747:7747 -p 7748:7748 -p 7749:7749 harbor.status.im/wakuorg/nwaku:v0.38.0-beta --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=7747 --rest-port=7745 --tcp-port=7746 --discv5-udp-port=7748 --rest-address=0.0.0.0 --nat=extip:172.18.1.58 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=e6c19eb4baf063e2edbbb8bc15221e0b1bccecc6191ee2003ca1e0dec2c4af07 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=7749 --metrics-logging=true --discv5-bootstrap-node=enr:-L24QI_Fvk_InQs2ubWRQmvVsooOSBKxa7mSHQpZ23APnn5GKCgfbrgsGwTEab8itlkZC8mifvrZ_KxxWVULy_OvxIICgmlkgnY0gmlwhKwSPCaKbXVsdGlhZGRyc5YACASsEjwmBrTFAAoErBI8Jga0xt0DgnJzhQADAQAAiXNlY3AyNTZrMaEDXqlrXFUZ_7zfcZoT6APHSRf1WGL4TUm3Il9Dz-jzcKyDdGNwgrTFg3VkcIK0x4V3YWt1MgM --storenode=/ip4/172.18.60.38/tcp/46277/p2p/16Uiu2HAmK2WWWTfcYjhjAKTUniaKxQuoqNWhQ6BkR61kj43nBF6s --store=true --relay=true
2026-03-05 15:13:49.305 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.1.58 waku af6f080b1c123c97366804830194fcc88e9e7bb8ad46551f9fcfb6f3e3a4bd27
2026-03-05 15:13:49.338 DEBUG [src.node.docker_mananger] Container started with ID af6f080b1c12. Setting up logs at ./log/docker/store_node1_2026-03-05_15-13-47__cc28a81d-d5b5-4360-895a-1c227671a422__harbor.status.im_wakuorg_nwaku:v0.38.0-beta.log
2026-03-05 15:13:49.338 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-beta. REST: 7745
2026-03-05 15:13:49.339 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-05 15:13:50.339 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:7745/health" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:13:50.342 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-05 15:13:50.342 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-05 15:13:50.342 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:7745/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:13:50.345 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.1.58/tcp/7746/p2p/16Uiu2HAmBkwQovnBvPuRmupFtCA5d1b1XCQk6mMbsH2bhE8GmAPC","/ip4/172.18.1.58/tcp/7747/ws/p2p/16Uiu2HAmBkwQovnBvPuRmupFtCA5d1b1XCQk6mMbsH2bhE8GmAPC"],"enrUri":"enr:-L24QGaOr_YQavoE9fcj7wdVuzRIdj_0xYzR0vc9zqvkvrwIHhmWhf9DgHhAXv7HXMxJAQv-X9SBGKQTWbpwYgAFqqQCgmlkgnY0gmlwhKwSATqKbXVsdGlhZGRyc5YACASsEgE6Bh5CAAoErBIBOgYeQ90DgnJzhQADAQAAiXNlY3AyNTZrMaEC8qp5dOSAxnVFuzpln_FtgwsHbNxqqoQQLU5B5MZBtruDdGNwgh5Cg3VkcIIeRIV3YWt1MgM"}'
2026-03-05 15:13:50.345 INFO [src.node.waku_node] REST service is ready !!
2026-03-05 15:13:50.345 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:7745/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.60.38/tcp/46277/p2p/16Uiu2HAmK2WWWTfcYjhjAKTUniaKxQuoqNWhQ6BkR61kj43nBF6s"]'
2026-03-05 15:13:50.348 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 15:13:50.348 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:46276/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-05 15:13:50.352 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 15:13:50.353 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:7745/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-05 15:13:50.359 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 15:13:50.361 DEBUG [src.steps.store] Relaying message
2026-03-05 15:13:50.361 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:46276/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-05 15:13:50.368 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 15:13:50.369 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-05 15:13:50.569 DEBUG [src.steps.store] Checking that peer harbor.status.im/wakuorg/nwaku:v0.38.0-beta can find the stored messages
2026-03-05 15:13:50.570 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:46276/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F1&pageSize=50&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:13:50.573 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0x80348fbc10be1f93e8460aca4684e9878adeeb3cfecdcc804a5bfa92a93478cb"}]}'
2026-03-05 15:13:50.573 DEBUG [src.steps.store] messages length is 1
2026-03-05 15:13:50.573 DEBUG [src.steps.store] Checking that peer harbor.status.im/wakuorg/nwaku:v0.38.0-beta can find the stored messages
2026-03-05 15:13:50.573 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:7745/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F1&pageSize=50&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:13:50.576 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0x80348fbc10be1f93e8460aca4684e9878adeeb3cfecdcc804a5bfa92a93478cb"}]}'
2026-03-05 15:13:50.576 DEBUG [src.steps.store] messages length is 1
2026-03-05 15:13:50.577 DEBUG [src.libs.common] Sleeping for 5 seconds
2026-03-05 15:13:55.577 DEBUG [src.steps.metrics] Checking metric: libp2p_peers has 1
2026-03-05 15:13:55.581 DEBUG [src.steps.metrics] Found metric: libp2p_peers with value 1.0
2026-03-05 15:13:55.581 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_peers has 1
2026-03-05 15:13:55.584 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_peers with value 1.0
2026-03-05 15:13:55.585 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_topics has 1
2026-03-05 15:13:55.587 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_topics with value 2.0
2026-03-05 15:13:55.588 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_subscriptions_total has 1
2026-03-05 15:13:55.591 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_subscriptions_total with value 2.0
2026-03-05 15:13:55.591 DEBUG [src.steps.metrics] Checking metric: waku_peer_store_size has 1
2026-03-05 15:13:55.594 DEBUG [src.steps.metrics] Found metric: waku_peer_store_size with value 1.0
2026-03-05 15:13:55.594 DEBUG [src.steps.metrics] Checking metric: waku_histogram_message_size_count has 1
2026-03-05 15:13:55.597 DEBUG [src.steps.metrics] Found metric: waku_histogram_message_size_count with value 1.0
2026-03-05 15:13:55.597 DEBUG [src.steps.metrics] Checking metric: waku_node_messages_total{type="relay"} has 1
2026-03-05 15:13:55.600 DEBUG [src.steps.metrics] Found metric: waku_node_messages_total{type="relay"} with value 1.0
2026-03-05 15:13:55.601 DEBUG [src.steps.metrics] Checking metric: waku_service_peers{protocol="/vac/waku/store/2.0.0-beta4",peerId="/ip4/172.18.60.38/tcp/46277"} has 1
2026-03-05 15:13:55.604 DEBUG [src.steps.metrics] Found metric: waku_service_peers{protocol="/vac/waku/store/2.0.0-beta4",peerId="/ip4/172.18.60.38/tcp/46277"} with value 1.0
2026-03-05 15:13:55.604 DEBUG [src.steps.metrics] Checking metric: waku_service_peers{protocol="/vac/waku/store-query/3.0.0",peerId="/ip4/172.18.60.38/tcp/46277"} has 1
2026-03-05 15:13:55.607 DEBUG [src.steps.metrics] Found metric: waku_service_peers{protocol="/vac/waku/store-query/3.0.0",peerId="/ip4/172.18.60.38/tcp/46277"} with value 1.0
2026-03-05 15:13:55.608 DEBUG [src.steps.metrics] Checking metric: libp2p_peers has 1
2026-03-05 15:13:55.611 DEBUG [src.steps.metrics] Found metric: libp2p_peers with value 1.0
2026-03-05 15:13:55.611 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_peers has 1
2026-03-05 15:13:55.614 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_peers with value 1.0
2026-03-05 15:13:55.614 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_topics has 1
2026-03-05 15:13:55.617 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_topics with value 2.0
2026-03-05 15:13:55.617 DEBUG [src.steps.metrics] Checking metric: libp2p_pubsub_subscriptions_total has 1
2026-03-05 15:13:55.620 DEBUG [src.steps.metrics] Found metric: libp2p_pubsub_subscriptions_total with value 2.0
2026-03-05 15:13:55.621 DEBUG [src.steps.metrics] Checking metric: waku_peer_store_size has 1
2026-03-05 15:13:55.624 DEBUG [src.steps.metrics] Found metric: waku_peer_store_size with value 1.0
2026-03-05 15:13:55.624 DEBUG [src.steps.metrics] Checking metric: waku_histogram_message_size_count has 1
2026-03-05 15:13:55.627 DEBUG [src.steps.metrics] Found metric: waku_histogram_message_size_count with value 1.0
2026-03-05 15:13:55.627 DEBUG [src.steps.metrics] Checking metric: waku_node_messages_total{type="relay"} has 1
2026-03-05 15:13:55.630 DEBUG [src.steps.metrics] Found metric: waku_node_messages_total{type="relay"} with value 1.0
2026-03-05 15:13:55.632 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-05 15:13:55.633 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-05 15:13:55.634 DEBUG [src.node.waku_node] Stopping container with id fa991f132516
2026-03-05 15:13:56.236 DEBUG [src.node.waku_node] Container stopped.
2026-03-05 15:13:56.237 DEBUG [src.node.waku_node] Stopping container with id af6f080b1c12
2026-03-05 15:13:56.803 DEBUG [src.node.waku_node] Container stopped.
2026-03-05 15:13:56.805 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-05 15:13:56.818 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-05 15:13:56.826 DEBUG [src.node.docker_mananger] No errors found in the waku logs.