209 lines
35 KiB
Plaintext

2026-03-09 04:41:43.081 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-09 04:41:43.082 DEBUG [tests.conftest] Running test: test_node_without_sync_or_relay_stays_empty with id: 2026-03-09_04-41-43__b70f52fc-ebcb-4420-8333-94dd9cfe83ad
2026-03-09 04:41:43.083 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-09 04:41:43.094 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:41:43.095 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-09_04-41-43__b70f52fc-ebcb-4420-8333-94dd9cfe83ad__wakuorg_nwaku:latest.log
2026-03-09 04:41:43.103 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:41:43.104 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-09_04-41-43__b70f52fc-ebcb-4420-8333-94dd9cfe83ad__wakuorg_nwaku:latest.log
2026-03-09 04:41:43.113 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:41:43.113 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-09_04-41-43__b70f52fc-ebcb-4420-8333-94dd9cfe83ad__wakuorg_nwaku:latest.log
2026-03-09 04:41:43.115 DEBUG [src.steps.store] Running fixture setup: store_setup
2026-03-09 04:41:43.116 DEBUG [src.node.waku_node] Starting Node...
2026-03-09 04:41:43.117 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-09 04:41:43.119 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-09 04:41:43.120 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.182.134
2026-03-09 04:41:43.120 DEBUG [src.node.docker_mananger] Generated ports ['28726', '28727', '28728', '28729', '28730']
2026-03-09 04:41:43.120 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-09 04:41:43.120 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-09 04:41:43.121 DEBUG [src.node.waku_node] Using volumes []
2026-03-09 04:41:43.121 DEBUG [src.node.docker_mananger] docker run -i -t -p 28726:28726 -p 28727:28727 -p 28728:28728 -p 28729:28729 -p 28730:28730 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=28728 --rest-port=28726 --tcp-port=28727 --discv5-udp-port=28729 --rest-address=0.0.0.0 --nat=extip:172.18.182.134 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=defb77471098cefde11c7cd668c324c13efbeeacceb7bcae97d3ba1cdcb967a1 --store-sync=true --store=true --store-sync-range=300 --store-sync-interval=6 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=28730 --metrics-logging=true --store-sync-relay-jitter=0 --relay=true --dns-discovery=false
2026-03-09 04:41:43.314 ERROR [src.node.docker_mananger] Max retries reached for container 892a6c795743. Exiting log stream.
2026-03-09 04:41:43.314 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.182.134 waku 579e98b422c9a24caa12f877747243ad4dda65a2e3f0c4385bfea16fe759c1a1
2026-03-09 04:41:43.351 DEBUG [src.node.docker_mananger] Container started with ID 579e98b422c9. Setting up logs at ./log/docker/node1_2026-03-09_04-41-43__b70f52fc-ebcb-4420-8333-94dd9cfe83ad__wakuorg_nwaku:latest.log
2026-03-09 04:41:43.352 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 28726
2026-03-09 04:41:43.352 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-09 04:41:43.938 ERROR [src.node.docker_mananger] Max retries reached for container 368157e4b191. Exiting log stream.
2026-03-09 04:41:44.354 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28726/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:41:44.357 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-09 04:41:44.357 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-09 04:41:44.357 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28726/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:41:44.359 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.182.134/tcp/28727/p2p/16Uiu2HAmPzQBs5NpVmmEcL613JKqHESw39YuGFHLn7Gk3sTXDHRC","/ip4/172.18.182.134/tcp/28728/ws/p2p/16Uiu2HAmPzQBs5NpVmmEcL613JKqHESw39YuGFHLn7Gk3sTXDHRC"],"enrUri":"enr:-L24QBVuzBxzYuDY_Im2kdBsQaC9Nz-KuEGXAMquQkqyYfnAT2g1L1QBI5CD5opSowfYcskd-K6eCQsMJkEy-0BMGD8CgmlkgnY0gmlwhKwStoaKbXVsdGlhZGRyc5YACASsEraGBnA3AAoErBK2hgZwON0DgnJzhQADAQAAiXNlY3AyNTZrMaEDqGmLcTL-WgsmB0ipi6Y5sAEYjqQi_Bj5YnYWp58wWjuDdGNwgnA3g3VkcIJwOYV3YWt1MhM"}'
2026-03-09 04:41:44.360 INFO [src.node.waku_node] REST service is ready !!
2026-03-09 04:41:44.360 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:44.360 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:44.376 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:44.377 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:44.587 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:44.588 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:44.592 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:44.592 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:44.803 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:44.803 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:44.807 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:44.807 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:45.018 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:45.018 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:45.023 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:45.023 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:45.234 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:45.234 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:45.238 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:45.238 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:45.449 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:45.449 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:45.454 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:45.454 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:45.665 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:45.665 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:45.669 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:45.669 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:45.880 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:45.880 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:45.885 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:45.885 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:46.096 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:46.096 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:46.100 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:46.100 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:46.311 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:46.311 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:46.316 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:46.316 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:46.527 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:46.527 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:46.531 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:46.531 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:46.742 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:46.742 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:46.746 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:46.747 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:46.957 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:46.958 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:46.961 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:46.962 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:47.172 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:47.173 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:47.177 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:47.177 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:47.388 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:47.388 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:47.392 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:47.392 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:47.603 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:47.603 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:47.608 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:47.608 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:47.819 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:47.819 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:47.823 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:47.823 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:48.034 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:48.034 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:48.038 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:48.038 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:48.249 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:48.249 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:48.254 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:48.254 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:48.465 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:48.465 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:48.471 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:48.471 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:48.682 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:48.682 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:48.687 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:48.687 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:48.898 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:48.898 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:48.902 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:48.902 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:49.113 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:49.114 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:49.118 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:49.118 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:49.329 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:49.329 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:49.333 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:49.333 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:49.544 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:49.544 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:49.548 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:49.549 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:49.759 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:49.760 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:49.764 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:49.764 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:49.975 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:49.975 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:49.979 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:49.979 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:50.190 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:50.190 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:50.195 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:50.195 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:50.406 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:50.406 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:50.410 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:50.410 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:50.621 DEBUG [src.steps.store] Relaying message
2026-03-09 04:41:50.621 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28726/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:41:50.625 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:41:50.625 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-09 04:41:50.836 DEBUG [src.node.waku_node] Starting Node...
2026-03-09 04:41:50.836 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-09 04:41:50.838 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-09 04:41:50.838 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.198.171
2026-03-09 04:41:50.838 DEBUG [src.node.docker_mananger] Generated ports ['43995', '43996', '43997', '43998', '43999']
2026-03-09 04:41:50.838 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-09 04:41:50.838 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-09 04:41:50.839 DEBUG [src.node.waku_node] Using volumes []
2026-03-09 04:41:50.839 DEBUG [src.node.docker_mananger] docker run -i -t -p 43995:43995 -p 43996:43996 -p 43997:43997 -p 43998:43998 -p 43999:43999 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=43997 --rest-port=43995 --tcp-port=43996 --discv5-udp-port=43998 --rest-address=0.0.0.0 --nat=extip:172.18.198.171 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=8fdafbdb5b2be1bc71ffc5f4ac5fcebab4c96ba8a1af5cafbd9afa9db7646e1c --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=43999 --metrics-logging=true --relay=false --dns-discovery=false --discv5-bootstrap-node=enr:-L24QBVuzBxzYuDY_Im2kdBsQaC9Nz-KuEGXAMquQkqyYfnAT2g1L1QBI5CD5opSowfYcskd-K6eCQsMJkEy-0BMGD8CgmlkgnY0gmlwhKwStoaKbXVsdGlhZGRyc5YACASsEraGBnA3AAoErBK2hgZwON0DgnJzhQADAQAAiXNlY3AyNTZrMaEDqGmLcTL-WgsmB0ipi6Y5sAEYjqQi_Bj5YnYWp58wWjuDdGNwgnA3g3VkcIJwOYV3YWt1MhM
2026-03-09 04:41:51.030 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.198.171 waku 7a6f0fa645e1e94f9fd13e5e53f7815c72d8a3acf67419506632ee0edb5bc047
2026-03-09 04:41:51.063 DEBUG [src.node.docker_mananger] Container started with ID 7a6f0fa645e1. Setting up logs at ./log/docker/node2_2026-03-09_04-41-43__b70f52fc-ebcb-4420-8333-94dd9cfe83ad__wakuorg_nwaku:latest.log
2026-03-09 04:41:51.063 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 43995
2026-03-09 04:41:51.064 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-09 04:41:52.065 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:43995/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:41:52.068 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-09 04:41:52.068 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-09 04:41:52.068 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:43995/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:41:52.071 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.198.171/tcp/43996/p2p/16Uiu2HAm1HF2Akn6iCwwnwiHKCexqZW1sk5rXtnM9ZJzMkcLRGhF","/ip4/172.18.198.171/tcp/43997/ws/p2p/16Uiu2HAm1HF2Akn6iCwwnwiHKCexqZW1sk5rXtnM9ZJzMkcLRGhF"],"enrUri":"enr:-L24QNonQlNbUAQddV8MX9vMN4VTsEUZA0SAH4Hxd0xejMtJbWm4u5gRkGCls3bcsKldoqzvfNFbdiw2EJMUBNrK-RcCgmlkgnY0gmlwhKwSxquKbXVsdGlhZGRyc5YACASsEsarBqvcAAoErBLGqwar3d0DgnJzhQADAQAAiXNlY3AyNTZrMaECVv1A-WD_Npxb-w86CJQzah8mReX2lMzunpYlVNtUAMqDdGNwgqvcg3VkcIKr3oV3YWt1MgA"}'
2026-03-09 04:41:52.071 INFO [src.node.waku_node] REST service is ready !!
2026-03-09 04:41:52.071 DEBUG [src.node.waku_node] Starting Node...
2026-03-09 04:41:52.071 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-09 04:41:52.073 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-09 04:41:52.073 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.120.155
2026-03-09 04:41:52.073 DEBUG [src.node.docker_mananger] Generated ports ['2695', '2696', '2697', '2698', '2699']
2026-03-09 04:41:52.073 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-09 04:41:52.073 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-09 04:41:52.073 DEBUG [src.node.waku_node] Using volumes []
2026-03-09 04:41:52.073 DEBUG [src.node.docker_mananger] docker run -i -t -p 2695:2695 -p 2696:2696 -p 2697:2697 -p 2698:2698 -p 2699:2699 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=2697 --rest-port=2695 --tcp-port=2696 --discv5-udp-port=2698 --rest-address=0.0.0.0 --nat=extip:172.18.120.155 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=ef7bea1dc857bdcd24c5bf8612f829cabfcefa3fdb153998fabd72daabb80966 --store-sync=true --store=true --store-sync-range=300 --store-sync-interval=6 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=2699 --metrics-logging=true --store-sync-relay-jitter=0 --relay=false --dns-discovery=false --discv5-bootstrap-node=enr:-L24QBVuzBxzYuDY_Im2kdBsQaC9Nz-KuEGXAMquQkqyYfnAT2g1L1QBI5CD5opSowfYcskd-K6eCQsMJkEy-0BMGD8CgmlkgnY0gmlwhKwStoaKbXVsdGlhZGRyc5YACASsEraGBnA3AAoErBK2hgZwON0DgnJzhQADAQAAiXNlY3AyNTZrMaEDqGmLcTL-WgsmB0ipi6Y5sAEYjqQi_Bj5YnYWp58wWjuDdGNwgnA3g3VkcIJwOYV3YWt1MhM
2026-03-09 04:41:52.265 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.120.155 waku 4d36102d2983cfb806a4175382f382f754865121a08e0fdfcfc755943c98b845
2026-03-09 04:41:52.306 DEBUG [src.node.docker_mananger] Container started with ID 4d36102d2983. Setting up logs at ./log/docker/node3_2026-03-09_04-41-43__b70f52fc-ebcb-4420-8333-94dd9cfe83ad__wakuorg_nwaku:latest.log
2026-03-09 04:41:52.307 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 2695
2026-03-09 04:41:52.308 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-09 04:41:53.308 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2695/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:41:53.311 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-09 04:41:53.311 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-09 04:41:53.312 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2695/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:41:53.314 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.120.155/tcp/2696/p2p/16Uiu2HAmAWhfDA8va4ncSMb5pcofrd2VeRicmwNqiLbrep6v2fJu","/ip4/172.18.120.155/tcp/2697/ws/p2p/16Uiu2HAmAWhfDA8va4ncSMb5pcofrd2VeRicmwNqiLbrep6v2fJu"],"enrUri":"enr:-L24QMJ9dQPv9H9tgrfEjCOs6VcTg-yCLpDWpxqla8jT8fG2QBOa_mc4wuHvvGGuZZU5VMXqeOZ3YYpuRRlEvAJJiyoCgmlkgnY0gmlwhKwSeJuKbXVsdGlhZGRyc5YACASsEnibBgqIAAoErBJ4mwYKid0DgnJzhQADAQAAiXNlY3AyNTZrMaEC4CkbaRxtbEw3bilZdp99bL7Uw5nvwKyXUnD5o8_HtT6DdGNwggqIg3VkcIIKioV3YWt1MhI"}'
2026-03-09 04:41:53.314 INFO [src.node.waku_node] REST service is ready !!
2026-03-09 04:41:53.315 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2695/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.182.134/tcp/28727/p2p/16Uiu2HAmPzQBs5NpVmmEcL613JKqHESw39YuGFHLn7Gk3sTXDHRC"]'
2026-03-09 04:41:53.354 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:42:11.356 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2695/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F0&cursor=&pageSize=200&ascending=true&peer_id=" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:42:11.360 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0x5bc58dec5be078aa38a83e9e4bcc472ec79ddbe5884ed2d88791d689323de5f1"},{"messageHash":"0x3c86b3e209e487a1fe9c9a11dd7ab1dc4afd5ebc32fab0178e5731971015e0fe"},{"messageHash":"0x2792b06455d8979bc43a052fe4bf250796f42801b3ca239915eaae94112fed6c"},{"messageHash":"0xc03d3b9fd1fe637e231be37a092cec2ea6725c7cce88d5e22bdab4e5c89fae0e"},{"messageHash":"0x012a5e2a257c9a7c035cac26dc8eb86cb0d4722949439bcab19dbe0bcc3c3ba6"},{"messageHash":"0xf00a1c0a5c9ebc73fed21ecc9a62ad804437fbc0a986b5b1ca91ecdeafa68f11"},{"messageHash":"0x6c7a835af5e5e0d86a443d33c12379df00656aa12d9312e7a7e2c9c62d83391b"},{"messageHash":"0x05bbc1c5c955b9346a4be20e9af1f2c45b3da923abfa4a61ed4e6530e862d544"},{"messageHash":"0x2a194bb1bf8ca87b0ba91d960f0c7720b3d56a27604c70ea953abf96638fbb94"},{"messageHash":"0x6f714b26b51b07eeacd74a539143382fefed622433ed0597021f5f6e94155b17"},{"messageHash":"0x9d6da731d3cd926aaf973b0a9ef8c66c67131ad1ec057a8e577a44bf0bbe3e64"},{"messageHash":"0x29fcbc94e5bc4d8c1ff30f2dd42f750e1368d2d7492d3363a6b7c7955db0218e"},{"messageHash":"0xc1e211147b9f1b5b768f99937a9b8a9d115cdacf479d5f97ca2e62b1faab98a0"},{"messageHash":"0x7dce626ddfb991d8f4a13d933f7f328174317c6d9538797904655baa562e9615"},{"messageHash":"0x6e02bc6fa0ca98b1c191fc4136e2802646b1c239476220e7224382b598afd3af"},{"messageHash":"0xa4e6c0434746b07149386bd45278ccf8c2e6ba6fba942c616c2805601240f22c"},{"messageHash":"0x74e93c4fda3677d66fe5ecb9b2218feb2d2c1b8a2c33f4c337b8f0af784e8f9a"},{"messageHash":"0x3703cfd757dc4aee9c6c5cf89eadc1a1712083e4747795a5dd0f6eba852d196e"},{"messageHash":"0x8a8f8af809d2aaf40d7b31eec0f5385bd7bfd6eb5034639673fe3bf026183eca"},{"messageHash":"0x7cd13a58affd29caac6e0f0fc5bf0d10207aa5e34d12559696a22e9d4087f767"},{"messageHash":"0xa622695eff2f1730db8078ab09871859cd1bea47255763f57a26cea29251f63c"},{"messageHash":"0xcb1cd564954b4c58d66d959a134d93df4cd8e610cdaef8496d5a96d537a48071"},{"messageHash":"0x58c5f3a6585700f74accc9d06fc39a0016b5f2b35f0feb99f173de75263f1b5f"},{"messageHash":"0x7da1adb93bd61347b436d509006e7656bb6ce38580be3590a0e656cdce723851"},{"messageHash":"0x425ab784ce1b45827777acdb185595562c2fde479c1694302024262e5fde8ca9"},{"messageHash":"0x49101130b6b93da6b319f120ef37b367207abeb655408e30385bdd363b4c5ea9"},{"messageHash":"0x65d7997a54f24504cb564ef3eb17efd76548c407b9f9474746007d7606b13c54"},{"messageHash":"0x8ee9811f98bb4af979a68e6c1050a47e295e4e8a64264871ce0ad36b80f97429"},{"messageHash":"0x0cd99795d34982cf3f4cec731acc24456e8b67d0a02447244af503fc1aead849"},{"messageHash":"0x6b4ea1ba6813c176de5632e4123c1f1563d2a090a6d2483687a8fa999d10f0f8"}]}'
2026-03-09 04:42:11.360 DEBUG [tests.store_sync.test_store_sync] Node3 store has 30 messages expected 30
2026-03-09 04:42:11.362 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-09 04:42:11.364 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-09 04:42:11.364 DEBUG [src.node.waku_node] Stopping container with id 579e98b422c9
2026-03-09 04:42:11.930 DEBUG [src.node.waku_node] Container stopped.
2026-03-09 04:42:11.931 DEBUG [src.node.waku_node] Stopping container with id 7a6f0fa645e1
2026-03-09 04:42:12.489 DEBUG [src.node.waku_node] Container stopped.
2026-03-09 04:42:12.491 DEBUG [src.node.waku_node] Stopping container with id 4d36102d2983
2026-03-09 04:42:12.744 ERROR [src.node.docker_mananger] Max retries reached for container 579e98b422c9. Exiting log stream.
2026-03-09 04:42:13.303 ERROR [src.node.docker_mananger] Max retries reached for container 7a6f0fa645e1. Exiting log stream.
2026-03-09 04:42:23.031 DEBUG [src.node.waku_node] Container stopped.
2026-03-09 04:42:23.034 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-09 04:42:23.053 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-09 04:42:23.056 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-09 04:42:23.070 DEBUG [src.node.docker_mananger] No errors found in the waku logs.