209 lines
35 KiB
Plaintext

2026-03-07 04:35:29.844 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-07 04:35:29.845 DEBUG [tests.conftest] Running test: test_node_without_sync_or_relay_stays_empty with id: 2026-03-07_04-35-29__cb313c5e-1e80-4372-a468-1dabbe00b1d8
2026-03-07 04:35:29.846 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-07 04:35:29.854 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-07 04:35:29.855 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-07_04-35-29__cb313c5e-1e80-4372-a468-1dabbe00b1d8__wakuorg_nwaku:latest.log
2026-03-07 04:35:29.863 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-07 04:35:29.863 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-07_04-35-29__cb313c5e-1e80-4372-a468-1dabbe00b1d8__wakuorg_nwaku:latest.log
2026-03-07 04:35:29.873 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-07 04:35:29.873 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-07_04-35-29__cb313c5e-1e80-4372-a468-1dabbe00b1d8__wakuorg_nwaku:latest.log
2026-03-07 04:35:29.874 DEBUG [src.steps.store] Running fixture setup: store_setup
2026-03-07 04:35:29.877 DEBUG [src.node.waku_node] Starting Node...
2026-03-07 04:35:29.878 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-07 04:35:29.880 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-07 04:35:29.880 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.42.199
2026-03-07 04:35:29.881 DEBUG [src.node.docker_mananger] Generated ports ['6517', '6518', '6519', '6520', '6521']
2026-03-07 04:35:29.881 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-07 04:35:29.882 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-07 04:35:29.882 DEBUG [src.node.waku_node] Using volumes []
2026-03-07 04:35:29.882 DEBUG [src.node.docker_mananger] docker run -i -t -p 6517:6517 -p 6518:6518 -p 6519:6519 -p 6520:6520 -p 6521:6521 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=6519 --rest-port=6517 --tcp-port=6518 --discv5-udp-port=6520 --rest-address=0.0.0.0 --nat=extip:172.18.42.199 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=42dadb3f288c1aee69efc27d7fbebcfd84bce51b7e7765167def2d1dcbfec5a7 --store-sync=true --store=true --store-sync-range=300 --store-sync-interval=6 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=6521 --metrics-logging=true --store-sync-relay-jitter=0 --relay=true --dns-discovery=false
2026-03-07 04:35:30.043 ERROR [src.node.docker_mananger] Max retries reached for container 6634bfe6ff1e. Exiting log stream.
2026-03-07 04:35:30.085 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.42.199 waku 995b6a1b5c533438635a9e835bca228980020fe0d814f0340cf4eed731b841cc
2026-03-07 04:35:30.121 DEBUG [src.node.docker_mananger] Container started with ID 995b6a1b5c53. Setting up logs at ./log/docker/node1_2026-03-07_04-35-29__cb313c5e-1e80-4372-a468-1dabbe00b1d8__wakuorg_nwaku:latest.log
2026-03-07 04:35:30.122 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 6517
2026-03-07 04:35:30.122 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-07 04:35:30.698 ERROR [src.node.docker_mananger] Max retries reached for container f5f763e8d291. Exiting log stream.
2026-03-07 04:35:31.123 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6517/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:35:31.125 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-07 04:35:31.126 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-07 04:35:31.126 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6517/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:35:31.128 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.42.199/tcp/6518/p2p/16Uiu2HAmStb7LyGEcDsnDEBAfHkgqhUehXTHUWDKVQWhbrwy6gdb","/ip4/172.18.42.199/tcp/6519/ws/p2p/16Uiu2HAmStb7LyGEcDsnDEBAfHkgqhUehXTHUWDKVQWhbrwy6gdb"],"enrUri":"enr:-L24QO8MB_Ea8FE8rndPlrkJ0rX5W_x67g2W6ZI9U3ZG8F4Jfe1ZVoPZZis3W3eFSewYCAWLeDfu6uc4dTiMynkJCgACgmlkgnY0gmlwhKwSKseKbXVsdGlhZGRyc5YACASsEirHBhl2AAoErBIqxwYZd90DgnJzhQADAQAAiXNlY3AyNTZrMaED0394YxLLBLB0DYQoQ2X2cizY1uNDaWLcPYwyWVXmhA6DdGNwghl2g3VkcIIZeIV3YWt1MhM"}'
2026-03-07 04:35:31.128 INFO [src.node.waku_node] REST service is ready !!
2026-03-07 04:35:31.129 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:31.129 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:31.148 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:31.148 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:31.359 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:31.359 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:31.363 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:31.363 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:31.574 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:31.574 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:31.578 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:31.579 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:31.789 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:31.790 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:31.794 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:31.794 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:32.004 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:32.005 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:32.008 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:32.009 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:32.219 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:32.220 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:32.224 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:32.224 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:32.435 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:32.435 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:32.439 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:32.439 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:32.650 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:32.650 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:32.654 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:32.654 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:32.865 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:32.865 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:32.869 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:32.869 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:33.079 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:33.080 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:33.083 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:33.084 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:33.294 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:33.294 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:33.298 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:33.298 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:33.509 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:33.509 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:33.513 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:33.513 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:33.724 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:33.724 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:33.728 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:33.729 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:33.939 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:33.940 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:33.944 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:33.944 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:34.155 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:34.155 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:34.159 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:34.159 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:34.370 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:34.370 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:34.374 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:34.374 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:34.585 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:34.585 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:34.589 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:34.589 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:34.800 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:34.800 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:34.804 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:34.804 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:35.015 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:35.015 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:35.019 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:35.019 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:35.229 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:35.230 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:35.233 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:35.234 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:35.444 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:35.444 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:35.448 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:35.448 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:35.659 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:35.659 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:35.663 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:35.663 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:35.874 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:35.874 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:35.878 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:35.878 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:36.089 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:36.089 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:36.093 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:36.093 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:36.304 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:36.304 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:36.310 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:36.310 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:36.520 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:36.521 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:36.524 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:36.525 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:36.735 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:36.736 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:36.739 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:36.740 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:36.950 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:36.950 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:36.954 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:36.955 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:37.165 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:37.166 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:37.170 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:37.170 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:37.381 DEBUG [src.steps.store] Relaying message
2026-03-07 04:35:37.381 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6517/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:35:37.384 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:37.385 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-07 04:35:37.595 DEBUG [src.node.waku_node] Starting Node...
2026-03-07 04:35:37.595 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-07 04:35:37.597 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-07 04:35:37.597 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.131.84
2026-03-07 04:35:37.597 DEBUG [src.node.docker_mananger] Generated ports ['54786', '54787', '54788', '54789', '54790']
2026-03-07 04:35:37.597 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-07 04:35:37.598 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-07 04:35:37.598 DEBUG [src.node.waku_node] Using volumes []
2026-03-07 04:35:37.598 DEBUG [src.node.docker_mananger] docker run -i -t -p 54786:54786 -p 54787:54787 -p 54788:54788 -p 54789:54789 -p 54790:54790 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=54788 --rest-port=54786 --tcp-port=54787 --discv5-udp-port=54789 --rest-address=0.0.0.0 --nat=extip:172.18.131.84 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=3e2c35d75dde70d35f8f71b581a3fe0fde5ba266ffe7dac7b4ace26ffaeee3ce --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=54790 --metrics-logging=true --relay=false --dns-discovery=false --discv5-bootstrap-node=enr:-L24QO8MB_Ea8FE8rndPlrkJ0rX5W_x67g2W6ZI9U3ZG8F4Jfe1ZVoPZZis3W3eFSewYCAWLeDfu6uc4dTiMynkJCgACgmlkgnY0gmlwhKwSKseKbXVsdGlhZGRyc5YACASsEirHBhl2AAoErBIqxwYZd90DgnJzhQADAQAAiXNlY3AyNTZrMaED0394YxLLBLB0DYQoQ2X2cizY1uNDaWLcPYwyWVXmhA6DdGNwghl2g3VkcIIZeIV3YWt1MhM
2026-03-07 04:35:37.787 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.131.84 waku 04253c41d3e04b89111f18f1221710f5f7424c6a61cd80cfde3a86ceb6e3c74d
2026-03-07 04:35:37.822 DEBUG [src.node.docker_mananger] Container started with ID 04253c41d3e0. Setting up logs at ./log/docker/node2_2026-03-07_04-35-29__cb313c5e-1e80-4372-a468-1dabbe00b1d8__wakuorg_nwaku:latest.log
2026-03-07 04:35:37.822 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 54786
2026-03-07 04:35:37.822 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-07 04:35:38.823 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54786/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:35:38.826 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-07 04:35:38.826 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-07 04:35:38.826 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54786/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:35:38.829 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.131.84/tcp/54787/p2p/16Uiu2HAm59jP39PQJ1pYZLwSiBPJuMqVPH3juo9gY1NsXNmcqk2M","/ip4/172.18.131.84/tcp/54788/ws/p2p/16Uiu2HAm59jP39PQJ1pYZLwSiBPJuMqVPH3juo9gY1NsXNmcqk2M"],"enrUri":"enr:-L24QFPgUlJqp5g703XIvEaYNwc3Bk1-Va8_c_qK5lSPjFCKHeBGZsw0ogVDiyhWUyTBIVL7eO69ccvP8MOcw9L0xDwCgmlkgnY0gmlwhKwSg1SKbXVsdGlhZGRyc5YACASsEoNUBtYDAAoErBKDVAbWBN0DgnJzhQADAQAAiXNlY3AyNTZrMaECkH9r7EFOWeNdnwDl23IZPl6yo6dyMDyqy8LyDtiqV8qDdGNwgtYDg3VkcILWBYV3YWt1MgA"}'
2026-03-07 04:35:38.829 INFO [src.node.waku_node] REST service is ready !!
2026-03-07 04:35:38.829 DEBUG [src.node.waku_node] Starting Node...
2026-03-07 04:35:38.829 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-07 04:35:38.831 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-07 04:35:38.831 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.108.99
2026-03-07 04:35:38.831 DEBUG [src.node.docker_mananger] Generated ports ['16571', '16572', '16573', '16574', '16575']
2026-03-07 04:35:38.831 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-07 04:35:38.831 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-07 04:35:38.831 DEBUG [src.node.waku_node] Using volumes []
2026-03-07 04:35:38.831 DEBUG [src.node.docker_mananger] docker run -i -t -p 16571:16571 -p 16572:16572 -p 16573:16573 -p 16574:16574 -p 16575:16575 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=16573 --rest-port=16571 --tcp-port=16572 --discv5-udp-port=16574 --rest-address=0.0.0.0 --nat=extip:172.18.108.99 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=09cb47fb0e5a9afbcfebace7d8a3a5ceb81cd5f420f674497aecdb7ab8cdf3b6 --store-sync=true --store=true --store-sync-range=300 --store-sync-interval=6 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=16575 --metrics-logging=true --store-sync-relay-jitter=0 --relay=false --dns-discovery=false --discv5-bootstrap-node=enr:-L24QO8MB_Ea8FE8rndPlrkJ0rX5W_x67g2W6ZI9U3ZG8F4Jfe1ZVoPZZis3W3eFSewYCAWLeDfu6uc4dTiMynkJCgACgmlkgnY0gmlwhKwSKseKbXVsdGlhZGRyc5YACASsEirHBhl2AAoErBIqxwYZd90DgnJzhQADAQAAiXNlY3AyNTZrMaED0394YxLLBLB0DYQoQ2X2cizY1uNDaWLcPYwyWVXmhA6DdGNwghl2g3VkcIIZeIV3YWt1MhM
2026-03-07 04:35:39.030 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.108.99 waku acf6df5957fa7b94afd057866c5dea459ca3d9a0a70ad6e831eaea401c58ad01
2026-03-07 04:35:39.073 DEBUG [src.node.docker_mananger] Container started with ID acf6df5957fa. Setting up logs at ./log/docker/node3_2026-03-07_04-35-29__cb313c5e-1e80-4372-a468-1dabbe00b1d8__wakuorg_nwaku:latest.log
2026-03-07 04:35:39.073 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 16571
2026-03-07 04:35:39.074 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-07 04:35:40.075 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16571/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:35:40.078 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-07 04:35:40.078 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-07 04:35:40.078 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16571/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:35:40.081 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.108.99/tcp/16572/p2p/16Uiu2HAmHK4u8NcRNgYSLBfctvBFzNbmfyaRYnraECxM1JF3VpxX","/ip4/172.18.108.99/tcp/16573/ws/p2p/16Uiu2HAmHK4u8NcRNgYSLBfctvBFzNbmfyaRYnraECxM1JF3VpxX"],"enrUri":"enr:-L24QC0GzlfQEz1zYQxP9bDFqapFz_3Q9LleFM6qbx_LMXLMHNnCd6FHUlwjVRFEDgHqykDc9DZm6j_qeA5f78KWNVYCgmlkgnY0gmlwhKwSbGOKbXVsdGlhZGRyc5YACASsEmxjBkC8AAoErBJsYwZAvd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDRS_2QfbY7E7X4_EYobg0T7mZWxBqtgTIfALdjYD587CDdGNwgkC8g3VkcIJAvoV3YWt1MhI"}'
2026-03-07 04:35:40.081 INFO [src.node.waku_node] REST service is ready !!
2026-03-07 04:35:40.081 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:16571/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.42.199/tcp/6518/p2p/16Uiu2HAmStb7LyGEcDsnDEBAfHkgqhUehXTHUWDKVQWhbrwy6gdb"]'
2026-03-07 04:35:40.117 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:35:58.119 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16571/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F0&cursor=&pageSize=200&ascending=true&peer_id=" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:35:58.123 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0x5c5ad04e89c09bd3b6dc9d185e8bf199267f64baf3cf9e9aedb97e1d56ea390a"},{"messageHash":"0xde71485134870828166db45851c93847aaecca6eeb6b241bba33ff7be3e88759"},{"messageHash":"0xeeccf3694fcbb2f7d79fef03c2832b61a89a53d2ddec00d3507b6062f0fe2f2c"},{"messageHash":"0xd231ab890b6adb88139c2742f2a25b6ae0bb8704ef6a351614751508b7c70d15"},{"messageHash":"0xc247c3d5dfad7c5cf90f9243163f7045f774b2417a767716be9e84d93902c6f6"},{"messageHash":"0xa31768041a6b2800793c314c6fd4510688cc99edd816c629c94a08f4fff60718"},{"messageHash":"0xf8391133c350f68812e3c6ee4689592fdc0d7bfaa9c0b3a738e5c90e042c7be1"},{"messageHash":"0x7d65c26e7212ec1428cf0e018e7165c5d8049c653cf7fb2d7802216bdde5f1c2"},{"messageHash":"0xa36ccf7598a9779bd10f442b9f39c6624f630d387c1a0cf4f45518ad73e13da7"},{"messageHash":"0x75b1519343fa20bbe41a752ca28fb2294d3a14e28af64f8ea0ab0b582b753b9a"},{"messageHash":"0x87e0b6aae9329c429a71483d7fd1b58c386d3fb05b6336ceb619e42f75f1a457"},{"messageHash":"0xc7dcb67325ed679cd330b045bde1a574d52c4322609020d5da51b1f358dad58e"},{"messageHash":"0x71a61365c69ea69c8fd94a8a236e6692ca086c125a7e8d6a2a587ba19b4e8ed3"},{"messageHash":"0x874fe709f11484861603d895b2e2b8308ff020e14659b394548a9f0e9cd5bed1"},{"messageHash":"0x4431b2b86edb4890a2a5c260da8f0f9494d5cabd175ce820880af6740e5f9c08"},{"messageHash":"0x3ca735b0e4fec02c40d5be25689d11cd01a12d1283ef28c6707370e0b34e7a4f"},{"messageHash":"0xfb0d7f376e6909be788550f9d89ee3feee2c72db89c496737a9edc3347bc7c79"},{"messageHash":"0x72f4fdda76278c286a6d9959ef7f0b277763a59902ddeaecc67f04a97cbb1288"},{"messageHash":"0x8826a9ffdb7e69a454946bf2d8a67087734c4e9df0fb8efb4feb9535007602cf"},{"messageHash":"0xad916959e49b66c6dc1b03d7413a3eb8c3782d6cf3721f0ee40f1eab842bdd14"},{"messageHash":"0x0ed6c73f969c92c3e74d511d6242660811dafa81c4e8fadeec09934897ca1fe5"},{"messageHash":"0xe607a7dbf52f5c7632938b78a2f1d066186b32910954487a0df0da39467ae26f"},{"messageHash":"0x2a25b624de7e6b6196b653a6091ff079792df291fd63a3da090194c1aa68e99e"},{"messageHash":"0xa2dfcb0bf554ec85041f2249f6ae3d4af285996d914278657abb4b55155b915b"},{"messageHash":"0xdd0df04d012c10c4102eb57abcf2ccb346c8e34d93b521fcdd1f207c0dab1b70"},{"messageHash":"0x95c53fcd8992e35b9e42c8e0491561857c2b393fc9051f73311dc70db581ae70"},{"messageHash":"0xe60e7f691eda0d57d12fd54a768e8a4fb6a6e118f952e4c84e892144f0f2696f"},{"messageHash":"0x6a4358dd60825193a196dfa6e98ef01ad007d925c8416e62d431a2d60502755d"},{"messageHash":"0xc3203932994faa7e40ce026e7b45ead974ff059473a9d1368a58594d1a2f8929"},{"messageHash":"0x44afc9696218c7eb3ec0a9e3688e4ea1cf72356bd7255c447452631f9fa97b1a"}]}'
2026-03-07 04:35:58.123 DEBUG [tests.store_sync.test_store_sync] Node3 store has 30 messages expected 30
2026-03-07 04:35:58.125 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-07 04:35:58.127 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-07 04:35:58.127 DEBUG [src.node.waku_node] Stopping container with id 995b6a1b5c53
2026-03-07 04:35:58.709 DEBUG [src.node.waku_node] Container stopped.
2026-03-07 04:35:58.709 DEBUG [src.node.waku_node] Stopping container with id 04253c41d3e0
2026-03-07 04:35:59.278 DEBUG [src.node.waku_node] Container stopped.
2026-03-07 04:35:59.279 DEBUG [src.node.waku_node] Stopping container with id acf6df5957fa
2026-03-07 04:35:59.523 ERROR [src.node.docker_mananger] Max retries reached for container 995b6a1b5c53. Exiting log stream.
2026-03-07 04:36:00.090 ERROR [src.node.docker_mananger] Max retries reached for container 04253c41d3e0. Exiting log stream.
2026-03-07 04:36:09.839 DEBUG [src.node.waku_node] Container stopped.
2026-03-07 04:36:09.840 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-07 04:36:09.857 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-07 04:36:09.859 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-07 04:36:09.872 DEBUG [src.node.docker_mananger] No errors found in the waku logs.