170 lines
32 KiB
Plaintext

2026-01-02 04:46:52.143 DEBUG [tests.conftest] Running fixture setup: test_id
2026-01-02 04:46:52.143 DEBUG [tests.conftest] Running test: test_auto_shard_relay with id: 2026-01-02_04-46-52__25099166-dace-42d8-8702-0dec4cf2220b
2026-01-02 04:46:52.143 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-01-02 04:46:52.143 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-01-02 04:46:52.144 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-01-02 04:46:52.151 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-01-02 04:46:52.151 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-01-02_04-46-52__25099166-dace-42d8-8702-0dec4cf2220b__wakuorg_nwaku:latest.log
2026-01-02 04:46:52.151 DEBUG [src.node.waku_node] Starting Node...
2026-01-02 04:46:52.151 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-01-02 04:46:52.153 DEBUG [src.node.docker_mananger] Network waku already exists
2026-01-02 04:46:52.153 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.87.21
2026-01-02 04:46:52.153 DEBUG [src.node.docker_mananger] Generated ports ['51174', '51175', '51176', '51177', '51178']
2026-01-02 04:46:52.153 DEBUG [src.node.waku_node] RLN credentials were not set
2026-01-02 04:46:52.153 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-01-02 04:46:52.153 DEBUG [src.node.waku_node] Using volumes []
2026-01-02 04:46:52.153 DEBUG [src.node.docker_mananger] docker run -i -t -p 51174:51174 -p 51175:51175 -p 51176:51176 -p 51177:51177 -p 51178:51178 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=51176 --rest-port=51174 --tcp-port=51175 --discv5-udp-port=51177 --rest-address=0.0.0.0 --nat=extip:172.18.87.21 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=3dac56a1a6acddccec8049a5eaf6eea1caacdc6aaacb92ca8f2bbb05f8d5acac --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=51178 --metrics-logging=true --relay=true --filter=true --content-topic=/myapp/1/latest/proto
2026-01-02 04:46:52.331 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.87.21 waku 45b81e2a1b5d067152b154906bfb32130e61793c10e1444498efce18d54ed2da
2026-01-02 04:46:52.363 DEBUG [src.node.docker_mananger] Container started with ID 45b81e2a1b5d. Setting up logs at ./log/docker/node1_2026-01-02_04-46-52__25099166-dace-42d8-8702-0dec4cf2220b__wakuorg_nwaku:latest.log
2026-01-02 04:46:52.364 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 51174
2026-01-02 04:46:52.364 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-01-02 04:46:52.414 ERROR [src.node.docker_mananger] Max retries reached for container 09f746eafcbe. Exiting log stream.
2026-01-02 04:46:52.950 ERROR [src.node.docker_mananger] Max retries reached for container 39be39448a48. Exiting log stream.
2026-01-02 04:46:53.365 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51174/health" -H "Content-Type: application/json" -d 'None'
2026-01-02 04:46:53.368 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"}]}'
2026-01-02 04:46:53.368 INFO [src.node.waku_node] Node protocols are initialized !!
2026-01-02 04:46:53.368 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51174/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-01-02 04:46:53.371 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.87.21/tcp/51175/p2p/16Uiu2HAmSTBcKhE8smeUm9CqZF3RfSoZAeyZmzFq1WdUJMt6xVMz","/ip4/172.18.87.21/tcp/51176/ws/p2p/16Uiu2HAmSTBcKhE8smeUm9CqZF3RfSoZAeyZmzFq1WdUJMt6xVMz"],"enrUri":"enr:-L24QHdeBtSql5SoTeV0uL84t8PgRGkUu4uObI9Fm44l5BPZFzyNbQ5IQ6kfrrr_vqcdFAqgpevxzsayzW1o3J8-vuECgmlkgnY0gmlwhKwSVxWKbXVsdGlhZGRyc5YACASsElcVBsfnAAoErBJXFQbH6N0DgnJzhQACAQAAiXNlY3AyNTZrMaEDzP1gyUSKKiNPbVKaMyhzbwOFAGzbLTSXQp5gLS9cSfmDdGNwgsfng3VkcILH6YV3YWt1MgU"}'
2026-01-02 04:46:53.371 INFO [src.node.waku_node] REST service is ready !!
2026-01-02 04:46:53.377 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-01-02 04:46:53.377 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-01-02_04-46-52__25099166-dace-42d8-8702-0dec4cf2220b__wakuorg_nwaku:latest.log
2026-01-02 04:46:53.378 DEBUG [src.node.waku_node] Starting Node...
2026-01-02 04:46:53.378 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-01-02 04:46:53.379 DEBUG [src.node.docker_mananger] Network waku already exists
2026-01-02 04:46:53.379 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.122.225
2026-01-02 04:46:53.379 DEBUG [src.node.docker_mananger] Generated ports ['17993', '17994', '17995', '17996', '17997']
2026-01-02 04:46:53.380 DEBUG [src.node.waku_node] RLN credentials were not set
2026-01-02 04:46:53.380 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-01-02 04:46:53.380 DEBUG [src.node.waku_node] Using volumes []
2026-01-02 04:46:53.380 DEBUG [src.node.docker_mananger] docker run -i -t -p 17993:17993 -p 17994:17994 -p 17995:17995 -p 17996:17996 -p 17997:17997 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=17995 --rest-port=17993 --tcp-port=17994 --discv5-udp-port=17996 --rest-address=0.0.0.0 --nat=extip:172.18.122.225 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=9bac8da8d05deddd2ba8afb1f9babc4a750fded7ceb4849929737efdbde242de --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=17997 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QHdeBtSql5SoTeV0uL84t8PgRGkUu4uObI9Fm44l5BPZFzyNbQ5IQ6kfrrr_vqcdFAqgpevxzsayzW1o3J8-vuECgmlkgnY0gmlwhKwSVxWKbXVsdGlhZGRyc5YACASsElcVBsfnAAoErBJXFQbH6N0DgnJzhQACAQAAiXNlY3AyNTZrMaEDzP1gyUSKKiNPbVKaMyhzbwOFAGzbLTSXQp5gLS9cSfmDdGNwgsfng3VkcILH6YV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-01-02 04:46:53.568 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.122.225 waku 327b7093801f9ab57dfec7c866c887e6ef18f73406f36590c97770dd3ab669d9
2026-01-02 04:46:53.600 DEBUG [src.node.docker_mananger] Container started with ID 327b7093801f. Setting up logs at ./log/docker/node2_2026-01-02_04-46-52__25099166-dace-42d8-8702-0dec4cf2220b__wakuorg_nwaku:latest.log
2026-01-02 04:46:53.600 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 17993
2026-01-02 04:46:53.600 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-01-02 04:46:54.601 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17993/health" -H "Content-Type: application/json" -d 'None'
2026-01-02 04:46:54.614 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"READY"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"}]}'
2026-01-02 04:46:54.617 INFO [src.node.waku_node] Node protocols are initialized !!
2026-01-02 04:46:54.617 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17993/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-01-02 04:46:54.620 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.122.225/tcp/17994/p2p/16Uiu2HAm1zXN1pY3RmFxPchtAkBL8tPqzRwN7JaiRLkzMKURnSdA","/ip4/172.18.122.225/tcp/17995/ws/p2p/16Uiu2HAm1zXN1pY3RmFxPchtAkBL8tPqzRwN7JaiRLkzMKURnSdA"],"enrUri":"enr:-L24QFdUJbFlqR7UjPv1BAHYuGxhjwHlM4ykBDbUHHqtFEoSRBNDyVmXhydBG21880Fo6R8QL4_i7EeNRqufKGrB__gCgmlkgnY0gmlwhKwSeuGKbXVsdGlhZGRyc5YACASsEnrhBkZKAAoErBJ64QZGS90DgnJzhQACAQAAiXNlY3AyNTZrMaECYZCLjZrwBnl_y_H6gHRite4R6rPHnZ95ORjXJUn-4J2DdGNwgkZKg3VkcIJGTIV3YWt1MgE"}'
2026-01-02 04:46:54.620 INFO [src.node.waku_node] REST service is ready !!
2026-01-02 04:46:54.621 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:17993/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.87.21/tcp/51175/p2p/16Uiu2HAmSTBcKhE8smeUm9CqZF3RfSoZAeyZmzFq1WdUJMt6xVMz"]'
2026-01-02 04:46:54.623 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-01-02 04:46:54.630 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-01-02 04:46:54.630 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-01-02_04-46-52__25099166-dace-42d8-8702-0dec4cf2220b__wakuorg_nwaku:latest.log
2026-01-02 04:46:54.631 DEBUG [src.node.waku_node] Starting Node...
2026-01-02 04:46:54.631 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-01-02 04:46:54.632 DEBUG [src.node.docker_mananger] Network waku already exists
2026-01-02 04:46:54.632 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.158.246
2026-01-02 04:46:54.632 DEBUG [src.node.docker_mananger] Generated ports ['60908', '60909', '60910', '60911', '60912']
2026-01-02 04:46:54.632 DEBUG [src.node.waku_node] RLN credentials were not set
2026-01-02 04:46:54.633 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-01-02 04:46:54.633 DEBUG [src.node.waku_node] Using volumes []
2026-01-02 04:46:54.633 DEBUG [src.node.docker_mananger] docker run -i -t -p 60908:60908 -p 60909:60909 -p 60910:60910 -p 60911:60911 -p 60912:60912 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=60910 --rest-port=60908 --tcp-port=60909 --discv5-udp-port=60911 --rest-address=0.0.0.0 --nat=extip:172.18.158.246 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=cbe7af65b916057c3abaddf91f86a99f8f7e7a1fedb85e58a3a4d2dabba46e60 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=60912 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QHdeBtSql5SoTeV0uL84t8PgRGkUu4uObI9Fm44l5BPZFzyNbQ5IQ6kfrrr_vqcdFAqgpevxzsayzW1o3J8-vuECgmlkgnY0gmlwhKwSVxWKbXVsdGlhZGRyc5YACASsElcVBsfnAAoErBJXFQbH6N0DgnJzhQACAQAAiXNlY3AyNTZrMaEDzP1gyUSKKiNPbVKaMyhzbwOFAGzbLTSXQp5gLS9cSfmDdGNwgsfng3VkcILH6YV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-01-02 04:46:54.819 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.158.246 waku 223c2247e3eeef6d2c4811c6f92bad75299bbb35600be62f5e7bb4059104b7db
2026-01-02 04:46:54.847 DEBUG [src.node.docker_mananger] Container started with ID 223c2247e3ee. Setting up logs at ./log/docker/node3_2026-01-02_04-46-52__25099166-dace-42d8-8702-0dec4cf2220b__wakuorg_nwaku:latest.log
2026-01-02 04:46:54.847 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 60908
2026-01-02 04:46:54.849 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-01-02 04:46:55.850 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:60908/health" -H "Content-Type: application/json" -d 'None'
2026-01-02 04:46:55.860 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"}]}'
2026-01-02 04:46:55.860 INFO [src.node.waku_node] Node protocols are initialized !!
2026-01-02 04:46:55.861 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:60908/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-01-02 04:46:55.866 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.158.246/tcp/60909/p2p/16Uiu2HAmS7qBSeZhXo3xSWzXvrfB8ffcaFVjwhfvueesdJ8j9N2z","/ip4/172.18.158.246/tcp/60910/ws/p2p/16Uiu2HAmS7qBSeZhXo3xSWzXvrfB8ffcaFVjwhfvueesdJ8j9N2z"],"enrUri":"enr:-L24QHIeYec-NXwbzWM__XpqHM5oIo6jr1B6gvWznfdSQZdgbHWvm9fR-4CqQ7c8UdyrZ8buYhL7tuGtYzfa1ANa9jACgmlkgnY0gmlwhKwSnvaKbXVsdGlhZGRyc5YACASsEp72Bu3tAAoErBKe9gbt7t0DgnJzhQACAQAAiXNlY3AyNTZrMaEDyAg-XAc2wkAUFcCQIVtOFQMVSFzExqvAsgPvwhlbPeeDdGNwgu3tg3VkcILt74V3YWt1MgE"}'
2026-01-02 04:46:55.867 INFO [src.node.waku_node] REST service is ready !!
2026-01-02 04:46:55.867 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:60908/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.87.21/tcp/51175/p2p/16Uiu2HAmSTBcKhE8smeUm9CqZF3RfSoZAeyZmzFq1WdUJMt6xVMz"]'
2026-01-02 04:46:55.871 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-01-02 04:46:55.878 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-01-02 04:46:55.878 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-01-02_04-46-52__25099166-dace-42d8-8702-0dec4cf2220b__wakuorg_nwaku:latest.log
2026-01-02 04:46:55.878 DEBUG [src.node.waku_node] Starting Node...
2026-01-02 04:46:55.878 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-01-02 04:46:55.880 DEBUG [src.node.docker_mananger] Network waku already exists
2026-01-02 04:46:55.880 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.223.196
2026-01-02 04:46:55.880 DEBUG [src.node.docker_mananger] Generated ports ['23702', '23703', '23704', '23705', '23706']
2026-01-02 04:46:55.880 DEBUG [src.node.waku_node] RLN credentials were not set
2026-01-02 04:46:55.880 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-01-02 04:46:55.880 DEBUG [src.node.waku_node] Using volumes []
2026-01-02 04:46:55.880 DEBUG [src.node.docker_mananger] docker run -i -t -p 23702:23702 -p 23703:23703 -p 23704:23704 -p 23705:23705 -p 23706:23706 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=23704 --rest-port=23702 --tcp-port=23703 --discv5-udp-port=23705 --rest-address=0.0.0.0 --nat=extip:172.18.223.196 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=be3d7e9cbba42fcdf2d6914eb5a5cc4c511e1ec5fdb07c8c4992a5df96ff3e9d --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=23706 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QHdeBtSql5SoTeV0uL84t8PgRGkUu4uObI9Fm44l5BPZFzyNbQ5IQ6kfrrr_vqcdFAqgpevxzsayzW1o3J8-vuECgmlkgnY0gmlwhKwSVxWKbXVsdGlhZGRyc5YACASsElcVBsfnAAoErBJXFQbH6N0DgnJzhQACAQAAiXNlY3AyNTZrMaEDzP1gyUSKKiNPbVKaMyhzbwOFAGzbLTSXQp5gLS9cSfmDdGNwgsfng3VkcILH6YV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-01-02 04:46:56.066 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.223.196 waku 107f875f4b8d4439afa5caab2e2150e93da3b1614b1defba3eab537200998302
2026-01-02 04:46:56.102 DEBUG [src.node.docker_mananger] Container started with ID 107f875f4b8d. Setting up logs at ./log/docker/node4_2026-01-02_04-46-52__25099166-dace-42d8-8702-0dec4cf2220b__wakuorg_nwaku:latest.log
2026-01-02 04:46:56.102 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 23702
2026-01-02 04:46:56.102 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-01-02 04:46:57.103 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23702/health" -H "Content-Type: application/json" -d 'None'
2026-01-02 04:46:57.109 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"READY"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"}]}'
2026-01-02 04:46:57.110 INFO [src.node.waku_node] Node protocols are initialized !!
2026-01-02 04:46:57.110 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23702/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-01-02 04:46:57.114 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.223.196/tcp/23703/p2p/16Uiu2HAmS9YZRJtCLcNY1otqYs9LzxUWnNTT6dkMvNJo16xsCyLJ","/ip4/172.18.223.196/tcp/23704/ws/p2p/16Uiu2HAmS9YZRJtCLcNY1otqYs9LzxUWnNTT6dkMvNJo16xsCyLJ"],"enrUri":"enr:-L24QAFqcS2ehQoZR6M91Pig9oijpljqn81oXPasc2c4af16PgFLYJIMW7j8LLTbceFwUcYBqX3hXAXwk47Xg7-TO6ECgmlkgnY0gmlwhKwS38SKbXVsdGlhZGRyc5YACASsEt_EBlyXAAoErBLfxAZcmN0DgnJzhQACAQAAiXNlY3AyNTZrMaEDyHicl3RKLEuQ2LwO23hqYOq3YHX3rHPxaee5Jqz8GbeDdGNwglyXg3VkcIJcmYV3YWt1MgE"}'
2026-01-02 04:46:57.116 INFO [src.node.waku_node] REST service is ready !!
2026-01-02 04:46:57.116 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:23702/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.87.21/tcp/51175/p2p/16Uiu2HAmSTBcKhE8smeUm9CqZF3RfSoZAeyZmzFq1WdUJMt6xVMz"]'
2026-01-02 04:46:57.119 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-01-02 04:46:57.125 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-01-02 04:46:57.126 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-01-02_04-46-52__25099166-dace-42d8-8702-0dec4cf2220b__wakuorg_nwaku:latest.log
2026-01-02 04:46:57.126 DEBUG [src.node.waku_node] Starting Node...
2026-01-02 04:46:57.126 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-01-02 04:46:57.127 DEBUG [src.node.docker_mananger] Network waku already exists
2026-01-02 04:46:57.127 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.177.220
2026-01-02 04:46:57.127 DEBUG [src.node.docker_mananger] Generated ports ['37883', '37884', '37885', '37886', '37887']
2026-01-02 04:46:57.128 DEBUG [src.node.waku_node] RLN credentials were not set
2026-01-02 04:46:57.128 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-01-02 04:46:57.128 DEBUG [src.node.waku_node] Using volumes []
2026-01-02 04:46:57.128 DEBUG [src.node.docker_mananger] docker run -i -t -p 37883:37883 -p 37884:37884 -p 37885:37885 -p 37886:37886 -p 37887:37887 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=37885 --rest-port=37883 --tcp-port=37884 --discv5-udp-port=37886 --rest-address=0.0.0.0 --nat=extip:172.18.177.220 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=e368ecafdeae63408f2cc7b1dda0e885ca50e23b75af0decaa33bfb36fb628b2 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=37887 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QHdeBtSql5SoTeV0uL84t8PgRGkUu4uObI9Fm44l5BPZFzyNbQ5IQ6kfrrr_vqcdFAqgpevxzsayzW1o3J8-vuECgmlkgnY0gmlwhKwSVxWKbXVsdGlhZGRyc5YACASsElcVBsfnAAoErBJXFQbH6N0DgnJzhQACAQAAiXNlY3AyNTZrMaEDzP1gyUSKKiNPbVKaMyhzbwOFAGzbLTSXQp5gLS9cSfmDdGNwgsfng3VkcILH6YV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-01-02 04:46:57.316 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.177.220 waku e1f601db02246740bb78488f79b9c03f66b2981feec1925ba9bdbaece66eb493
2026-01-02 04:46:57.347 DEBUG [src.node.docker_mananger] Container started with ID e1f601db0224. Setting up logs at ./log/docker/node5_2026-01-02_04-46-52__25099166-dace-42d8-8702-0dec4cf2220b__wakuorg_nwaku:latest.log
2026-01-02 04:46:57.347 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 37883
2026-01-02 04:46:57.347 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-01-02 04:46:58.348 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:37883/health" -H "Content-Type: application/json" -d 'None'
2026-01-02 04:46:58.360 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"READY"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"}]}'
2026-01-02 04:46:58.360 INFO [src.node.waku_node] Node protocols are initialized !!
2026-01-02 04:46:58.361 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:37883/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-01-02 04:46:58.365 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.177.220/tcp/37884/p2p/16Uiu2HAmHwcrNcECW5bwzc83pYY3x3sBVKze5HbLjndXTz5aDDAT","/ip4/172.18.177.220/tcp/37885/ws/p2p/16Uiu2HAmHwcrNcECW5bwzc83pYY3x3sBVKze5HbLjndXTz5aDDAT"],"enrUri":"enr:-L24QMAkxtY3L-IxZDKno6-lj29xDr4WBg_kDz4TH-asecZEPb_lvWMKKABw7SkkEqaxjdsqChue7nxuSwZilPHsiDkCgmlkgnY0gmlwhKwSsdyKbXVsdGlhZGRyc5YACASsErHcBpP8AAoErBKx3AaT_d0DgnJzhQACAQAAiXNlY3AyNTZrMaEDToz_-IdJRMo7Zg7IMr58ki13uyVqoJ2rNVZ3B1BiLoSDdGNwgpP8g3VkcIKT_oV3YWt1MgE"}'
2026-01-02 04:46:58.365 INFO [src.node.waku_node] REST service is ready !!
2026-01-02 04:46:58.366 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:37883/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.87.21/tcp/51175/p2p/16Uiu2HAmSTBcKhE8smeUm9CqZF3RfSoZAeyZmzFq1WdUJMt6xVMz"]'
2026-01-02 04:46:58.368 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-01-02 04:46:58.369 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:51174/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-01-02 04:46:58.371 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-01-02 04:46:58.372 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:17993/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-01-02 04:46:58.374 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-01-02 04:46:58.375 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:60908/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-01-02 04:46:58.377 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-01-02 04:46:58.377 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:23702/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-01-02 04:46:58.380 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-01-02 04:46:58.380 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:37883/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-01-02 04:46:58.382 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-01-02 04:46:58.383 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:51174/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-01-02 04:46:58.390 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-01-02 04:46:58.393 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-01-02 04:46:58.494 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-01-02 04:46:58.494 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51174/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-01-02 04:46:58.496 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1767329218383051308,"ephemeral":false,"proof":""}]'
2026-01-02 04:46:58.498 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-01-02 04:46:58.498 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17993/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-01-02 04:46:58.500 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1767329218383051308,"ephemeral":false,"proof":""}]'
2026-01-02 04:46:58.502 DEBUG [src.steps.sharding] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-01-02 04:46:58.502 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:60908/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-01-02 04:46:58.504 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1767329218383051308,"ephemeral":false,"proof":""}]'
2026-01-02 04:46:58.505 DEBUG [src.steps.sharding] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-01-02 04:46:58.506 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23702/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-01-02 04:46:58.508 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1767329218383051308,"ephemeral":false,"proof":""}]'
2026-01-02 04:46:58.509 DEBUG [src.steps.sharding] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-01-02 04:46:58.509 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:37883/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-01-02 04:46:58.511 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1767329218383051308,"ephemeral":false,"proof":""}]'
2026-01-02 04:46:58.514 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-01-02 04:46:58.515 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-01-02 04:46:58.515 DEBUG [src.node.waku_node] Stopping container with id 45b81e2a1b5d
2026-01-02 04:46:59.081 DEBUG [src.node.waku_node] Container stopped.
2026-01-02 04:46:59.081 DEBUG [src.node.waku_node] Stopping container with id 327b7093801f
2026-01-02 04:46:59.623 DEBUG [src.node.waku_node] Container stopped.
2026-01-02 04:46:59.624 DEBUG [src.node.waku_node] Stopping container with id 223c2247e3ee
2026-01-02 04:46:59.895 ERROR [src.node.docker_mananger] Max retries reached for container 45b81e2a1b5d. Exiting log stream.
2026-01-02 04:47:00.145 DEBUG [src.node.waku_node] Container stopped.
2026-01-02 04:47:00.145 DEBUG [src.node.waku_node] Stopping container with id 107f875f4b8d
2026-01-02 04:47:00.435 ERROR [src.node.docker_mananger] Max retries reached for container 327b7093801f. Exiting log stream.
2026-01-02 04:47:00.663 DEBUG [src.node.waku_node] Container stopped.
2026-01-02 04:47:00.663 DEBUG [src.node.waku_node] Stopping container with id e1f601db0224
2026-01-02 04:47:00.958 ERROR [src.node.docker_mananger] Max retries reached for container 223c2247e3ee. Exiting log stream.
2026-01-02 04:47:01.185 DEBUG [src.node.waku_node] Container stopped.
2026-01-02 04:47:01.186 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-01-02 04:47:01.200 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-01-02 04:47:01.205 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-01-02 04:47:01.211 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-01-02 04:47:01.216 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-01-02 04:47:01.221 DEBUG [src.node.docker_mananger] No errors found in the waku logs.