176 lines
32 KiB
Plaintext

2026-03-30 05:04:02.251 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-30 05:04:02.251 DEBUG [tests.conftest] Running test: test_auto_shard_relay with id: 2026-03-30_05-04-02__11b41263-4a63-4aea-9c8e-9a67734b2fff
2026-03-30 05:04:02.252 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-30 05:04:02.252 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-30 05:04:02.252 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-30 05:04:02.259 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-30 05:04:02.260 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-30_05-04-02__11b41263-4a63-4aea-9c8e-9a67734b2fff__wakuorg_nwaku:latest.log
2026-03-30 05:04:02.260 DEBUG [src.node.waku_node] Starting Node...
2026-03-30 05:04:02.260 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-30 05:04:02.262 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-30 05:04:02.262 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.249.250
2026-03-30 05:04:02.262 DEBUG [src.node.docker_mananger] Generated ports ['36982', '36983', '36984', '36985', '36986']
2026-03-30 05:04:02.262 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-30 05:04:02.262 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-30 05:04:02.262 DEBUG [src.node.waku_node] Using volumes []
2026-03-30 05:04:02.262 DEBUG [src.node.docker_mananger] docker run -i -t -p 36982:36982 -p 36983:36983 -p 36984:36984 -p 36985:36985 -p 36986:36986 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=36984 --rest-port=36982 --tcp-port=36983 --discv5-udp-port=36985 --rest-address=0.0.0.0 --nat=extip:172.18.249.250 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=cfe95bfda8c712bb0413d5a307c51408cc6c64acbccd6ad9aab2e9ac18c58aec --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=36986 --metrics-logging=true --relay=true --filter=true --content-topic=/myapp/1/latest/proto
2026-03-30 05:04:02.448 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.249.250 waku c5e63f2baa74d56d25bf46769874e924a909098ca528beb540e80ae638b62a62
2026-03-30 05:04:02.485 DEBUG [src.node.docker_mananger] Container started with ID c5e63f2baa74. Setting up logs at ./log/docker/node1_2026-03-30_05-04-02__11b41263-4a63-4aea-9c8e-9a67734b2fff__wakuorg_nwaku:latest.log
2026-03-30 05:04:02.487 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 36982
2026-03-30 05:04:02.487 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-30 05:04:02.603 ERROR [src.node.docker_mananger] Max retries reached for container 1236f991d6f2. Exiting log stream.
2026-03-30 05:04:03.049 ERROR [src.node.docker_mananger] Max retries reached for container 3955b7d22c30. Exiting log stream.
2026-03-30 05:04:03.487 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:36982/health" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:03.490 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-30 05:04:03.591 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:36982/health" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:03.594 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-30 05:04:03.594 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-30 05:04:03.594 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:36982/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:03.597 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.249.250/tcp/36983/p2p/16Uiu2HAm7v6syYw9XYMoukZAHK5jhqYR9Jy6zCZhdFuXXgRgbQrA","/ip4/172.18.249.250/tcp/36984/ws/p2p/16Uiu2HAm7v6syYw9XYMoukZAHK5jhqYR9Jy6zCZhdFuXXgRgbQrA"],"enrUri":"enr:-L24QK5QzOqD-UFWJ9e0XCOXlsfLXHlODJfaSjFt1vxkcfVUdc0pHdpRQ2-k7qfckkTq5y1NsRpqQvkiYO7RPNiUpgkCgmlkgnY0gmlwhKwS-fqKbXVsdGlhZGRyc5YACASsEvn6BpB3AAoErBL5-gaQeN0DgnJzhQACAQAAiXNlY3AyNTZrMaECuZSo8itYNEbkmas7-jtD9T1LDj_0L8hgdlqCS2wQo9-DdGNwgpB3g3VkcIKQeYV3YWt1MgU"}'
2026-03-30 05:04:03.597 INFO [src.node.waku_node] REST service is ready !!
2026-03-30 05:04:03.604 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-30 05:04:03.604 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-30_05-04-02__11b41263-4a63-4aea-9c8e-9a67734b2fff__wakuorg_nwaku:latest.log
2026-03-30 05:04:03.604 DEBUG [src.node.waku_node] Starting Node...
2026-03-30 05:04:03.604 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-30 05:04:03.606 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-30 05:04:03.606 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.53.229
2026-03-30 05:04:03.606 DEBUG [src.node.docker_mananger] Generated ports ['32949', '32950', '32951', '32952', '32953']
2026-03-30 05:04:03.606 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-30 05:04:03.606 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-30 05:04:03.606 DEBUG [src.node.waku_node] Using volumes []
2026-03-30 05:04:03.607 DEBUG [src.node.docker_mananger] docker run -i -t -p 32949:32949 -p 32950:32950 -p 32951:32951 -p 32952:32952 -p 32953:32953 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=32951 --rest-port=32949 --tcp-port=32950 --discv5-udp-port=32952 --rest-address=0.0.0.0 --nat=extip:172.18.53.229 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=b12ddabeb0abc00f14ffa2aecd14294f3affbdbad6ae130b68dc3c2866fbc5d6 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=32953 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QK5QzOqD-UFWJ9e0XCOXlsfLXHlODJfaSjFt1vxkcfVUdc0pHdpRQ2-k7qfckkTq5y1NsRpqQvkiYO7RPNiUpgkCgmlkgnY0gmlwhKwS-fqKbXVsdGlhZGRyc5YACASsEvn6BpB3AAoErBL5-gaQeN0DgnJzhQACAQAAiXNlY3AyNTZrMaECuZSo8itYNEbkmas7-jtD9T1LDj_0L8hgdlqCS2wQo9-DdGNwgpB3g3VkcIKQeYV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-03-30 05:04:03.808 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.53.229 waku 6013d48b6674eb4806e3825c3255bbbc172a20c7a1b57088ec7d983b0e0971ee
2026-03-30 05:04:03.842 DEBUG [src.node.docker_mananger] Container started with ID 6013d48b6674. Setting up logs at ./log/docker/node2_2026-03-30_05-04-02__11b41263-4a63-4aea-9c8e-9a67734b2fff__wakuorg_nwaku:latest.log
2026-03-30 05:04:03.843 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 32949
2026-03-30 05:04:03.844 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-30 05:04:04.844 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:32949/health" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:04.847 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-30 05:04:04.847 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-30 05:04:04.848 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:32949/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:04.850 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.53.229/tcp/32950/p2p/16Uiu2HAmT35D4cdzSpszQo7u22T6sQp8TZVHFivrN7hxiNjqsGC9","/ip4/172.18.53.229/tcp/32951/ws/p2p/16Uiu2HAmT35D4cdzSpszQo7u22T6sQp8TZVHFivrN7hxiNjqsGC9"],"enrUri":"enr:-L24QAUNUEoBE-o81UJJ3UWdx0Bnw91mMhxvEJzEdbAHo6WhB96aCNnaFia9MI2xgyyDK5yBreBy-kf5tKSOGhIhsnwCgmlkgnY0gmlwhKwSNeWKbXVsdGlhZGRyc5YACASsEjXlBoC2AAoErBI15QaAt90DgnJzhQACAQAAiXNlY3AyNTZrMaED1avjTeKpBworY2iQwzQev4hu92Bl5twLJfbKgRHUi3KDdGNwgoC2g3VkcIKAuIV3YWt1MgE"}'
2026-03-30 05:04:04.850 INFO [src.node.waku_node] REST service is ready !!
2026-03-30 05:04:04.851 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:32949/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.249.250/tcp/36983/p2p/16Uiu2HAm7v6syYw9XYMoukZAHK5jhqYR9Jy6zCZhdFuXXgRgbQrA"]'
2026-03-30 05:04:04.878 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-30 05:04:04.891 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-30 05:04:04.891 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-30_05-04-02__11b41263-4a63-4aea-9c8e-9a67734b2fff__wakuorg_nwaku:latest.log
2026-03-30 05:04:04.891 DEBUG [src.node.waku_node] Starting Node...
2026-03-30 05:04:04.891 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-30 05:04:04.892 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-30 05:04:04.892 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.5.93
2026-03-30 05:04:04.893 DEBUG [src.node.docker_mananger] Generated ports ['63755', '63756', '63757', '63758', '63759']
2026-03-30 05:04:04.893 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-30 05:04:04.893 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-30 05:04:04.893 DEBUG [src.node.waku_node] Using volumes []
2026-03-30 05:04:04.893 DEBUG [src.node.docker_mananger] docker run -i -t -p 63755:63755 -p 63756:63756 -p 63757:63757 -p 63758:63758 -p 63759:63759 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=63757 --rest-port=63755 --tcp-port=63756 --discv5-udp-port=63758 --rest-address=0.0.0.0 --nat=extip:172.18.5.93 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=9c8fd1def40c2cb70fb7d87f33dbd3f83aee36a4dbab10c8f801873fb6b5bfba --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=63759 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QK5QzOqD-UFWJ9e0XCOXlsfLXHlODJfaSjFt1vxkcfVUdc0pHdpRQ2-k7qfckkTq5y1NsRpqQvkiYO7RPNiUpgkCgmlkgnY0gmlwhKwS-fqKbXVsdGlhZGRyc5YACASsEvn6BpB3AAoErBL5-gaQeN0DgnJzhQACAQAAiXNlY3AyNTZrMaECuZSo8itYNEbkmas7-jtD9T1LDj_0L8hgdlqCS2wQo9-DdGNwgpB3g3VkcIKQeYV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-03-30 05:04:05.100 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.5.93 waku f6a5979f57ce3d501c97ef10293ca77c85a19208ee8a55e0a2c0fea161125d21
2026-03-30 05:04:05.133 DEBUG [src.node.docker_mananger] Container started with ID f6a5979f57ce. Setting up logs at ./log/docker/node3_2026-03-30_05-04-02__11b41263-4a63-4aea-9c8e-9a67734b2fff__wakuorg_nwaku:latest.log
2026-03-30 05:04:05.134 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 63755
2026-03-30 05:04:05.135 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-30 05:04:06.135 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:63755/health" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:06.139 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-30 05:04:06.240 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:63755/health" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:06.242 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-30 05:04:06.243 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-30 05:04:06.243 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:63755/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:06.246 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.5.93/tcp/63756/p2p/16Uiu2HAmQS38ooR3SCpuwHjYnPGqM7kg1fSkEKpeQwoDqgcAt4FZ","/ip4/172.18.5.93/tcp/63757/ws/p2p/16Uiu2HAmQS38ooR3SCpuwHjYnPGqM7kg1fSkEKpeQwoDqgcAt4FZ"],"enrUri":"enr:-L24QKFXuXrWLxpKG-sLPbHSxxG1vqHTmlBilWy9DO_4HUFfHTqzyLgLNka2nQ97UgUfM75yWrF_K-JS1K-yK8q3Y-UCgmlkgnY0gmlwhKwSBV2KbXVsdGlhZGRyc5YACASsEgVdBvkMAAoErBIFXQb5Dd0DgnJzhQACAQAAiXNlY3AyNTZrMaEDrvrXZBLyXPXnItMIHMqACl-BYKt1T0nt4Y8iJ2YUjuCDdGNwgvkMg3VkcIL5DoV3YWt1MgE"}'
2026-03-30 05:04:06.246 INFO [src.node.waku_node] REST service is ready !!
2026-03-30 05:04:06.246 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:63755/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.249.250/tcp/36983/p2p/16Uiu2HAm7v6syYw9XYMoukZAHK5jhqYR9Jy6zCZhdFuXXgRgbQrA"]'
2026-03-30 05:04:06.275 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-30 05:04:06.285 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-30 05:04:06.285 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-30_05-04-02__11b41263-4a63-4aea-9c8e-9a67734b2fff__wakuorg_nwaku:latest.log
2026-03-30 05:04:06.285 DEBUG [src.node.waku_node] Starting Node...
2026-03-30 05:04:06.285 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-30 05:04:06.286 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-30 05:04:06.287 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.233.131
2026-03-30 05:04:06.287 DEBUG [src.node.docker_mananger] Generated ports ['30899', '30900', '30901', '30902', '30903']
2026-03-30 05:04:06.287 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-30 05:04:06.287 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-30 05:04:06.287 DEBUG [src.node.waku_node] Using volumes []
2026-03-30 05:04:06.287 DEBUG [src.node.docker_mananger] docker run -i -t -p 30899:30899 -p 30900:30900 -p 30901:30901 -p 30902:30902 -p 30903:30903 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=30901 --rest-port=30899 --tcp-port=30900 --discv5-udp-port=30902 --rest-address=0.0.0.0 --nat=extip:172.18.233.131 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=dd54e5a4ce5d0d0dced63a345e7bea490facefbfa0a7de6a0c3f0a60affa7d57 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=30903 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QK5QzOqD-UFWJ9e0XCOXlsfLXHlODJfaSjFt1vxkcfVUdc0pHdpRQ2-k7qfckkTq5y1NsRpqQvkiYO7RPNiUpgkCgmlkgnY0gmlwhKwS-fqKbXVsdGlhZGRyc5YACASsEvn6BpB3AAoErBL5-gaQeN0DgnJzhQACAQAAiXNlY3AyNTZrMaECuZSo8itYNEbkmas7-jtD9T1LDj_0L8hgdlqCS2wQo9-DdGNwgpB3g3VkcIKQeYV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-03-30 05:04:06.493 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.233.131 waku 29d9bb770964cbf428ff192348b7d2bcef9ef0ec200441b4e2eba3ae2b11fc69
2026-03-30 05:04:06.530 DEBUG [src.node.docker_mananger] Container started with ID 29d9bb770964. Setting up logs at ./log/docker/node4_2026-03-30_05-04-02__11b41263-4a63-4aea-9c8e-9a67734b2fff__wakuorg_nwaku:latest.log
2026-03-30 05:04:06.531 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 30899
2026-03-30 05:04:06.531 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-30 05:04:07.532 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30899/health" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:07.535 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-30 05:04:07.536 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-30 05:04:07.536 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30899/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:07.538 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.233.131/tcp/30900/p2p/16Uiu2HAkwNi7mBTQxseByNsTqn1eVgX8GrWMPpdJUHZGX8dazdyK","/ip4/172.18.233.131/tcp/30901/ws/p2p/16Uiu2HAkwNi7mBTQxseByNsTqn1eVgX8GrWMPpdJUHZGX8dazdyK"],"enrUri":"enr:-L24QET0qlDaXbbNo6lmAcDkEU5xQAnCZtFyGkIIjq4H17OPS9_N4px2yon2t5mMnFMUhAbWsNyZ9UeSOMDQG8k9i3ACgmlkgnY0gmlwhKwS6YOKbXVsdGlhZGRyc5YACASsEumDBni0AAoErBLpgwZ4td0DgnJzhQACAQAAiXNlY3AyNTZrMaECHPUMVgyz4qmLRZdNRhO6dJtgF30Ydt7hsIGXm3Bv8AqDdGNwgni0g3VkcIJ4toV3YWt1MgE"}'
2026-03-30 05:04:07.538 INFO [src.node.waku_node] REST service is ready !!
2026-03-30 05:04:07.539 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:30899/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.249.250/tcp/36983/p2p/16Uiu2HAm7v6syYw9XYMoukZAHK5jhqYR9Jy6zCZhdFuXXgRgbQrA"]'
2026-03-30 05:04:07.577 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-30 05:04:07.587 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-30 05:04:07.588 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-30_05-04-02__11b41263-4a63-4aea-9c8e-9a67734b2fff__wakuorg_nwaku:latest.log
2026-03-30 05:04:07.588 DEBUG [src.node.waku_node] Starting Node...
2026-03-30 05:04:07.588 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-30 05:04:07.589 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-30 05:04:07.589 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.164.85
2026-03-30 05:04:07.590 DEBUG [src.node.docker_mananger] Generated ports ['52618', '52619', '52620', '52621', '52622']
2026-03-30 05:04:07.590 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-30 05:04:07.590 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-30 05:04:07.590 DEBUG [src.node.waku_node] Using volumes []
2026-03-30 05:04:07.590 DEBUG [src.node.docker_mananger] docker run -i -t -p 52618:52618 -p 52619:52619 -p 52620:52620 -p 52621:52621 -p 52622:52622 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=52620 --rest-port=52618 --tcp-port=52619 --discv5-udp-port=52621 --rest-address=0.0.0.0 --nat=extip:172.18.164.85 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=a26bd60eeb7b44cbafda20f8c4fccf613183ee1b0fddc5bbf7e86c3dfb523708 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=52622 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QK5QzOqD-UFWJ9e0XCOXlsfLXHlODJfaSjFt1vxkcfVUdc0pHdpRQ2-k7qfckkTq5y1NsRpqQvkiYO7RPNiUpgkCgmlkgnY0gmlwhKwS-fqKbXVsdGlhZGRyc5YACASsEvn6BpB3AAoErBL5-gaQeN0DgnJzhQACAQAAiXNlY3AyNTZrMaECuZSo8itYNEbkmas7-jtD9T1LDj_0L8hgdlqCS2wQo9-DdGNwgpB3g3VkcIKQeYV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-03-30 05:04:07.800 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.164.85 waku e5d8805ba274affd23385630f77e9e8e1a71054ebfad310750e121183d9c4bf3
2026-03-30 05:04:07.841 DEBUG [src.node.docker_mananger] Container started with ID e5d8805ba274. Setting up logs at ./log/docker/node5_2026-03-30_05-04-02__11b41263-4a63-4aea-9c8e-9a67734b2fff__wakuorg_nwaku:latest.log
2026-03-30 05:04:07.842 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 52618
2026-03-30 05:04:07.842 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-30 05:04:08.842 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:52618/health" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:08.858 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-30 05:04:08.959 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:52618/health" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:08.967 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-30 05:04:08.967 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-30 05:04:08.967 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:52618/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:08.973 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.164.85/tcp/52619/p2p/16Uiu2HAm5RmqATkfxJecEtkS9nybRpPPSvwbQL6eZVXj6jzAbf6o","/ip4/172.18.164.85/tcp/52620/ws/p2p/16Uiu2HAm5RmqATkfxJecEtkS9nybRpPPSvwbQL6eZVXj6jzAbf6o"],"enrUri":"enr:-L24QNxyPv2GenioCFb6OlIE_DobZZfcCAMBZQxrLDcEDvZjGyZASkOF1yXo_K1nAgJoN3ma2SOlkLnldkLc1nXv91cCgmlkgnY0gmlwhKwSpFWKbXVsdGlhZGRyc5YACASsEqRVBs2LAAoErBKkVQbNjN0DgnJzhQACAQAAiXNlY3AyNTZrMaEClJt8OYKApPzZpAaEGp8O2sqNCkLVB4L41XbCYFwVJSiDdGNwgs2Lg3VkcILNjYV3YWt1MgE"}'
2026-03-30 05:04:08.973 INFO [src.node.waku_node] REST service is ready !!
2026-03-30 05:04:08.974 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:52618/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.249.250/tcp/36983/p2p/16Uiu2HAm7v6syYw9XYMoukZAHK5jhqYR9Jy6zCZhdFuXXgRgbQrA"]'
2026-03-30 05:04:08.996 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-30 05:04:09.000 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:36982/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-30 05:04:09.013 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-30 05:04:09.014 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:32949/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-30 05:04:09.019 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-30 05:04:09.020 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:63755/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-30 05:04:09.031 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-30 05:04:09.031 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:30899/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-30 05:04:09.040 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-30 05:04:09.040 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:52618/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-30 05:04:09.045 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-30 05:04:09.046 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:36982/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-30 05:04:09.066 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-30 05:04:09.067 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-30 05:04:09.168 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-30 05:04:09.168 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:36982/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:09.171 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774847049046035675,"ephemeral":false,"proof":""}]'
2026-03-30 05:04:09.173 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-30 05:04:09.174 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:32949/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:09.177 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774847049046035675,"ephemeral":false,"proof":""}]'
2026-03-30 05:04:09.179 DEBUG [src.steps.sharding] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-03-30 05:04:09.179 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:63755/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:09.181 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774847049046035675,"ephemeral":false,"proof":""}]'
2026-03-30 05:04:09.183 DEBUG [src.steps.sharding] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-03-30 05:04:09.183 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30899/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:09.186 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774847049046035675,"ephemeral":false,"proof":""}]'
2026-03-30 05:04:09.187 DEBUG [src.steps.sharding] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-03-30 05:04:09.187 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:52618/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-30 05:04:09.191 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774847049046035675,"ephemeral":false,"proof":""}]'
2026-03-30 05:04:09.200 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-30 05:04:09.201 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-30 05:04:09.201 DEBUG [src.node.waku_node] Stopping container with id c5e63f2baa74
2026-03-30 05:04:09.738 DEBUG [src.node.waku_node] Container stopped.
2026-03-30 05:04:09.740 DEBUG [src.node.waku_node] Stopping container with id 6013d48b6674
2026-03-30 05:04:10.252 DEBUG [src.node.waku_node] Container stopped.
2026-03-30 05:04:10.252 DEBUG [src.node.waku_node] Stopping container with id f6a5979f57ce
2026-03-30 05:04:10.554 ERROR [src.node.docker_mananger] Max retries reached for container c5e63f2baa74. Exiting log stream.
2026-03-30 05:04:10.771 DEBUG [src.node.waku_node] Container stopped.
2026-03-30 05:04:10.771 DEBUG [src.node.waku_node] Stopping container with id 29d9bb770964
2026-03-30 05:04:11.066 ERROR [src.node.docker_mananger] Max retries reached for container 6013d48b6674. Exiting log stream.
2026-03-30 05:04:11.282 DEBUG [src.node.waku_node] Container stopped.
2026-03-30 05:04:11.283 DEBUG [src.node.waku_node] Stopping container with id e5d8805ba274
2026-03-30 05:04:11.585 ERROR [src.node.docker_mananger] Max retries reached for container f6a5979f57ce. Exiting log stream.
2026-03-30 05:04:11.779 DEBUG [src.node.waku_node] Container stopped.
2026-03-30 05:04:11.786 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-30 05:04:11.830 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-30 05:04:11.861 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-30 05:04:11.892 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-30 05:04:11.927 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-30 05:04:11.973 DEBUG [src.node.docker_mananger] No errors found in the waku logs.