86 lines
14 KiB
Plaintext

2026-03-21 04:33:26.439 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-21 04:33:26.440 DEBUG [tests.conftest] Running test: test_2_nodes_different_cluster_different_shard with id: 2026-03-21_04-33-26__48bdc672-86b0-44ff-9d14-5f4ab7ae1bd3
2026-03-21 04:33:26.440 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-21 04:33:26.441 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-21 04:33:26.441 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-21 04:33:26.458 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-21 04:33:26.459 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-21_04-33-26__48bdc672-86b0-44ff-9d14-5f4ab7ae1bd3__wakuorg_nwaku:latest.log
2026-03-21 04:33:26.460 DEBUG [src.node.waku_node] Starting Node...
2026-03-21 04:33:26.461 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-21 04:33:26.464 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-21 04:33:26.465 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.64.179
2026-03-21 04:33:26.466 DEBUG [src.node.docker_mananger] Generated ports ['23847', '23848', '23849', '23850', '23851']
2026-03-21 04:33:26.466 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-21 04:33:26.466 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-21 04:33:26.467 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-21 04:33:26.467 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-21 04:33:26.467 DEBUG [src.node.waku_node] Using volumes []
2026-03-21 04:33:26.467 DEBUG [src.node.docker_mananger] docker run -i -t -p 23847:23847 -p 23848:23848 -p 23849:23849 -p 23850:23850 -p 23851:23851 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=23849 --rest-port=23847 --tcp-port=23848 --discv5-udp-port=23850 --rest-address=0.0.0.0 --nat=extip:172.18.64.179 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=625fdb12a5e1ea71a1aebdf295148cc71a21f91cffe7faface4fdbab3cfafb56 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=23851 --metrics-logging=true --relay=true --filter=true
2026-03-21 04:33:26.667 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.64.179 waku 1ed091059a075455f65a91df5ec23a17027fc48f1738cc0e9b9bd07f1d946200
2026-03-21 04:33:26.681 ERROR [src.node.docker_mananger] Max retries reached for container b8aaa72be440. Exiting log stream.
2026-03-21 04:33:26.708 DEBUG [src.node.docker_mananger] Container started with ID 1ed091059a07. Setting up logs at ./log/docker/node1_2026-03-21_04-33-26__48bdc672-86b0-44ff-9d14-5f4ab7ae1bd3__wakuorg_nwaku:latest.log
2026-03-21 04:33:26.709 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 23847
2026-03-21 04:33:26.709 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-21 04:33:27.369 ERROR [src.node.docker_mananger] Max retries reached for container 147eeac1a05a. Exiting log stream.
2026-03-21 04:33:27.710 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23847/health" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:33:27.712 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-21 04:33:27.813 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23847/health" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:33:27.815 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-21 04:33:27.916 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23847/health" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:33:27.918 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-21 04:33:28.019 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23847/health" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:33:28.021 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-21 04:33:28.122 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23847/health" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:33:28.125 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-21 04:33:28.125 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-21 04:33:28.125 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23847/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:33:28.128 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.64.179/tcp/23848/p2p/16Uiu2HAm7sr58SiRRmFJuPXuS3pnshfEFJ3qhzNjLzLZiyJBnkDL","/ip4/172.18.64.179/tcp/23849/ws/p2p/16Uiu2HAm7sr58SiRRmFJuPXuS3pnshfEFJ3qhzNjLzLZiyJBnkDL"],"enrUri":"enr:-L24QKlwNBfa4_uCmkH8ocueJFyDMNgVdtlavdQ3z-W76mDFHYvSbsw4DTc5Iv24HQLG6B0maeFytuQaT8hyEnqq1gYCgmlkgnY0gmlwhKwSQLOKbXVsdGlhZGRyc5YACASsEkCzBl0oAAoErBJAswZdKd0DgnJzhQACAQAAiXNlY3AyNTZrMaECuQDBVSrh2AK6dpSOedhgjHeJQgm9uSrv7eZqQkS02Z-DdGNwgl0og3VkcIJdKoV3YWt1MgU"}'
2026-03-21 04:33:28.128 INFO [src.node.waku_node] REST service is ready !!
2026-03-21 04:33:28.135 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-21 04:33:28.135 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-21_04-33-26__48bdc672-86b0-44ff-9d14-5f4ab7ae1bd3__wakuorg_nwaku:latest.log
2026-03-21 04:33:28.135 DEBUG [src.node.waku_node] Starting Node...
2026-03-21 04:33:28.135 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-21 04:33:28.137 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-21 04:33:28.137 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.102.244
2026-03-21 04:33:28.137 DEBUG [src.node.docker_mananger] Generated ports ['25144', '25145', '25146', '25147', '25148']
2026-03-21 04:33:28.137 DEBUG [src.node.waku_node] Cluster id was resolved to: 4
2026-03-21 04:33:28.138 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-21 04:33:28.138 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-21 04:33:28.138 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-21 04:33:28.138 DEBUG [src.node.waku_node] Using volumes []
2026-03-21 04:33:28.138 DEBUG [src.node.docker_mananger] docker run -i -t -p 25144:25144 -p 25145:25145 -p 25146:25146 -p 25147:25147 -p 25148:25148 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=25146 --rest-port=25144 --tcp-port=25145 --discv5-udp-port=25147 --rest-address=0.0.0.0 --nat=extip:172.18.102.244 --peer-exchange=true --discv5-discovery=true --cluster-id=4 --nodekey=bddcdbbf4c87a72ac2bf9d12326f1c25c688959ad2e3dffbd1e771efc6be5e9e --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=25148 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QKlwNBfa4_uCmkH8ocueJFyDMNgVdtlavdQ3z-W76mDFHYvSbsw4DTc5Iv24HQLG6B0maeFytuQaT8hyEnqq1gYCgmlkgnY0gmlwhKwSQLOKbXVsdGlhZGRyc5YACASsEkCzBl0oAAoErBJAswZdKd0DgnJzhQACAQAAiXNlY3AyNTZrMaECuQDBVSrh2AK6dpSOedhgjHeJQgm9uSrv7eZqQkS02Z-DdGNwgl0og3VkcIJdKoV3YWt1MgU
2026-03-21 04:33:28.347 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.102.244 waku 02369a57869235c473036f85186e5d6578572bf4e9b26a75e35dce2504b21cc7
2026-03-21 04:33:28.384 DEBUG [src.node.docker_mananger] Container started with ID 02369a578692. Setting up logs at ./log/docker/node2_2026-03-21_04-33-26__48bdc672-86b0-44ff-9d14-5f4ab7ae1bd3__wakuorg_nwaku:latest.log
2026-03-21 04:33:28.385 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 25144
2026-03-21 04:33:28.385 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-21 04:33:29.387 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25144/health" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:33:29.403 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-21 04:33:29.404 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-21 04:33:29.404 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25144/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:33:29.407 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.102.244/tcp/25145/p2p/16Uiu2HAmFYVbv3WNqabctYKkT5662AruEPEaqkuhyBvUGv7ikrk3","/ip4/172.18.102.244/tcp/25146/ws/p2p/16Uiu2HAmFYVbv3WNqabctYKkT5662AruEPEaqkuhyBvUGv7ikrk3"],"enrUri":"enr:-L24QGYjQLvRehr8tjC_NRWbk-HlCKzFMnvUOrI5RqRakcj6bFH3TmYcF8n_KdSuKR1EdU3xnBEFOa2t2-VFQ1HrcL4CgmlkgnY0gmlwhKwSZvSKbXVsdGlhZGRyc5YACASsEmb0BmI5AAoErBJm9AZiOt0DgnJzhQAEAQAAiXNlY3AyNTZrMaEDKukSFZrLyYeQcUMb_NsjzC9Jo0z6i8kRa8aUSxd9mWyDdGNwgmI5g3VkcIJiO4V3YWt1MgE"}'
2026-03-21 04:33:29.407 INFO [src.node.waku_node] REST service is ready !!
2026-03-21 04:33:29.408 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:25144/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.64.179/tcp/23848/p2p/16Uiu2HAm7sr58SiRRmFJuPXuS3pnshfEFJ3qhzNjLzLZiyJBnkDL"]'
2026-03-21 04:33:29.435 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-21 04:33:29.438 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:23847/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-21 04:33:29.444 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-21 04:33:29.448 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:25144/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/4/1"]'
2026-03-21 04:33:29.453 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-21 04:33:29.454 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:23847/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-21 04:33:29.469 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-21 04:33:29.470 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-21 04:33:29.570 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-21 04:33:29.571 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23847/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:33:29.573 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774067609454516284,"ephemeral":false,"proof":""}]'
2026-03-21 04:33:29.575 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-21 04:33:29.575 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25144/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-21 04:33:29.578 ERROR [src.node.api_clients.base_client] HTTP error occurred: 404 Client Error: Not Found for url: http://127.0.0.1:25144/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0. Response content: b''
2026-03-21 04:33:29.581 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-21 04:33:29.582 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-21 04:33:29.582 DEBUG [src.node.waku_node] Stopping container with id 1ed091059a07
2026-03-21 04:33:30.218 DEBUG [src.node.waku_node] Container stopped.
2026-03-21 04:33:30.220 DEBUG [src.node.waku_node] Stopping container with id 02369a578692
2026-03-21 04:33:30.784 DEBUG [src.node.waku_node] Container stopped.
2026-03-21 04:33:30.787 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-21 04:33:30.817 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-21 04:33:30.824 DEBUG [src.node.docker_mananger] No errors found in the waku logs.