78 lines
13 KiB
Plaintext

2026-03-15 04:51:44.796 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-15 04:51:44.797 DEBUG [tests.conftest] Running test: test_2_nodes_different_cluster_same_shard with id: 2026-03-15_04-51-44__bc1afa96-b220-40bf-9ae7-8976405f350d
2026-03-15 04:51:44.797 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-15 04:51:44.797 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-15 04:51:44.798 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-15 04:51:44.805 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-15 04:51:44.806 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-15_04-51-44__bc1afa96-b220-40bf-9ae7-8976405f350d__wakuorg_nwaku:latest.log
2026-03-15 04:51:44.806 DEBUG [src.node.waku_node] Starting Node...
2026-03-15 04:51:44.806 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-15 04:51:44.807 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-15 04:51:44.808 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.167.203
2026-03-15 04:51:44.808 DEBUG [src.node.docker_mananger] Generated ports ['51042', '51043', '51044', '51045', '51046']
2026-03-15 04:51:44.808 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-15 04:51:44.808 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-15 04:51:44.808 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-15 04:51:44.808 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-15 04:51:44.808 DEBUG [src.node.waku_node] Using volumes []
2026-03-15 04:51:44.809 DEBUG [src.node.docker_mananger] docker run -i -t -p 51042:51042 -p 51043:51043 -p 51044:51044 -p 51045:51045 -p 51046:51046 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=51044 --rest-port=51042 --tcp-port=51043 --discv5-udp-port=51045 --rest-address=0.0.0.0 --nat=extip:172.18.167.203 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=ef4c11ad8cdfabb34d1c70a7bd870b529a78c6cfd3eddd1f734fb019bfda8c8f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=51046 --metrics-logging=true --relay=true --filter=true
2026-03-15 04:51:45.008 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.167.203 waku 864340040c4a0f78725860f1c54a587a9f49309c9d177db7590f7f001f8bc7d3
2026-03-15 04:51:45.034 ERROR [src.node.docker_mananger] Max retries reached for container 1c41192536c9. Exiting log stream.
2026-03-15 04:51:45.047 DEBUG [src.node.docker_mananger] Container started with ID 864340040c4a. Setting up logs at ./log/docker/node1_2026-03-15_04-51-44__bc1afa96-b220-40bf-9ae7-8976405f350d__wakuorg_nwaku:latest.log
2026-03-15 04:51:45.048 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 51042
2026-03-15 04:51:45.048 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-15 04:51:45.500 ERROR [src.node.docker_mananger] Max retries reached for container 2da54629325b. Exiting log stream.
2026-03-15 04:51:46.048 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51042/health" -H "Content-Type: application/json" -d 'None'
2026-03-15 04:51:46.052 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-15 04:51:46.052 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-15 04:51:46.053 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51042/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-15 04:51:46.058 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.167.203/tcp/51043/p2p/16Uiu2HAm16mgAxEg1LmteQen5TRJMXMDh2V7FYhEKkSjDikTpWDa","/ip4/172.18.167.203/tcp/51044/ws/p2p/16Uiu2HAm16mgAxEg1LmteQen5TRJMXMDh2V7FYhEKkSjDikTpWDa"],"enrUri":"enr:-L24QM_m2qSmnWIQ0K53ZbFnZPJuLwsC-EjtCprzgF6P2jOrZ6fZzwfFDMGjwbLFuZy344WCJnlbg0mteDKh38fhLk4CgmlkgnY0gmlwhKwSp8uKbXVsdGlhZGRyc5YACASsEqfLBsdjAAoErBKnywbHZN0DgnJzhQACAQAAiXNlY3AyNTZrMaECVE6G4fwDAtNQfkhXTSjiFOYdj0H5HRmvXwWzLgrQYQWDdGNwgsdjg3VkcILHZYV3YWt1MgU"}'
2026-03-15 04:51:46.058 INFO [src.node.waku_node] REST service is ready !!
2026-03-15 04:51:46.068 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-15 04:51:46.068 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-15_04-51-44__bc1afa96-b220-40bf-9ae7-8976405f350d__wakuorg_nwaku:latest.log
2026-03-15 04:51:46.069 DEBUG [src.node.waku_node] Starting Node...
2026-03-15 04:51:46.069 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-15 04:51:46.072 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-15 04:51:46.073 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.254.188
2026-03-15 04:51:46.074 DEBUG [src.node.docker_mananger] Generated ports ['27817', '27818', '27819', '27820', '27821']
2026-03-15 04:51:46.074 DEBUG [src.node.waku_node] Cluster id was resolved to: 4
2026-03-15 04:51:46.074 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-15 04:51:46.075 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-15 04:51:46.075 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-15 04:51:46.075 DEBUG [src.node.waku_node] Using volumes []
2026-03-15 04:51:46.076 DEBUG [src.node.docker_mananger] docker run -i -t -p 27817:27817 -p 27818:27818 -p 27819:27819 -p 27820:27820 -p 27821:27821 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=27819 --rest-port=27817 --tcp-port=27818 --discv5-udp-port=27820 --rest-address=0.0.0.0 --nat=extip:172.18.254.188 --peer-exchange=true --discv5-discovery=true --cluster-id=4 --nodekey=af2ee28943dff7cde6a2dbf02ddfebfffb6ad95fd21df0f1de9fdef18de9ccaf --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=27821 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QM_m2qSmnWIQ0K53ZbFnZPJuLwsC-EjtCprzgF6P2jOrZ6fZzwfFDMGjwbLFuZy344WCJnlbg0mteDKh38fhLk4CgmlkgnY0gmlwhKwSp8uKbXVsdGlhZGRyc5YACASsEqfLBsdjAAoErBKnywbHZN0DgnJzhQACAQAAiXNlY3AyNTZrMaECVE6G4fwDAtNQfkhXTSjiFOYdj0H5HRmvXwWzLgrQYQWDdGNwgsdjg3VkcILHZYV3YWt1MgU
2026-03-15 04:51:46.289 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.254.188 waku a6a8628703ffe9ade027666bbf1309bbd771fcd51c52c85c4b607c83e106e80b
2026-03-15 04:51:46.323 DEBUG [src.node.docker_mananger] Container started with ID a6a8628703ff. Setting up logs at ./log/docker/node2_2026-03-15_04-51-44__bc1afa96-b220-40bf-9ae7-8976405f350d__wakuorg_nwaku:latest.log
2026-03-15 04:51:46.324 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 27817
2026-03-15 04:51:46.324 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-15 04:51:47.327 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:27817/health" -H "Content-Type: application/json" -d 'None'
2026-03-15 04:51:47.340 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-15 04:51:47.343 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-15 04:51:47.346 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:27817/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-15 04:51:47.353 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.254.188/tcp/27818/p2p/16Uiu2HAmGfUpdPxW9Gkty5ZThWopbqC3pEFQV26M7iCbHxLkRTWv","/ip4/172.18.254.188/tcp/27819/ws/p2p/16Uiu2HAmGfUpdPxW9Gkty5ZThWopbqC3pEFQV26M7iCbHxLkRTWv"],"enrUri":"enr:-L24QNu2yIgExBDu2s6gSWiDN4pRWgdcnaAOuLDn-PrwmtC5CeIivvX4i9lwiqCM69U4WU90S7IML4GFiU0IhBb3IH8CgmlkgnY0gmlwhKwS_ryKbXVsdGlhZGRyc5YACASsEv68BmyqAAoErBL-vAZsq90DgnJzhQAEAQAAiXNlY3AyNTZrMaEDO47wu9vM3K0I6vca-sb41E7xD7fJ_DrN9CLP417jMH-DdGNwgmyqg3VkcIJsrIV3YWt1MgE"}'
2026-03-15 04:51:47.354 INFO [src.node.waku_node] REST service is ready !!
2026-03-15 04:51:47.354 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:27817/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.167.203/tcp/51043/p2p/16Uiu2HAm16mgAxEg1LmteQen5TRJMXMDh2V7FYhEKkSjDikTpWDa"]'
2026-03-15 04:51:47.380 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-15 04:51:47.383 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:51042/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-15 04:51:47.388 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-15 04:51:47.389 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:27817/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/4/0"]'
2026-03-15 04:51:47.394 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-15 04:51:47.395 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:51042/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-15 04:51:47.406 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-15 04:51:47.406 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-15 04:51:47.506 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-15 04:51:47.507 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51042/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-15 04:51:47.509 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773550307395363877,"ephemeral":false,"proof":""}]'
2026-03-15 04:51:47.511 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-15 04:51:47.511 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:27817/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-15 04:51:47.514 ERROR [src.node.api_clients.base_client] HTTP error occurred: 404 Client Error: Not Found for url: http://127.0.0.1:27817/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0. Response content: b''
2026-03-15 04:51:47.516 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-15 04:51:47.517 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-15 04:51:47.518 DEBUG [src.node.waku_node] Stopping container with id 864340040c4a
2026-03-15 04:51:48.161 DEBUG [src.node.waku_node] Container stopped.
2026-03-15 04:51:48.163 DEBUG [src.node.waku_node] Stopping container with id a6a8628703ff
2026-03-15 04:51:48.717 DEBUG [src.node.waku_node] Container stopped.
2026-03-15 04:51:48.719 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-15 04:51:48.782 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-15 04:51:48.790 DEBUG [src.node.docker_mananger] No errors found in the waku logs.