95 lines
16 KiB
Plaintext

2026-03-13 04:42:44.249 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-13 04:42:44.249 DEBUG [tests.conftest] Running test: test_subscribe_and_publish_on_another_shard with id: 2026-03-13_04-42-44__b1690c6a-f27a-4bcb-ac57-a9d65f5d25f2
2026-03-13 04:42:44.249 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-13 04:42:44.249 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-13 04:42:44.250 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-13 04:42:44.257 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:42:44.258 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-13_04-42-44__b1690c6a-f27a-4bcb-ac57-a9d65f5d25f2__wakuorg_nwaku:latest.log
2026-03-13 04:42:44.258 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:42:44.258 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:42:44.259 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:42:44.259 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.212.232
2026-03-13 04:42:44.259 DEBUG [src.node.docker_mananger] Generated ports ['5039', '5040', '5041', '5042', '5043']
2026-03-13 04:42:44.260 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-13 04:42:44.260 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-13 04:42:44.260 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:42:44.260 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:42:44.260 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:42:44.260 DEBUG [src.node.docker_mananger] docker run -i -t -p 5039:5039 -p 5040:5040 -p 5041:5041 -p 5042:5042 -p 5043:5043 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=5041 --rest-port=5039 --tcp-port=5040 --discv5-udp-port=5042 --rest-address=0.0.0.0 --nat=extip:172.18.212.232 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=4973bddbd84e6caad8c25ae1de16bc0ba3dc728d3b01102d89ef8f026fffef45 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=5043 --metrics-logging=true --relay=true --filter=true
2026-03-13 04:42:44.282 ERROR [src.node.docker_mananger] Max retries reached for container b9281d067025. Exiting log stream.
2026-03-13 04:42:44.455 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.212.232 waku 8cfb3ffa2de03f0eb8070eb1a8dda7c124ff1742e74b8c6938a4e8d7b878eaf9
2026-03-13 04:42:44.492 DEBUG [src.node.docker_mananger] Container started with ID 8cfb3ffa2de0. Setting up logs at ./log/docker/node1_2026-03-13_04-42-44__b1690c6a-f27a-4bcb-ac57-a9d65f5d25f2__wakuorg_nwaku:latest.log
2026-03-13 04:42:44.492 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 5039
2026-03-13 04:42:44.492 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:42:44.906 ERROR [src.node.docker_mananger] Max retries reached for container f54f94197263. Exiting log stream.
2026-03-13 04:42:45.492 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:5039/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:42:45.496 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:42:45.596 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:5039/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:42:45.598 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:42:45.699 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:5039/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:42:45.701 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:42:45.802 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:5039/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:42:45.845 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:42:45.945 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:5039/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:42:45.951 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:42:45.951 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:42:45.952 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:5039/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:42:45.954 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.212.232/tcp/5040/p2p/16Uiu2HAkzC3y2gwzjAeZ1ittKd4BZ67J2ykrW8fcRjVdxuwF8RRp","/ip4/172.18.212.232/tcp/5041/ws/p2p/16Uiu2HAkzC3y2gwzjAeZ1ittKd4BZ67J2ykrW8fcRjVdxuwF8RRp"],"enrUri":"enr:-L24QMeGmLbXl7cKaeIGo33JF3znIjMFHLYnx5idnioIw6CLbsoFHxnSQUr_9kILVHv4Iq5MbDL0oC3eW8P0d7-Tg5wCgmlkgnY0gmlwhKwS1OiKbXVsdGlhZGRyc5YACASsEtToBhOwAAoErBLU6AYTsd0DgnJzhQACAQAAiXNlY3AyNTZrMaECRs0p9TiiI2q5qqB1_fbCiX6HcnVzDP9TKrq1VFUGGTeDdGNwghOwg3VkcIITsoV3YWt1MgU"}'
2026-03-13 04:42:45.954 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:42:45.961 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:42:45.961 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-13_04-42-44__b1690c6a-f27a-4bcb-ac57-a9d65f5d25f2__wakuorg_nwaku:latest.log
2026-03-13 04:42:45.961 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:42:45.962 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:42:45.963 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:42:45.963 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.241.211
2026-03-13 04:42:45.963 DEBUG [src.node.docker_mananger] Generated ports ['18806', '18807', '18808', '18809', '18810']
2026-03-13 04:42:45.963 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-13 04:42:45.963 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-13 04:42:45.964 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:42:45.964 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:42:45.964 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:42:45.964 DEBUG [src.node.docker_mananger] docker run -i -t -p 18806:18806 -p 18807:18807 -p 18808:18808 -p 18809:18809 -p 18810:18810 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=18808 --rest-port=18806 --tcp-port=18807 --discv5-udp-port=18809 --rest-address=0.0.0.0 --nat=extip:172.18.241.211 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=b5db12af86ba0deaa9c3df0d5bacbde6b7fe96a4bd0eaec116bc9d1f9f106d6f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=18810 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QMeGmLbXl7cKaeIGo33JF3znIjMFHLYnx5idnioIw6CLbsoFHxnSQUr_9kILVHv4Iq5MbDL0oC3eW8P0d7-Tg5wCgmlkgnY0gmlwhKwS1OiKbXVsdGlhZGRyc5YACASsEtToBhOwAAoErBLU6AYTsd0DgnJzhQACAQAAiXNlY3AyNTZrMaECRs0p9TiiI2q5qqB1_fbCiX6HcnVzDP9TKrq1VFUGGTeDdGNwghOwg3VkcIITsoV3YWt1MgU
2026-03-13 04:42:46.166 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.241.211 waku ffadbac2cd9eaeddfd2163d4597c17970a117fb9d2675b2fb83b99fda794a49d
2026-03-13 04:42:46.204 DEBUG [src.node.docker_mananger] Container started with ID ffadbac2cd9e. Setting up logs at ./log/docker/node2_2026-03-13_04-42-44__b1690c6a-f27a-4bcb-ac57-a9d65f5d25f2__wakuorg_nwaku:latest.log
2026-03-13 04:42:46.204 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 18806
2026-03-13 04:42:46.205 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:42:47.205 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18806/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:42:47.208 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:42:47.208 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:42:47.209 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18806/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:42:47.211 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.241.211/tcp/18807/p2p/16Uiu2HAkzZX9v1pSVHuzeXfteQmBD4Nzp55D5DbV5NqxQ3ju65Zu","/ip4/172.18.241.211/tcp/18808/ws/p2p/16Uiu2HAkzZX9v1pSVHuzeXfteQmBD4Nzp55D5DbV5NqxQ3ju65Zu"],"enrUri":"enr:-L24QOqacJhrOsVfBgkldr0ByeNNhtHTB2gFUeVMtTlzeKeDHMRPzgDcj6fKm9Gxjui_9Q7uzxoXhQLr-McJKiLDVr0CgmlkgnY0gmlwhKwS8dOKbXVsdGlhZGRyc5YACASsEvHTBkl3AAoErBLx0wZJeN0DgnJzhQACAQAAiXNlY3AyNTZrMaECTE0aIlLr7RStn6h4Q-i7_aHF5vOHCZh3J1VEo8wDO0yDdGNwgkl3g3VkcIJJeYV3YWt1MgE"}'
2026-03-13 04:42:47.211 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:42:47.211 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:18806/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.212.232/tcp/5040/p2p/16Uiu2HAkzC3y2gwzjAeZ1ittKd4BZ67J2ykrW8fcRjVdxuwF8RRp"]'
2026-03-13 04:42:47.234 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:42:47.235 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:5039/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/1"]'
2026-03-13 04:42:47.246 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:42:47.247 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:18806/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/1"]'
2026-03-13 04:42:47.251 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:42:47.252 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:5039/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F1" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:42:47.262 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:42:47.263 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-13 04:42:47.363 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-13 04:42:47.364 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:5039/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:42:47.367 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773376967252392173,"ephemeral":false,"proof":""}]'
2026-03-13 04:42:47.369 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-13 04:42:47.369 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18806/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:42:47.371 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773376967252392173,"ephemeral":false,"proof":""}]'
2026-03-13 04:42:47.374 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:5039/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:42:47.384 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:42:47.385 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-13 04:42:47.485 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-13 04:42:47.486 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:5039/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:42:47.488 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773376967374656485,"ephemeral":false,"proof":""}]'
2026-03-13 04:42:47.490 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-13 04:42:47.490 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18806/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:42:47.492 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773376967374656485,"ephemeral":false,"proof":""}]'
2026-03-13 04:42:47.495 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-13 04:42:47.496 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-13 04:42:47.496 DEBUG [src.node.waku_node] Stopping container with id 8cfb3ffa2de0
2026-03-13 04:42:48.104 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:42:48.104 DEBUG [src.node.waku_node] Stopping container with id ffadbac2cd9e
2026-03-13 04:42:48.731 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:42:48.732 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-13 04:42:48.765 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:42:48.792 DEBUG [src.node.docker_mananger] No errors found in the waku logs.