89 lines
17 KiB
Plaintext

2026-03-22 04:46:02.760 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-22 04:46:02.760 DEBUG [tests.conftest] Running test: test_message_relayed_while_store_node_is_stopped_without_removing with id: 2026-03-22_04-46-02__c66d2049-8f58-4a3b-a1a5-f1e1caec9199
2026-03-22 04:46:02.760 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-22 04:46:02.761 DEBUG [src.steps.store] Running fixture setup: store_setup
2026-03-22 04:46:02.769 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-22 04:46:02.769 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/publishing_node1_2026-03-22_04-46-02__c66d2049-8f58-4a3b-a1a5-f1e1caec9199__wakuorg_nwaku:latest.log
2026-03-22 04:46:02.770 DEBUG [src.node.waku_node] Starting Node...
2026-03-22 04:46:02.770 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-22 04:46:02.771 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-22 04:46:02.772 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.41.171
2026-03-22 04:46:02.772 DEBUG [src.node.docker_mananger] Generated ports ['20543', '20544', '20545', '20546', '20547']
2026-03-22 04:46:02.772 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-22 04:46:02.772 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-22 04:46:02.772 DEBUG [src.node.waku_node] Using volumes []
2026-03-22 04:46:02.772 DEBUG [src.node.docker_mananger] docker run -i -t -p 20543:20543 -p 20544:20544 -p 20545:20545 -p 20546:20546 -p 20547:20547 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=20545 --rest-port=20543 --tcp-port=20544 --discv5-udp-port=20546 --rest-address=0.0.0.0 --nat=extip:172.18.41.171 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=e4fdfed1f3bd5dd132ddad299761f831d5ca83da4dacac20a1acb5981bd3c6ad --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=20547 --metrics-logging=true --store=true --relay=true
2026-03-22 04:46:02.910 ERROR [src.node.docker_mananger] Max retries reached for container cbdc6c1b20b4. Exiting log stream.
2026-03-22 04:46:02.983 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.41.171 waku 21495a0662e030e9f2a46b72f30b1dac1a44feb939400f692c198ef283396d51
2026-03-22 04:46:03.020 DEBUG [src.node.docker_mananger] Container started with ID 21495a0662e0. Setting up logs at ./log/docker/publishing_node1_2026-03-22_04-46-02__c66d2049-8f58-4a3b-a1a5-f1e1caec9199__wakuorg_nwaku:latest.log
2026-03-22 04:46:03.021 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 20543
2026-03-22 04:46:03.023 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-22 04:46:03.561 ERROR [src.node.docker_mananger] Max retries reached for container a00770dc22e2. Exiting log stream.
2026-03-22 04:46:04.025 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:20543/health" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:46:04.028 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-22 04:46:04.028 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-22 04:46:04.028 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:20543/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:46:04.031 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.41.171/tcp/20544/p2p/16Uiu2HAmERf8wSCBkXpHVRyQKeHx35YheWLGMPC2U1FVzFUUzScC","/ip4/172.18.41.171/tcp/20545/ws/p2p/16Uiu2HAmERf8wSCBkXpHVRyQKeHx35YheWLGMPC2U1FVzFUUzScC"],"enrUri":"enr:-L24QP_px-x3EhwAJ0hemvhYV98Zd1wLxUu2DU3plmQlaJEgWECgUvyvFk9byCS86bx56e6lpvrL5VnW2ZF1Nfx6RhECgmlkgnY0gmlwhKwSKauKbXVsdGlhZGRyc5YACASsEimrBlBAAAoErBIpqwZQQd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDGk0ZZMVyeDfFEdmGzsHjKdHWANjSeE4w2VfSed926fWDdGNwglBAg3VkcIJQQoV3YWt1MgM"}'
2026-03-22 04:46:04.032 INFO [src.node.waku_node] REST service is ready !!
2026-03-22 04:46:04.040 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-22 04:46:04.040 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/store_node1_2026-03-22_04-46-02__c66d2049-8f58-4a3b-a1a5-f1e1caec9199__wakuorg_nwaku:latest.log
2026-03-22 04:46:04.040 DEBUG [src.node.waku_node] Starting Node...
2026-03-22 04:46:04.040 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-22 04:46:04.042 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-22 04:46:04.042 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.160.199
2026-03-22 04:46:04.042 DEBUG [src.node.docker_mananger] Generated ports ['9032', '9033', '9034', '9035', '9036']
2026-03-22 04:46:04.042 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-22 04:46:04.042 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-22 04:46:04.043 DEBUG [src.node.waku_node] Using volumes []
2026-03-22 04:46:04.043 DEBUG [src.node.docker_mananger] docker run -i -t -p 9032:9032 -p 9033:9033 -p 9034:9034 -p 9035:9035 -p 9036:9036 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=9034 --rest-port=9032 --tcp-port=9033 --discv5-udp-port=9035 --rest-address=0.0.0.0 --nat=extip:172.18.160.199 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=f95fdd6888871d1e8b3fb4da907d1d881dd68f851d86f60de1ca39fff69b9c5b --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=9036 --metrics-logging=true --discv5-bootstrap-node=enr:-L24QP_px-x3EhwAJ0hemvhYV98Zd1wLxUu2DU3plmQlaJEgWECgUvyvFk9byCS86bx56e6lpvrL5VnW2ZF1Nfx6RhECgmlkgnY0gmlwhKwSKauKbXVsdGlhZGRyc5YACASsEimrBlBAAAoErBIpqwZQQd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDGk0ZZMVyeDfFEdmGzsHjKdHWANjSeE4w2VfSed926fWDdGNwglBAg3VkcIJQQoV3YWt1MgM --storenode=/ip4/172.18.41.171/tcp/20544/p2p/16Uiu2HAmERf8wSCBkXpHVRyQKeHx35YheWLGMPC2U1FVzFUUzScC --store=false --relay=false
2026-03-22 04:46:04.255 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.160.199 waku fe6f33c40d318617f6190e291986f9a576bc5166d2f662e14171c17f46cb0e81
2026-03-22 04:46:04.291 DEBUG [src.node.docker_mananger] Container started with ID fe6f33c40d31. Setting up logs at ./log/docker/store_node1_2026-03-22_04-46-02__c66d2049-8f58-4a3b-a1a5-f1e1caec9199__wakuorg_nwaku:latest.log
2026-03-22 04:46:04.292 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 9032
2026-03-22 04:46:04.292 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-22 04:46:05.293 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:9032/health" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:46:05.296 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-22 04:46:05.296 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-22 04:46:05.297 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:9032/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:46:05.299 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.160.199/tcp/9033/p2p/16Uiu2HAmFPXS1ftJB9X2pAZUxc5bXcztvRFeU5z5zwvN5k8SVjjU","/ip4/172.18.160.199/tcp/9034/ws/p2p/16Uiu2HAmFPXS1ftJB9X2pAZUxc5bXcztvRFeU5z5zwvN5k8SVjjU"],"enrUri":"enr:-L24QPeTBbMCWzfXIcPh83x5y_1myaLauvYFvKLS4Rs4iAbIeXS8U1q1-SKNSxqn8se8CZDIpNnOQ_RoxZqw9TxmkYACgmlkgnY0gmlwhKwSoMeKbXVsdGlhZGRyc5YACASsEqDHBiNJAAoErBKgxwYjSt0DgnJzhQADAQAAiXNlY3AyNTZrMaEDKJzpZdcUPRtM6_eYkxREivq99njZIwp3zWn7mtVfWZeDdGNwgiNJg3VkcIIjS4V3YWt1MgA"}'
2026-03-22 04:46:05.299 INFO [src.node.waku_node] REST service is ready !!
2026-03-22 04:46:05.299 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:9032/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.41.171/tcp/20544/p2p/16Uiu2HAmERf8wSCBkXpHVRyQKeHx35YheWLGMPC2U1FVzFUUzScC"]'
2026-03-22 04:46:05.337 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:46:05.339 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:20543/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/0"]'
2026-03-22 04:46:05.356 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:46:05.360 DEBUG [src.steps.store] Relaying message
2026-03-22 04:46:05.360 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:20543/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-22 04:46:05.366 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:46:05.367 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-22 04:46:05.568 DEBUG [src.steps.store] Checking that peer wakuorg/nwaku:latest can find the stored messages
2026-03-22 04:46:05.568 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:20543/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F0&pageSize=5" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:46:05.571 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0xee386ba5674a5dc67b2f8a70cc9d7f9a1645aac676d3fb35114388bc9b775a46"}]}'
2026-03-22 04:46:05.572 DEBUG [src.steps.store] messages length is 1
2026-03-22 04:46:05.572 DEBUG [src.steps.store] Checking that peer wakuorg/nwaku:latest can find the stored messages
2026-03-22 04:46:05.572 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:9032/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F0&pageSize=5" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:46:05.582 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"6905d6aaa78553337a8e","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0xee386ba5674a5dc67b2f8a70cc9d7f9a1645aac676d3fb35114388bc9b775a46"}]}'
2026-03-22 04:46:05.583 DEBUG [src.steps.store] messages length is 1
2026-03-22 04:46:06.181 DEBUG [src.steps.store] Relaying message
2026-03-22 04:46:06.181 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:20543/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-22 04:46:06.189 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:46:06.189 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-22 04:46:06.611 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:9032/health" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:46:06.716 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:9032/health" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:46:06.720 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-22 04:46:06.720 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-22 04:46:06.720 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:9032/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:46:06.723 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.160.199/tcp/9033/p2p/16Uiu2HAmFPXS1ftJB9X2pAZUxc5bXcztvRFeU5z5zwvN5k8SVjjU","/ip4/172.18.160.199/tcp/9034/ws/p2p/16Uiu2HAmFPXS1ftJB9X2pAZUxc5bXcztvRFeU5z5zwvN5k8SVjjU"],"enrUri":"enr:-L24QPeTBbMCWzfXIcPh83x5y_1myaLauvYFvKLS4Rs4iAbIeXS8U1q1-SKNSxqn8se8CZDIpNnOQ_RoxZqw9TxmkYACgmlkgnY0gmlwhKwSoMeKbXVsdGlhZGRyc5YACASsEqDHBiNJAAoErBKgxwYjSt0DgnJzhQADAQAAiXNlY3AyNTZrMaEDKJzpZdcUPRtM6_eYkxREivq99njZIwp3zWn7mtVfWZeDdGNwgiNJg3VkcIIjS4V3YWt1MgA"}'
2026-03-22 04:46:06.723 INFO [src.node.waku_node] REST service is ready !!
2026-03-22 04:46:06.724 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:20543/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F0&pageSize=5&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:46:06.727 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0xee386ba5674a5dc67b2f8a70cc9d7f9a1645aac676d3fb35114388bc9b775a46"},{"messageHash":"0x3496e46c64c984dca0a61f9dc424d38f79a987248c2729c371912607eae2cf58"}]}'
2026-03-22 04:46:06.728 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:9032/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F0&pageSize=5&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:46:06.776 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"5808a517e4aa99b4b88c","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0xee386ba5674a5dc67b2f8a70cc9d7f9a1645aac676d3fb35114388bc9b775a46"},{"messageHash":"0x3496e46c64c984dca0a61f9dc424d38f79a987248c2729c371912607eae2cf58"}]}'
2026-03-22 04:46:06.781 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-22 04:46:06.788 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-22 04:46:06.788 DEBUG [src.node.waku_node] Stopping container with id 21495a0662e0
2026-03-22 04:46:07.374 DEBUG [src.node.waku_node] Container stopped.
2026-03-22 04:46:07.374 DEBUG [src.node.waku_node] Stopping container with id fe6f33c40d31
2026-03-22 04:46:07.941 DEBUG [src.node.waku_node] Container stopped.
2026-03-22 04:46:07.944 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-22 04:46:07.953 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-22 04:46:08.023 DEBUG [src.node.docker_mananger] No errors found in the waku logs.