190 lines
32 KiB
Plaintext

2026-03-10 04:36:03.423 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-10 04:36:03.423 DEBUG [tests.conftest] Running test: test_node_uses_both_auto_and_regular_apis with id: 2026-03-10_04-36-03__2c97660b-b0c8-452a-939d-4af5f3f0dc47
2026-03-10 04:36:03.423 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-10 04:36:03.424 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-10 04:36:03.424 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-10 04:36:03.432 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-10 04:36:03.433 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-10_04-36-03__2c97660b-b0c8-452a-939d-4af5f3f0dc47__wakuorg_nwaku:latest.log
2026-03-10 04:36:03.433 DEBUG [src.node.waku_node] Starting Node...
2026-03-10 04:36:03.433 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-10 04:36:03.434 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-10 04:36:03.435 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.69.111
2026-03-10 04:36:03.435 DEBUG [src.node.docker_mananger] Generated ports ['35301', '35302', '35303', '35304', '35305']
2026-03-10 04:36:03.435 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-10 04:36:03.435 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-10 04:36:03.435 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-10 04:36:03.435 DEBUG [src.node.waku_node] Using volumes []
2026-03-10 04:36:03.435 DEBUG [src.node.docker_mananger] docker run -i -t -p 35301:35301 -p 35302:35302 -p 35303:35303 -p 35304:35304 -p 35305:35305 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=35303 --rest-port=35301 --tcp-port=35302 --discv5-udp-port=35304 --rest-address=0.0.0.0 --nat=extip:172.18.69.111 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=5a07a3b2980306f0fafbddcdbf076c1c63afcb7cfeb1d61b4bcb615b5a5003cb --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=35305 --metrics-logging=true --relay=true --filter=true
2026-03-10 04:36:03.619 ERROR [src.node.docker_mananger] Max retries reached for container 623e5dd07396. Exiting log stream.
2026-03-10 04:36:03.649 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.69.111 waku 0d336e6bd04a63206aae9953122925272bee98365653193d50393098d9fb8877
2026-03-10 04:36:03.686 DEBUG [src.node.docker_mananger] Container started with ID 0d336e6bd04a. Setting up logs at ./log/docker/node1_2026-03-10_04-36-03__2c97660b-b0c8-452a-939d-4af5f3f0dc47__wakuorg_nwaku:latest.log
2026-03-10 04:36:03.687 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 35301
2026-03-10 04:36:03.687 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-10 04:36:04.188 ERROR [src.node.docker_mananger] Max retries reached for container 5e5cf90ebd2e. Exiting log stream.
2026-03-10 04:36:04.688 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:35301/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:04.692 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-10 04:36:04.793 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:35301/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:04.796 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-10 04:36:04.796 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-10 04:36:04.796 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:35301/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:04.798 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.69.111/tcp/35302/p2p/16Uiu2HAm4Rf7E3SrAqA2YoWMAs7rrTs4HDgKX8kmerzxxU3QZYZr","/ip4/172.18.69.111/tcp/35303/ws/p2p/16Uiu2HAm4Rf7E3SrAqA2YoWMAs7rrTs4HDgKX8kmerzxxU3QZYZr"],"enrUri":"enr:-L24QPujuCRncDe2P5sxBUrP3Xvv5xCPAHdHet7c7S3v-RISRzaTweZxQq_EvMc303QC7zJbivQu60QHf9Q39-2GjAwCgmlkgnY0gmlwhKwSRW-KbXVsdGlhZGRyc5YACASsEkVvBonmAAoErBJFbwaJ590DgnJzhQACAQAAiXNlY3AyNTZrMaEChbgyT1FNk7jANZVCAepkwY21U6_qcWX6mcWOdfx_0r2DdGNwgonmg3VkcIKJ6IV3YWt1MgU"}'
2026-03-10 04:36:04.799 INFO [src.node.waku_node] REST service is ready !!
2026-03-10 04:36:04.806 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-10 04:36:04.806 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-10_04-36-03__2c97660b-b0c8-452a-939d-4af5f3f0dc47__wakuorg_nwaku:latest.log
2026-03-10 04:36:04.806 DEBUG [src.node.waku_node] Starting Node...
2026-03-10 04:36:04.806 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-10 04:36:04.808 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-10 04:36:04.808 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.47.120
2026-03-10 04:36:04.808 DEBUG [src.node.docker_mananger] Generated ports ['12410', '12411', '12412', '12413', '12414']
2026-03-10 04:36:04.808 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-10 04:36:04.808 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-10 04:36:04.809 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-10 04:36:04.809 DEBUG [src.node.waku_node] Using volumes []
2026-03-10 04:36:04.809 DEBUG [src.node.docker_mananger] docker run -i -t -p 12410:12410 -p 12411:12411 -p 12412:12412 -p 12413:12413 -p 12414:12414 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=12412 --rest-port=12410 --tcp-port=12411 --discv5-udp-port=12413 --rest-address=0.0.0.0 --nat=extip:172.18.47.120 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=faea05a2738491279c59481e6a7ecf6b5fbd6a31ee73f146d3a9b4a4cd1fccba --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=12414 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QPujuCRncDe2P5sxBUrP3Xvv5xCPAHdHet7c7S3v-RISRzaTweZxQq_EvMc303QC7zJbivQu60QHf9Q39-2GjAwCgmlkgnY0gmlwhKwSRW-KbXVsdGlhZGRyc5YACASsEkVvBonmAAoErBJFbwaJ590DgnJzhQACAQAAiXNlY3AyNTZrMaEChbgyT1FNk7jANZVCAepkwY21U6_qcWX6mcWOdfx_0r2DdGNwgonmg3VkcIKJ6IV3YWt1MgU
2026-03-10 04:36:05.024 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.47.120 waku 7331d949e259d3b514aaa94ac0c10dfad20d03aeee9e4820cacf8e2481a85a3a
2026-03-10 04:36:05.064 DEBUG [src.node.docker_mananger] Container started with ID 7331d949e259. Setting up logs at ./log/docker/node2_2026-03-10_04-36-03__2c97660b-b0c8-452a-939d-4af5f3f0dc47__wakuorg_nwaku:latest.log
2026-03-10 04:36:05.065 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 12410
2026-03-10 04:36:05.065 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-10 04:36:06.066 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:12410/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:06.069 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-10 04:36:06.170 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:12410/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:06.173 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-10 04:36:06.274 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:12410/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:06.277 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-10 04:36:06.277 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-10 04:36:06.278 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:12410/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:06.281 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.47.120/tcp/12411/p2p/16Uiu2HAkzsfQUFGvWC4r2hjwWBUkZVJ3uqTbwiB2KktVnxJUcgKB","/ip4/172.18.47.120/tcp/12412/ws/p2p/16Uiu2HAkzsfQUFGvWC4r2hjwWBUkZVJ3uqTbwiB2KktVnxJUcgKB"],"enrUri":"enr:-L24QOdHwdt2rdJdy6qMDUu3aIn1uLVYfmQHP0Be9klYCAd3BqlcfgL2uU-WXJNUIlAxl-RHXUcWMJc77wPS4E3MRLYCgmlkgnY0gmlwhKwSL3iKbXVsdGlhZGRyc5YACASsEi94BjB7AAoErBIveAYwfN0DgnJzhQACAQAAiXNlY3AyNTZrMaECUPLi04-3QTSPE6ARkNE_RhZUduKV5snsSi052JvfDqKDdGNwgjB7g3VkcIIwfYV3YWt1MgE"}'
2026-03-10 04:36:06.282 INFO [src.node.waku_node] REST service is ready !!
2026-03-10 04:36:06.282 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:12410/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.69.111/tcp/35302/p2p/16Uiu2HAm4Rf7E3SrAqA2YoWMAs7rrTs4HDgKX8kmerzxxU3QZYZr"]'
2026-03-10 04:36:06.319 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:06.324 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:35301/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/toychat/2/huilong/proto"]'
2026-03-10 04:36:06.341 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:06.343 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:12410/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/toychat/2/huilong/proto"]'
2026-03-10 04:36:06.357 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:06.360 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:35301/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/toychat/2/huilong/proto", "timestamp": '$(date +%s%N)'}'
2026-03-10 04:36:06.373 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:06.374 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-10 04:36:06.475 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-10 04:36:06.475 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:35301/relay/v1/auto/messages/%2Ftoychat%2F2%2Fhuilong%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:06.479 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/toychat/2/huilong/proto","version":0,"timestamp":1773117366360297514,"ephemeral":false,"proof":""}]'
2026-03-10 04:36:06.481 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-10 04:36:06.481 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:12410/relay/v1/auto/messages/%2Ftoychat%2F2%2Fhuilong%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:06.484 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/toychat/2/huilong/proto","version":0,"timestamp":1773117366360297514,"ephemeral":false,"proof":""}]'
2026-03-10 04:36:06.486 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:35301/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-10 04:36:06.488 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:06.489 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:12410/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-10 04:36:06.491 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:06.492 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:35301/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-10 04:36:06.501 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:06.501 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-10 04:36:06.602 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-10 04:36:06.603 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:35301/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:06.605 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773117365194974328,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773117366492076262,"ephemeral":false,"proof":""}]'
2026-03-10 04:36:06.614 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-10 04:36:06.615 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-10 04:36:06.615 DEBUG [src.node.waku_node] Stopping container with id 0d336e6bd04a
2026-03-10 04:36:07.242 DEBUG [src.node.waku_node] Container stopped.
2026-03-10 04:36:07.242 DEBUG [src.node.waku_node] Stopping container with id 7331d949e259
2026-03-10 04:36:07.845 DEBUG [src.node.waku_node] Container stopped.
2026-03-10 04:36:07.847 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-10 04:36:07.891 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-10 04:36:07.937 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-10 04:36:07.941 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-10 04:36:07.942 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-10 04:36:07.944 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-10_04-36-03__2c97660b-b0c8-452a-939d-4af5f3f0dc47__wakuorg_nwaku:latest.log
2026-03-10 04:36:07.950 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-10_04-36-03__2c97660b-b0c8-452a-939d-4af5f3f0dc47__wakuorg_nwaku:latest.log
2026-03-10 04:36:07.965 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-10 04:36:07.966 DEBUG [tests.conftest] Running test: test_node_uses_both_auto_and_regular_apis with id: 2026-03-10_04-36-07__ed154964-9602-4708-8264-1bf3f59dfc50
2026-03-10 04:36:07.966 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-10 04:36:07.966 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-10 04:36:07.967 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-10 04:36:07.979 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-10 04:36:07.979 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-10_04-36-07__ed154964-9602-4708-8264-1bf3f59dfc50__wakuorg_nwaku:latest.log
2026-03-10 04:36:07.980 DEBUG [src.node.waku_node] Starting Node...
2026-03-10 04:36:07.980 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-10 04:36:07.983 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-10 04:36:07.983 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.250.39
2026-03-10 04:36:07.984 DEBUG [src.node.docker_mananger] Generated ports ['23737', '23738', '23739', '23740', '23741']
2026-03-10 04:36:07.984 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-10 04:36:07.984 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-10 04:36:07.984 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-10 04:36:07.984 DEBUG [src.node.waku_node] Using volumes []
2026-03-10 04:36:07.985 DEBUG [src.node.docker_mananger] docker run -i -t -p 23737:23737 -p 23738:23738 -p 23739:23739 -p 23740:23740 -p 23741:23741 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=23739 --rest-port=23737 --tcp-port=23738 --discv5-udp-port=23740 --rest-address=0.0.0.0 --nat=extip:172.18.250.39 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=f9c05d8dd8834a009c39eb77a2dacfce54d5dfd7f9daada57beaa9dc5bddeebf --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=23741 --metrics-logging=true --relay=true --filter=true
2026-03-10 04:36:08.142 ERROR [src.node.docker_mananger] Max retries reached for container 0d336e6bd04a. Exiting log stream.
2026-03-10 04:36:08.208 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.250.39 waku 330412efbe0981eb210f0ebcea2eb23c6b602946776cff63417a0af688fbae02
2026-03-10 04:36:08.251 DEBUG [src.node.docker_mananger] Container started with ID 330412efbe09. Setting up logs at ./log/docker/node1_2026-03-10_04-36-07__ed154964-9602-4708-8264-1bf3f59dfc50__wakuorg_nwaku:latest.log
2026-03-10 04:36:08.251 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 23737
2026-03-10 04:36:08.251 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-10 04:36:08.940 ERROR [src.node.docker_mananger] Max retries reached for container 7331d949e259. Exiting log stream.
2026-03-10 04:36:09.252 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23737/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:09.254 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-10 04:36:09.355 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23737/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:09.358 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-10 04:36:09.358 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-10 04:36:09.358 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23737/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:09.361 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.250.39/tcp/23738/p2p/16Uiu2HAmJF22sHUS7U1wu72b3ssvfbv4nbHeH9rt398PiKBi9ZRu","/ip4/172.18.250.39/tcp/23739/ws/p2p/16Uiu2HAmJF22sHUS7U1wu72b3ssvfbv4nbHeH9rt398PiKBi9ZRu"],"enrUri":"enr:-L24QJUivlMnvW0F8RftU5qCpNctcNxflxirmuJmV-QI3cpeFnjRKjZPUrMW04-h83BIrIz_MALC6JfmqGePwWgmjN8CgmlkgnY0gmlwhKwS-ieKbXVsdGlhZGRyc5YACASsEvonBly6AAoErBL6JwZcu90DgnJzhQACAQAAiXNlY3AyNTZrMaEDUwIVm_LOxtE9I3O-xZn5zvPkifiXnVqjLMTZ0DTutjSDdGNwgly6g3VkcIJcvIV3YWt1MgU"}'
2026-03-10 04:36:09.361 INFO [src.node.waku_node] REST service is ready !!
2026-03-10 04:36:09.369 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-10 04:36:09.369 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-10_04-36-07__ed154964-9602-4708-8264-1bf3f59dfc50__wakuorg_nwaku:latest.log
2026-03-10 04:36:09.369 DEBUG [src.node.waku_node] Starting Node...
2026-03-10 04:36:09.369 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-10 04:36:09.370 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-10 04:36:09.371 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.79.228
2026-03-10 04:36:09.371 DEBUG [src.node.docker_mananger] Generated ports ['33179', '33180', '33181', '33182', '33183']
2026-03-10 04:36:09.371 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-10 04:36:09.371 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-10 04:36:09.371 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-10 04:36:09.371 DEBUG [src.node.waku_node] Using volumes []
2026-03-10 04:36:09.371 DEBUG [src.node.docker_mananger] docker run -i -t -p 33179:33179 -p 33180:33180 -p 33181:33181 -p 33182:33182 -p 33183:33183 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=33181 --rest-port=33179 --tcp-port=33180 --discv5-udp-port=33182 --rest-address=0.0.0.0 --nat=extip:172.18.79.228 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=d27aa84d515e19ea7ecc4bfed5030f995decb39fbddc23e391c6dff17bcf1aed --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=33183 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QJUivlMnvW0F8RftU5qCpNctcNxflxirmuJmV-QI3cpeFnjRKjZPUrMW04-h83BIrIz_MALC6JfmqGePwWgmjN8CgmlkgnY0gmlwhKwS-ieKbXVsdGlhZGRyc5YACASsEvonBly6AAoErBL6JwZcu90DgnJzhQACAQAAiXNlY3AyNTZrMaEDUwIVm_LOxtE9I3O-xZn5zvPkifiXnVqjLMTZ0DTutjSDdGNwgly6g3VkcIJcvIV3YWt1MgU
2026-03-10 04:36:09.586 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.79.228 waku e183efd71a6720cdc29474374af59bceb9a8d326dcad1d667dfa5e4ff16eea69
2026-03-10 04:36:09.622 DEBUG [src.node.docker_mananger] Container started with ID e183efd71a67. Setting up logs at ./log/docker/node2_2026-03-10_04-36-07__ed154964-9602-4708-8264-1bf3f59dfc50__wakuorg_nwaku:latest.log
2026-03-10 04:36:09.622 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 33179
2026-03-10 04:36:09.622 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-10 04:36:10.623 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33179/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:10.626 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-10 04:36:10.726 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33179/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:10.750 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-10 04:36:10.850 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33179/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:10.853 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-10 04:36:10.853 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-10 04:36:10.853 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33179/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:10.856 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.79.228/tcp/33180/p2p/16Uiu2HAm2QPNDHNzH4qMxFUmufQL9EPgUUUa8vTdSfAsfkC5hLSB","/ip4/172.18.79.228/tcp/33181/ws/p2p/16Uiu2HAm2QPNDHNzH4qMxFUmufQL9EPgUUUa8vTdSfAsfkC5hLSB"],"enrUri":"enr:-L24QNFfGMJH42FsoJpk1KxE5NCUiiG2Qtjg6NDodvodkU6UX3JR-wI4EkOia57SCvr3xm5-bI5qC4yNVBoVl97moD4CgmlkgnY0gmlwhKwST-SKbXVsdGlhZGRyc5YACASsEk_kBoGcAAoErBJP5AaBnd0DgnJzhQACAQAAiXNlY3AyNTZrMaECZ61xWXXhK0BXiDBsvaSz-13KgA9M9hCvascmhX4BD4CDdGNwgoGcg3VkcIKBnoV3YWt1MgE"}'
2026-03-10 04:36:10.856 INFO [src.node.waku_node] REST service is ready !!
2026-03-10 04:36:10.856 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:33179/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.250.39/tcp/23738/p2p/16Uiu2HAmJF22sHUS7U1wu72b3ssvfbv4nbHeH9rt398PiKBi9ZRu"]'
2026-03-10 04:36:10.878 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:10.883 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:23737/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/toychat/2/huilong/proto"]'
2026-03-10 04:36:10.890 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:10.891 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:33179/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/toychat/2/huilong/proto"]'
2026-03-10 04:36:10.897 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:10.898 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:23737/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/toychat/2/huilong/proto", "timestamp": '$(date +%s%N)'}'
2026-03-10 04:36:10.916 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:10.916 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-10 04:36:11.017 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-10 04:36:11.017 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23737/relay/v1/auto/messages/%2Ftoychat%2F2%2Fhuilong%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:11.020 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/toychat/2/huilong/proto","version":0,"timestamp":1773117370898444773,"ephemeral":false,"proof":""}]'
2026-03-10 04:36:11.022 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-10 04:36:11.022 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33179/relay/v1/auto/messages/%2Ftoychat%2F2%2Fhuilong%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:11.024 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/toychat/2/huilong/proto","version":0,"timestamp":1773117370898444773,"ephemeral":false,"proof":""}]'
2026-03-10 04:36:11.026 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:23737/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-10 04:36:11.028 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:11.029 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:33179/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-10 04:36:11.031 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:11.032 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:23737/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-10 04:36:11.042 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:11.042 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-10 04:36:11.142 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-10 04:36:11.143 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:23737/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:11.146 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773117371031983307,"ephemeral":false,"proof":""}]'
2026-03-10 04:36:11.147 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-10 04:36:11.148 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33179/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:11.150 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773117371031983307,"ephemeral":false,"proof":""}]'
2026-03-10 04:36:11.152 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-10 04:36:11.154 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-10 04:36:11.154 DEBUG [src.node.waku_node] Stopping container with id 330412efbe09
2026-03-10 04:36:11.758 DEBUG [src.node.waku_node] Container stopped.
2026-03-10 04:36:11.760 DEBUG [src.node.waku_node] Stopping container with id e183efd71a67
2026-03-10 04:36:12.336 DEBUG [src.node.waku_node] Container stopped.
2026-03-10 04:36:12.338 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-10 04:36:12.392 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-10 04:36:12.421 DEBUG [src.node.docker_mananger] No errors found in the waku logs.