171 lines
30 KiB
Plaintext

2026-03-10 04:35:49.594 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-10 04:35:49.595 DEBUG [tests.conftest] Running test: test_multiple_pubsub_topics_same_cluster with id: 2026-03-10_04-35-49__d0aeefb4-7b2f-4029-9bf7-0d5c7d117481
2026-03-10 04:35:49.596 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-10 04:35:49.597 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-10 04:35:49.597 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-10 04:35:49.609 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-10 04:35:49.609 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-10_04-35-49__d0aeefb4-7b2f-4029-9bf7-0d5c7d117481__wakuorg_nwaku:latest.log
2026-03-10 04:35:49.610 DEBUG [src.node.waku_node] Starting Node...
2026-03-10 04:35:49.611 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-10 04:35:49.613 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-10 04:35:49.613 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.255.59
2026-03-10 04:35:49.613 DEBUG [src.node.docker_mananger] Generated ports ['17526', '17527', '17528', '17529', '17530']
2026-03-10 04:35:49.614 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-10 04:35:49.614 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-10 04:35:49.614 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-10 04:35:49.614 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-10 04:35:49.614 DEBUG [src.node.waku_node] Using volumes []
2026-03-10 04:35:49.614 DEBUG [src.node.docker_mananger] docker run -i -t -p 17526:17526 -p 17527:17527 -p 17528:17528 -p 17529:17529 -p 17530:17530 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=17528 --rest-port=17526 --tcp-port=17527 --discv5-udp-port=17529 --rest-address=0.0.0.0 --nat=extip:172.18.255.59 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=a60ba4298dcd556e1e123d4c651c8bf9df2f708ce7af4cc4baa5a27f0d36746f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=17530 --metrics-logging=true --relay=true --filter=true
2026-03-10 04:35:49.795 ERROR [src.node.docker_mananger] Max retries reached for container 2c38242be17f. Exiting log stream.
2026-03-10 04:35:49.802 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.255.59 waku 176fe4d6274bdb5fb982b5c7ee975c2ce97c2bcf0d203bea0020f253abcd5694
2026-03-10 04:35:49.843 DEBUG [src.node.docker_mananger] Container started with ID 176fe4d6274b. Setting up logs at ./log/docker/node1_2026-03-10_04-35-49__d0aeefb4-7b2f-4029-9bf7-0d5c7d117481__wakuorg_nwaku:latest.log
2026-03-10 04:35:49.844 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 17526
2026-03-10 04:35:49.845 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-10 04:35:50.427 ERROR [src.node.docker_mananger] Max retries reached for container 646325f00215. Exiting log stream.
2026-03-10 04:35:50.845 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17526/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:50.848 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-10 04:35:50.948 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17526/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:50.951 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-10 04:35:51.051 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17526/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:51.054 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-10 04:35:51.054 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-10 04:35:51.055 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17526/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:51.057 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.255.59/tcp/17527/p2p/16Uiu2HAm18ZwaFb6owz6LeeBTDEW2hwhRd4uLLTUU5N2siky9Hpt","/ip4/172.18.255.59/tcp/17528/ws/p2p/16Uiu2HAm18ZwaFb6owz6LeeBTDEW2hwhRd4uLLTUU5N2siky9Hpt"],"enrUri":"enr:-L24QLztzvkXubShIGB2tyorvv5C9XapC8q8kgDGRQOcxItYHMon0DehK1FgUjuaO6W7pzXDERXhX6j5IDfLO_NcUdgCgmlkgnY0gmlwhKwS_zuKbXVsdGlhZGRyc5YACASsEv87BkR3AAoErBL_OwZEeN0DgnJzhQACAQAAiXNlY3AyNTZrMaECVMRrmcg3pWzGLI0-azURRwftUBQo3-kKffnJpEFbk_mDdGNwgkR3g3VkcIJEeYV3YWt1MgU"}'
2026-03-10 04:35:51.057 INFO [src.node.waku_node] REST service is ready !!
2026-03-10 04:35:51.064 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-10 04:35:51.065 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-10_04-35-49__d0aeefb4-7b2f-4029-9bf7-0d5c7d117481__wakuorg_nwaku:latest.log
2026-03-10 04:35:51.065 DEBUG [src.node.waku_node] Starting Node...
2026-03-10 04:35:51.065 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-10 04:35:51.066 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-10 04:35:51.066 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.232.243
2026-03-10 04:35:51.067 DEBUG [src.node.docker_mananger] Generated ports ['33332', '33333', '33334', '33335', '33336']
2026-03-10 04:35:51.067 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-10 04:35:51.067 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-10 04:35:51.067 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-10 04:35:51.067 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-10 04:35:51.067 DEBUG [src.node.waku_node] Using volumes []
2026-03-10 04:35:51.067 DEBUG [src.node.docker_mananger] docker run -i -t -p 33332:33332 -p 33333:33333 -p 33334:33334 -p 33335:33335 -p 33336:33336 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=33334 --rest-port=33332 --tcp-port=33333 --discv5-udp-port=33335 --rest-address=0.0.0.0 --nat=extip:172.18.232.243 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=8589fbfbbf18c4ca3c568db5bf9199644c3c200ebcf74e7a6e72dd47cae18c10 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=33336 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QLztzvkXubShIGB2tyorvv5C9XapC8q8kgDGRQOcxItYHMon0DehK1FgUjuaO6W7pzXDERXhX6j5IDfLO_NcUdgCgmlkgnY0gmlwhKwS_zuKbXVsdGlhZGRyc5YACASsEv87BkR3AAoErBL_OwZEeN0DgnJzhQACAQAAiXNlY3AyNTZrMaECVMRrmcg3pWzGLI0-azURRwftUBQo3-kKffnJpEFbk_mDdGNwgkR3g3VkcIJEeYV3YWt1MgU
2026-03-10 04:35:51.269 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.232.243 waku c89b551e906277dac1a40257be54a051a9422d928d2074c63a843303a0970fad
2026-03-10 04:35:51.307 DEBUG [src.node.docker_mananger] Container started with ID c89b551e9062. Setting up logs at ./log/docker/node2_2026-03-10_04-35-49__d0aeefb4-7b2f-4029-9bf7-0d5c7d117481__wakuorg_nwaku:latest.log
2026-03-10 04:35:51.307 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 33332
2026-03-10 04:35:51.307 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-10 04:35:52.308 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33332/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:52.311 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-10 04:35:52.412 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33332/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:52.415 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-10 04:35:52.515 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33332/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:52.518 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-10 04:35:52.618 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33332/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:52.621 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-10 04:35:52.621 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-10 04:35:52.621 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33332/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:52.623 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.232.243/tcp/33333/p2p/16Uiu2HAmP8NdgbkNw41f2aZLtASte6stVD5NkTbXL1BLLQBFc7uh","/ip4/172.18.232.243/tcp/33334/ws/p2p/16Uiu2HAmP8NdgbkNw41f2aZLtASte6stVD5NkTbXL1BLLQBFc7uh"],"enrUri":"enr:-L24QNIlsYcnluS0TknoMTOKJ0e4qUhWIF6xDAsy8AV0faMbRBnSSAwVN0-iVfn0JZpGJD725q54iCH7668oRZTPnqACgmlkgnY0gmlwhKwS6POKbXVsdGlhZGRyc5YACASsEujzBoI1AAoErBLo8waCNt0DgnJzhQACAQAAiXNlY3AyNTZrMaEDm5i_BbY9MWl5QMLd_WpvbqnU6R5nvjwDnaGfT2C5A6CDdGNwgoI1g3VkcIKCN4V3YWt1MgE"}'
2026-03-10 04:35:52.624 INFO [src.node.waku_node] REST service is ready !!
2026-03-10 04:35:52.624 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:33332/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.255.59/tcp/17527/p2p/16Uiu2HAm18ZwaFb6owz6LeeBTDEW2hwhRd4uLLTUU5N2siky9Hpt"]'
2026-03-10 04:35:52.662 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:35:52.665 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:17526/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0", "/waku/2/rs/2/1", "/waku/2/rs/2/2", "/waku/2/rs/2/3", "/waku/2/rs/2/4", "/waku/2/rs/2/5", "/waku/2/rs/2/6", "/waku/2/rs/2/7"]'
2026-03-10 04:35:52.673 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:35:52.675 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:33332/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0", "/waku/2/rs/2/1", "/waku/2/rs/2/2", "/waku/2/rs/2/3", "/waku/2/rs/2/4", "/waku/2/rs/2/5", "/waku/2/rs/2/6", "/waku/2/rs/2/7"]'
2026-03-10 04:35:52.685 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:35:52.688 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:17526/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-10 04:35:52.699 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:35:52.700 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-10 04:35:52.800 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-10 04:35:52.800 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17526/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:52.803 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"L3dha3UvMi9ycy8yLzA=","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773117348506057417,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773117349390720887,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773117352688129991,"ephemeral":false,"proof":""}]'
2026-03-10 04:35:52.812 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-10 04:35:52.813 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-10 04:35:52.813 DEBUG [src.node.waku_node] Stopping container with id 176fe4d6274b
2026-03-10 04:35:53.496 DEBUG [src.node.waku_node] Container stopped.
2026-03-10 04:35:53.498 DEBUG [src.node.waku_node] Stopping container with id c89b551e9062
2026-03-10 04:35:54.091 DEBUG [src.node.waku_node] Container stopped.
2026-03-10 04:35:54.093 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-10 04:35:54.172 DEBUG [src.node.docker_mananger] Found matches for keyword 'stacktrace': ['FAT 2026-03-10 04:35:52.822+00:00 Shutting down after receiving SIGSEGV topics="wakunode main" tid=1 file=wakunode2.nim:91 stacktrace="/app/vendor/nim-chronicles/chronicles/formats.nim(20) _ZN9wakunode213handleSigsegvE5int32\\n"', '/app/vendor/nimbus-build-system/vendor/Nim/lib/system/excpt.nim(530) _ZN6system15writeStackTraceE', '/app/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(59) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI6stringE', 'FAT 2026-03-10 04:35:52.822+00:00 Shutting down after receiving SIGSEGV topics="wakunode main" tid=1 file=wakunode2.nim:91 stacktrace="/app/vendor/nim-chronicles/chronicles/formats.nim(20) _ZN9wakunode213handleSigsegvE5int32\\n"', '/app/vendor/nimbus-build-system/vendor/Nim/lib/system/excpt.nim(530) _ZN6system15writeStackTraceE', '/app/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(59) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI6stringE']
2026-03-10 04:35:54.172 DEBUG [src.node.docker_mananger] Found matches for keyword 'SIGSEGV': ['FAT 2026-03-10 04:35:52.822+00:00 Shutting down after receiving SIGSEGV topics="wakunode main" tid=1 file=wakunode2.nim:91 stacktrace="/app/vendor/nim-chronicles/chronicles/formats.nim(20) _ZN9wakunode213handleSigsegvE5int32\\n"', '/app/apps/wakunode2/wakunode2.nim(94) _ZN9wakunode213handleSigsegvE5int32', 'FAT 2026-03-10 04:35:52.822+00:00 Shutting down after receiving SIGSEGV topics="wakunode main" tid=1 file=wakunode2.nim:91 stacktrace="/app/vendor/nim-chronicles/chronicles/formats.nim(20) _ZN9wakunode213handleSigsegvE5int32\\n"', '/app/apps/wakunode2/wakunode2.nim(94) _ZN9wakunode213handleSigsegvE5int32']
2026-03-10 04:35:54.175 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-10 04:35:54.175 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-10 04:35:54.176 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-10_04-35-49__d0aeefb4-7b2f-4029-9bf7-0d5c7d117481__wakuorg_nwaku:latest.log
2026-03-10 04:35:54.181 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-10_04-35-49__d0aeefb4-7b2f-4029-9bf7-0d5c7d117481__wakuorg_nwaku:latest.log
2026-03-10 04:35:54.212 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-10 04:35:54.213 DEBUG [tests.conftest] Running test: test_multiple_pubsub_topics_same_cluster with id: 2026-03-10_04-35-54__397749d5-aab6-4892-ba14-f695d6c3d971
2026-03-10 04:35:54.213 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-10 04:35:54.214 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-10 04:35:54.215 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-10 04:35:54.227 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-10 04:35:54.228 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-10_04-35-54__397749d5-aab6-4892-ba14-f695d6c3d971__wakuorg_nwaku:latest.log
2026-03-10 04:35:54.229 DEBUG [src.node.waku_node] Starting Node...
2026-03-10 04:35:54.229 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-10 04:35:54.233 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-10 04:35:54.233 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.207.244
2026-03-10 04:35:54.234 DEBUG [src.node.docker_mananger] Generated ports ['45921', '45922', '45923', '45924', '45925']
2026-03-10 04:35:54.234 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-10 04:35:54.235 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-10 04:35:54.235 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-10 04:35:54.235 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-10 04:35:54.235 DEBUG [src.node.waku_node] Using volumes []
2026-03-10 04:35:54.236 DEBUG [src.node.docker_mananger] docker run -i -t -p 45921:45921 -p 45922:45922 -p 45923:45923 -p 45924:45924 -p 45925:45925 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=45923 --rest-port=45921 --tcp-port=45922 --discv5-udp-port=45924 --rest-address=0.0.0.0 --nat=extip:172.18.207.244 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=9a5579f3e897dea56f6fec0db2ce07ead3fafb203cec9b6a6bd10077e6a7f84d --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=45925 --metrics-logging=true --relay=true --filter=true
2026-03-10 04:35:54.444 ERROR [src.node.docker_mananger] Max retries reached for container 176fe4d6274b. Exiting log stream.
2026-03-10 04:35:54.446 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.207.244 waku 299e4b7f0e2a5e054bf42f1cebdb03435063038cd406461054e9e62ec0a7bf31
2026-03-10 04:35:54.481 DEBUG [src.node.docker_mananger] Container started with ID 299e4b7f0e2a. Setting up logs at ./log/docker/node1_2026-03-10_04-35-54__397749d5-aab6-4892-ba14-f695d6c3d971__wakuorg_nwaku:latest.log
2026-03-10 04:35:54.482 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 45921
2026-03-10 04:35:54.482 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-10 04:35:55.148 ERROR [src.node.docker_mananger] Max retries reached for container c89b551e9062. Exiting log stream.
2026-03-10 04:35:55.482 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45921/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:55.485 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-10 04:35:55.586 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45921/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:55.589 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-10 04:35:55.689 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45921/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:55.692 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-10 04:35:55.692 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-10 04:35:55.692 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45921/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:55.694 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.207.244/tcp/45922/p2p/16Uiu2HAmHqF9ANBFXVQTe1VGKgx9f9ASguosHERVFPCQ5gcJruQi","/ip4/172.18.207.244/tcp/45923/ws/p2p/16Uiu2HAmHqF9ANBFXVQTe1VGKgx9f9ASguosHERVFPCQ5gcJruQi"],"enrUri":"enr:-L24QGAMordpRPWOD2hreHxqOf2SnwO-GhDGN_BdI1KlCzKwMK9RCjRsmjNoacCUWAIKQyY-RdLvdIVa2ChL1pDIIRwCgmlkgnY0gmlwhKwSz_SKbXVsdGlhZGRyc5YACASsEs_0BrNiAAoErBLP9AazY90DgnJzhQACAQAAiXNlY3AyNTZrMaEDTOr3g4PiDDqIOGMlkiKwK6QFH9pksiqEPXSfGftM4AeDdGNwgrNig3VkcIKzZIV3YWt1MgU"}'
2026-03-10 04:35:55.695 INFO [src.node.waku_node] REST service is ready !!
2026-03-10 04:35:55.701 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-10 04:35:55.702 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-10_04-35-54__397749d5-aab6-4892-ba14-f695d6c3d971__wakuorg_nwaku:latest.log
2026-03-10 04:35:55.702 DEBUG [src.node.waku_node] Starting Node...
2026-03-10 04:35:55.702 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-10 04:35:55.703 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-10 04:35:55.703 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.122.189
2026-03-10 04:35:55.704 DEBUG [src.node.docker_mananger] Generated ports ['47062', '47063', '47064', '47065', '47066']
2026-03-10 04:35:55.704 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-10 04:35:55.704 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-10 04:35:55.704 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-10 04:35:55.704 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-10 04:35:55.704 DEBUG [src.node.waku_node] Using volumes []
2026-03-10 04:35:55.704 DEBUG [src.node.docker_mananger] docker run -i -t -p 47062:47062 -p 47063:47063 -p 47064:47064 -p 47065:47065 -p 47066:47066 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=47064 --rest-port=47062 --tcp-port=47063 --discv5-udp-port=47065 --rest-address=0.0.0.0 --nat=extip:172.18.122.189 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=5916bff1e6f21ea3cc884ec8dfc0d8ec6f3b2657fcefde7b3fbbbf6eb2933cd9 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=47066 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QGAMordpRPWOD2hreHxqOf2SnwO-GhDGN_BdI1KlCzKwMK9RCjRsmjNoacCUWAIKQyY-RdLvdIVa2ChL1pDIIRwCgmlkgnY0gmlwhKwSz_SKbXVsdGlhZGRyc5YACASsEs_0BrNiAAoErBLP9AazY90DgnJzhQACAQAAiXNlY3AyNTZrMaEDTOr3g4PiDDqIOGMlkiKwK6QFH9pksiqEPXSfGftM4AeDdGNwgrNig3VkcIKzZIV3YWt1MgU
2026-03-10 04:35:55.927 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.122.189 waku a88ac82b0b09bd7b70466a2e3513b2d29fa99103c346b8b598daab2c65c57538
2026-03-10 04:35:55.965 DEBUG [src.node.docker_mananger] Container started with ID a88ac82b0b09. Setting up logs at ./log/docker/node2_2026-03-10_04-35-54__397749d5-aab6-4892-ba14-f695d6c3d971__wakuorg_nwaku:latest.log
2026-03-10 04:35:55.966 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 47062
2026-03-10 04:35:55.966 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-10 04:35:56.967 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:47062/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:56.973 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-10 04:35:56.973 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-10 04:35:56.974 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:47062/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:56.981 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.122.189/tcp/47063/p2p/16Uiu2HAmJyDFpf94dSoWWdNkWFKN9J34o3mU7Krq6iryQcRMHskv","/ip4/172.18.122.189/tcp/47064/ws/p2p/16Uiu2HAmJyDFpf94dSoWWdNkWFKN9J34o3mU7Krq6iryQcRMHskv"],"enrUri":"enr:-L24QBhzV1LOTH6xLulmYip8swnIUSkMX9F_n4vK6lJYrxoFLwW_OQczKocgcxl6JDL0gof1E1X6c3gjvvL8ENUacA8CgmlkgnY0gmlwhKwSer2KbXVsdGlhZGRyc5YACASsEnq9BrfXAAoErBJ6vQa32N0DgnJzhQACAQAAiXNlY3AyNTZrMaEDXdErLsUsi7c4k81FMRl7_kDPWsMifLWWqPKT-3l4E_uDdGNwgrfXg3VkcIK32YV3YWt1MgE"}'
2026-03-10 04:35:56.982 INFO [src.node.waku_node] REST service is ready !!
2026-03-10 04:35:56.982 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:47062/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.207.244/tcp/45922/p2p/16Uiu2HAmHqF9ANBFXVQTe1VGKgx9f9ASguosHERVFPCQ5gcJruQi"]'
2026-03-10 04:35:57.010 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:35:57.012 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:45921/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0", "/waku/2/rs/2/1", "/waku/2/rs/2/2", "/waku/2/rs/2/3", "/waku/2/rs/2/4", "/waku/2/rs/2/5", "/waku/2/rs/2/6", "/waku/2/rs/2/7"]'
2026-03-10 04:35:57.030 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:35:57.032 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:47062/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0", "/waku/2/rs/2/1", "/waku/2/rs/2/2", "/waku/2/rs/2/3", "/waku/2/rs/2/4", "/waku/2/rs/2/5", "/waku/2/rs/2/6", "/waku/2/rs/2/7"]'
2026-03-10 04:35:57.038 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:35:57.041 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:45921/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-10 04:35:57.052 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:35:57.053 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-10 04:35:57.154 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-10 04:35:57.155 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45921/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:35:57.158 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773117353578959668,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773117353118795295,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773117357040106624,"ephemeral":false,"proof":""}]'
2026-03-10 04:35:57.166 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-10 04:35:57.167 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-10 04:35:57.167 DEBUG [src.node.waku_node] Stopping container with id 299e4b7f0e2a
2026-03-10 04:35:57.739 DEBUG [src.node.waku_node] Container stopped.
2026-03-10 04:35:57.741 DEBUG [src.node.waku_node] Stopping container with id a88ac82b0b09
2026-03-10 04:35:58.342 DEBUG [src.node.waku_node] Container stopped.
2026-03-10 04:35:58.344 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-10 04:35:58.383 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-10 04:35:58.419 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-10 04:35:58.421 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-10 04:35:58.421 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-10 04:35:58.422 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-10_04-35-54__397749d5-aab6-4892-ba14-f695d6c3d971__wakuorg_nwaku:latest.log
2026-03-10 04:35:58.423 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-10_04-35-54__397749d5-aab6-4892-ba14-f695d6c3d971__wakuorg_nwaku:latest.log