110 lines
18 KiB
Plaintext

2026-02-26 04:41:53.390 DEBUG [tests.conftest] Running fixture setup: test_id
2026-02-26 04:41:53.390 DEBUG [tests.conftest] Running test: test_sync_flags_node2_start_later with id: 2026-02-26_04-41-53__4b7f6921-1ff0-4a58-97d1-b5654c9a20a5
2026-02-26 04:41:53.391 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-02-26 04:41:53.397 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-26 04:41:53.397 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-02-26_04-41-53__4b7f6921-1ff0-4a58-97d1-b5654c9a20a5__wakuorg_nwaku:latest.log
2026-02-26 04:41:53.404 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-26 04:41:53.404 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-02-26_04-41-53__4b7f6921-1ff0-4a58-97d1-b5654c9a20a5__wakuorg_nwaku:latest.log
2026-02-26 04:41:53.410 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-26 04:41:53.410 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-02-26_04-41-53__4b7f6921-1ff0-4a58-97d1-b5654c9a20a5__wakuorg_nwaku:latest.log
2026-02-26 04:41:53.410 DEBUG [src.steps.store] Running fixture setup: store_setup
2026-02-26 04:41:53.411 DEBUG [src.node.waku_node] Starting Node...
2026-02-26 04:41:53.411 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-26 04:41:53.414 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-26 04:41:53.414 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.14.131
2026-02-26 04:41:53.415 DEBUG [src.node.docker_mananger] Generated ports ['59743', '59744', '59745', '59746', '59747']
2026-02-26 04:41:53.415 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-26 04:41:53.415 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-26 04:41:53.415 DEBUG [src.node.waku_node] Using volumes []
2026-02-26 04:41:53.415 DEBUG [src.node.docker_mananger] docker run -i -t -p 59743:59743 -p 59744:59744 -p 59745:59745 -p 59746:59746 -p 59747:59747 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=59745 --rest-port=59743 --tcp-port=59744 --discv5-udp-port=59746 --rest-address=0.0.0.0 --nat=extip:172.18.14.131 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=aacf1cccf91dcba4543eea0cbc177eafa9126afa6b40bbc7a2fffaebcc926d1e --store-sync=true --store=true --store-sync-range=45 --store-sync-interval=10 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=59747 --metrics-logging=true --store-sync-relay-jitter=0 --relay=true
2026-02-26 04:41:53.610 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.14.131 waku 3b76af528fcccaebf09582706505718eb4442ab5cadfeb5125092bc461135320
2026-02-26 04:41:53.618 ERROR [src.node.docker_mananger] Max retries reached for container 46a692f615bf. Exiting log stream.
2026-02-26 04:41:53.645 DEBUG [src.node.docker_mananger] Container started with ID 3b76af528fcc. Setting up logs at ./log/docker/node1_2026-02-26_04-41-53__4b7f6921-1ff0-4a58-97d1-b5654c9a20a5__wakuorg_nwaku:latest.log
2026-02-26 04:41:53.645 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 59743
2026-02-26 04:41:53.645 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-26 04:41:54.163 ERROR [src.node.docker_mananger] Max retries reached for container 97cb002c6511. Exiting log stream.
2026-02-26 04:41:54.647 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59743/health" -H "Content-Type: application/json" -d 'None'
2026-02-26 04:41:54.649 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-26 04:41:54.650 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-26 04:41:54.650 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59743/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-26 04:41:54.652 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.14.131/tcp/59744/p2p/16Uiu2HAmCuc7NVBvzFcbLRYCUwjKz7QRShVfssGpWzJquD7wpTth","/ip4/172.18.14.131/tcp/59745/ws/p2p/16Uiu2HAmCuc7NVBvzFcbLRYCUwjKz7QRShVfssGpWzJquD7wpTth"],"enrUri":"enr:-L24QOD5oneXL5oEce10FDecBLvNc8Rr8PbYDiJUQBbEEddkOnJja4BXytn5uCGDBt0tZaW3G78qtlCBs9th2jVtJ4gCgmlkgnY0gmlwhKwSDoOKbXVsdGlhZGRyc5YACASsEg6DBulgAAoErBIOgwbpYd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDA76QuHxePBFSf_PTzkUkbGKhEOycYaKGTDtbEjmSpNaDdGNwgulgg3VkcILpYoV3YWt1MhM"}'
2026-02-26 04:41:54.652 INFO [src.node.waku_node] REST service is ready !!
2026-02-26 04:41:54.653 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59743/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/0"]'
2026-02-26 04:41:54.668 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-26 04:41:54.668 DEBUG [src.steps.store] Relaying message
2026-02-26 04:41:54.669 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59743/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-02-26 04:41:54.672 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-26 04:41:54.672 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-02-26 04:41:54.873 DEBUG [src.steps.store] Relaying message
2026-02-26 04:41:54.873 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59743/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-02-26 04:41:54.877 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-26 04:41:54.877 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-02-26 04:41:55.077 DEBUG [src.steps.store] Relaying message
2026-02-26 04:41:55.078 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59743/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-02-26 04:41:55.081 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-26 04:41:55.081 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-02-26 04:41:55.282 DEBUG [src.steps.store] Relaying message
2026-02-26 04:41:55.282 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59743/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-02-26 04:41:55.285 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-26 04:41:55.286 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-02-26 04:41:55.486 DEBUG [src.steps.store] Relaying message
2026-02-26 04:41:55.486 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59743/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-02-26 04:41:55.490 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-26 04:41:55.490 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-02-26 04:41:55.690 DEBUG [src.steps.store] Relaying message
2026-02-26 04:41:55.691 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59743/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-02-26 04:41:55.694 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-26 04:41:55.695 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-02-26 04:41:55.895 DEBUG [src.steps.store] Relaying message
2026-02-26 04:41:55.895 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59743/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-02-26 04:41:55.899 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-26 04:41:55.899 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-02-26 04:41:56.100 DEBUG [src.steps.store] Relaying message
2026-02-26 04:41:56.100 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59743/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-02-26 04:41:56.103 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-26 04:41:56.104 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-02-26 04:41:56.304 DEBUG [src.steps.store] Relaying message
2026-02-26 04:41:56.304 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59743/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-02-26 04:41:56.308 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-26 04:41:56.308 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-02-26 04:41:56.508 DEBUG [src.steps.store] Relaying message
2026-02-26 04:41:56.509 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59743/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "U3RvcmUgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-02-26 04:41:56.512 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-26 04:41:56.512 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-02-26 04:41:56.713 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-26 04:41:57.713 DEBUG [src.node.waku_node] Starting Node...
2026-02-26 04:41:57.713 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-26 04:41:57.715 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-26 04:41:57.715 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.117.248
2026-02-26 04:41:57.715 DEBUG [src.node.docker_mananger] Generated ports ['51307', '51308', '51309', '51310', '51311']
2026-02-26 04:41:57.716 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-26 04:41:57.716 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-26 04:41:57.716 DEBUG [src.node.waku_node] Using volumes []
2026-02-26 04:41:57.716 DEBUG [src.node.docker_mananger] docker run -i -t -p 51307:51307 -p 51308:51308 -p 51309:51309 -p 51310:51310 -p 51311:51311 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=51309 --rest-port=51307 --tcp-port=51308 --discv5-udp-port=51310 --rest-address=0.0.0.0 --nat=extip:172.18.117.248 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=ce7dde75e6fc1861631ca1405d1ffc1aaf62da98db8dacd1e1b3908caa2178bd --store-sync=true --store=true --store-sync-range=45 --store-sync-interval=10 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=51311 --metrics-logging=true --store-sync-relay-jitter=0 --relay=false --discv5-bootstrap-node=enr:-L24QOD5oneXL5oEce10FDecBLvNc8Rr8PbYDiJUQBbEEddkOnJja4BXytn5uCGDBt0tZaW3G78qtlCBs9th2jVtJ4gCgmlkgnY0gmlwhKwSDoOKbXVsdGlhZGRyc5YACASsEg6DBulgAAoErBIOgwbpYd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDA76QuHxePBFSf_PTzkUkbGKhEOycYaKGTDtbEjmSpNaDdGNwgulgg3VkcILpYoV3YWt1MhM
2026-02-26 04:41:57.916 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.117.248 waku ec2c3c221120ae4d5269ced8b16fedc7c97d5d827a820c8622370cb492fe70ba
2026-02-26 04:41:57.961 DEBUG [src.node.docker_mananger] Container started with ID ec2c3c221120. Setting up logs at ./log/docker/node2_2026-02-26_04-41-53__4b7f6921-1ff0-4a58-97d1-b5654c9a20a5__wakuorg_nwaku:latest.log
2026-02-26 04:41:57.961 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 51307
2026-02-26 04:41:57.962 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-26 04:41:58.962 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51307/health" -H "Content-Type: application/json" -d 'None'
2026-02-26 04:41:58.965 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-26 04:41:58.965 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-26 04:41:58.965 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51307/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-26 04:41:58.968 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.117.248/tcp/51308/p2p/16Uiu2HAmAPL7bQdm92uYLu8MkcwstFTote7dXugcuEUrDGubZQZe","/ip4/172.18.117.248/tcp/51309/ws/p2p/16Uiu2HAmAPL7bQdm92uYLu8MkcwstFTote7dXugcuEUrDGubZQZe"],"enrUri":"enr:-L24QPeQ-PxuT5uF-1v3MwZEwcJqVi8CQXd1Xj23mBncBbn5N-CeX516erFdknotwlNtEqXuBXlW5nwQ0v6mcXHWkEcCgmlkgnY0gmlwhKwSdfiKbXVsdGlhZGRyc5YACASsEnX4BshsAAoErBJ1-AbIbd0DgnJzhQADAQAAiXNlY3AyNTZrMaEC3kWuJonNOmtCFP2ijS0-UP0ZhJRpHRP0FkFgYoVuSgGDdGNwgshsg3VkcILIboV3YWt1MhI"}'
2026-02-26 04:41:58.968 INFO [src.node.waku_node] REST service is ready !!
2026-02-26 04:41:58.968 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:51307/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.14.131/tcp/59744/p2p/16Uiu2HAmCuc7NVBvzFcbLRYCUwjKz7QRShVfssGpWzJquD7wpTth"]'
2026-02-26 04:41:58.971 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-26 04:41:58.971 DEBUG [src.libs.common] Sleeping for 65 seconds
2026-02-26 04:43:03.972 DEBUG [src.steps.store] Checking that peer wakuorg/nwaku:latest can find the stored messages
2026-02-26 04:43:03.972 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51307/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F0&pageSize=100&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-02-26 04:43:03.976 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0x87589bb9afad154e230842eb2312cb0757c2b6a637fd93689636c8757ff511b0"},{"messageHash":"0xa36722790795c8b51e73db4f377adeb05863c95c4f1ce067f3b97592a198cc56"},{"messageHash":"0x2f95319df404c79db3353296c5ec84faf37d3a557e2301bab0a198586e2b9cb2"},{"messageHash":"0xa376340bb5611fa26df401e279e17a29fc52d66b8ae4e4ab4cf2048a692d1c0b"},{"messageHash":"0xa64741bc4239344e2796bdde4e64a49eb2dd668df60588c56ab98d0884ab0259"},{"messageHash":"0x9363bd0d1b672c48a2bcd624d5995586c7348db86297bc17bb625da48e81f21c"},{"messageHash":"0xbd8d01299aa0661030597154ffd7bb93cc431b73a4a7d177a840190500cb6460"},{"messageHash":"0xa88f12c7a0ec5ae32f0f149addb34d571dc2666c21926efb7e06e7ed4dbdb9d6"},{"messageHash":"0x998e19067f50c8f8b80d90bb4f12759976d69bf88edf9c4c3b9a29f6a1502347"},{"messageHash":"0x262fbe7f5c33d875632e3e4b83d4ee045de06039b636854329c7960497434348"}]}'
2026-02-26 04:43:03.976 DEBUG [src.steps.store] messages length is 10
2026-02-26 04:43:03.978 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-02-26 04:43:03.979 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-02-26 04:43:03.980 DEBUG [src.node.waku_node] Stopping container with id 3b76af528fcc
2026-02-26 04:43:04.537 DEBUG [src.node.waku_node] Container stopped.
2026-02-26 04:43:04.537 DEBUG [src.node.waku_node] Stopping container with id ec2c3c221120
2026-02-26 04:43:05.107 DEBUG [src.node.waku_node] Container stopped.
2026-02-26 04:43:05.110 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-02-26 04:43:05.144 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-26 04:43:05.172 DEBUG [src.node.docker_mananger] No errors found in the waku logs.