109 lines
18 KiB
Plaintext

2026-03-10 04:36:06.473 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-10 04:36:06.474 DEBUG [tests.conftest] Running test: test_cursor_pointing_to_deleted_message with id: 2026-03-10_04-36-06__6f956993-9663-44cf-a973-e1921440f23a
2026-03-10 04:36:06.474 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-10 04:36:06.474 DEBUG [src.steps.store] Running fixture setup: store_setup
2026-03-10 04:36:06.474 DEBUG [src.steps.store] Running fixture setup: node_setup
2026-03-10 04:36:06.481 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-10 04:36:06.481 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/publishing_node1_2026-03-10_04-36-06__6f956993-9663-44cf-a973-e1921440f23a__wakuorg_nwaku:latest.log
2026-03-10 04:36:06.481 DEBUG [src.node.waku_node] Starting Node...
2026-03-10 04:36:06.482 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-10 04:36:06.483 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-10 04:36:06.483 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.212.48
2026-03-10 04:36:06.483 DEBUG [src.node.docker_mananger] Generated ports ['48619', '48620', '48621', '48622', '48623']
2026-03-10 04:36:06.484 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-10 04:36:06.484 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-10 04:36:06.484 DEBUG [src.node.waku_node] Using volumes []
2026-03-10 04:36:06.484 DEBUG [src.node.docker_mananger] docker run -i -t -p 48619:48619 -p 48620:48620 -p 48621:48621 -p 48622:48622 -p 48623:48623 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=48621 --rest-port=48619 --tcp-port=48620 --discv5-udp-port=48622 --rest-address=0.0.0.0 --nat=extip:172.18.212.48 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=cf8cbbb75aed5704b6d40ba159dba6844baddf3eadcd99b97a4f9d30cf9f7f48 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=48623 --metrics-logging=true --store=true --relay=true
2026-03-10 04:36:06.678 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.212.48 waku 83ade48cf8e93b596b80cf125b46cf8c5991bd647504cad11dd5c83ad8c887ce
2026-03-10 04:36:06.713 DEBUG [src.node.docker_mananger] Container started with ID 83ade48cf8e9. Setting up logs at ./log/docker/publishing_node1_2026-03-10_04-36-06__6f956993-9663-44cf-a973-e1921440f23a__wakuorg_nwaku:latest.log
2026-03-10 04:36:06.713 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 48619
2026-03-10 04:36:06.713 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-10 04:36:06.728 ERROR [src.node.docker_mananger] Max retries reached for container 40c16b59e269. Exiting log stream.
2026-03-10 04:36:07.269 ERROR [src.node.docker_mananger] Max retries reached for container ff4697b344f9. Exiting log stream.
2026-03-10 04:36:07.714 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:48619/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:07.717 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-10 04:36:07.717 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-10 04:36:07.717 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:48619/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:07.720 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.212.48/tcp/48620/p2p/16Uiu2HAmUatNKgUwirHRV4ULGJBUxvWHpFtXkBVN5m3V1nJBFBrp","/ip4/172.18.212.48/tcp/48621/ws/p2p/16Uiu2HAmUatNKgUwirHRV4ULGJBUxvWHpFtXkBVN5m3V1nJBFBrp"],"enrUri":"enr:-L24QGbaYhs06M8Um26qEeWYQbcyGY_mB5Lu-7C0fWGBs1ZmW2keBNHggroD242y38rafpx9NiBlo_FPb9zRsK2tTKYCgmlkgnY0gmlwhKwS1DCKbXVsdGlhZGRyc5YACASsEtQwBr3sAAoErBLUMAa97d0DgnJzhQADAQAAiXNlY3AyNTZrMaED7K3nS-4oEIRhAnOD9KIXV7DjDN_Y0IcpLkmBAZxDeGGDdGNwgr3sg3VkcIK97oV3YWt1MgM"}'
2026-03-10 04:36:07.720 INFO [src.node.waku_node] REST service is ready !!
2026-03-10 04:36:07.727 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-10 04:36:07.727 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/store_node1_2026-03-10_04-36-06__6f956993-9663-44cf-a973-e1921440f23a__wakuorg_nwaku:latest.log
2026-03-10 04:36:07.727 DEBUG [src.node.waku_node] Starting Node...
2026-03-10 04:36:07.728 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-10 04:36:07.729 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-10 04:36:07.729 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.229.105
2026-03-10 04:36:07.729 DEBUG [src.node.docker_mananger] Generated ports ['54150', '54151', '54152', '54153', '54154']
2026-03-10 04:36:07.729 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-10 04:36:07.730 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-10 04:36:07.730 DEBUG [src.node.waku_node] Using volumes []
2026-03-10 04:36:07.730 DEBUG [src.node.docker_mananger] docker run -i -t -p 54150:54150 -p 54151:54151 -p 54152:54152 -p 54153:54153 -p 54154:54154 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=54152 --rest-port=54150 --tcp-port=54151 --discv5-udp-port=54153 --rest-address=0.0.0.0 --nat=extip:172.18.229.105 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=6374b0addb7ea79de7456add1bdce9aeb7aeaec5fba7ef2765161c279feffbaf --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=54154 --metrics-logging=true --discv5-bootstrap-node=enr:-L24QGbaYhs06M8Um26qEeWYQbcyGY_mB5Lu-7C0fWGBs1ZmW2keBNHggroD242y38rafpx9NiBlo_FPb9zRsK2tTKYCgmlkgnY0gmlwhKwS1DCKbXVsdGlhZGRyc5YACASsEtQwBr3sAAoErBLUMAa97d0DgnJzhQADAQAAiXNlY3AyNTZrMaED7K3nS-4oEIRhAnOD9KIXV7DjDN_Y0IcpLkmBAZxDeGGDdGNwgr3sg3VkcIK97oV3YWt1MgM --storenode=/ip4/172.18.212.48/tcp/48620/p2p/16Uiu2HAmUatNKgUwirHRV4ULGJBUxvWHpFtXkBVN5m3V1nJBFBrp --store=true --relay=true
2026-03-10 04:36:07.923 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.229.105 waku 0fbe71f4c0be6718f63c8ee8623e5dcb832ac162a38416bf48f625bbfeafd843
2026-03-10 04:36:07.958 DEBUG [src.node.docker_mananger] Container started with ID 0fbe71f4c0be. Setting up logs at ./log/docker/store_node1_2026-03-10_04-36-06__6f956993-9663-44cf-a973-e1921440f23a__wakuorg_nwaku:latest.log
2026-03-10 04:36:07.958 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 54150
2026-03-10 04:36:07.959 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-10 04:36:08.960 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54150/health" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:08.963 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-10 04:36:08.964 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-10 04:36:08.964 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54150/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:08.966 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.229.105/tcp/54151/p2p/16Uiu2HAmHPWe2QNj1AR2PrC2QNSwJeBBp64RStKcpXqUiUiU69LK","/ip4/172.18.229.105/tcp/54152/ws/p2p/16Uiu2HAmHPWe2QNj1AR2PrC2QNSwJeBBp64RStKcpXqUiUiU69LK"],"enrUri":"enr:-L24QC5Oq6wrYOMWlcAG8WqyPsf91353E1NdgpV1P7ecL_yfF04TjODPsld3uSOxIuR0lBIwrQycxL00I9uhf0ecqcoCgmlkgnY0gmlwhKwS5WmKbXVsdGlhZGRyc5YACASsEuVpBtOHAAoErBLlaQbTiN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDRlNjmGbuH1Ur5xBkqsYQvWpVgssf2kES9KIFcq7dmdiDdGNwgtOHg3VkcILTiYV3YWt1MgM"}'
2026-03-10 04:36:08.967 INFO [src.node.waku_node] REST service is ready !!
2026-03-10 04:36:08.967 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:54150/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.212.48/tcp/48620/p2p/16Uiu2HAmUatNKgUwirHRV4ULGJBUxvWHpFtXkBVN5m3V1nJBFBrp"]'
2026-03-10 04:36:08.970 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:08.970 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48619/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/0"]'
2026-03-10 04:36:08.973 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:08.973 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:54150/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/0"]'
2026-03-10 04:36:08.975 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:08.976 DEBUG [src.steps.store] Relaying message
2026-03-10 04:36:08.976 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48619/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "TWVzc2FnZV8w", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-10 04:36:08.981 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:08.981 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-10 04:36:09.182 DEBUG [src.steps.store] Relaying message
2026-03-10 04:36:09.182 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48619/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "TWVzc2FnZV8x", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-10 04:36:09.188 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:09.189 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-10 04:36:09.390 DEBUG [src.steps.store] Relaying message
2026-03-10 04:36:09.390 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48619/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "TWVzc2FnZV8y", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-10 04:36:09.396 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:09.397 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-10 04:36:09.598 DEBUG [src.steps.store] Relaying message
2026-03-10 04:36:09.598 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48619/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "TWVzc2FnZV8z", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-10 04:36:09.604 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:09.604 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-10 04:36:09.805 DEBUG [src.steps.store] Relaying message
2026-03-10 04:36:09.806 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48619/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "TWVzc2FnZV80", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-10 04:36:09.812 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:09.812 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-10 04:36:10.013 DEBUG [src.steps.store] Relaying message
2026-03-10 04:36:10.013 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48619/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "TWVzc2FnZV81", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-10 04:36:10.019 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:10.019 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-10 04:36:10.220 DEBUG [src.steps.store] Relaying message
2026-03-10 04:36:10.220 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48619/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "TWVzc2FnZV82", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-10 04:36:10.226 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:10.226 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-10 04:36:10.428 DEBUG [src.steps.store] Relaying message
2026-03-10 04:36:10.428 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48619/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "TWVzc2FnZV83", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-10 04:36:10.435 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:10.436 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-10 04:36:10.636 DEBUG [src.steps.store] Relaying message
2026-03-10 04:36:10.637 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48619/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "TWVzc2FnZV84", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-10 04:36:10.642 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:10.643 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-10 04:36:10.843 DEBUG [src.steps.store] Relaying message
2026-03-10 04:36:10.844 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48619/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d '{"payload": "TWVzc2FnZV85", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-10 04:36:10.850 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-10 04:36:10.851 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-10 04:36:11.052 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:48619/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F0&contentTopics=%2Fmyapp%2F1%2Flatest%2Fproto&cursor=0xb6c1942990909c53f072ab49569b42be4498accca6fe9ef0530966ed88fd5056&pageSize=100&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:11.055 ERROR [src.node.api_clients.base_client] HTTP error occurred: 500 Server Error: Internal Server Error for url: http://127.0.0.1:48619/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F0&contentTopics=%2Fmyapp%2F1%2Flatest%2Fproto&cursor=0xb6c1942990909c53f072ab49569b42be4498accca6fe9ef0530966ed88fd5056&pageSize=100&ascending=true. Response content: b'error in handleSelfStoreRequest: BAD_RESPONSE: archive error: DRIVER_ERROR: cursor not found'
2026-03-10 04:36:11.055 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54150/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F0&contentTopics=%2Fmyapp%2F1%2Flatest%2Fproto&cursor=0xb6c1942990909c53f072ab49569b42be4498accca6fe9ef0530966ed88fd5056&pageSize=100&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-03-10 04:36:11.058 ERROR [src.node.api_clients.base_client] HTTP error occurred: 500 Server Error: Internal Server Error for url: http://127.0.0.1:54150/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F0&contentTopics=%2Fmyapp%2F1%2Flatest%2Fproto&cursor=0xb6c1942990909c53f072ab49569b42be4498accca6fe9ef0530966ed88fd5056&pageSize=100&ascending=true. Response content: b'error in handleSelfStoreRequest: BAD_RESPONSE: archive error: DRIVER_ERROR: cursor not found'
2026-03-10 04:36:11.060 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-10 04:36:11.061 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-10 04:36:11.061 DEBUG [src.node.waku_node] Stopping container with id 83ade48cf8e9
2026-03-10 04:36:11.603 DEBUG [src.node.waku_node] Container stopped.
2026-03-10 04:36:11.604 DEBUG [src.node.waku_node] Stopping container with id 0fbe71f4c0be
2026-03-10 04:36:12.133 DEBUG [src.node.waku_node] Container stopped.
2026-03-10 04:36:12.136 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-10 04:36:12.145 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-10 04:36:12.152 DEBUG [src.node.docker_mananger] No errors found in the waku logs.