741 Commits

Author SHA1 Message Date
Ben
87e800db34
testing: chronos stream reads and writes 2025-01-09 11:52:48 +01:00
benbierens
d9a821e7be
messages numbers and send/recv logs 2025-01-08 15:43:45 +01:00
Ben
6e2d601ecd
more testing logs 2025-01-07 15:16:08 +01:00
benbierens
4048daa64b
testing in progress 2025-01-06 14:17:17 +01:00
benbierens
9d1471ce05
wiptesting 2024-12-20 10:16:14 +01:00
benbierens
23c0f30977
Review by Dmitriy: Remember peerWants only if we don't have them. 2024-12-20 09:10:30 +01:00
Ben Bierens
55dbcbe459
Merge branch 'master' into feature/blkexc-peer-selection 2024-12-20 08:28:58 +01:00
Adam Uhlíř
4b99b58645
fix: unhide log-format cli flag (#1049) 2024-12-19 12:03:50 +00:00
Ben Bierens
edb05f8959
Merge branch 'master' into feature/blkexc-peer-selection 2024-12-18 10:27:16 +01:00
Ben Bierens
a4dc802c9c
Merge branch 'master' into feature/blkexc-peer-selection 2024-12-18 10:07:54 +01:00
Eric
6ff4d30b43
fix(tests/integration): fix bootstrap node spr (#1048)
After a change in PR #1031, bootstrap node sprs may not work when Hardhat nodes are started with the tests. This fixes it by appending all started client's and provider's SPR to a sequence, and using that sequence of SPRs to start the next node. This means all subsequently started nodes will be connected to its previously started peers.

This also fixes the case when bootstrap SPRs would not be present if no clients were started.
2024-12-18 08:10:36 +00:00
Eric
8645d336ff
refactor(trackedfutures): remove return of future from tracked futures api (#1046)
- cleans up all instances of `.track` to use the `module.trackedfutures.track(future)` procedure, for better readability
- removes the `track` override that is no longer used in the codebase
2024-12-18 07:39:03 +00:00
Ben
4b5c35534d
Merge branch 'master' into feature/blkexc-peer-selection
# Conflicts:
#	tests/integration/testsales.nim
2024-12-17 15:13:59 +01:00
Ben Bierens
20bb5e5a38
Applies multinodessuite in twonodessuite (#1031)
* Sets timeout for codexclient httpClient. Adds reliable transfer test.

* disable new test to check timeout setting in CI

* restores new test

* adds heartbeat log and logfile to ci output

* fixes suite

* fixes blocked output stream by switching to multinode fixture

* new twonodessuite based on multinodesuite

* Applies updated twonodessuite

* removes heartbeat log

* applies multinodesuite in testsales

* applies multinodesuite in testmarketplace

* fixes account fetch for host and client in testmarketplace

* adds waitTillNextPeriod at end of marketplace test

* Uses marketplacesuite in testmarketplace
2024-12-17 13:01:41 +00:00
Ben Bierens
f0b1e098fc
Merge branch 'master' into feature/blkexc-peer-selection 2024-12-17 12:56:13 +01:00
Eric
c498e2f53b
fix(nodeprocess): asyncspawn capture output (#1045)
- Ensures no exceptions are raised from `captureOutput`
- Asyncspawns the future to ensure errors are not silently swallowed
2024-12-17 09:51:38 +00:00
Eric
f0f04ddf1d
refactor(then): removes then util (#1047)
- removes then util as it is no longer being used in the codebase
2024-12-17 09:51:05 +00:00
Ben Bierens
1ecd784d7b
Merge branch 'master' into feature/blkexc-peer-selection 2024-12-16 16:15:18 +01:00
Eric
bef1160799
fix(validation): asyncSpawns run (#1039)
- annotates run with raises: []
- asyncSpawns run, to ensure there are no escaping exceptions
2024-12-16 06:55:19 +00:00
Eric
b0cc27f563
fix(blockexchange): ensures futures are asyncSpawned (#1037)
* fix(blockexchange): asyncSpawn advertising of local store blocks

* fix(blockexchange): asyncSpawn discoveryQueueLoop

- prevents silently swallowing async errors

* fix(blockexchange): asyncSpawns block exchange tasks

- prevents silently swallow future exceptions
2024-12-16 06:01:49 +00:00
Eric
5f2ba14281
fix(codexnode): ensure timer loop is asyncSpawned (#1038)
* fix(codexnode): stop clock after validator stops

* fix(timer): ensure timer loop is asyncSpawned
2024-12-16 04:24:57 +00:00
Slava
6d415b0ace
ci: split nim-matrix workflow (#1041) 2024-12-15 23:31:55 +00:00
Eric
01fb685bf6
fix(sales): replaces then with asyncSpawn (#1036)
- ensures `addSlotToQueue` does not raise exceptions as it is now asyncSpawned
2024-12-15 23:19:31 +00:00
Marcin Czenko
92a0eda79a
Validator historical state restoration (#922)
* adds a new overload of queryPastEvents allowing to query past events based on timestamp in the past

* adds state restoration to validator

* refactors a bit to get the tests back to work

* replaces deprecated generic methods from Market with methods for specific event types

* Refactors binary search

* adds market tests for querying past SlotFilled events and binary search

* Takes into account that <<earliest>> block available is not necessarily the genesis block

* Adds more logging and makes testing earliest block boundary more reliable

* adds validation tests for historical state restoration

* adds mockprovider to simplify and improve testing of the edge conditions

* adds slot reservation to the new tests after rebasing

* adds validation groups and group index in logs of validator

* adds integration test with two validators

* adds comment on how to enable logging in integration test executable itself

* testIntegration: makes list is running nodes injected and available in the body of the test

* validation: adds integration test for historical state

* adds more logging to validator

* integration test: validator only looks 30 days back for historical state

* adds logging of the slotState when removing slots during validation

* review and refactor validator integration tests

* adds validation to the set of integration tests

* Fixes mistyped name of the mock provider module in testMarket

* Fixes a typo in the name of the validation suite in integration tests

* Makes validation unit test a bit easier to follow

* better use of logScopes to reduce duplication

* improves timing and clarifies the test conditions

* uses http as default RPC provider for nodes running in integration tests as a workaround for dropped subscriptions

* simplifies the validation integration tests by waiting for failed request instead of tracking slots

* adds config option allowing selectively to set different provider url

* Brings back the default settings for RPC provider in integration tests

* use http RPC provider for clients in validation integration tests

* fine-tune the tests

* Makes validator integration test more robust - adds extra tracking

* brings tracking of marketplace event back to validator integration test

* refactors integration tests

* deletes tmp file

* adds <<return>> after forcing integration test to fail preliminarily

* re-enables all integration tests and matrix

* stops debug output in CI

* allows to choose a different RPC provider for a given integration test suite

* fixes signature of <<getBlock>> method in mockProvider

* adds missing import which seem to be braking integration tests on windows

* makes sure that clients, SPs, and validators use the same provider url

* makes validator integration tests using http at 127.0.0.1:8545

* testvalidator: stop resubscribing as we are now using http polling as rpc provider

* applying review comments

* groups queryPastStorage overrides together (review comment)

* groups the historical validation tests into a sub suite

* removes the temporary extensions in marketplacesuite and multinodesuite allowing to specify provider url

* simplifies validation integration tests

* Removes debug logs when waiting for request to fail

* Renaming waitForRequestFailed => waitForRequestToFail

* renames blockNumberForBlocksAgo to pastBlockTag and makes it private

* removes redundant debugging logs

* refines logging in validation

* removes dev logging from mockmarket

* improves exception handling in provider helper procs and prepares for extraction to a separate module

* Uses chronos instead of std/times for Duration

* extracts provider and binary search helpers to a separate module

* removes redundant log entry params from validator

* unifies the notation to consistently use method call syntax

* reuses ProviderError from nim-ethers in the provider extension

* clarifies the comment in multinodesuite

* uses == operator to check the predefined tags and raises exception when `BlockTag.pending` is requested.

* when waiting for request to fail, we break on any request state that is not Started

* removes tests that were moved to testProvider from testMarket

* extracts tests that use MockProvider to a separate async suite

* improves performance of the historical state restoration

* removing redundant log messages in validator (groupIndex and groups)

* adds testProvider to testContracts group

* removes unused import in testMarket
2024-12-14 05:07:55 +00:00
Ben Bierens
9c2f3faa4d
Merge branch 'master' into feature/blkexc-peer-selection 2024-12-13 08:31:00 +01:00
Eric
1f49f86131
fix(slotqueue): asyncSpawns futures correctly (#1034)
- asyncSpawns `run` and worker `dispatch` in slotqueue.
- removes usage of `then` from slotqueue.
2024-12-13 06:42:05 +00:00
Eric
7c804b0ec9
fix(asyncstatemachine): fixes not awaiting or asyncSpawning futures (#1033)
- adds a break in scheduler when CancelledError is caught
- tracks asyncSpawned state.run, so that it can be cancelled during stop
- removes usages of `then`
- ensures that no exceptions are leaked from async procs
2024-12-13 02:35:39 +00:00
Adam Uhlíř
19af79786e
feat: repair is rewarded (#1022)
* feat: repair is rewarded

* chore: update contracts repo

* feat: proving loop handles repair case

* test: assert repair state

* chore: update contracts repo

* fix: upon unknown state of repair go to error
2024-12-12 20:19:56 +00:00
Adam Uhlíř
d10072bf67
refactor: marketplace configuration is cached (#1029) 2024-12-12 12:57:34 +00:00
Eric
da234d503b
fix(trackedfutures): removes usage of then from tracked futures (#1032)
- removes usage of `then`, simplifying the logic, and allowing `then` to be removed completely
- updates annotations to reflect that all procs (sync and async) raise no exceptions
2024-12-12 12:31:51 +00:00
Vaclav Pavlin
855b973811
chore: fix inconsistent metric naming (#1027) 2024-12-12 10:45:47 +00:00
Ben
960c0107f0
fixtest: testsales enable logging 2024-12-10 11:05:59 +01:00
Ben Bierens
cc1121c9f0
Merge branch 'master' into feature/blkexc-peer-selection 2024-12-10 09:30:14 +01:00
Marko Burčul
0c6784da7e
nix: make derivation and update shell (#1003)
* nix: make derivation and update shell

Create a structure for nix files. Add the derivation file which is using
system Nim to compile Codex.

Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>

* nim-circom-compat: update

Include commit which allows building circom-compat-ffi using Nix(doesn't
affect current usage of the submodule).

Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>

* makefile: fix for detecting linux arch

Signed-off-by: markoburcul <marko@status.im>

---------

Signed-off-by: markoburcul <marko@status.im>
2024-12-09 17:07:01 +00:00
Ben
c3c04e84df
Marks cancel-presence situation with todo comment. 2024-12-09 09:14:49 +01:00
Ben Bierens
7e4182ef12
Merge branch 'master' into feature/blkexc-peer-selection 2024-12-07 13:09:49 +01:00
benbierens
dcf43d3227
fast 2024-12-06 12:41:17 +01:00
Eric
fb4577f25c
chore(tests): fix unneeded async (#1021)
Removes an unneeded `check eventually` inside a helper proc, removing the need for the proc to be async.
2024-12-06 08:02:57 +00:00
benbierens
1f063fe163
prevents duplicate scheduling. Fixes cancellation 2024-12-05 19:52:57 +01:00
benbierens
532b83a52b
consistent logging of addresses 2024-12-05 15:20:25 +01:00
Ben Bierens
d32e052deb
Merge branch 'master' into feature/blkexc-peer-selection 2024-12-05 14:40:31 +01:00
Ben Bierens
f51eae30fe
Fixes raceconditions in testadvertiser (#1008) 2024-12-05 09:30:01 +00:00
benbierens
ed3e91cbde
Review comments by Dmitriy 2024-12-05 10:21:32 +01:00
Ben
64e691b824
Fixes issue where peerWants are only stored for type wantBlock. 2024-12-04 16:04:08 +01:00
Ben
7c2bff79c0
Cleanup cheapestPeer. Fixes test for peers lists 2024-12-04 16:04:08 +01:00
Ben
3607b88e9d
sends wantBlock to peers with block. wantHave to everyone else 2024-12-04 16:04:08 +01:00
Ben Bierens
8e29939cf8
Send pluralized wantBlock messages (#1016)
* don't unroll wantCids when sending wantBlock message in blockPresenceHandler

* workaround logging

* Fixes logformatting upraises for sequences.

* Applies upraises rule for setProperty of textmode for sequences.

* Replaces upraises with raises

* Removes redundant log in sendWantHave
2024-12-04 13:33:48 +00:00
Slava
921159f87f
Run release tests for docker images (#1017)
* Refactor Docker reusable workflow and add release tests support

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Add an option to run release tests for Docker images

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Use bigger instance for arm builds

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Pass repository and branch to release tests workflow

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Do not use computation job because run_release_tests is a string

https://github.com/codex-storage/cs-codex-dist-tests/issues/108
2024-12-04 11:52:51 +00:00
Ben Bierens
63e54d135c
Fixes race condition in initial-proving tests (#1007) 2024-12-04 08:56:53 +00:00
Adam Uhlíř
0707446cdd
feat: expose underlying nim-ethers errors to logs (#985)
* feat: expose underlying nim-ethers errors to logs

* chore: bump nim-ethers

* test: fix testproof compilation

* test: raise defects on results error
2024-12-03 11:16:24 +00:00