Commit Graph

745 Commits

Author SHA1 Message Date
Arnaud 56d2186655
Move to miniupnp version 2.2.4 to avoid symlink error 2025-01-03 17:53:09 +01:00
Arnaud 696f66d002
Set nim version to 2.0.x for CI 2025-01-02 10:00:10 +01:00
Arnaud d9eca3b93e
Set nim version to 2.0.x for CI 2025-01-02 09:43:47 +01:00
Arnaud b9222d0415
Merge branch 'master' into chore/update-nim-version
# Conflicts:
#	codex/contracts/clock.nim
#	codex/contracts/market.nim
#	codex/utils/asyncstatemachine.nim
#	tests/integration/nodeprocess.nim
#	tests/integration/testecbug.nim
#	tests/integration/testmarketplace.nim
#	tests/integration/testproofs.nim
2024-12-30 18:33:31 +01:00
Arnaud d2388a32fc
Fix cast for value which is already an option 2024-12-27 12:09:40 +01:00
Arnaud 3f9f421c74
Update the tests using ValidIpAddress to IpAddress 2024-12-26 07:21:57 +01:00
Arnaud ab39da8230
Extract node components to a variable to make Nim 2 happy 2024-12-26 07:20:33 +01:00
Arnaud 51a40548d1
Fix camel case 2024-12-26 07:19:59 +01:00
Arnaud 627e6ae5a6
Fix enums in tests 2024-12-26 07:19:44 +01:00
Arnaud 6b64c253f1
Fix camel case 2024-12-26 07:19:06 +01:00
Arnaud 5c80a84982
Copy LPStreamReadError code from libp2p which was removed 2024-12-26 07:18:34 +01:00
Arnaud 6ad496dcba
Update method signatures 2024-12-26 07:18:00 +01:00
Arnaud 0724d7f8f1
Restore maintenance parameter deleted by mistake when removing esasure coding fix attempt code 2024-12-26 07:15:18 +01:00
Arnaud d6b1781b34
Add gcsafe pragma 2024-12-26 07:14:28 +01:00
Adam Uhlíř 4b99b58645
fix: unhide log-format cli flag (#1049) 2024-12-19 12:03:50 +00:00
Arnaud daadffe30a
Replace ValidIpAddress by IpAddress 2024-12-18 17:17:49 +01:00
Arnaud 83b4943c17
newLPStreamReadError does not exist anymore so use another error 2024-12-18 16:51:12 +01:00
Arnaud 902c6dc692
Use alreadySeen because need is not a bool anymore 2024-12-18 16:50:26 +01:00
Arnaud a2f09b2296
Fix camelcase function names 2024-12-18 16:49:48 +01:00
Arnaud a2f0528dc4
Duplicate code from libp2p because it is not exported anymore 2024-12-18 16:48:56 +01:00
Arnaud 599e868bc1
More gcsafe pragma 2024-12-18 16:48:15 +01:00
Arnaud a1c8c94ada
Revert erasure coding attempt to fix bug 2024-12-18 16:47:21 +01:00
Arnaud 4935494c29
Fix method signatures 2024-12-18 16:45:25 +01:00
Arnaud 0ecf3f7a78
Update specific conf for Nim 2.x 2024-12-18 16:43:38 +01:00
Arnaud 0e00f0c7cb
Add missing import 2024-12-18 16:41:42 +01:00
Arnaud 7c9b607e38
Add gcsafe pragma 2024-12-18 16:37:12 +01:00
Arnaud 34f5e0071c
Update Nim version to 2.0.12 2024-12-18 16:28:49 +01:00
Arnaud ef3274b290
Update dependencies 2024-12-18 16:25:52 +01:00
Eric 6ff4d30b43
fix(tests/integration): fix bootstrap node spr (#1048)
After a change in PR #1031, bootstrap node sprs may not work when Hardhat nodes are started with the tests. This fixes it by appending all started client's and provider's SPR to a sequence, and using that sequence of SPRs to start the next node. This means all subsequently started nodes will be connected to its previously started peers.

This also fixes the case when bootstrap SPRs would not be present if no clients were started.
2024-12-18 08:10:36 +00:00
Eric 8645d336ff
refactor(trackedfutures): remove return of future from tracked futures api (#1046)
- cleans up all instances of `.track` to use the `module.trackedfutures.track(future)` procedure, for better readability
- removes the `track` override that is no longer used in the codebase
2024-12-18 07:39:03 +00:00
Ben Bierens 20bb5e5a38
Applies multinodessuite in twonodessuite (#1031)
* Sets timeout for codexclient httpClient. Adds reliable transfer test.

* disable new test to check timeout setting in CI

* restores new test

* adds heartbeat log and logfile to ci output

* fixes suite

* fixes blocked output stream by switching to multinode fixture

* new twonodessuite based on multinodesuite

* Applies updated twonodessuite

* removes heartbeat log

* applies multinodesuite in testsales

* applies multinodesuite in testmarketplace

* fixes account fetch for host and client in testmarketplace

* adds waitTillNextPeriod at end of marketplace test

* Uses marketplacesuite in testmarketplace
2024-12-17 13:01:41 +00:00
Eric c498e2f53b
fix(nodeprocess): asyncspawn capture output (#1045)
- Ensures no exceptions are raised from `captureOutput`
- Asyncspawns the future to ensure errors are not silently swallowed
2024-12-17 09:51:38 +00:00
Eric f0f04ddf1d
refactor(then): removes then util (#1047)
- removes then util as it is no longer being used in the codebase
2024-12-17 09:51:05 +00:00
Eric bef1160799
fix(validation): asyncSpawns run (#1039)
- annotates run with raises: []
- asyncSpawns run, to ensure there are no escaping exceptions
2024-12-16 06:55:19 +00:00
Eric b0cc27f563
fix(blockexchange): ensures futures are asyncSpawned (#1037)
* fix(blockexchange): asyncSpawn advertising of local store blocks

* fix(blockexchange): asyncSpawn discoveryQueueLoop

- prevents silently swallowing async errors

* fix(blockexchange): asyncSpawns block exchange tasks

- prevents silently swallow future exceptions
2024-12-16 06:01:49 +00:00
Eric 5f2ba14281
fix(codexnode): ensure timer loop is asyncSpawned (#1038)
* fix(codexnode): stop clock after validator stops

* fix(timer): ensure timer loop is asyncSpawned
2024-12-16 04:24:57 +00:00
Slava 6d415b0ace
ci: split nim-matrix workflow (#1041) 2024-12-15 23:31:55 +00:00
Eric 01fb685bf6
fix(sales): replaces `then` with `asyncSpawn` (#1036)
- ensures `addSlotToQueue` does not raise exceptions as it is now asyncSpawned
2024-12-15 23:19:31 +00:00
Marcin Czenko 92a0eda79a
Validator historical state restoration (#922)
* adds a new overload of queryPastEvents allowing to query past events based on timestamp in the past

* adds state restoration to validator

* refactors a bit to get the tests back to work

* replaces deprecated generic methods from Market with methods for specific event types

* Refactors binary search

* adds market tests for querying past SlotFilled events and binary search

* Takes into account that <<earliest>> block available is not necessarily the genesis block

* Adds more logging and makes testing earliest block boundary more reliable

* adds validation tests for historical state restoration

* adds mockprovider to simplify and improve testing of the edge conditions

* adds slot reservation to the new tests after rebasing

* adds validation groups and group index in logs of validator

* adds integration test with two validators

* adds comment on how to enable logging in integration test executable itself

* testIntegration: makes list is running nodes injected and available in the body of the test

* validation: adds integration test for historical state

* adds more logging to validator

* integration test: validator only looks 30 days back for historical state

* adds logging of the slotState when removing slots during validation

* review and refactor validator integration tests

* adds validation to the set of integration tests

* Fixes mistyped name of the mock provider module in testMarket

* Fixes a typo in the name of the validation suite in integration tests

* Makes validation unit test a bit easier to follow

* better use of logScopes to reduce duplication

* improves timing and clarifies the test conditions

* uses http as default RPC provider for nodes running in integration tests as a workaround for dropped subscriptions

* simplifies the validation integration tests by waiting for failed request instead of tracking slots

* adds config option allowing selectively to set different provider url

* Brings back the default settings for RPC provider in integration tests

* use http RPC provider for clients in validation integration tests

* fine-tune the tests

* Makes validator integration test more robust - adds extra tracking

* brings tracking of marketplace event back to validator integration test

* refactors integration tests

* deletes tmp file

* adds <<return>> after forcing integration test to fail preliminarily

* re-enables all integration tests and matrix

* stops debug output in CI

* allows to choose a different RPC provider for a given integration test suite

* fixes signature of <<getBlock>> method in mockProvider

* adds missing import which seem to be braking integration tests on windows

* makes sure that clients, SPs, and validators use the same provider url

* makes validator integration tests using http at 127.0.0.1:8545

* testvalidator: stop resubscribing as we are now using http polling as rpc provider

* applying review comments

* groups queryPastStorage overrides together (review comment)

* groups the historical validation tests into a sub suite

* removes the temporary extensions in marketplacesuite and multinodesuite allowing to specify provider url

* simplifies validation integration tests

* Removes debug logs when waiting for request to fail

* Renaming waitForRequestFailed => waitForRequestToFail

* renames blockNumberForBlocksAgo to pastBlockTag and makes it private

* removes redundant debugging logs

* refines logging in validation

* removes dev logging from mockmarket

* improves exception handling in provider helper procs and prepares for extraction to a separate module

* Uses chronos instead of std/times for Duration

* extracts provider and binary search helpers to a separate module

* removes redundant log entry params from validator

* unifies the notation to consistently use method call syntax

* reuses ProviderError from nim-ethers in the provider extension

* clarifies the comment in multinodesuite

* uses == operator to check the predefined tags and raises exception when `BlockTag.pending` is requested.

* when waiting for request to fail, we break on any request state that is not Started

* removes tests that were moved to testProvider from testMarket

* extracts tests that use MockProvider to a separate async suite

* improves performance of the historical state restoration

* removing redundant log messages in validator (groupIndex and groups)

* adds testProvider to testContracts group

* removes unused import in testMarket
2024-12-14 05:07:55 +00:00
Eric 1f49f86131
fix(slotqueue): asyncSpawns futures correctly (#1034)
- asyncSpawns `run` and worker `dispatch` in slotqueue.
- removes usage of `then` from slotqueue.
2024-12-13 06:42:05 +00:00
Eric 7c804b0ec9
fix(asyncstatemachine): fixes not awaiting or asyncSpawning futures (#1033)
- adds a break in scheduler when CancelledError is caught
- tracks asyncSpawned state.run, so that it can be cancelled during stop
- removes usages of `then`
- ensures that no exceptions are leaked from async procs
2024-12-13 02:35:39 +00:00
Adam Uhlíř 19af79786e
feat: repair is rewarded (#1022)
* feat: repair is rewarded

* chore: update contracts repo

* feat: proving loop handles repair case

* test: assert repair state

* chore: update contracts repo

* fix: upon unknown state of repair go to error
2024-12-12 20:19:56 +00:00
Adam Uhlíř d10072bf67
refactor: marketplace configuration is cached (#1029) 2024-12-12 12:57:34 +00:00
Eric da234d503b
fix(trackedfutures): removes usage of `then` from tracked futures (#1032)
- removes usage of `then`, simplifying the logic, and allowing `then` to be removed completely
- updates annotations to reflect that all procs (sync and async) raise no exceptions
2024-12-12 12:31:51 +00:00
Vaclav Pavlin 855b973811
chore: fix inconsistent metric naming (#1027) 2024-12-12 10:45:47 +00:00
Marko Burčul 0c6784da7e
nix: make derivation and update shell (#1003)
* nix: make derivation and update shell

Create a structure for nix files. Add the derivation file which is using
system Nim to compile Codex.

Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>

* nim-circom-compat: update

Include commit which allows building circom-compat-ffi using Nix(doesn't
affect current usage of the submodule).

Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>

* makefile: fix for detecting linux arch

Signed-off-by: markoburcul <marko@status.im>

---------

Signed-off-by: markoburcul <marko@status.im>
2024-12-09 17:07:01 +00:00
Eric fb4577f25c
chore(tests): fix unneeded async (#1021)
Removes an unneeded `check eventually` inside a helper proc, removing the need for the proc to be async.
2024-12-06 08:02:57 +00:00
Ben Bierens f51eae30fe
Fixes raceconditions in testadvertiser (#1008) 2024-12-05 09:30:01 +00:00
Ben Bierens 8e29939cf8
Send pluralized wantBlock messages (#1016)
* don't unroll wantCids when sending wantBlock message in blockPresenceHandler

* workaround logging

* Fixes logformatting upraises for sequences.

* Applies upraises rule for setProperty of textmode for sequences.

* Replaces upraises with raises

* Removes redundant log in sendWantHave
2024-12-04 13:33:48 +00:00
Slava 921159f87f
Run release tests for docker images (#1017)
* Refactor Docker reusable workflow and add release tests support

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Add an option to run release tests for Docker images

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Use bigger instance for arm builds

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Pass repository and branch to release tests workflow

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Do not use computation job because run_release_tests is a string

https://github.com/codex-storage/cs-codex-dist-tests/issues/108
2024-12-04 11:52:51 +00:00