123 Commits

Author SHA1 Message Date
Slava
2f4948e46f
ci: use macos-14-xlarge for macos integration tests
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2025-06-05 16:42:28 +03:00
Eric
e1e25f7342
Move macos runners back to arm64 2025-06-05 16:12:34 +10:00
Eric
3215a3c0ef
fix(CI): change all macos runners to amd64 2025-06-05 12:06:58 +10:00
Eric
072be525ad
fix(CI): change macos runner arch type to amd64 2025-06-05 11:56:13 +10:00
Eric
79564ce72f
Update ci.yml to be in line with master 2025-06-04 17:32:07 +10:00
Eric
2d8f05cc25
disable ccache symlink
Permission denied creating the symlink in CI
2025-06-04 17:23:07 +10:00
Eric
af5759b053
re-enable all CI tests
Also adds a condition such that a hardhat node is not started for parallel integration tests
2025-06-04 16:04:46 +10:00
Eric
bf13173e58
CI yml fix for gcc-14 2025-06-04 15:58:29 +10:00
Eric
8e6eca6bc0
Update NBS action.yml in CI 2025-06-04 15:58:28 +10:00
Slava
e0b222a10c
ci: check runner for parallel integration tests and use value from vars 2025-06-04 15:56:02 +10:00
Eric
58153ee921
CI: enable artefact upload for parallel tests 2025-06-04 15:55:16 +10:00
Eric
865403d877
run npm install for parallel tests in ci 2025-06-04 15:55:16 +10:00
Eric
b6b595aab5
Allow debug runs from the github UI 2025-06-04 15:55:16 +10:00
Eric
fc63ee188e
clean up 2025-06-04 15:55:16 +10:00
Eric
c196920d20
update Make flag to simply debug
DEBUG -> enables DebugCodexNodes, DebugTestHarness, and ShowContinuousStatusUpdates
DEBUG_HARDHAT -> enables DebugHardhat
PARALLEL -> enables EnableParallelTests

Additionally, when DEBUG is enabled, all integration tests debug configs are enabled for Codex nodes, the Codex node output is printed with the test output (not interleaved), and the Codex node output is logged to file in `tests/integrations/logs/<starttime>__IntegrationTests/<integration_test_name>/<suite_name>/<testname>/<role>_<idx>.log`.
When DEBUG_HARDHAT is enabled, all hardhat output is printed with the test output (not interleaved), and the output is also written to a log file in `tests/integrations/logs/<starttime>__IntegrationTests/<integration_test_name>/hardhat.log
2025-06-04 15:55:15 +10:00
Eric
43fd465ec9
Allow parallel integration tests to be disabled
Disable with `make ENABLE_PARALLEL_TESTS=0 testIntegration`
2025-06-04 15:54:28 +10:00
Eric
f278d11b70
swap linux large runner 2025-06-04 15:54:28 +10:00
Eric
69b9bb0030
re-add windows to CI 2025-06-04 15:54:27 +10:00
Eric
5dea609386
use macos-14-large
windows is failing with SIGSEGV so remove it for now
2025-06-04 15:54:26 +10:00
Eric
951da79a4e
add windows large runner 32vcpu 2025-06-04 15:54:26 +10:00
Eric
1e4e8c8a8d
trying a self-hosted runner to see if it has more resources 2025-06-04 15:54:26 +10:00
Eric
3e86cb47f2
bump to larger runner, linux only 2025-06-04 15:54:25 +10:00
Eric
36cc3b2a8f
temp remove coverage from ci workflow 2025-06-04 15:53:45 +10:00
Eric
c695a3fee0
try large runners for linux/windows 2025-06-04 15:52:32 +10:00
Eric
3737e147c4
use large linux runner and disable other tests 2025-06-04 15:52:09 +10:00
Eric
29394329b3
temporarily remove linux test to see if macos works in ci 2025-06-04 15:50:46 +10:00
Eric
627bdecee0
enable CI debugging 2025-06-04 15:49:17 +10:00
Eric
7eac5fe06b
disable windows integration tests for now 2025-06-04 15:49:17 +10:00
Eric
58cca7ce65
npm install to install the hardhat binary in CI 2025-06-04 15:49:17 +10:00
Eric
b2c0e77f02
CI: do not start hardhat for integration tests 2025-06-04 15:49:16 +10:00
Giuliano Mega
d59c5b023c
chore: bump Nim to 2.2.4 (#1242)
* chore: bump Nim to 2.2.4

* fix: resolve symbol ambiguity and drop auto type

* fix: use reference to task instead of pointer or the compiler will deallocate `task` before the encoding/decoding is done

* fix: convention that maxCollateralPerByte equals totalRemainingCollateral when freeSize is 0 to avoid DivByZeroDefect

* fix: bump compiler version in CI pipeline as well
2025-05-29 16:37:38 -07:00
Slava
13811825b3
ci: use macos arm runners (#1174)
* ci: use inputs instead of matrix in a ccache key

* ci: switch to arm runners for macos

* ci: use node 20

* ci: pass cpu to a composite action
2025-05-29 10:17:46 +00:00
Adam Uhlíř
d220e53fe1
ci: trigger python generator upon release (#1208) 2025-04-22 14:46:03 +00:00
Slava
22f5150d1d
ci: add compatible contracts image for nim-codex dist-tests docker image (#1204) 2025-04-18 14:21:24 +00:00
Adam Uhlíř
7c7871ac75
ci: add compatible contracts image for nim-codex docker image (#1186)
* ci: add compatible contracts image for nim-codex docker image

* ci: with submodules

* ci: with submodules on correct place

* ci: remove double dash

* ci: avoiding artifact conflicts

* ci: add labels to arch images

* ci: correct way to add label to arch images

* ci: correct contract label

* ci: avoid building contracts image and use contracts commit hash

* refactor: better way to get the hash
2025-04-15 13:52:19 +00:00
Slava
1213377ac4
ci: switch out from ubuntu 20.04 (#1184)
* ci: use ubuntu-latest for coverage (#1141)

* ci: pass --keep-going to lcov and genhtml (#1141)

* ci: use ubuntu-22.04 for release workflow (#1141)

* ci: install gcc-14 on linux (#1141)

* chore: bump nim-leveldbstatic to 0.2.1
2025-04-02 09:09:43 +00:00
Arnaud
eb09e610d5
fix(ci): handle coverage as a string to enable gcc 14 on linux (#1140)
* Handle coverage as a string not a boolean

* Update ubuntu version to latest
2025-03-05 08:35:46 +00:00
Slava
20f6fef7ab
fix: use ubuntu-24.04 runners for docker workflows (#1102)
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
2025-02-11 21:49:37 +00:00
Marcin Czenko
11888e78d7
deploy openapi spec only when tagged (#1106) 2025-02-11 15:16:45 +00:00
Csaba Kiraly
e62a09d9b1
add ccache and sccache to speed up CI (#1074)
* add ccache and sccache to speed up CI

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* include testname and nim version in cache separation

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* Make sure ccache has precedence over custom clang/llvm

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* enable ccache for windows

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* ccache: evict old files

Make sure old unused cache files are not lingering around for long

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

---------

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
2025-02-06 21:36:14 +00:00
Adam Uhlíř
e5df8c50d3
style: nph formatting (#1067)
* style: nph setup

* chore: formates codex/ and tests/ folder with nph 0.6.1
2025-01-21 20:54:46 +00:00
Slava
833e253baa
feat: switch to github arm runners (#1077)
* feat: switch to github arm runners

* chore: fabiocaccamo/create-matrix-action
2025-01-20 14:42:01 +00:00
Arnaud
f25c555d59
Chore/update nim version (#1052)
* Move to version 2.0.6

* Update nim-confutils submodule to latest version

* Update dependencies

* Update Nim version to 2.0.12

* Add gcsafe pragma

* Add missing import

* Update specific conf for Nim 2.x

* Fix method signatures

* Revert erasure coding attempt to fix bug

* More gcsafe pragma

* Duplicate code from libp2p because it is not exported anymore

* Fix camelcase function names

* Use alreadySeen because need is not a bool anymore

* newLPStreamReadError does not exist anymore so use another error

* Replace ValidIpAddress by IpAddress

* Add gcsafe pragma

* Restore maintenance parameter deleted by mistake when removing esasure coding fix attempt code

* Update method signatures

* Copy LPStreamReadError code from libp2p which was removed

* Fix camel case

* Fix enums in tests

* Fix camel case

* Extract node components to a variable to make Nim 2 happy

* Update the tests using ValidIpAddress to IpAddress

* Fix cast for value which is already an option

* Set nim version to 2.0.x for CI

* Set nim version to 2.0.x for CI

* Move to miniupnp version 2.2.4 to avoid symlink error

* Set core.symlinks to false for Windows for miniupnp >= 2.2.5 support

* Update to Nim 2.0.14

* Update CI nim versions to 2.0.14

* Try with GCC 14

* Replace apt-fast by apt-get

* Update ubuntu runner to latest

* Use Ubuntu 20.04 for coverage

* Disable CI cache for coverage

* Add coverage property description

* Remove commented test

* Check the node value of seen instead of using alreadySeen

* Fix the merge. The taskpool work was reverted.

* Update nim-ethers submodule

* Remove deprecated ValidIpAddress. Fix missing case and imports.

* Fix a weird issue where nim-confutils cannot find NatAny

* Fix tests and remove useless static keyword
2025-01-10 14:12:37 +00:00
Slava
6d415b0ace
ci: split nim-matrix workflow (#1041) 2024-12-15 23:31:55 +00:00
Marcin Czenko
92a0eda79a
Validator historical state restoration (#922)
* adds a new overload of queryPastEvents allowing to query past events based on timestamp in the past

* adds state restoration to validator

* refactors a bit to get the tests back to work

* replaces deprecated generic methods from Market with methods for specific event types

* Refactors binary search

* adds market tests for querying past SlotFilled events and binary search

* Takes into account that <<earliest>> block available is not necessarily the genesis block

* Adds more logging and makes testing earliest block boundary more reliable

* adds validation tests for historical state restoration

* adds mockprovider to simplify and improve testing of the edge conditions

* adds slot reservation to the new tests after rebasing

* adds validation groups and group index in logs of validator

* adds integration test with two validators

* adds comment on how to enable logging in integration test executable itself

* testIntegration: makes list is running nodes injected and available in the body of the test

* validation: adds integration test for historical state

* adds more logging to validator

* integration test: validator only looks 30 days back for historical state

* adds logging of the slotState when removing slots during validation

* review and refactor validator integration tests

* adds validation to the set of integration tests

* Fixes mistyped name of the mock provider module in testMarket

* Fixes a typo in the name of the validation suite in integration tests

* Makes validation unit test a bit easier to follow

* better use of logScopes to reduce duplication

* improves timing and clarifies the test conditions

* uses http as default RPC provider for nodes running in integration tests as a workaround for dropped subscriptions

* simplifies the validation integration tests by waiting for failed request instead of tracking slots

* adds config option allowing selectively to set different provider url

* Brings back the default settings for RPC provider in integration tests

* use http RPC provider for clients in validation integration tests

* fine-tune the tests

* Makes validator integration test more robust - adds extra tracking

* brings tracking of marketplace event back to validator integration test

* refactors integration tests

* deletes tmp file

* adds <<return>> after forcing integration test to fail preliminarily

* re-enables all integration tests and matrix

* stops debug output in CI

* allows to choose a different RPC provider for a given integration test suite

* fixes signature of <<getBlock>> method in mockProvider

* adds missing import which seem to be braking integration tests on windows

* makes sure that clients, SPs, and validators use the same provider url

* makes validator integration tests using http at 127.0.0.1:8545

* testvalidator: stop resubscribing as we are now using http polling as rpc provider

* applying review comments

* groups queryPastStorage overrides together (review comment)

* groups the historical validation tests into a sub suite

* removes the temporary extensions in marketplacesuite and multinodesuite allowing to specify provider url

* simplifies validation integration tests

* Removes debug logs when waiting for request to fail

* Renaming waitForRequestFailed => waitForRequestToFail

* renames blockNumberForBlocksAgo to pastBlockTag and makes it private

* removes redundant debugging logs

* refines logging in validation

* removes dev logging from mockmarket

* improves exception handling in provider helper procs and prepares for extraction to a separate module

* Uses chronos instead of std/times for Duration

* extracts provider and binary search helpers to a separate module

* removes redundant log entry params from validator

* unifies the notation to consistently use method call syntax

* reuses ProviderError from nim-ethers in the provider extension

* clarifies the comment in multinodesuite

* uses == operator to check the predefined tags and raises exception when `BlockTag.pending` is requested.

* when waiting for request to fail, we break on any request state that is not Started

* removes tests that were moved to testProvider from testMarket

* extracts tests that use MockProvider to a separate async suite

* improves performance of the historical state restoration

* removing redundant log messages in validator (groupIndex and groups)

* adds testProvider to testContracts group

* removes unused import in testMarket
2024-12-14 05:07:55 +00:00
Slava
921159f87f
Run release tests for docker images (#1017)
* Refactor Docker reusable workflow and add release tests support

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Add an option to run release tests for Docker images

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Use bigger instance for arm builds

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Pass repository and branch to release tests workflow

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Do not use computation job because run_release_tests is a string

https://github.com/codex-storage/cs-codex-dist-tests/issues/108
2024-12-04 11:52:51 +00:00
Slava
d12de20868
ci: use rust 1.7.9 for release workflow and dockerfile (#999)
* ci: use rust 1.7.9 for release workflow (#996)

* docker: use rust 1.7.9 for docker (#996)
2024-11-25 16:13:14 +00:00
markspanbroek
29433bad9a
Fix concurrency issues (#993)
* Use http subscriptions instead of websocket for tests

To work around this issue when subscriptions are
inactive for more than 5 minutes:
https://github.com/NomicFoundation/hardhat/issues/2053

Use 100 millisecond polling; default polling interval
of 4 seconds is too close to the 5 second timeout for
`check eventually`.

* use .confirm(1) instead of confirm(0)

confirm(0) doesn't wait at all, confirm(1) waits
for the transaction to be mined

* speed up partial payout integration test

* update nim-ethers to version 0.10.0

includes fixes for http polling and .confirm()

* fix timing of marketplace tests

allow for a bit more time to withdraw funds

* use .confirm(1) in marketplace tests

to ensure that the transaction has been processed
before continuing with the test

* fix timing issue in validation unit test

* fix proof integration test

there were two logic errors in this test:
- a slot is freed anyway at the end of the contract
- when starting the request takes a long time, the
  first slot can already be freed because there were
  too many missing proofs

* fix intermittent error in contract tests

currentTime() doesn't always correctly reflect
the time of the next transaction

* reduce number of slots in integration test

otherwise the windows runner in the CI won't
be able to start the request before it expires

* fix timing in purchasing test

allow for a bit more time for a request to
be submitted

* fix timing of request submission in test

windows ci is so slow, it can take up to 40 seconds
just to submit a storage request to hardhat

* increase proof period to 90 seconds

* adjust timing of integration tests

reason: with the increased period length of 90 seconds, it
can take longer to wait for a stable challenge at the
beginning of a period.

* increase CI timeout to 2 hours

* Fix slow builds on windows

apparently it takes windows 2-3 seconds to
resolve "localhost" to 127.0.0.1 for every
json-rpc connection that we make 🤦
2024-11-25 11:23:04 +00:00
Slava
6038fb456e
ci: split linux and macos tests (#997)
* Split Linux and macOS tests

* Make jobs names more readable
2024-11-22 12:05:00 +00:00
Slava
71b8a95d12
ci: install rust 1.7.9 as required by packages (#996)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-11-22 06:34:10 +00:00