nim-codex/.github/workflows/ci.yml

76 lines
2.5 KiB
YAML
Raw Permalink Normal View History

name: CI
on:
push:
branches:
- master
pull_request:
workflow_dispatch:
Speed up development (#303) * [ci] Extract setup of nimbus build system into separate file * [ci] Cleanup NodeJS setup * [ci] Use amd64 by default * [ci] Separate steps for unit, contract and integration tests * [ci] Separate build job for coverage that is allowed to fail * [ci] Separate build job for Nim 1.2 * [ci] Calculate coverage on unit tests only This enables faster development cycles at the expense of code coverage accuracy. * [PoR] Test proofs of retrievability with a small block size To get a reasonable test execution time. * [ci] Set NIM_COMMIT environment variable To prevent subsequent `make` command to build a different version of Nim. * [ci] Checkout submodules recursively So that nimbus-build-system doesn't have to do it afterwards. * [ci] Update caches on every run Will automatically update caches when a new minor version of Nim is released. See also: https://github.com/actions/cache/blob/main/workarounds.md#update-a-cache * [ci] Set cache_nonce for all jobs, not just those in the matrix * [build] update to latest nimbus-build-system Requires an update to latest nim-nat-traversal * [erasure] Test erasure coding with a small block size To get a reasonable test execution time. * [erasure] fix typo * [PoR] Test PoR store with a small block size To get a reasonable test execution time. * [PoR] Test PoR network with a small block size To get a reasonable test execution time. * [ci] Ensure that unit tests are not dependent on Ethereum Start Ethereum node *after* the unit tests are run. * [ci] Cancel all other jobs when one fails Allows for faster feedback loops; a next attempt to run the CI will start sooner.
2022-11-14 13:50:00 +00:00
env:
cache_nonce: 0 # Allows for easily busting actions/cache caches
nim_version: pinned
ci: update GitHub Actions CI workflow to use msys2/setup-msys2@v2 Main goal is to update the nim-codex CI workflow to use the [`msys2/setup-msys2@v2`][setmsys2] GitHub Action, as used by the [nimbus-eth2 workflow][ne2w] per changes made to it several months ago. The `msys2/setup-msys2@v2` action has been used by other Status-org projects prior to this commit, e.g. nim-leopard, nim-datastore, nim-status. A fix is included for the failing macOS builds, related to [actions/virtual-environments#5819][ave5819]. See [L151][L151]. All builds [succeed][succeed] (with build arch-targets verified as far as possible via `ldd`, `otool`, `ntldd`), including `linux-i386` and `windows-i386`, though the `i386` builds are presently disabled (commented out). The `i386` builds can be enabled simply by uncommenting: ``` # - os: linux # cpu: i386 ... # - os: windows # cpu: i386 ``` The resulting `.github/workflows/ci.yml` is a "remix" of the current workflows for nimbus-eth2 and nim-codex (i.e. prior to this commit) along with techniques learned from developing workflows for other Status-org repos. Some comments and code-reorg help to clarify/explain what's done in the `Derive environment variables` step. `-d:limitStackUsage` has been adopted for `linux-amd64` builds from [nimbus-eth2's workflow][ne2wL155] and [related code][ne2config] has been copied into `config.nims` `-d:limitStackUsage` can easily be dropped if it's not desirable for Codex. Build targets use `-latest` for `runs-on`, i.e. `macos-latest`, `ubuntu-latest`, `windows-latest`. Through a combination of local testing and iterative pushes to GitHub, the workflow's embedded Bash scripts have been revised to include only the necessary steps for all builds to succeed, including `linux-i386` and `windows-i386`. The GitHub Actions workflow `.github/workflows/codecov.yml` has been removed, while coverage data generation/upload steps have been added to `.github/workflows/ci.yml` as the final steps conditional on `if: runner.os == 'Linux' && matrix.target.cpu == 'amd64' && matrix.nim_branch == matrix.cov_branch`. A redundant `--passC:'-m32 -mno-adx'` is used for `linux-i386` builds; the redundant flags do not affect the build, but can be helpful when eyeballing GitHub Actions builds with increased compile-time verbosity. Some variable expansions used in `github/workflows/ci.yml` could result in compilation failures if related paths include whitespace. It's not a problem for this commit but could be a problem for a user copy-pasting from the workflow; solving that problem is left as an exercise for the reader. [setmsys2]: https://github.com/msys2/setup-msys2#readme [ne2w]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml [ave5819]: https://github.com/actions/virtual-environments/pull/5819 [L151]: https://github.com/status-im/nim-codex/blob/ci/msys2/.github/workflows/ci.yml#L151 [succeed]: https://github.com/status-im/nim-codex/actions/runs/2606854455 [ne2wL155]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml#L155-L159 [ne2config]: https://github.com/status-im/nimbus-eth2/blob/stable/config.nims#L43-L49
2022-07-04 02:37:34 +00:00
concurrency:
group: ${{ github.workflow }}-${{ github.ref || github.run_id }}
cancel-in-progress: true
ci: update GitHub Actions CI workflow to use msys2/setup-msys2@v2 Main goal is to update the nim-codex CI workflow to use the [`msys2/setup-msys2@v2`][setmsys2] GitHub Action, as used by the [nimbus-eth2 workflow][ne2w] per changes made to it several months ago. The `msys2/setup-msys2@v2` action has been used by other Status-org projects prior to this commit, e.g. nim-leopard, nim-datastore, nim-status. A fix is included for the failing macOS builds, related to [actions/virtual-environments#5819][ave5819]. See [L151][L151]. All builds [succeed][succeed] (with build arch-targets verified as far as possible via `ldd`, `otool`, `ntldd`), including `linux-i386` and `windows-i386`, though the `i386` builds are presently disabled (commented out). The `i386` builds can be enabled simply by uncommenting: ``` # - os: linux # cpu: i386 ... # - os: windows # cpu: i386 ``` The resulting `.github/workflows/ci.yml` is a "remix" of the current workflows for nimbus-eth2 and nim-codex (i.e. prior to this commit) along with techniques learned from developing workflows for other Status-org repos. Some comments and code-reorg help to clarify/explain what's done in the `Derive environment variables` step. `-d:limitStackUsage` has been adopted for `linux-amd64` builds from [nimbus-eth2's workflow][ne2wL155] and [related code][ne2config] has been copied into `config.nims` `-d:limitStackUsage` can easily be dropped if it's not desirable for Codex. Build targets use `-latest` for `runs-on`, i.e. `macos-latest`, `ubuntu-latest`, `windows-latest`. Through a combination of local testing and iterative pushes to GitHub, the workflow's embedded Bash scripts have been revised to include only the necessary steps for all builds to succeed, including `linux-i386` and `windows-i386`. The GitHub Actions workflow `.github/workflows/codecov.yml` has been removed, while coverage data generation/upload steps have been added to `.github/workflows/ci.yml` as the final steps conditional on `if: runner.os == 'Linux' && matrix.target.cpu == 'amd64' && matrix.nim_branch == matrix.cov_branch`. A redundant `--passC:'-m32 -mno-adx'` is used for `linux-i386` builds; the redundant flags do not affect the build, but can be helpful when eyeballing GitHub Actions builds with increased compile-time verbosity. Some variable expansions used in `github/workflows/ci.yml` could result in compilation failures if related paths include whitespace. It's not a problem for this commit but could be a problem for a user copy-pasting from the workflow; solving that problem is left as an exercise for the reader. [setmsys2]: https://github.com/msys2/setup-msys2#readme [ne2w]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml [ave5819]: https://github.com/actions/virtual-environments/pull/5819 [L151]: https://github.com/status-im/nim-codex/blob/ci/msys2/.github/workflows/ci.yml#L151 [succeed]: https://github.com/status-im/nim-codex/actions/runs/2606854455 [ne2wL155]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml#L155-L159 [ne2config]: https://github.com/status-im/nimbus-eth2/blob/stable/config.nims#L43-L49
2022-07-04 02:37:34 +00:00
jobs:
matrix:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.matrix.outputs.matrix }}
cache_nonce: ${{ env.cache_nonce }}
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
steps:
- name: Compute matrix
id: matrix
uses: fabiocaccamo/create-matrix-action@v4
with:
matrix: |
os {linux}, cpu {amd64}, builder {ubuntu-20.04}, tests {all}, nim_version {${{ env.nim_version }}}, shell {bash --noprofile --norc -e -o pipefail}
os {macos}, cpu {amd64}, builder {macos-13}, tests {all}, nim_version {${{ env.nim_version }}}, shell {bash --noprofile --norc -e -o pipefail}
os {windows}, cpu {amd64}, builder {windows-latest}, tests {unittest}, nim_version {${{ env.nim_version }}}, shell {msys2}
os {windows}, cpu {amd64}, builder {windows-latest}, tests {contract}, nim_version {${{ env.nim_version }}}, shell {msys2}
os {windows}, cpu {amd64}, builder {windows-latest}, tests {integration}, nim_version {${{ env.nim_version }}}, shell {msys2}
Release v0.1.4 (#912) * fix: createReservation lock (#825) * fix: createReservation lock * fix: additional locking places * fix: acquire lock * chore: feedback Co-authored-by: markspanbroek <mark@spanbroek.net> Signed-off-by: Adam Uhlíř <adam@uhlir.dev> * feat: withLock template and fixed tests * fix: use proc for MockReservations constructor * chore: feedback Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com> Signed-off-by: Adam Uhlíř <adam@uhlir.dev> * chore: feedback implementation --------- Signed-off-by: Adam Uhlíř <adam@uhlir.dev> Co-authored-by: markspanbroek <mark@spanbroek.net> Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com> * Block deletion with ref count & repostore refactor (#631) * Fix StoreStream so it doesn't return parity bytes (#838) * fix storestream so it doesn\'t return parity bits for protected/verifiable manifests * use Cid.example instead of creating a mock manually * Fix verifiable manifest initialization (#839) * fix verifiable manifest initialization * fix linearstrategy, use verifiableStrategy to select blocks for slots * check for both strategies in attribute inheritance test * ci: add verify_circuit=true to the releases (#840) * provisional fix so EC errors do not crash the node on download (#841) * prevent node crashing with `not val.isNil` (#843) * bump nim-leopard to handle no parity data (#845) * Fix verifiable manifest constructor (#844) * Fix verifiable manifest constructor * Add integration test for verifiable manifest download Add integration test for testing download of verifiable dataset after creating request for storage * add missing import * add testecbug to integration suite * Remove hardhat instance from integration test * change description, drop echo --------- Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com> Co-authored-by: gmega <giuliano.mega@gmail.com> * Bump Nim to 1.6.21 (#851) * bump Nim to 1.6.21 (range type reset fixes) * remove incompatible versions from compiler matrix * feat(rest): adds erasure coding constraints when requesting storage (#848) * Rest API: add erasure coding constraints when requesting storage * clean up * Make error message for "dataset too small" more informative. * fix API integration test --------- Co-authored-by: gmega <giuliano.mega@gmail.com> * Prover workshop band-aid (#853) * add prover bandaid * Improve error message text Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com> Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com> --------- Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com> Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com> * Bandaid for failing erasure coding (#855) * Update Release workflow (#858) Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com> * Fixes prover behavior with singleton proof trees (#859) * add logs and test * add Merkle proof checks * factor out Circom input normalization, fix proof input serialization * add test and update existing ones * update circuit assets * add back trace message * switch contracts to fix branch * update codex-contracts-eth to latest * do not expose prove with prenormalized inputs * Chronos v4 Update (v3 Compat Mode) (#814) * add changes to use chronos v4 in compat mode * switch chronos to compat fix branch * use nimbus-build-system with configurable Nim repo * add missing imports * add missing await * bump compat * pin nim version in Makefile * add await instead of asyncSpawn to advertisement queue loop * bump DHT to v0.5.0 * allow error state of `onBatch` to propagate upwards in test code * pin Nim compiler commit to avoid fetching stale branch * make CI build against branch head instead of merge * fix handling of return values in testslotqueue * Downgrade to gcc 13 on Windows (#874) * Downgrade to gcc 13 on Windows Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com> * Increase build job timeout to 90 minutes Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com> --------- Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com> * Add MIT/Apache licenses (#861) * Add MIT/Apache licenses * Center "Apache License" Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com> * remove wrong legal entity; rename apache license file --------- Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com> * Add OPTIONS endpoint to allow the content-type header for the upload endpoint (#869) * Add OPTIONS endpoint to allow the content-type header exec git commit --amend --no-edit -S * Remove useless header "Access-Control-Headers" and add cache Signed-off-by: Arnaud <arnaud@status.im> --------- Signed-off-by: Arnaud <arnaud@status.im> Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com> * chore: add `downtimeProduct` config parameter (#867) * chore: add `downtimeProduct` config parameter * bump codex-contracts-eth to master * Support CORS preflight requests when the storage request api returns an error (#878) * Add CORS headers when the REST API is returning an error * Use the allowedOrigin instead of the wilcard when setting the origin Signed-off-by: Arnaud <arnaud@status.im> --------- Signed-off-by: Arnaud <arnaud@status.im> * refactor(marketplace): generic querying of historical marketplace events (#872) * refactor(marketplace): move marketplace events to the Market abstraction Move marketplace contract events to the Market abstraction so the types can be shared across all modules that call the Market abstraction. * Remove unneeded conversion * Switch to generic implementation of event querying * change parent type to MarketplaceEvent * Remove extra license file (#876) * remove extra license * center "apache license" * Update advertising (#862) * Setting up advertiser * Wires up advertiser * cleanup * test compiles * tests pass * setting up test for advertiser * Finishes advertiser tests * fixes commonstore tests * Review comments by Giuliano * Race condition found by Giuliano * Review comment by Dmitriy Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com> Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com> * fixes tests --------- Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com> Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com> * feat: add `--payout-address` (#870) * feat: add `--payout-address` Allows SPs to be paid out to a separate address, keeping their profits secure. Supports https://github.com/codex-storage/codex-contracts-eth/pull/144 in the nim-codex client. * Remove optional payoutAddress Change --payout-address so that it is no longer optional. There is no longer an overload in `Marketplace.sol` for `fillSlot` accepting no `payoutAddress`. * Update integration tests to include --payout-address * move payoutAddress from fillSlot to freeSlot * Update integration tests to use required payoutAddress - to make payoutAddress required, the integration tests needed to avoid building the cli params until just before starting the node, otherwise if cli params were added ad-hoc, there would be an error after a non-required parameter was added before a required parameter. * support client payout address - withdrawFunds requires a withdrawAddress parameter, directs payouts for withdrawing of client funds (for a cancelled request) to go to that address. * fix integration test adds --payout-address to validators * refactor: support withdrawFunds and freeSlot optional parameters - withdrawFunds has an optional parameter for withdrawRecipient - freeSlot has optional parameters for rewardRecipient and collateralRecipient - change --payout-address to --reward-recipient to match contract signature naming * Revert "Update integration tests to include --payout-address" This reverts commit 8f9535cf35b0f2b183ac4013a7ed11b246486964. There are some valid improvements to the integration tests, but they can be handled in a separate PR. * small fix * bump contracts to fix marketplace spec * bump codex-contracts-eth, now rebased on master * bump codex-contracts-eth now that feat/reward-address has been merged to master * clean up, comments * Rework circuit downloader (#882) * Introduces a start method to prover * Moves backend creation into start method * sets up three paths for backend initialization * Extracts backend initialization to backend-factory * Implements loading backend from cli files or previously downloaded local files * Wires up downloading and unzipping * functional implementation * Fixes testprover.nim * Sets up tests for backendfactory * includes libzip-dev * pulls in updated contracts * removes integration cli tests for r1cs, wasm, and zkey file arguments. * Fixes issue where inner-scope values are lost before returning * sets local proof verification for dist-test images * Adds two traces and bumps nim-ethers * Adds separate path for circuit files * Create circuit dir if not exists * fix: make sure requestStorage is mined * fix: correct place to plug confirm * test: fixing contracts tests * Restores gitmodules * restores nim-datastore reference * Sets up downloader exe * sets up tool skeleton * implements getting of circuit hash * Implements downloader tool * sets up test skeleton * Implements test for cirdl * includes testTools in testAll * Cleanup building.md * cleans up previous downloader implementation * cleans up testbackendfactory * moves start of prover into node.nim * Fills in arguments in example command * Initializes backend in prover constructor * Restores tests * Restores tests for cli instructions * Review comments by Dmitriy, part 1 * Quotes path in download instruction. * replaces curl with chronos http session * Moves cirdl build output to 'build' folder. * Fixes chronicles log output * Add cirdl support to the codex Dockerfile Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com> * Add cirdl support to the docker entrypoint Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com> * Add cirdl support to the release workflow Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com> * Disable verify_circuit flag for releases Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com> * Removes backendFactory placeholder type * wip * Replaces zip library with status-im/zippy library (which supports zip and tar) * Updates cirdl to not change circuitdir folder * Switches from zip to tar.gz * Review comments by Dmitriy * updates codex-contracts-eth * Adds testTools to CI * Adds check for access to config.circuitdir * Update fixture circuit zkey * Update matrix to run tools tests on Windows * Adds 'deps' dependency for cirdl * Adjust docker-entrypoint.sh to use CODEX_CIRCUIT_DIR env var * Review comments by Giuliano --------- Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com> Co-authored-by: Adam Uhlíř <adam@uhlir.dev> Co-authored-by: Veaceslav Doina <20563034+veaceslavdoina@users.noreply.github.com> * Support CORS for POST and PATCH availability endpoints (#897) * Adds testnet marketplace address to known deployments (#911) * API tweaks for OpenAPI, errors and endpoints (#886) * All sort of tweaks * docs: availability's minPrice doc * Revert changes to the two node test example * Change default EC params in REST API Change default EC params in REST API to 3 nodes and 1 tolerance. Adjust integration tests to honour these settings. --------- Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com> --------- Signed-off-by: Adam Uhlíř <adam@uhlir.dev> Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com> Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com> Signed-off-by: Arnaud <arnaud@status.im> Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com> Co-authored-by: Adam Uhlíř <adam@uhlir.dev> Co-authored-by: markspanbroek <mark@spanbroek.net> Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com> Co-authored-by: Tomasz Bekas <tomasz.bekas@gmail.com> Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com> Co-authored-by: Arnaud <arno.deville@gmail.com> Co-authored-by: Ben Bierens <39762930+benbierens@users.noreply.github.com> Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com> Co-authored-by: Arnaud <arnaud@status.im>
2024-09-24 10:19:58 +00:00
os {windows}, cpu {amd64}, builder {windows-latest}, tests {tools}, nim_version {${{ env.nim_version }}}, shell {msys2}
Poc 2 (#7) * moving protobuf into bitswap * adding block type * reworking bitswap * adding chunker * adding license header * use 1.2.6 * adding fixed size chunker * add blockstore * add iterator to chunker * more bitswap changes * rename ipfs to dagger * rename to dagger * blockstore inherits from BlockProvider * wip - add core block handling logic * smal changes * use proper block store methods * adding asynq heapqueue * wip prepare for bitswap task runner * adding `$` * adding memory store and tests * fixed chunking * extracted desicion engine from bitswap * added helper random funcs * adding testing helpers * only handle seqs * add peer events * cleanup pending blocks on blockstore event * allow nil handlers * move protobuf type helpers * allow initializing block from Cid * testing and fixes * small fixes * expose `<` * spelling * default value * spelling * pending blocks manager * adding stores manager * more tests a wip around bitswap * small changes * merge bitswap and engine for now * for now run only the new poc's tests * add a more complete ci setup * use template in map * remove p2pd * remove go * dont use asyncCheck * few small changes * adding ability to update items * adding multiple task runners * handle cancelation properly * use Result instead of throwing * wip bitswap tests * moving things around * split out engine again * add request and handlers interface * fix tests * wip - engine tests * remove unused imports * fix tests * cleanup block requesting logic * add block request tests * more block requests * add support for max heap * don't use result * use max heap & send block presence in task handler * add task handler tests * rename store to localStore * cleanup & logging * cancel task on stop * don't depend on local store for events * dont use heap queue for wants * add chronicles * fix issue with peer wants * add test for delayed block sends * remove obsolete tests * wip chunker * run all tests * add todo * misc * remove irrelevant files * removing more files * adding helpers for bitswap tests * moved bitswap file * misc * make blocks timeout longer * adjust block timeout * speedup test * compile with threads * import missing crypto * misc * disable threads for now * fix 32 bit platforms * re-enable threads support in tests
2021-02-26 00:23:22 +00:00
build:
needs: matrix
uses: ./.github/workflows/ci-reusable.yml
with:
matrix: ${{ needs.matrix.outputs.matrix }}
cache_nonce: ${{ needs.matrix.outputs.cache_nonce }}
ci: update GitHub Actions CI workflow to use msys2/setup-msys2@v2 Main goal is to update the nim-codex CI workflow to use the [`msys2/setup-msys2@v2`][setmsys2] GitHub Action, as used by the [nimbus-eth2 workflow][ne2w] per changes made to it several months ago. The `msys2/setup-msys2@v2` action has been used by other Status-org projects prior to this commit, e.g. nim-leopard, nim-datastore, nim-status. A fix is included for the failing macOS builds, related to [actions/virtual-environments#5819][ave5819]. See [L151][L151]. All builds [succeed][succeed] (with build arch-targets verified as far as possible via `ldd`, `otool`, `ntldd`), including `linux-i386` and `windows-i386`, though the `i386` builds are presently disabled (commented out). The `i386` builds can be enabled simply by uncommenting: ``` # - os: linux # cpu: i386 ... # - os: windows # cpu: i386 ``` The resulting `.github/workflows/ci.yml` is a "remix" of the current workflows for nimbus-eth2 and nim-codex (i.e. prior to this commit) along with techniques learned from developing workflows for other Status-org repos. Some comments and code-reorg help to clarify/explain what's done in the `Derive environment variables` step. `-d:limitStackUsage` has been adopted for `linux-amd64` builds from [nimbus-eth2's workflow][ne2wL155] and [related code][ne2config] has been copied into `config.nims` `-d:limitStackUsage` can easily be dropped if it's not desirable for Codex. Build targets use `-latest` for `runs-on`, i.e. `macos-latest`, `ubuntu-latest`, `windows-latest`. Through a combination of local testing and iterative pushes to GitHub, the workflow's embedded Bash scripts have been revised to include only the necessary steps for all builds to succeed, including `linux-i386` and `windows-i386`. The GitHub Actions workflow `.github/workflows/codecov.yml` has been removed, while coverage data generation/upload steps have been added to `.github/workflows/ci.yml` as the final steps conditional on `if: runner.os == 'Linux' && matrix.target.cpu == 'amd64' && matrix.nim_branch == matrix.cov_branch`. A redundant `--passC:'-m32 -mno-adx'` is used for `linux-i386` builds; the redundant flags do not affect the build, but can be helpful when eyeballing GitHub Actions builds with increased compile-time verbosity. Some variable expansions used in `github/workflows/ci.yml` could result in compilation failures if related paths include whitespace. It's not a problem for this commit but could be a problem for a user copy-pasting from the workflow; solving that problem is left as an exercise for the reader. [setmsys2]: https://github.com/msys2/setup-msys2#readme [ne2w]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml [ave5819]: https://github.com/actions/virtual-environments/pull/5819 [L151]: https://github.com/status-im/nim-codex/blob/ci/msys2/.github/workflows/ci.yml#L151 [succeed]: https://github.com/status-im/nim-codex/actions/runs/2606854455 [ne2wL155]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml#L155-L159 [ne2config]: https://github.com/status-im/nimbus-eth2/blob/stable/config.nims#L43-L49
2022-07-04 02:37:34 +00:00
Speed up development (#303) * [ci] Extract setup of nimbus build system into separate file * [ci] Cleanup NodeJS setup * [ci] Use amd64 by default * [ci] Separate steps for unit, contract and integration tests * [ci] Separate build job for coverage that is allowed to fail * [ci] Separate build job for Nim 1.2 * [ci] Calculate coverage on unit tests only This enables faster development cycles at the expense of code coverage accuracy. * [PoR] Test proofs of retrievability with a small block size To get a reasonable test execution time. * [ci] Set NIM_COMMIT environment variable To prevent subsequent `make` command to build a different version of Nim. * [ci] Checkout submodules recursively So that nimbus-build-system doesn't have to do it afterwards. * [ci] Update caches on every run Will automatically update caches when a new minor version of Nim is released. See also: https://github.com/actions/cache/blob/main/workarounds.md#update-a-cache * [ci] Set cache_nonce for all jobs, not just those in the matrix * [build] update to latest nimbus-build-system Requires an update to latest nim-nat-traversal * [erasure] Test erasure coding with a small block size To get a reasonable test execution time. * [erasure] fix typo * [PoR] Test PoR store with a small block size To get a reasonable test execution time. * [PoR] Test PoR network with a small block size To get a reasonable test execution time. * [ci] Ensure that unit tests are not dependent on Ethereum Start Ethereum node *after* the unit tests are run. * [ci] Cancel all other jobs when one fails Allows for faster feedback loops; a next attempt to run the CI will start sooner.
2022-11-14 13:50:00 +00:00
coverage:
runs-on: ubuntu-20.04
Speed up development (#303) * [ci] Extract setup of nimbus build system into separate file * [ci] Cleanup NodeJS setup * [ci] Use amd64 by default * [ci] Separate steps for unit, contract and integration tests * [ci] Separate build job for coverage that is allowed to fail * [ci] Separate build job for Nim 1.2 * [ci] Calculate coverage on unit tests only This enables faster development cycles at the expense of code coverage accuracy. * [PoR] Test proofs of retrievability with a small block size To get a reasonable test execution time. * [ci] Set NIM_COMMIT environment variable To prevent subsequent `make` command to build a different version of Nim. * [ci] Checkout submodules recursively So that nimbus-build-system doesn't have to do it afterwards. * [ci] Update caches on every run Will automatically update caches when a new minor version of Nim is released. See also: https://github.com/actions/cache/blob/main/workarounds.md#update-a-cache * [ci] Set cache_nonce for all jobs, not just those in the matrix * [build] update to latest nimbus-build-system Requires an update to latest nim-nat-traversal * [erasure] Test erasure coding with a small block size To get a reasonable test execution time. * [erasure] fix typo * [PoR] Test PoR store with a small block size To get a reasonable test execution time. * [PoR] Test PoR network with a small block size To get a reasonable test execution time. * [ci] Ensure that unit tests are not dependent on Ethereum Start Ethereum node *after* the unit tests are run. * [ci] Cancel all other jobs when one fails Allows for faster feedback loops; a next attempt to run the CI will start sooner.
2022-11-14 13:50:00 +00:00
steps:
- name: Checkout sources
uses: actions/checkout@v4
Speed up development (#303) * [ci] Extract setup of nimbus build system into separate file * [ci] Cleanup NodeJS setup * [ci] Use amd64 by default * [ci] Separate steps for unit, contract and integration tests * [ci] Separate build job for coverage that is allowed to fail * [ci] Separate build job for Nim 1.2 * [ci] Calculate coverage on unit tests only This enables faster development cycles at the expense of code coverage accuracy. * [PoR] Test proofs of retrievability with a small block size To get a reasonable test execution time. * [ci] Set NIM_COMMIT environment variable To prevent subsequent `make` command to build a different version of Nim. * [ci] Checkout submodules recursively So that nimbus-build-system doesn't have to do it afterwards. * [ci] Update caches on every run Will automatically update caches when a new minor version of Nim is released. See also: https://github.com/actions/cache/blob/main/workarounds.md#update-a-cache * [ci] Set cache_nonce for all jobs, not just those in the matrix * [build] update to latest nimbus-build-system Requires an update to latest nim-nat-traversal * [erasure] Test erasure coding with a small block size To get a reasonable test execution time. * [erasure] fix typo * [PoR] Test PoR store with a small block size To get a reasonable test execution time. * [PoR] Test PoR network with a small block size To get a reasonable test execution time. * [ci] Ensure that unit tests are not dependent on Ethereum Start Ethereum node *after* the unit tests are run. * [ci] Cancel all other jobs when one fails Allows for faster feedback loops; a next attempt to run the CI will start sooner.
2022-11-14 13:50:00 +00:00
with:
submodules: recursive
ref: ${{ github.event.pull_request.head.sha }}
Speed up development (#303) * [ci] Extract setup of nimbus build system into separate file * [ci] Cleanup NodeJS setup * [ci] Use amd64 by default * [ci] Separate steps for unit, contract and integration tests * [ci] Separate build job for coverage that is allowed to fail * [ci] Separate build job for Nim 1.2 * [ci] Calculate coverage on unit tests only This enables faster development cycles at the expense of code coverage accuracy. * [PoR] Test proofs of retrievability with a small block size To get a reasonable test execution time. * [ci] Set NIM_COMMIT environment variable To prevent subsequent `make` command to build a different version of Nim. * [ci] Checkout submodules recursively So that nimbus-build-system doesn't have to do it afterwards. * [ci] Update caches on every run Will automatically update caches when a new minor version of Nim is released. See also: https://github.com/actions/cache/blob/main/workarounds.md#update-a-cache * [ci] Set cache_nonce for all jobs, not just those in the matrix * [build] update to latest nimbus-build-system Requires an update to latest nim-nat-traversal * [erasure] Test erasure coding with a small block size To get a reasonable test execution time. * [erasure] fix typo * [PoR] Test PoR store with a small block size To get a reasonable test execution time. * [PoR] Test PoR network with a small block size To get a reasonable test execution time. * [ci] Ensure that unit tests are not dependent on Ethereum Start Ethereum node *after* the unit tests are run. * [ci] Cancel all other jobs when one fails Allows for faster feedback loops; a next attempt to run the CI will start sooner.
2022-11-14 13:50:00 +00:00
- name: Setup Nimbus Build System
uses: ./.github/actions/nimbus-build-system
with:
os: linux
nim_version: ${{ env.nim_version }}
Speed up development (#303) * [ci] Extract setup of nimbus build system into separate file * [ci] Cleanup NodeJS setup * [ci] Use amd64 by default * [ci] Separate steps for unit, contract and integration tests * [ci] Separate build job for coverage that is allowed to fail * [ci] Separate build job for Nim 1.2 * [ci] Calculate coverage on unit tests only This enables faster development cycles at the expense of code coverage accuracy. * [PoR] Test proofs of retrievability with a small block size To get a reasonable test execution time. * [ci] Set NIM_COMMIT environment variable To prevent subsequent `make` command to build a different version of Nim. * [ci] Checkout submodules recursively So that nimbus-build-system doesn't have to do it afterwards. * [ci] Update caches on every run Will automatically update caches when a new minor version of Nim is released. See also: https://github.com/actions/cache/blob/main/workarounds.md#update-a-cache * [ci] Set cache_nonce for all jobs, not just those in the matrix * [build] update to latest nimbus-build-system Requires an update to latest nim-nat-traversal * [erasure] Test erasure coding with a small block size To get a reasonable test execution time. * [erasure] fix typo * [PoR] Test PoR store with a small block size To get a reasonable test execution time. * [PoR] Test PoR network with a small block size To get a reasonable test execution time. * [ci] Ensure that unit tests are not dependent on Ethereum Start Ethereum node *after* the unit tests are run. * [ci] Cancel all other jobs when one fails Allows for faster feedback loops; a next attempt to run the CI will start sooner.
2022-11-14 13:50:00 +00:00
- name: Generate coverage data
run: |
# make -j${ncpu} coverage
make -j${ncpu} coverage-script
Speed up development (#303) * [ci] Extract setup of nimbus build system into separate file * [ci] Cleanup NodeJS setup * [ci] Use amd64 by default * [ci] Separate steps for unit, contract and integration tests * [ci] Separate build job for coverage that is allowed to fail * [ci] Separate build job for Nim 1.2 * [ci] Calculate coverage on unit tests only This enables faster development cycles at the expense of code coverage accuracy. * [PoR] Test proofs of retrievability with a small block size To get a reasonable test execution time. * [ci] Set NIM_COMMIT environment variable To prevent subsequent `make` command to build a different version of Nim. * [ci] Checkout submodules recursively So that nimbus-build-system doesn't have to do it afterwards. * [ci] Update caches on every run Will automatically update caches when a new minor version of Nim is released. See also: https://github.com/actions/cache/blob/main/workarounds.md#update-a-cache * [ci] Set cache_nonce for all jobs, not just those in the matrix * [build] update to latest nimbus-build-system Requires an update to latest nim-nat-traversal * [erasure] Test erasure coding with a small block size To get a reasonable test execution time. * [erasure] fix typo * [PoR] Test PoR store with a small block size To get a reasonable test execution time. * [PoR] Test PoR network with a small block size To get a reasonable test execution time. * [ci] Ensure that unit tests are not dependent on Ethereum Start Ethereum node *after* the unit tests are run. * [ci] Cancel all other jobs when one fails Allows for faster feedback loops; a next attempt to run the CI will start sooner.
2022-11-14 13:50:00 +00:00
shell: bash
- name: Upload coverage data to Codecov
uses: codecov/codecov-action@v4
ci: update GitHub Actions CI workflow to use msys2/setup-msys2@v2 Main goal is to update the nim-codex CI workflow to use the [`msys2/setup-msys2@v2`][setmsys2] GitHub Action, as used by the [nimbus-eth2 workflow][ne2w] per changes made to it several months ago. The `msys2/setup-msys2@v2` action has been used by other Status-org projects prior to this commit, e.g. nim-leopard, nim-datastore, nim-status. A fix is included for the failing macOS builds, related to [actions/virtual-environments#5819][ave5819]. See [L151][L151]. All builds [succeed][succeed] (with build arch-targets verified as far as possible via `ldd`, `otool`, `ntldd`), including `linux-i386` and `windows-i386`, though the `i386` builds are presently disabled (commented out). The `i386` builds can be enabled simply by uncommenting: ``` # - os: linux # cpu: i386 ... # - os: windows # cpu: i386 ``` The resulting `.github/workflows/ci.yml` is a "remix" of the current workflows for nimbus-eth2 and nim-codex (i.e. prior to this commit) along with techniques learned from developing workflows for other Status-org repos. Some comments and code-reorg help to clarify/explain what's done in the `Derive environment variables` step. `-d:limitStackUsage` has been adopted for `linux-amd64` builds from [nimbus-eth2's workflow][ne2wL155] and [related code][ne2config] has been copied into `config.nims` `-d:limitStackUsage` can easily be dropped if it's not desirable for Codex. Build targets use `-latest` for `runs-on`, i.e. `macos-latest`, `ubuntu-latest`, `windows-latest`. Through a combination of local testing and iterative pushes to GitHub, the workflow's embedded Bash scripts have been revised to include only the necessary steps for all builds to succeed, including `linux-i386` and `windows-i386`. The GitHub Actions workflow `.github/workflows/codecov.yml` has been removed, while coverage data generation/upload steps have been added to `.github/workflows/ci.yml` as the final steps conditional on `if: runner.os == 'Linux' && matrix.target.cpu == 'amd64' && matrix.nim_branch == matrix.cov_branch`. A redundant `--passC:'-m32 -mno-adx'` is used for `linux-i386` builds; the redundant flags do not affect the build, but can be helpful when eyeballing GitHub Actions builds with increased compile-time verbosity. Some variable expansions used in `github/workflows/ci.yml` could result in compilation failures if related paths include whitespace. It's not a problem for this commit but could be a problem for a user copy-pasting from the workflow; solving that problem is left as an exercise for the reader. [setmsys2]: https://github.com/msys2/setup-msys2#readme [ne2w]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml [ave5819]: https://github.com/actions/virtual-environments/pull/5819 [L151]: https://github.com/status-im/nim-codex/blob/ci/msys2/.github/workflows/ci.yml#L151 [succeed]: https://github.com/status-im/nim-codex/actions/runs/2606854455 [ne2wL155]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml#L155-L159 [ne2config]: https://github.com/status-im/nimbus-eth2/blob/stable/config.nims#L43-L49
2022-07-04 02:37:34 +00:00
with:
directory: ./coverage/
fail_ci_if_error: true
files: ./coverage/coverage.f.info
flags: unittests
name: codecov-umbrella
token: ${{ secrets.CODECOV_TOKEN }}
ci: update GitHub Actions CI workflow to use msys2/setup-msys2@v2 Main goal is to update the nim-codex CI workflow to use the [`msys2/setup-msys2@v2`][setmsys2] GitHub Action, as used by the [nimbus-eth2 workflow][ne2w] per changes made to it several months ago. The `msys2/setup-msys2@v2` action has been used by other Status-org projects prior to this commit, e.g. nim-leopard, nim-datastore, nim-status. A fix is included for the failing macOS builds, related to [actions/virtual-environments#5819][ave5819]. See [L151][L151]. All builds [succeed][succeed] (with build arch-targets verified as far as possible via `ldd`, `otool`, `ntldd`), including `linux-i386` and `windows-i386`, though the `i386` builds are presently disabled (commented out). The `i386` builds can be enabled simply by uncommenting: ``` # - os: linux # cpu: i386 ... # - os: windows # cpu: i386 ``` The resulting `.github/workflows/ci.yml` is a "remix" of the current workflows for nimbus-eth2 and nim-codex (i.e. prior to this commit) along with techniques learned from developing workflows for other Status-org repos. Some comments and code-reorg help to clarify/explain what's done in the `Derive environment variables` step. `-d:limitStackUsage` has been adopted for `linux-amd64` builds from [nimbus-eth2's workflow][ne2wL155] and [related code][ne2config] has been copied into `config.nims` `-d:limitStackUsage` can easily be dropped if it's not desirable for Codex. Build targets use `-latest` for `runs-on`, i.e. `macos-latest`, `ubuntu-latest`, `windows-latest`. Through a combination of local testing and iterative pushes to GitHub, the workflow's embedded Bash scripts have been revised to include only the necessary steps for all builds to succeed, including `linux-i386` and `windows-i386`. The GitHub Actions workflow `.github/workflows/codecov.yml` has been removed, while coverage data generation/upload steps have been added to `.github/workflows/ci.yml` as the final steps conditional on `if: runner.os == 'Linux' && matrix.target.cpu == 'amd64' && matrix.nim_branch == matrix.cov_branch`. A redundant `--passC:'-m32 -mno-adx'` is used for `linux-i386` builds; the redundant flags do not affect the build, but can be helpful when eyeballing GitHub Actions builds with increased compile-time verbosity. Some variable expansions used in `github/workflows/ci.yml` could result in compilation failures if related paths include whitespace. It's not a problem for this commit but could be a problem for a user copy-pasting from the workflow; solving that problem is left as an exercise for the reader. [setmsys2]: https://github.com/msys2/setup-msys2#readme [ne2w]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml [ave5819]: https://github.com/actions/virtual-environments/pull/5819 [L151]: https://github.com/status-im/nim-codex/blob/ci/msys2/.github/workflows/ci.yml#L151 [succeed]: https://github.com/status-im/nim-codex/actions/runs/2606854455 [ne2wL155]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml#L155-L159 [ne2config]: https://github.com/status-im/nimbus-eth2/blob/stable/config.nims#L43-L49
2022-07-04 02:37:34 +00:00
verbose: true