Commit Graph

54 Commits

Author SHA1 Message Date
Slava 1e957da109
CI update (#695)
* fix: update codecov action

* fix: remove nim v2 from Nim-matrix
2024-02-07 19:47:40 +00:00
Slava 403b9baf9f
Add Nim-matrix workflow to run on merge queue (#693)
* Add Nim-matrix workflow to run on merge queue

* Use reusable workflows for CI and Nim-matrix
2024-02-06 12:56:27 +02:00
Slava ae61c29b3f
ci: fix docker workflow after actions update (#689)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-01-29 20:12:07 +02:00
Slava 34095461ae
ci: update actions to lates major versions (#688) 2024-01-29 17:21:52 +02:00
Eric df6b9c6760
Workaround for Hardhat last block timestamp bug (#644)
* Workaround for Hardhat timestamp bug

Likely due to a Hardhat bug in which the callbacks for subscription events are called and awaited before updating its local understanding of the last block time, Hardhat will report a block time in the `newHeads` event that is generally 1 second before the time reported from `getLatestBlock.timestamp`. This was causing issues with the OnChainClock's offset and therefore the `now()` used by the `OnChainClock` would sometimes be off by a second (or more), causing tests to fail.

This commit introduce a `codex_use_hardhat` compilation flag, that when set, will always get the latest block timestamp from Hardhat via the `getLatestBlock.timestamp` RPC call for `OnChainClock.now` calls. Otherwise, the last block timestamp reported in the `newHeads` event will be used.

Update the docker dist tests compilation flag for simulated proof failures (it was not correct), and explicitly add the `codex_use_hardhat=false` for clarity.

* enable simulated proof failures for coverage

* comment out failing test on linux -- will be replaced

* bump codex contracts eth

* add back clock offset for non-hardhat cases

* bump codex-contracts-eth

increases pointer by 67 blocks each period increase

* Add `codex_use_hardhat` flag to coverage tests
2023-12-20 08:06:24 +11:00
Eric 1186616381
Updates reference to continuous tests workflow in cs-codex-dist-tests (#658) 2023-12-19 06:47:23 +00:00
Slava a805af02c1
Run C-Tests on master commits (#636)
https://github.com/codex-storage/cs-codex-dist-tests/pull/84
2023-11-27 13:32:13 +02:00
Slava 5b30bfd427
Use token for coverage data upload to Codecov (#583)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2023-10-18 14:35:46 +03:00
Ben Bierens fede106118
Numbered log entries (#559)
* add numbered log entries (hack)

* Changed from CLI argument to compiler define

* Use uint64 for counting

---------

Co-authored-by: gmega <giuliano.mega@gmail.com>
2023-10-03 11:53:58 +02:00
Slava fc3eac9dbc
Use latest tag for Tags and Default branch only (#540) 2023-08-30 13:09:52 +03:00
Slava ba41b5a232
Add reusable workflow for Docker (codex-storage/cs-codex-dist-tests/issues/42) (#518) 2023-08-17 06:04:50 +03:00
Jaremy Creechley 7efa9177df
Bump deps take2 (#492)
* extra utilities and tweaks
* add atlas lock
* update ignores
* break build into it's own script
* update url rules
* base off codexdht's
* compile fixes for Nim 1.6.14
* update submodules
* convert mapFailure to procs to work around type resolution issues
* add toml parser for multiaddress
* change error type on keyutils
* bump nimbus build to use 1.6.14
* update gitignore
* adding new deps submodules
* bump nim ci version
* even more fixes
* more libp2p changes
* update keys
* fix eventually function
* adding coverage test file
* move coverage to build.nims
* use nimcache/coverage
* move libp2p import for tests into helper.nim
* remove named bin
* bug fixes for networkpeers (from Dmitriy)

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2023-08-01 16:47:57 -07:00
Jaremy Creechley 8bebc90428
split windows CI tests into parts (#485)
* setup to use env variable for parallel tests
* use explicit targets
* use target in includes
* test windows split
* try simpler logic
* only use include in ci.yml
* add cpu
* fix name
* re-add unit tests!
* restore coverage
* not sure why env broke there
* startup node for part 1 & 2 tests
* fixup part 1 & 2 tests
* split windows into 3!
2023-07-19 16:00:34 -07:00
Slava 6dd7e55719
Enable codex_enable_api_debug_peers (#471) (#474)
- Enable codex_enable_api_debug_peers

 - Add NAT_IP_AUTO for Dist-Tests
2023-07-10 16:15:06 +03:00
Slava f053135f68
feat: add docker auto-builds (#461)
https://github.com/codex-storage/nim-codex/issues/412
2023-07-04 14:46:49 +03:00
Slava 04ea8c36bf
Update CI workflow trigger branch (#464)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2023-07-03 17:12:21 +03:00
Slava e0d127f1d5
Add Feature request template (#465) 2023-06-29 08:44:04 +03:00
Ben Bierens 8c232b6759
Docker multiarch debug (#447)
* Uses correct string for marketplace address

* first steps towards support for arm64

* Applies multiarch ubuntu dockerfile as codex-dockerfile.

* Add `--simulate-proof-failures` env variable, update docker-compose to point to slimmed image

* Add image to CI, and update startCodex.sh

* Sets up separate docker build for arm

* Update arm64 arm of docker ci

* [docker] modify startCodex.sh

- include overridden node name in log output if specified in test
- quote `—log-level` value so that multiple log levels can be specified
- ensure any CLI parameter env vars are passed through to the codex binary, instead of conditionally including them
- add `—persistence`
- add `—validator`

* fixes load and push for amd docker build

---------

Co-authored-by: Eric Mastro <github@egonat.me>
2023-06-19 08:28:27 +02:00
Slava cfe9225be1
Fix Open API workflow branch (codex-storage/infra-codex/issues/5) (#442)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2023-06-12 12:34:44 +03:00
Slava 3e7ce137a4
Update links to codex-storage organization (#420) 2023-05-23 23:01:13 +03:00
markspanbroek 5ce655db47
Fix CI (#408)
* [CI] workaround for hardhat crash

See: https://github.com/NomicFoundation/hardhat/issues/3877

* [ci] use nodejs from windows PATH
2023-05-01 16:20:50 +02:00
Ben Bierens 80106cd3d2
Creates marketplace_address.json during docker start. (#397)
* Creates marketplace_address.json during docker start.

Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* Enables manual start for docker workflow

---------

Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
2023-04-18 14:17:24 +02:00
Adam Uhlíř 067c1c9625
ci: lint openapi on all prs (#381) 2023-03-23 12:19:33 +01:00
Adam Uhlíř 5ae545bc3c
docs: openapi specification (#355)
Co-authored-by: Eric Mastro <github@egonat.me>
2023-03-15 14:10:53 +01:00
Ben Bierens c3095f78a2
Adds linux/arm/v7 (#360) 2023-03-12 11:27:33 +01:00
markspanbroek 7a0a48e4a5
Fix warnings (drops Nim 1.2) (#348)
* [build] disable XCannotRaiseY hint

There are too many {.raises:[Defect].} in the
libraries that we use, drowning out all other
warnings and hints

* [build] disable BareExcept warning

Not yet enabled in a released version of Nim,
so libraries that we depend on have not fixed
this yet, drowning out our own hints and warnings

* [build] disable DotLikeOps warning

dot-like ops were an experiment that is not going
land in Nim

* [build] compile log statements in tests

When running tests, all log statements are compiled.
They are filtered out at runtime during a test run.

* [build] do not build executable when running unit test

It's already built in the integration test

* [build] Fix warnings

- remove unused code
- remove unused imports
- stop using deprecated stuff

* [build] Put compiler flags behind nim version checks

* [CI] remove Nim 1.2 compatibility
2023-03-09 12:23:45 +01:00
Ben Bierens d263ca0735
Docker build (#354)
* Sets up working dockerized build and codex docker image creation

* Making codex configurable from the docker environment

* Sets up two networks with three codex nodes

* enables and exposes metrics endpoint for first node

* Manually performed two-client test scenario with docker containers

* Sets up docker-ignore and docker github workflow

* Wires up all codex CLI arguments to docker env vars

* Makes API_PORT variable optional as well

* Removes duplicate docker-login step

* Fixes path to docker file

* Switches target dockerhub for debugging

* Adds git tag info to --version output

* Exposes version information via debug endpoint

* Debugging docker image

* specifies target platforms for docker build

* specifies platform for QEMU and buildx steps

* Attempt to debug line endings

* Disables march-native in config.nims as test

* Applies make argument to disable architecture optimization during docker build

* Removes subset version tags from docker build

* Restore multi-arch build

* Removes docker-build test branch from CI branches
2023-03-08 12:45:55 +01:00
markspanbroek 82e7a2dde6
Fix CI (#339)
* [ci] Use Nim version 1.6.10 instead of version-1-6

* [ci] use nimbus-build-system branch support-shallow-checkout
2023-02-06 17:15:02 +01:00
Eric Mastro df729be261
[marketplace] support latest contracts changes (#327)
* [marketplace] support removal of Storage contract

* [marketplace] change submod dep dagger-contracts to codex-contracts-eth
2023-01-19 16:58:04 +11:00
markspanbroek 9143a25fa1
[ci] Add tips for maintaining CI (#311)
* [ci] Add tips for maintaining CI

Lessons learned from speeding up the CI in
https://github.com/status-im/nim-codex/pull/303

* [ci] Readme: add tip about timing of every test

As suggested by @Menduist

* [ci] Readme: add link to github actions usage limits

As suggested by @Menduist
2022-11-16 14:58:49 +01:00
markspanbroek ccf3d04dc8
Speed up development (#303)
* [ci] Extract setup of nimbus build system into separate file

* [ci] Cleanup NodeJS setup

* [ci] Use amd64 by default

* [ci] Separate steps for unit, contract and integration tests

* [ci] Separate build job for coverage that is allowed to fail

* [ci] Separate build job for Nim 1.2

* [ci] Calculate coverage on unit tests only

This enables faster development cycles at
the expense of code coverage accuracy.

* [PoR] Test proofs of retrievability with a small block size

To get a reasonable test execution time.

* [ci] Set NIM_COMMIT environment variable

To prevent subsequent `make` command to build a
different version of Nim.

* [ci] Checkout submodules recursively

So that nimbus-build-system doesn't have to
do it afterwards.

* [ci] Update caches on every run

Will automatically update caches when a new minor version
of Nim is released.

See also:
https://github.com/actions/cache/blob/main/workarounds.md#update-a-cache

* [ci] Set cache_nonce for all jobs, not just those in the matrix

* [build] update to latest nimbus-build-system

Requires an update to latest nim-nat-traversal

* [erasure] Test erasure coding with a small block size

To get a reasonable test execution time.

* [erasure] fix typo

* [PoR] Test PoR store with a small block size

To get a reasonable test execution time.

* [PoR] Test PoR network with a small block size

To get a reasonable test execution time.

* [ci] Ensure that unit tests are not dependent on Ethereum

Start Ethereum node *after* the unit tests are run.

* [ci] Cancel all other jobs when one fails

Allows for faster feedback loops; a next attempt to run
the CI will start sooner.
2022-11-14 14:50:00 +01:00
Dmitriy Ryajov fd56875667
updating bug report template environment section 2022-09-08 12:35:07 -06:00
Dmitriy Ryajov d074d3b351
Adding bug report template 2022-09-08 12:32:32 -06:00
Michael Bradley, Jr c64259b0d5 ci: more comments added to `.github/workflows/ci.yml` 2022-07-06 19:03:10 -05:00
Michael Bradley, Jr 2b3195518d ci: update GitHub Actions CI workflow to use msys2/setup-msys2@v2
Main goal is to update the nim-codex CI workflow to use the
[`msys2/setup-msys2@v2`][setmsys2] GitHub Action, as used by the
[nimbus-eth2 workflow][ne2w] per changes made to it several months ago. The
`msys2/setup-msys2@v2` action has been used by other Status-org projects prior
to this commit, e.g. nim-leopard, nim-datastore, nim-status.

A fix is included for the failing macOS builds, related to
[actions/virtual-environments#5819][ave5819]. See [L151][L151].

All builds [succeed][succeed] (with build arch-targets verified as far as
possible via `ldd`, `otool`, `ntldd`), including `linux-i386` and
`windows-i386`, though the `i386` builds are presently disabled (commented
out). The `i386` builds can be enabled simply by uncommenting:

 ```
 # - os: linux
 #   cpu: i386
 ...
 # - os: windows
 #   cpu: i386
 ```

The resulting `.github/workflows/ci.yml` is a "remix" of the current workflows
for nimbus-eth2 and nim-codex (i.e. prior to this commit) along with techniques
learned from developing workflows for other Status-org repos. Some comments and
code-reorg help to clarify/explain what's done in the
`Derive environment variables` step.

`-d:limitStackUsage` has been adopted for `linux-amd64` builds from
[nimbus-eth2's workflow][ne2wL155] and [related code][ne2config] has been
copied into `config.nims`

`-d:limitStackUsage` can easily be dropped if it's not desirable for Codex.

Build targets use `-latest` for `runs-on`, i.e. `macos-latest`,
`ubuntu-latest`, `windows-latest`.

Through a combination of local testing and iterative pushes to GitHub, the
workflow's embedded Bash scripts have been revised to include only the
necessary steps for all builds to succeed, including `linux-i386` and
`windows-i386`.

The GitHub Actions workflow `.github/workflows/codecov.yml` has been removed,
while coverage data generation/upload steps have been added to
`.github/workflows/ci.yml` as the final steps conditional on
`if: runner.os == 'Linux' && matrix.target.cpu == 'amd64' && matrix.nim_branch == matrix.cov_branch`.

A redundant `--passC:'-m32 -mno-adx'` is used for `linux-i386` builds; the
redundant flags do not affect the build, but can be helpful when eyeballing
GitHub Actions builds with increased compile-time verbosity.

Some variable expansions used in `github/workflows/ci.yml` could result in
compilation failures if related paths include whitespace. It's not a problem
for this commit but could be a problem for a user copy-pasting from the
workflow; solving that problem is left as an exercise for the reader.

[setmsys2]: https://github.com/msys2/setup-msys2#readme
[ne2w]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml
[ave5819]: https://github.com/actions/virtual-environments/pull/5819
[L151]: https://github.com/status-im/nim-codex/blob/ci/msys2/.github/workflows/ci.yml#L151
[succeed]: https://github.com/status-im/nim-codex/actions/runs/2606854455
[ne2wL155]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml#L155-L159
[ne2config]: https://github.com/status-im/nimbus-eth2/blob/stable/config.nims#L43-L49
2022-07-06 19:03:10 -05:00
Dmitriy Ryajov 56b80d6f6d
Por serialize (#106)
* move por into storage proofs

* use SeekableStream

* adding serialization for por

* remove leftovers

* add empty block support

* add basic por test

* rename block exchange for consistency

* add storageproofstests

* moving timing to storageproofs

* fix imports

* fix imports

* fix imports

* add top level exports

* move delete blocks helper to helpers

* more import/export fixes

* cleanup

* more import fixes

* fix unused warnings

* detect corrupt blocks tests

* add serialization tests

* move init method around

* bump asynctest

* fix CID version

* get rid of warning

* wip: fix CI

* increase CI timeout
2022-05-23 23:24:15 -06:00
Michael Bradley 209343087c
Change every dagger to codex (#102) 2022-05-19 13:56:03 -06:00
Mark Spanbroek 2992229c3a [ci] limit CI runs to 40 minutes 2022-05-19 10:38:19 +02:00
Michael Bradley c2d7fe3fd7
[ci] generate coverage data and upload to Codecov (#93) 2022-05-10 14:41:44 -06:00
Tanguy c06c9b578c
Various dev-ops things (#85)
* Add metrics server
* Add --version
* Fixes for chronicles=trace + ci
2022-04-14 12:49:03 +02:00
markspanbroek 168468a3fd
[ci] Speed up CI by eliminating unnecessary runs (#79) 2022-04-11 12:01:32 -06:00
Michael Bradley e92fbd4ad1
[ci] use matrix.branch to specify NIM_COMMIT make var of nimbus-build-system (#65)
See: https://github.com/status-im/nimbus-build-system#nim_commit
2022-04-01 10:22:10 -06:00
Michael Bradley 1e5b41a447
add github.com/status-im/nim-leopard to vendor (#61)
* [leopard] add github.com/status-im/nim-leopard to vendor

* [leopard] specify absolute path for nimcache dir

* [leopard] make adjustments to nim-dagger CI GHA workflow
2022-03-30 09:34:47 -06:00
Mark Spanbroek 03c8ceccf9 [contracts] Separate nimble task to run all tests
A simple "nimble test" won't require you to run a hardhat
node with the contracts, but a "nimble testAll" will.
2022-03-28 11:59:01 +02:00
Mark Spanbroek 28294d431a [contracts] Tests now also work in Windows 2022-03-28 11:59:01 +02:00
Mark Spanbroek 4fd92b52eb [contracts] Install NodeJS in CI environment 2022-03-28 11:59:01 +02:00
Dmitriy Ryajov 1663c1d71d don't run win i386 tests 2021-11-25 09:37:12 +01:00
Dmitriy Ryajov aa5260df73 update binaries cache 2021-11-25 09:37:12 +01:00
Dmitriy Ryajov 879a76fa41 make test 2021-11-25 09:37:12 +01:00
Dmitriy Ryajov 4d136a2e4c make ci use new build system 2021-11-25 09:37:12 +01:00