Commit Graph

41 Commits

Author SHA1 Message Date
Slava 6dd7e55719
Enable codex_enable_api_debug_peers (#471) (#474)
- Enable codex_enable_api_debug_peers

 - Add NAT_IP_AUTO for Dist-Tests
2023-07-10 16:15:06 +03:00
Slava f053135f68
feat: add docker auto-builds (#461)
https://github.com/codex-storage/nim-codex/issues/412
2023-07-04 14:46:49 +03:00
Slava 04ea8c36bf
Update CI workflow trigger branch (#464)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2023-07-03 17:12:21 +03:00
Slava e0d127f1d5
Add Feature request template (#465) 2023-06-29 08:44:04 +03:00
Ben Bierens 8c232b6759
Docker multiarch debug (#447)
* Uses correct string for marketplace address

* first steps towards support for arm64

* Applies multiarch ubuntu dockerfile as codex-dockerfile.

* Add `--simulate-proof-failures` env variable, update docker-compose to point to slimmed image

* Add image to CI, and update startCodex.sh

* Sets up separate docker build for arm

* Update arm64 arm of docker ci

* [docker] modify startCodex.sh

- include overridden node name in log output if specified in test
- quote `—log-level` value so that multiple log levels can be specified
- ensure any CLI parameter env vars are passed through to the codex binary, instead of conditionally including them
- add `—persistence`
- add `—validator`

* fixes load and push for amd docker build

---------

Co-authored-by: Eric Mastro <github@egonat.me>
2023-06-19 08:28:27 +02:00
Slava cfe9225be1
Fix Open API workflow branch (codex-storage/infra-codex/issues/5) (#442)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2023-06-12 12:34:44 +03:00
Slava 3e7ce137a4
Update links to codex-storage organization (#420) 2023-05-23 23:01:13 +03:00
markspanbroek 5ce655db47
Fix CI (#408)
* [CI] workaround for hardhat crash

See: https://github.com/NomicFoundation/hardhat/issues/3877

* [ci] use nodejs from windows PATH
2023-05-01 16:20:50 +02:00
Ben Bierens 80106cd3d2
Creates marketplace_address.json during docker start. (#397)
* Creates marketplace_address.json during docker start.

Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* Enables manual start for docker workflow

---------

Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
2023-04-18 14:17:24 +02:00
Adam Uhlíř 067c1c9625
ci: lint openapi on all prs (#381) 2023-03-23 12:19:33 +01:00
Adam Uhlíř 5ae545bc3c
docs: openapi specification (#355)
Co-authored-by: Eric Mastro <github@egonat.me>
2023-03-15 14:10:53 +01:00
Ben Bierens c3095f78a2
Adds linux/arm/v7 (#360) 2023-03-12 11:27:33 +01:00
markspanbroek 7a0a48e4a5
Fix warnings (drops Nim 1.2) (#348)
* [build] disable XCannotRaiseY hint

There are too many {.raises:[Defect].} in the
libraries that we use, drowning out all other
warnings and hints

* [build] disable BareExcept warning

Not yet enabled in a released version of Nim,
so libraries that we depend on have not fixed
this yet, drowning out our own hints and warnings

* [build] disable DotLikeOps warning

dot-like ops were an experiment that is not going
land in Nim

* [build] compile log statements in tests

When running tests, all log statements are compiled.
They are filtered out at runtime during a test run.

* [build] do not build executable when running unit test

It's already built in the integration test

* [build] Fix warnings

- remove unused code
- remove unused imports
- stop using deprecated stuff

* [build] Put compiler flags behind nim version checks

* [CI] remove Nim 1.2 compatibility
2023-03-09 12:23:45 +01:00
Ben Bierens d263ca0735
Docker build (#354)
* Sets up working dockerized build and codex docker image creation

* Making codex configurable from the docker environment

* Sets up two networks with three codex nodes

* enables and exposes metrics endpoint for first node

* Manually performed two-client test scenario with docker containers

* Sets up docker-ignore and docker github workflow

* Wires up all codex CLI arguments to docker env vars

* Makes API_PORT variable optional as well

* Removes duplicate docker-login step

* Fixes path to docker file

* Switches target dockerhub for debugging

* Adds git tag info to --version output

* Exposes version information via debug endpoint

* Debugging docker image

* specifies target platforms for docker build

* specifies platform for QEMU and buildx steps

* Attempt to debug line endings

* Disables march-native in config.nims as test

* Applies make argument to disable architecture optimization during docker build

* Removes subset version tags from docker build

* Restore multi-arch build

* Removes docker-build test branch from CI branches
2023-03-08 12:45:55 +01:00
markspanbroek 82e7a2dde6
Fix CI (#339)
* [ci] Use Nim version 1.6.10 instead of version-1-6

* [ci] use nimbus-build-system branch support-shallow-checkout
2023-02-06 17:15:02 +01:00
Eric Mastro df729be261
[marketplace] support latest contracts changes (#327)
* [marketplace] support removal of Storage contract

* [marketplace] change submod dep dagger-contracts to codex-contracts-eth
2023-01-19 16:58:04 +11:00
markspanbroek 9143a25fa1
[ci] Add tips for maintaining CI (#311)
* [ci] Add tips for maintaining CI

Lessons learned from speeding up the CI in
https://github.com/status-im/nim-codex/pull/303

* [ci] Readme: add tip about timing of every test

As suggested by @Menduist

* [ci] Readme: add link to github actions usage limits

As suggested by @Menduist
2022-11-16 14:58:49 +01:00
markspanbroek ccf3d04dc8
Speed up development (#303)
* [ci] Extract setup of nimbus build system into separate file

* [ci] Cleanup NodeJS setup

* [ci] Use amd64 by default

* [ci] Separate steps for unit, contract and integration tests

* [ci] Separate build job for coverage that is allowed to fail

* [ci] Separate build job for Nim 1.2

* [ci] Calculate coverage on unit tests only

This enables faster development cycles at
the expense of code coverage accuracy.

* [PoR] Test proofs of retrievability with a small block size

To get a reasonable test execution time.

* [ci] Set NIM_COMMIT environment variable

To prevent subsequent `make` command to build a
different version of Nim.

* [ci] Checkout submodules recursively

So that nimbus-build-system doesn't have to
do it afterwards.

* [ci] Update caches on every run

Will automatically update caches when a new minor version
of Nim is released.

See also:
https://github.com/actions/cache/blob/main/workarounds.md#update-a-cache

* [ci] Set cache_nonce for all jobs, not just those in the matrix

* [build] update to latest nimbus-build-system

Requires an update to latest nim-nat-traversal

* [erasure] Test erasure coding with a small block size

To get a reasonable test execution time.

* [erasure] fix typo

* [PoR] Test PoR store with a small block size

To get a reasonable test execution time.

* [PoR] Test PoR network with a small block size

To get a reasonable test execution time.

* [ci] Ensure that unit tests are not dependent on Ethereum

Start Ethereum node *after* the unit tests are run.

* [ci] Cancel all other jobs when one fails

Allows for faster feedback loops; a next attempt to run
the CI will start sooner.
2022-11-14 14:50:00 +01:00
Dmitriy Ryajov fd56875667
updating bug report template environment section 2022-09-08 12:35:07 -06:00
Dmitriy Ryajov d074d3b351
Adding bug report template 2022-09-08 12:32:32 -06:00
Michael Bradley, Jr c64259b0d5 ci: more comments added to `.github/workflows/ci.yml` 2022-07-06 19:03:10 -05:00
Michael Bradley, Jr 2b3195518d ci: update GitHub Actions CI workflow to use msys2/setup-msys2@v2
Main goal is to update the nim-codex CI workflow to use the
[`msys2/setup-msys2@v2`][setmsys2] GitHub Action, as used by the
[nimbus-eth2 workflow][ne2w] per changes made to it several months ago. The
`msys2/setup-msys2@v2` action has been used by other Status-org projects prior
to this commit, e.g. nim-leopard, nim-datastore, nim-status.

A fix is included for the failing macOS builds, related to
[actions/virtual-environments#5819][ave5819]. See [L151][L151].

All builds [succeed][succeed] (with build arch-targets verified as far as
possible via `ldd`, `otool`, `ntldd`), including `linux-i386` and
`windows-i386`, though the `i386` builds are presently disabled (commented
out). The `i386` builds can be enabled simply by uncommenting:

 ```
 # - os: linux
 #   cpu: i386
 ...
 # - os: windows
 #   cpu: i386
 ```

The resulting `.github/workflows/ci.yml` is a "remix" of the current workflows
for nimbus-eth2 and nim-codex (i.e. prior to this commit) along with techniques
learned from developing workflows for other Status-org repos. Some comments and
code-reorg help to clarify/explain what's done in the
`Derive environment variables` step.

`-d:limitStackUsage` has been adopted for `linux-amd64` builds from
[nimbus-eth2's workflow][ne2wL155] and [related code][ne2config] has been
copied into `config.nims`

`-d:limitStackUsage` can easily be dropped if it's not desirable for Codex.

Build targets use `-latest` for `runs-on`, i.e. `macos-latest`,
`ubuntu-latest`, `windows-latest`.

Through a combination of local testing and iterative pushes to GitHub, the
workflow's embedded Bash scripts have been revised to include only the
necessary steps for all builds to succeed, including `linux-i386` and
`windows-i386`.

The GitHub Actions workflow `.github/workflows/codecov.yml` has been removed,
while coverage data generation/upload steps have been added to
`.github/workflows/ci.yml` as the final steps conditional on
`if: runner.os == 'Linux' && matrix.target.cpu == 'amd64' && matrix.nim_branch == matrix.cov_branch`.

A redundant `--passC:'-m32 -mno-adx'` is used for `linux-i386` builds; the
redundant flags do not affect the build, but can be helpful when eyeballing
GitHub Actions builds with increased compile-time verbosity.

Some variable expansions used in `github/workflows/ci.yml` could result in
compilation failures if related paths include whitespace. It's not a problem
for this commit but could be a problem for a user copy-pasting from the
workflow; solving that problem is left as an exercise for the reader.

[setmsys2]: https://github.com/msys2/setup-msys2#readme
[ne2w]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml
[ave5819]: https://github.com/actions/virtual-environments/pull/5819
[L151]: https://github.com/status-im/nim-codex/blob/ci/msys2/.github/workflows/ci.yml#L151
[succeed]: https://github.com/status-im/nim-codex/actions/runs/2606854455
[ne2wL155]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml#L155-L159
[ne2config]: https://github.com/status-im/nimbus-eth2/blob/stable/config.nims#L43-L49
2022-07-06 19:03:10 -05:00
Dmitriy Ryajov 56b80d6f6d
Por serialize (#106)
* move por into storage proofs

* use SeekableStream

* adding serialization for por

* remove leftovers

* add empty block support

* add basic por test

* rename block exchange for consistency

* add storageproofstests

* moving timing to storageproofs

* fix imports

* fix imports

* fix imports

* add top level exports

* move delete blocks helper to helpers

* more import/export fixes

* cleanup

* more import fixes

* fix unused warnings

* detect corrupt blocks tests

* add serialization tests

* move init method around

* bump asynctest

* fix CID version

* get rid of warning

* wip: fix CI

* increase CI timeout
2022-05-23 23:24:15 -06:00
Michael Bradley 209343087c
Change every dagger to codex (#102) 2022-05-19 13:56:03 -06:00
Mark Spanbroek 2992229c3a [ci] limit CI runs to 40 minutes 2022-05-19 10:38:19 +02:00
Michael Bradley c2d7fe3fd7
[ci] generate coverage data and upload to Codecov (#93) 2022-05-10 14:41:44 -06:00
Tanguy c06c9b578c
Various dev-ops things (#85)
* Add metrics server
* Add --version
* Fixes for chronicles=trace + ci
2022-04-14 12:49:03 +02:00
markspanbroek 168468a3fd
[ci] Speed up CI by eliminating unnecessary runs (#79) 2022-04-11 12:01:32 -06:00
Michael Bradley e92fbd4ad1
[ci] use matrix.branch to specify NIM_COMMIT make var of nimbus-build-system (#65)
See: https://github.com/status-im/nimbus-build-system#nim_commit
2022-04-01 10:22:10 -06:00
Michael Bradley 1e5b41a447
add github.com/status-im/nim-leopard to vendor (#61)
* [leopard] add github.com/status-im/nim-leopard to vendor

* [leopard] specify absolute path for nimcache dir

* [leopard] make adjustments to nim-dagger CI GHA workflow
2022-03-30 09:34:47 -06:00
Mark Spanbroek 03c8ceccf9 [contracts] Separate nimble task to run all tests
A simple "nimble test" won't require you to run a hardhat
node with the contracts, but a "nimble testAll" will.
2022-03-28 11:59:01 +02:00
Mark Spanbroek 28294d431a [contracts] Tests now also work in Windows 2022-03-28 11:59:01 +02:00
Mark Spanbroek 4fd92b52eb [contracts] Install NodeJS in CI environment 2022-03-28 11:59:01 +02:00
Dmitriy Ryajov 1663c1d71d don't run win i386 tests 2021-11-25 09:37:12 +01:00
Dmitriy Ryajov aa5260df73 update binaries cache 2021-11-25 09:37:12 +01:00
Dmitriy Ryajov 879a76fa41 make test 2021-11-25 09:37:12 +01:00
Dmitriy Ryajov 4d136a2e4c make ci use new build system 2021-11-25 09:37:12 +01:00
Dmitriy Ryajov ce66e43440
fix compilation issues with latest libp2p (#27)
* fix compilation issues with latest libp2p

* change package description

* fix CI compiler

* try with nim 1.4.2

* try with 1.4.2

* use more standardized ci config

* set reasonable log level

* remove nim version dependency

* reverting to 1.4.6 as otherwise it fails on amd64
2021-10-29 13:30:52 -06:00
Mark Spanbroek b049e54d9e Update CI script
Copied from lib-p2p master; fixes error with windows dependencies.
2021-06-07 13:21:34 +02:00
Dmitriy Ryajov 6c92b3dc25
Poc 2 (#7)
* moving protobuf into bitswap

* adding block type

* reworking bitswap

* adding chunker

* adding license header

* use 1.2.6

* adding fixed size chunker

* add blockstore

* add iterator to chunker

* more bitswap changes

* rename ipfs to dagger

* rename to dagger

* blockstore inherits from BlockProvider

* wip - add core block handling logic

* smal changes

* use proper block store methods

* adding asynq heapqueue

* wip prepare for bitswap task runner

* adding `$`

* adding memory store and tests

* fixed chunking

* extracted desicion engine from bitswap

* added helper random funcs

* adding testing helpers

* only handle seqs

* add peer events

* cleanup pending blocks on blockstore event

* allow nil handlers

* move protobuf type helpers

* allow initializing block from Cid

* testing and fixes

* small fixes

* expose `<`

* spelling

* default value

* spelling

* pending blocks manager

* adding stores manager

* more tests a wip around bitswap

* small changes

* merge bitswap and engine for now

* for now run only the new poc's tests

* add a more complete ci setup

* use template in map

* remove p2pd

* remove go

* dont use asyncCheck

* few small changes

* adding ability to update items

* adding multiple task runners

* handle cancelation properly

* use Result instead of throwing

* wip bitswap tests

* moving things around

* split out engine again

* add request and handlers interface

* fix tests

* wip - engine tests

* remove unused imports

* fix tests

* cleanup block requesting logic

* add block request tests

* more block requests

* add support for max heap

* don't use result

* use max heap & send block presence in task handler

* add task handler tests

* rename store to localStore

* cleanup & logging

* cancel task on stop

* don't depend on local store for events

* dont use heap queue for wants

* add chronicles

* fix issue with peer wants

* add test for delayed block sends

* remove obsolete tests

* wip chunker

* run all tests

* add todo

* misc

* remove irrelevant files

* removing more files

* adding helpers for bitswap tests

* moved bitswap file

* misc

* make blocks timeout longer

* adjust block timeout

* speedup test

* compile with threads

* import missing crypto

* misc

* disable threads for now

* fix 32 bit platforms

* re-enable threads support in tests
2021-02-25 18:23:22 -06:00
Mark Spanbroek bd29797aba Project setup 2021-02-02 19:29:52 +01:00