Commit Graph

703 Commits

Author SHA1 Message Date
Eric dc244a36bd
fix(asyncstatemachine): fixes not awaiting or asyncSpawning futures
- adds a break in scheduler when CancelledError is caught
- tracks asyncSpawned state.run, so that it can be cancelled during stop
- removes usages of `then`
- ensures that no exceptions are leaked from async procs
2024-12-13 11:21:05 +11:00
Adam Uhlíř 19af79786e
feat: repair is rewarded (#1022)
* feat: repair is rewarded

* chore: update contracts repo

* feat: proving loop handles repair case

* test: assert repair state

* chore: update contracts repo

* fix: upon unknown state of repair go to error
2024-12-12 20:19:56 +00:00
Adam Uhlíř d10072bf67
refactor: marketplace configuration is cached (#1029) 2024-12-12 12:57:34 +00:00
Eric da234d503b
fix(trackedfutures): removes usage of `then` from tracked futures (#1032)
- removes usage of `then`, simplifying the logic, and allowing `then` to be removed completely
- updates annotations to reflect that all procs (sync and async) raise no exceptions
2024-12-12 12:31:51 +00:00
Vaclav Pavlin 855b973811
chore: fix inconsistent metric naming (#1027) 2024-12-12 10:45:47 +00:00
Marko Burčul 0c6784da7e
nix: make derivation and update shell (#1003)
* nix: make derivation and update shell

Create a structure for nix files. Add the derivation file which is using
system Nim to compile Codex.

Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>

* nim-circom-compat: update

Include commit which allows building circom-compat-ffi using Nix(doesn't
affect current usage of the submodule).

Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>

* makefile: fix for detecting linux arch

Signed-off-by: markoburcul <marko@status.im>

---------

Signed-off-by: markoburcul <marko@status.im>
2024-12-09 17:07:01 +00:00
Eric fb4577f25c
chore(tests): fix unneeded async (#1021)
Removes an unneeded `check eventually` inside a helper proc, removing the need for the proc to be async.
2024-12-06 08:02:57 +00:00
Ben Bierens f51eae30fe
Fixes raceconditions in testadvertiser (#1008) 2024-12-05 09:30:01 +00:00
Ben Bierens 8e29939cf8
Send pluralized wantBlock messages (#1016)
* don't unroll wantCids when sending wantBlock message in blockPresenceHandler

* workaround logging

* Fixes logformatting upraises for sequences.

* Applies upraises rule for setProperty of textmode for sequences.

* Replaces upraises with raises

* Removes redundant log in sendWantHave
2024-12-04 13:33:48 +00:00
Slava 921159f87f
Run release tests for docker images (#1017)
* Refactor Docker reusable workflow and add release tests support

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Add an option to run release tests for Docker images

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Use bigger instance for arm builds

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Pass repository and branch to release tests workflow

https://github.com/codex-storage/cs-codex-dist-tests/issues/108

* Do not use computation job because run_release_tests is a string

https://github.com/codex-storage/cs-codex-dist-tests/issues/108
2024-12-04 11:52:51 +00:00
Ben Bierens 63e54d135c
Fixes race condition in initial-proving tests (#1007) 2024-12-04 08:56:53 +00:00
Adam Uhlíř 0707446cdd
feat: expose underlying nim-ethers errors to logs (#985)
* feat: expose underlying nim-ethers errors to logs

* chore: bump nim-ethers

* test: fix testproof compilation

* test: raise defects on results error
2024-12-03 11:16:24 +00:00
Marcin Czenko 3f510eb501
fixes incomplete async annotation (#1015) 2024-12-02 15:07:43 +00:00
Ben Bierens ab019a08ae
Enables stacktrace in docker images (#1013)
* Enables libbacktrace in docker images

* Make libbacktrace configurable in docker via build-arg

---------

Co-authored-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-12-02 09:32:53 +00:00
Ben Bierens 21249968d4
Requesting the same CID sometimes causes a worker to discard the request if it's already inflight by another worker. (#1002) 2024-11-27 07:13:34 +00:00
Ben Bierens d47ce38894
Fixes race in testsales (#995)
* Adds isWaiting to mockClock to remove sleep in testsales

* Review comments by Eric. Also replaced two more sleeps with check-eventually
2024-11-26 10:48:52 +00:00
Slava 5c6bbb0cee
chore: update testnet marketplace address (#1000) (#1001)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-11-26 08:16:41 +00:00
Ben Bierens 024c75e4f9
Bumps leveldbstatic to v0.1.6 (#998) 2024-11-25 17:15:05 +00:00
Slava d12de20868
ci: use rust 1.7.9 for release workflow and dockerfile (#999)
* ci: use rust 1.7.9 for release workflow (#996)

* docker: use rust 1.7.9 for docker (#996)
2024-11-25 16:13:14 +00:00
markspanbroek 29433bad9a
Fix concurrency issues (#993)
* Use http subscriptions instead of websocket for tests

To work around this issue when subscriptions are
inactive for more than 5 minutes:
https://github.com/NomicFoundation/hardhat/issues/2053

Use 100 millisecond polling; default polling interval
of 4 seconds is too close to the 5 second timeout for
`check eventually`.

* use .confirm(1) instead of confirm(0)

confirm(0) doesn't wait at all, confirm(1) waits
for the transaction to be mined

* speed up partial payout integration test

* update nim-ethers to version 0.10.0

includes fixes for http polling and .confirm()

* fix timing of marketplace tests

allow for a bit more time to withdraw funds

* use .confirm(1) in marketplace tests

to ensure that the transaction has been processed
before continuing with the test

* fix timing issue in validation unit test

* fix proof integration test

there were two logic errors in this test:
- a slot is freed anyway at the end of the contract
- when starting the request takes a long time, the
  first slot can already be freed because there were
  too many missing proofs

* fix intermittent error in contract tests

currentTime() doesn't always correctly reflect
the time of the next transaction

* reduce number of slots in integration test

otherwise the windows runner in the CI won't
be able to start the request before it expires

* fix timing in purchasing test

allow for a bit more time for a request to
be submitted

* fix timing of request submission in test

windows ci is so slow, it can take up to 40 seconds
just to submit a storage request to hardhat

* increase proof period to 90 seconds

* adjust timing of integration tests

reason: with the increased period length of 90 seconds, it
can take longer to wait for a stable challenge at the
beginning of a period.

* increase CI timeout to 2 hours

* Fix slow builds on windows

apparently it takes windows 2-3 seconds to
resolve "localhost" to 127.0.0.1 for every
json-rpc connection that we make 🤦
2024-11-25 11:23:04 +00:00
Slava 6038fb456e
ci: split linux and macos tests (#997)
* Split Linux and macOS tests

* Make jobs names more readable
2024-11-22 12:05:00 +00:00
Slava 71b8a95d12
ci: install rust 1.7.9 as required by packages (#996)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-11-22 06:34:10 +00:00
Arnaud 9b7f3f4aaf
chore(openapi): update the openapi spec for the manifest object (#972)
* Update the manifest openapi spec

* Make metadata nullable

Signed-off-by: Arnaud <arnaud@status.im>

---------

Signed-off-by: Arnaud <arnaud@status.im>
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
2024-11-15 13:21:18 +00:00
Slava d7ae8b734a
makefile: Use do not use -mssse3 instructions on arm (#940) (#990) 2024-11-09 11:39:39 +00:00
markspanbroek a6f0311b50
change default proof period to 2 minutes (#989)
on hardhat the proof period remains 1 minute
2024-11-08 08:03:16 +00:00
Marko Burčul 2151e02838
nix-flake: Add shell definition (#954)
Initialized flake file and added development shell definition.
Exporting of default compiler flags is moved to makefile.

Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>
2024-11-04 07:46:22 +00:00
Slava 86257054ee
chore: update testnet marketplace address (#983) (#984)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-11-04 00:41:03 +00:00
Slava 96459188c9
Add ETH_PRIVATE_KEY to Docker entrypoint (#982)
* Add ETH_PRIVATE_KEY to Docker entrypoint

* Add deprecation warning for PRIV_KEY variable

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

---------

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-11-03 06:51:57 +00:00
Ben Bierens b8dd68063f
fix: bumps ethers to fix missing nonce error (#980)
* fix: bumps ethers to fix missing nonce error

* fix was merged in nim-ethers

---------

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-11-01 17:23:43 +00:00
Eric 2b5a40559e
chore: update dependencies, especially nim-ethers to chronos v4 compatible version (#968)
* chore: bump dependencies, including nim-ethers with chronos v4 support

Bumps the following dependencies:
- nim-ethers to commit 507ac6a4cc71cec9be7693fa393db4a49b52baf9 which contains a pinned nim-eth version. This is to be replaced by a versioned library, so it will be pinned to a particular version. There is a crucial fix in this version of ethers that fixes nonce management which is causing issues in the Codex testnet.
- nim-json-rpc to v0.4.4
- nim-json-serialization to v0.2.8
- nim-serde to v1.2.2
- nim-serialization to v0.2.4

Currently, one of the integration tests is failing.

* fix integration test

- When a state's run was cancelled, it was being caught as an error due to catching all CatchableErrors. This caused a state transition to SaleErrored, however cancellation of run was not actually an error. Handling this correctly fixed the issue.
- Stopping of the clock was moved to after `HostInteractions` (sales) which avoided an assertion around getting time when the clock was not started.

* bump ethers to include nonce fix and filter not found fix

* bump ethers: fixes missing symbol not exported in ethers

* Fix cirdl test imports/exports

* Debugging in ci

* Handle CancelledErrors for state.run in one place only

* Rename `config` to `configuration`

There was a symbol clash preventing compilation and it was easiest to rename `config` to `configuration` in the contracts. Not even remotely ideal, but it was the only way.

* bump ethers to latest

Prevents an issue were `JsonNode.items` symbol could not be found

* More changes to support `config` > `configuration`

* cleanup

* testing to see if this fixes failure in ci

* bumps contracts

- ensures slot is free before allowing reservation
- renames config to configuration to avoid symbol clash
2024-10-30 10:40:17 +00:00
Arnaud 942f940c92
Move the upload headers to the POST method (#978) 2024-10-29 14:40:43 +00:00
Slava a2ac7453fa
Build Postman Collection (#973)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-10-28 13:53:41 +00:00
Arnaud 2fb7031ec6
feat: add metadata to the manifest (#960)
* Add metadata to the manifest

* Remove useless import

* Fix the openapi documentation

* Use optional fields instead of default values

* Remove testRestApi target

* Return failure when the protobuf cannot get the field

* Set download headers and fix cors headers when an error is returned

* Add tests to verify the download headers

* Try to adjust the content length header

* Fix convertion to string

* Remove the content length header

* Remove testRestApi target

* Removing debug messages
2024-10-25 13:43:19 +00:00
Arnaud bcc1468130
Remove duplicated header (#970) 2024-10-25 09:23:35 +00:00
Arnaud 40068512a6
Complete documentation for debug endpoint (#969) 2024-10-25 07:20:00 +00:00
Eric 0157ca4c57
fix(slot-reservations): Avoid slot filled cancellations (#963)
* Avoid cancelling states when slot is filled

* improve logging

Improves logging for situations where a Sale should be ignored instead of being considered an error, including when reservation is not allowed and when a slot was filled by another host.

* remove onSlotFilled unit tests from states
2024-10-24 05:56:12 +00:00
Slava 3a2d0926f1
chore: new marketplace address for testnet (#961)
https://github.com/codex-storage/infra-codex/issues/248

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-10-21 13:01:56 +03:00
Ben Bierens 562e4329e2
Download API upgrade (#955)
* Adds API for fetching manifest only and downloading dataset without stream

* Updates openapi.yaml

* Adds tests for downloading manifest-only and without stream.

* review comments by Giuliano

* updates test clients
2024-10-17 16:54:28 +00:00
Adam Uhlíř 436baef20a
docs: openapi node fix (#950) 2024-10-14 17:26:58 +00:00
Adam Uhlíř 7c33473c88
ci: linux ci runs on ubuntu-20.04 (#953)
* ci: linux ci runs uses ubuntu-20.04

* ci: use ubuntu-20.04 for nim-matrix

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

---------

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
Co-authored-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-10-14 11:24:53 +00:00
Ben Bierens 93960033f4
Adds log for cirdl download URL (#948) 2024-10-10 12:05:03 +00:00
Adam Uhlíř ffa203b04f
feat: partial rewards and withdraws (#880)
* feat: partial rewards and withdraws

* test: missing reserve slot

* test: fix contracts
2024-10-10 11:53:33 +00:00
Ben Bierens 3699601393
Handles LPStreamError in chunker (#947)
* Handles LPStreamError in chunker

* Adds test for lpstream exception

* Adds tests for other stream exceptions. Cleanup.
2024-10-10 11:22:36 +00:00
Arnaud 1fe3abfd03
fix(restapi): Add cors headers when the request is returning errors (#942)
* Add cors headers when the request is returning errors

* Prevent nim presto to send multiple cors headers
2024-10-10 08:25:07 +00:00
Eric 7e0ec3c233
Support enforcement of slot reservations before filling slot (#934) 2024-10-09 04:44:07 +00:00
Slava 44f21b8a68
Update Codex Testnet marketplace contract address (#944)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-10-08 12:10:42 +03:00
Arnaud f02de34f77
docs(openapi): provider better documentation for space endpoint parameters (#921)
* Trying to improve documentation

* Update openapi.yaml

Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Signed-off-by: Arnaud <arno.deville@gmail.com>

* Update openapi.yaml

Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Signed-off-by: Arnaud <arno.deville@gmail.com>

* Update openapi.yaml

Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Signed-off-by: Arnaud <arno.deville@gmail.com>

---------

Signed-off-by: Arnaud <arno.deville@gmail.com>
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
2024-10-08 06:37:10 +00:00
Ben Bierens 17f0988fc7
Fix: null-ref in networkPeer (#937)
* Fixes nullref in networkPeer

* Removes inflight semaphore

* Revert "Removes inflight semaphore"

This reverts commit 26ec15c6f7.
2024-10-07 08:50:54 +00:00
Slava 0ea8cfb085
Remove moved docs (#935)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-10-07 08:33:10 +00:00
Eric 566db2fa30
feat(slot-reservations): support SlotReservationsFull event (#926) 2024-10-05 01:37:08 +00:00