697 Commits

Author SHA1 Message Date
Mark Spanbroek
6879e3d871
attempt to speed up windows CI
https://github.com/actions/runner-images/issues/7320#issuecomment-1862639391
2024-11-12 12:46:49 +01:00
Mark Spanbroek
9a2e5478a5
fix proof integration test
there were two logic errors in this test:
- a slot is freed anyway at the end of the contract
- when starting the request takes a long time, the
  first slot can already be freed because there were
  too many missing proofs
2024-11-12 11:57:31 +01:00
Mark Spanbroek
3cfa205f2f
use 100 millisecond polling for multinodes tests
default polling interval of 4 seconds is too close
to the 5 second timeout for `check eventually`.

uses the same polling interval as ethersuite does
2024-11-12 11:03:27 +01:00
Mark Spanbroek
388cd5166e
fix timing issue in validation unit test 2024-11-12 10:50:38 +01:00
Mark Spanbroek
1169a08747
WIP: update polling fix from nim-ethers 2024-11-12 08:53:06 +01:00
Mark Spanbroek
33da7fcbad
use .confirm(1) in marketplace tests
to ensure that the transaction has been processed
before continuing with the test
2024-11-12 08:52:36 +01:00
Mark Spanbroek
5d3756a735
fix timing of marketplace tests
allow for a bit more time to withdraw funds
2024-11-11 17:11:47 +01:00
Mark Spanbroek
86264f3ff0
WIP: include polling fix from nim-ethers 2024-11-11 16:32:25 +01:00
Mark Spanbroek
f79e5ecf52
speed up partial payout integration test 2024-11-11 11:16:47 +01:00
Mark Spanbroek
2e520e0e0e
use .confirm(1) instead of confirm(0)
confirm(0) doesn't wait at all, confirm(1) waits
for the transaction to be mined
2024-11-10 15:27:56 +01:00
Mark Spanbroek
fbbfd38d6c
Use http subscriptions instead of websocket for tests 2024-11-10 15:24:27 +01:00
Mark Spanbroek
1462332655
Revert "Use websockets for integration tests"
This reverts commit 9539d0ebba1b80c0e41e838c3ff062d2341ea414.
2024-11-08 20:04:34 +01:00
Mark Spanbroek
9539d0ebba
Use websockets for integration tests
Use workaround in hardhat to ensure that
subscriptions do not time out after 5
minutes
2024-11-08 17:09:28 +01:00
Mark Spanbroek
703b943175
more adjustments to integration test
filling 3 slots takes longer now that we
limited the amount of slot queue workers to 1
2024-11-08 10:28:57 +01:00
Mark Spanbroek
d53f531656
Revert change to ethersuite
reason: broke the contract tests
2024-11-08 10:28:57 +01:00
Mark Spanbroek
5e02b0c1b8
Do not use websocket connections to hardhat
To work around this issue when subscriptions are
inactive for more than 5 minutes:
https://github.com/NomicFoundation/hardhat/issues/2053
2024-11-08 10:28:57 +01:00
Mark Spanbroek
3701ba2b68
adjust integration tests for 1 slot queue worker
reason: with only 1 slot queue worker, downloading
of slots no longer happens in parallel and
therefore takes longer
2024-11-08 10:28:57 +01:00
Mark Spanbroek
320a5b3ac8
limit amount of slot queue workers to 1
reason: proof generation does not happen in
separate threads yet, so to avoid taking too
long to provide initial storage proofs we only
pick up one slot at a time
2024-11-08 10:28:53 +01:00
markspanbroek
a6f0311b50
change default proof period to 2 minutes (#989)
on hardhat the proof period remains 1 minute
2024-11-08 08:03:16 +00:00
Marko Burčul
2151e02838
nix-flake: Add shell definition (#954)
Initialized flake file and added development shell definition.
Exporting of default compiler flags is moved to makefile.

Referenced issue: https://github.com/codex-storage/nim-codex/issues/940

Signed-off-by: markoburcul <marko@status.im>
2024-11-04 07:46:22 +00:00
Slava
86257054ee
chore: update testnet marketplace address (#983) (#984)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-11-04 00:41:03 +00:00
Slava
96459188c9
Add ETH_PRIVATE_KEY to Docker entrypoint (#982)
* Add ETH_PRIVATE_KEY to Docker entrypoint

* Add deprecation warning for PRIV_KEY variable

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

---------

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-11-03 06:51:57 +00:00
Ben Bierens
b8dd68063f
fix: bumps ethers to fix missing nonce error (#980)
* fix: bumps ethers to fix missing nonce error

* fix was merged in nim-ethers

---------

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-11-01 17:23:43 +00:00
Eric
2b5a40559e
chore: update dependencies, especially nim-ethers to chronos v4 compatible version (#968)
* chore: bump dependencies, including nim-ethers with chronos v4 support

Bumps the following dependencies:
- nim-ethers to commit 507ac6a4cc71cec9be7693fa393db4a49b52baf9 which contains a pinned nim-eth version. This is to be replaced by a versioned library, so it will be pinned to a particular version. There is a crucial fix in this version of ethers that fixes nonce management which is causing issues in the Codex testnet.
- nim-json-rpc to v0.4.4
- nim-json-serialization to v0.2.8
- nim-serde to v1.2.2
- nim-serialization to v0.2.4

Currently, one of the integration tests is failing.

* fix integration test

- When a state's run was cancelled, it was being caught as an error due to catching all CatchableErrors. This caused a state transition to SaleErrored, however cancellation of run was not actually an error. Handling this correctly fixed the issue.
- Stopping of the clock was moved to after `HostInteractions` (sales) which avoided an assertion around getting time when the clock was not started.

* bump ethers to include nonce fix and filter not found fix

* bump ethers: fixes missing symbol not exported in ethers

* Fix cirdl test imports/exports

* Debugging in ci

* Handle CancelledErrors for state.run in one place only

* Rename `config` to `configuration`

There was a symbol clash preventing compilation and it was easiest to rename `config` to `configuration` in the contracts. Not even remotely ideal, but it was the only way.

* bump ethers to latest

Prevents an issue were `JsonNode.items` symbol could not be found

* More changes to support `config` > `configuration`

* cleanup

* testing to see if this fixes failure in ci

* bumps contracts

- ensures slot is free before allowing reservation
- renames config to configuration to avoid symbol clash
2024-10-30 10:40:17 +00:00
Arnaud
942f940c92
Move the upload headers to the POST method (#978) 2024-10-29 14:40:43 +00:00
Slava
a2ac7453fa
Build Postman Collection (#973)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-10-28 13:53:41 +00:00
Arnaud
2fb7031ec6
feat: add metadata to the manifest (#960)
* Add metadata to the manifest

* Remove useless import

* Fix the openapi documentation

* Use optional fields instead of default values

* Remove testRestApi target

* Return failure when the protobuf cannot get the field

* Set download headers and fix cors headers when an error is returned

* Add tests to verify the download headers

* Try to adjust the content length header

* Fix convertion to string

* Remove the content length header

* Remove testRestApi target

* Removing debug messages
2024-10-25 13:43:19 +00:00
Arnaud
bcc1468130
Remove duplicated header (#970) 2024-10-25 09:23:35 +00:00
Arnaud
40068512a6
Complete documentation for debug endpoint (#969) 2024-10-25 07:20:00 +00:00
Eric
0157ca4c57
fix(slot-reservations): Avoid slot filled cancellations (#963)
* Avoid cancelling states when slot is filled

* improve logging

Improves logging for situations where a Sale should be ignored instead of being considered an error, including when reservation is not allowed and when a slot was filled by another host.

* remove onSlotFilled unit tests from states
2024-10-24 05:56:12 +00:00
Slava
3a2d0926f1
chore: new marketplace address for testnet (#961)
https://github.com/codex-storage/infra-codex/issues/248

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-10-21 13:01:56 +03:00
Ben Bierens
562e4329e2
Download API upgrade (#955)
* Adds API for fetching manifest only and downloading dataset without stream

* Updates openapi.yaml

* Adds tests for downloading manifest-only and without stream.

* review comments by Giuliano

* updates test clients
2024-10-17 16:54:28 +00:00
Adam Uhlíř
436baef20a
docs: openapi node fix (#950) 2024-10-14 17:26:58 +00:00
Adam Uhlíř
7c33473c88
ci: linux ci runs on ubuntu-20.04 (#953)
* ci: linux ci runs uses ubuntu-20.04

* ci: use ubuntu-20.04 for nim-matrix

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

---------

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
Co-authored-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-10-14 11:24:53 +00:00
Ben Bierens
93960033f4
Adds log for cirdl download URL (#948) 2024-10-10 12:05:03 +00:00
Adam Uhlíř
ffa203b04f
feat: partial rewards and withdraws (#880)
* feat: partial rewards and withdraws

* test: missing reserve slot

* test: fix contracts
2024-10-10 11:53:33 +00:00
Ben Bierens
3699601393
Handles LPStreamError in chunker (#947)
* Handles LPStreamError in chunker

* Adds test for lpstream exception

* Adds tests for other stream exceptions. Cleanup.
2024-10-10 11:22:36 +00:00
Arnaud
1fe3abfd03
fix(restapi): Add cors headers when the request is returning errors (#942)
* Add cors headers when the request is returning errors

* Prevent nim presto to send multiple cors headers
2024-10-10 08:25:07 +00:00
Eric
7e0ec3c233
Support enforcement of slot reservations before filling slot (#934) 2024-10-09 04:44:07 +00:00
Slava
44f21b8a68
Update Codex Testnet marketplace contract address (#944)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-10-08 12:10:42 +03:00
Arnaud
f02de34f77
docs(openapi): provider better documentation for space endpoint parameters (#921)
* Trying to improve documentation

* Update openapi.yaml

Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Signed-off-by: Arnaud <arno.deville@gmail.com>

* Update openapi.yaml

Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Signed-off-by: Arnaud <arno.deville@gmail.com>

* Update openapi.yaml

Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Signed-off-by: Arnaud <arno.deville@gmail.com>

---------

Signed-off-by: Arnaud <arno.deville@gmail.com>
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
2024-10-08 06:37:10 +00:00
Ben Bierens
17f0988fc7
Fix: null-ref in networkPeer (#937)
* Fixes nullref in networkPeer

* Removes inflight semaphore

* Revert "Removes inflight semaphore"

This reverts commit 26ec15c6f788df3adb6ff3b912a0c4b5d3139358.
2024-10-07 08:50:54 +00:00
Slava
0ea8cfb085
Remove moved docs (#935)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-10-07 08:33:10 +00:00
Eric
566db2fa30
feat(slot-reservations): support SlotReservationsFull event (#926) 2024-10-05 01:37:08 +00:00
Slava
91e4e368de
Use Ubuntu 20.04 for Linux amd64 releases (#939)
* Use Ubuntu 20.04 for Linux amd64 releases (#932)

* Accept branches with the slash in the name for release workflow (#932)

* Increase artifacts retention-days for release workflow (#932)
2024-10-04 11:07:44 +00:00
Eric
4c51dca299
feat(slot-reservations): Add SaleSlotReserving state (#917)
* convert EthersError to MarketError

* change `canReserveSlot` and `reserveSlot` parameters

Parameters for `canReserveSlot` and `reserveSlot` were changed from `SlotId` to `RequestId` and `UInt256 slotIndex`.

* Add SaleSlotReserving

Adds a new state, SaleSlotReserving, that attempts to reserve a slot before downloading.
If the slot cannot be reserved, the state moves to SaleIgnored.
On error, the state moves to SaleErrored.

SaleIgnored is also updated to pass in `reprocessSlot` and `returnBytes`, controlling the behaviour in the Sales module after the slot is ignored. This is because previously it was assumed that SaleIgnored was only reached when there was no Availability. This is no longer the case, since SaleIgnored can now be reached when a slot cannot be reserved.

* Update SalePreparing

Specify `reprocessSlot` and `returnBytes` when moving to `SaleIgnored` from `SalePreparing`.

Update tests to include test for a raised CatchableError.

* Fix unit test

* Modify `canReserveSlot` and `reverseSlot` params after rebase

* Update MockMarket with new `canReserveSlot` and `reserveSlot` params

* fix after rebase

also bump codex-contracts-eth to master
2024-10-04 06:16:11 +00:00
Eric
b5ee57afd7
feat(slot-reservations): Support reserving slots (#907)
* feat(slot-reservations): Support reserving slots

Closes #898.

Wire up reserveSlot and canReserveSlot contract calls, but don't call them

* Remove return value from `reserveSlot`

* convert EthersError to MarketError

* Move convertEthersError to reserveSlot

* bump codex-contracts-eth after rebase

* change `canReserveSlot` and `reserveSlot` parameters

Parameters for `canReserveSlot` and `reserveSlot` were changed from `SlotId` to `RequestId` and `UInt256 slotIndex`.

* bump codex-contracts-eth after rebase

* bump codex-contracts-eth to master after codex-contracts-eth/pull/177 merged
2024-10-04 03:21:51 +00:00
Slava
b491906005
Remove moved docs (#930)
* Remove moved document

* Update main Readme and point links to the documentation site
2024-10-03 08:55:54 +00:00
Marcin Czenko
5ace105a66
Validator - support partitioning of the slot id space (#890)
* Adds validatorPartitionSize and validatorPartitionIndex config options

* adds partitioning options to the validation type

* adds partitioning logic to the validator

* ignores partitionIndex when partitionSize is either 0 or 1

* clips the partition index to <<partitionIndex mod partitionSize>>

* handles negative values for the validation partition index

* updates long description of the new validator cli options

* makes default partitionSize to be 0 for better backward compatibility

* Improving formatting on validator CLI

* reactors validation params into a separate type and simplifies validation of validation params

* removes suspected duplication

* fixes typo in validator CLI help

* updates README

* Applies review comments - using optionals and range types to handle validation params

* Adds initializer to the configFactory for validatorMaxSlots

* [Review] update validator CLI description and README

* [Review]: renaming validationParams to validationConfig (config)

* [Review]: move validationconfig.nim to a higher level (next to validation.nim)

* changes backing type of MaxSlots to be int and makes sure slots are validated without limit when maxSlots is set to 0

* adds more end-to-end test for the validator and the groups

* fixes typo in README and conf.nim

* makes `maxSlotsConstraintRespected` and `shouldValidateSlot` private + updates the tests

* fixes public address of the signer account in the marketplace tutorial

* applies review comments - removes two tests
2024-10-02 22:00:40 +00:00
Slava
e1a02c8b76
Use CLI args when passed for cirdl in Docker entrypoint (#927)
* Use CLI args when passed for cirdl in Docker entrypoint

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

* Increase CI timeout

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

---------

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-10-01 13:05:22 +00:00