Commit Graph

656 Commits

Author SHA1 Message Date
Ben Bierens 17f0988fc7
Fix: null-ref in networkPeer (#937)
* Fixes nullref in networkPeer

* Removes inflight semaphore

* Revert "Removes inflight semaphore"

This reverts commit 26ec15c6f7.
2024-10-07 08:50:54 +00:00
Slava 0ea8cfb085
Remove moved docs (#935)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-10-07 08:33:10 +00:00
Eric 566db2fa30
feat(slot-reservations): support SlotReservationsFull event (#926) 2024-10-05 01:37:08 +00:00
Slava 91e4e368de
Use Ubuntu 20.04 for Linux amd64 releases (#939)
* Use Ubuntu 20.04 for Linux amd64 releases (#932)

* Accept branches with the slash in the name for release workflow (#932)

* Increase artifacts retention-days for release workflow (#932)
2024-10-04 11:07:44 +00:00
Eric 4c51dca299
feat(slot-reservations): Add SaleSlotReserving state (#917)
* convert EthersError to MarketError

* change `canReserveSlot` and `reserveSlot` parameters

Parameters for `canReserveSlot` and `reserveSlot` were changed from `SlotId` to `RequestId` and `UInt256 slotIndex`.

* Add SaleSlotReserving

Adds a new state, SaleSlotReserving, that attempts to reserve a slot before downloading.
If the slot cannot be reserved, the state moves to SaleIgnored.
On error, the state moves to SaleErrored.

SaleIgnored is also updated to pass in `reprocessSlot` and `returnBytes`, controlling the behaviour in the Sales module after the slot is ignored. This is because previously it was assumed that SaleIgnored was only reached when there was no Availability. This is no longer the case, since SaleIgnored can now be reached when a slot cannot be reserved.

* Update SalePreparing

Specify `reprocessSlot` and `returnBytes` when moving to `SaleIgnored` from `SalePreparing`.

Update tests to include test for a raised CatchableError.

* Fix unit test

* Modify `canReserveSlot` and `reverseSlot` params after rebase

* Update MockMarket with new `canReserveSlot` and `reserveSlot` params

* fix after rebase

also bump codex-contracts-eth to master
2024-10-04 06:16:11 +00:00
Eric b5ee57afd7
feat(slot-reservations): Support reserving slots (#907)
* feat(slot-reservations): Support reserving slots

Closes #898.

Wire up reserveSlot and canReserveSlot contract calls, but don't call them

* Remove return value from `reserveSlot`

* convert EthersError to MarketError

* Move convertEthersError to reserveSlot

* bump codex-contracts-eth after rebase

* change `canReserveSlot` and `reserveSlot` parameters

Parameters for `canReserveSlot` and `reserveSlot` were changed from `SlotId` to `RequestId` and `UInt256 slotIndex`.

* bump codex-contracts-eth after rebase

* bump codex-contracts-eth to master after codex-contracts-eth/pull/177 merged
2024-10-04 03:21:51 +00:00
Slava b491906005
Remove moved docs (#930)
* Remove moved document

* Update main Readme and point links to the documentation site
2024-10-03 08:55:54 +00:00
Marcin Czenko 5ace105a66
Validator - support partitioning of the slot id space (#890)
* Adds validatorPartitionSize and validatorPartitionIndex config options

* adds partitioning options to the validation type

* adds partitioning logic to the validator

* ignores partitionIndex when partitionSize is either 0 or 1

* clips the partition index to <<partitionIndex mod partitionSize>>

* handles negative values for the validation partition index

* updates long description of the new validator cli options

* makes default partitionSize to be 0 for better backward compatibility

* Improving formatting on validator CLI

* reactors validation params into a separate type and simplifies validation of validation params

* removes suspected duplication

* fixes typo in validator CLI help

* updates README

* Applies review comments - using optionals and range types to handle validation params

* Adds initializer to the configFactory for validatorMaxSlots

* [Review] update validator CLI description and README

* [Review]: renaming validationParams to validationConfig (config)

* [Review]: move validationconfig.nim to a higher level (next to validation.nim)

* changes backing type of MaxSlots to be int and makes sure slots are validated without limit when maxSlots is set to 0

* adds more end-to-end test for the validator and the groups

* fixes typo in README and conf.nim

* makes `maxSlotsConstraintRespected` and `shouldValidateSlot` private + updates the tests

* fixes public address of the signer account in the marketplace tutorial

* applies review comments - removes two tests
2024-10-02 22:00:40 +00:00
Slava e1a02c8b76
Use CLI args when passed for cirdl in Docker entrypoint (#927)
* Use CLI args when passed for cirdl in Docker entrypoint

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

* Increase CI timeout

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

---------

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-10-01 13:05:22 +00:00
Marcin Czenko 264bfa17f5
updates Marketplace tutorial documentation (#888)
* updates Marketplace tutorial documentation

* Applies review comments to marketplace-tutorial

* Final formatting touches

* moved `Prerequisites` around

* Fixes indentation in one JSON snippet
2024-09-29 19:18:01 +00:00
Slava dcfc249945
Move Building Codex guide to the main docs site (#893) 2024-09-29 14:38:57 +00:00
Eric 8b374d5acb
remove erasure and por parameters from openapi spec (#915) 2024-09-25 10:37:19 +00:00
Adam Uhlíř 4b9336ec07
API tweaks for OpenAPI, errors and endpoints (#886)
* All sort of tweaks

* docs: availability's minPrice doc

* Revert changes to the two node test example

* Change default EC params in REST API

Change default EC params in REST API to 3 nodes and 1 tolerance.

Adjust integration tests to honour these settings.

---------

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-09-24 08:37:08 +00:00
Ben Bierens 098c4bb6e9
Adds testnet marketplace address to known deployments (#911) 2024-09-24 11:52:07 +03:00
Arnaud 032d7e7495
Support CORS for POST and PATCH availability endpoints (#897) 2024-09-23 15:08:56 +00:00
Ben Bierens 4e8630791a
Rework circuit downloader (#882)
* Introduces a start method to prover

* Moves backend creation into start method

* sets up three paths for backend initialization

* Extracts backend initialization to backend-factory

* Implements loading backend from cli files or previously downloaded local files

* Wires up downloading and unzipping

* functional implementation

* Fixes testprover.nim

* Sets up tests for backendfactory

* includes libzip-dev

* pulls in updated contracts

* removes integration cli tests for r1cs, wasm, and zkey file arguments.

* Fixes issue where inner-scope values are lost before returning

* sets local proof verification for dist-test images

* Adds two traces and bumps nim-ethers

* Adds separate path for circuit files

* Create circuit dir if not exists

* fix: make sure requestStorage is mined

* fix: correct place to plug confirm

* test: fixing contracts tests

* Restores gitmodules

* restores nim-datastore reference

* Sets up downloader exe

* sets up tool skeleton

* implements getting of circuit hash

* Implements downloader tool

* sets up test skeleton

* Implements test for cirdl

* includes testTools in testAll

* Cleanup building.md

* cleans up previous downloader implementation

* cleans up testbackendfactory

* moves start of prover into node.nim

* Fills in arguments in example command

* Initializes backend in prover constructor

* Restores tests

* Restores tests for cli instructions

* Review comments by Dmitriy, part 1

* Quotes path in download instruction.

* replaces curl with chronos http session

* Moves cirdl build output to 'build' folder.

* Fixes chronicles log output

* Add cirdl support to the codex Dockerfile

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

* Add cirdl support to the docker entrypoint

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

* Add cirdl support to the release workflow

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

* Disable verify_circuit flag for releases

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

* Removes backendFactory placeholder type

* wip

* Replaces zip library with status-im/zippy library (which supports zip and tar)

* Updates cirdl to not change circuitdir folder

* Switches from zip to tar.gz

* Review comments by Dmitriy

* updates codex-contracts-eth

* Adds testTools to CI

* Adds check for access to config.circuitdir

* Update fixture circuit zkey

* Update matrix to run tools tests on Windows

* Adds 'deps' dependency for cirdl

* Adjust docker-entrypoint.sh to use CODEX_CIRCUIT_DIR env var

* Review comments by Giuliano

---------

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Veaceslav Doina <20563034+veaceslavdoina@users.noreply.github.com>
2024-09-23 14:37:17 +00:00
Eric e8e9820d5b
feat: add `--payout-address` (#870)
* feat: add `--payout-address`

Allows SPs to be paid out to a separate address, keeping their profits secure.
Supports https://github.com/codex-storage/codex-contracts-eth/pull/144 in the nim-codex client.

* Remove optional payoutAddress

Change --payout-address so that it is no longer optional. There is no longer an overload in `Marketplace.sol` for `fillSlot` accepting no `payoutAddress`.

* Update integration tests to include --payout-address

* move payoutAddress from fillSlot to freeSlot

* Update integration tests to use required payoutAddress

- to make payoutAddress required, the integration tests needed to avoid building the cli params until just before starting the node, otherwise if cli params were added ad-hoc, there would be an error after a non-required parameter was added before a required parameter.

* support client payout address

- withdrawFunds requires a withdrawAddress parameter, directs payouts for withdrawing of client funds (for a cancelled request) to go to that address.

* fix integration test

adds --payout-address to validators

* refactor: support withdrawFunds and freeSlot optional parameters

- withdrawFunds has an optional parameter for withdrawRecipient
- freeSlot has optional parameters for rewardRecipient and collateralRecipient
- change --payout-address to --reward-recipient to match contract signature naming

* Revert "Update integration tests to include --payout-address"

This reverts commit 8f9535cf35.
There are some valid improvements to the integration tests, but they can be handled in a separate PR.

* small fix

* bump contracts to fix marketplace spec

* bump codex-contracts-eth, now rebased on master

* bump codex-contracts-eth

now that feat/reward-address has been merged to master

* clean up, comments
2024-09-17 04:18:15 +00:00
Ben Bierens 1e2ad95659
Update advertising (#862)
* Setting up advertiser

* Wires up advertiser

* cleanup

* test compiles

* tests pass

* setting up test for advertiser

* Finishes advertiser tests

* fixes commonstore tests

* Review comments by Giuliano

* Race condition found by Giuliano

* Review comment by Dmitriy

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* fixes tests

---------

Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2024-08-26 13:18:59 +00:00
Giuliano Mega e017b05cf1
Remove extra license file (#876)
* remove extra license

* center "apache license"
2024-08-19 09:48:03 +00:00
Eric 63ee4ca0f9
refactor(marketplace): generic querying of historical marketplace events (#872)
* refactor(marketplace): move marketplace events to the Market abstraction

Move marketplace contract events to the Market abstraction so the types can be shared across all modules that call the Market abstraction.

* Remove unneeded conversion

* Switch to generic implementation of event querying

* change parent type to MarketplaceEvent
2024-08-16 02:55:25 +00:00
Arnaud 15303125f6
Support CORS preflight requests when the storage request api returns an error (#878)
* Add CORS headers when the REST API is returning an error

* Use the allowedOrigin instead of the wilcard when setting the origin

Signed-off-by: Arnaud <arnaud@status.im>

---------

Signed-off-by: Arnaud <arnaud@status.im>
2024-08-15 23:57:50 +00:00
Eric eeb048e386
chore: add `downtimeProduct` config parameter (#867)
* chore: add `downtimeProduct` config parameter

* bump codex-contracts-eth to master
2024-08-15 02:31:02 +00:00
Arnaud bb9a5fbe92
Add OPTIONS endpoint to allow the content-type header for the upload endpoint (#869)
* Add OPTIONS endpoint to allow the content-type header
exec git commit --amend --no-edit -S

* Remove useless header "Access-Control-Headers" and add cache

Signed-off-by: Arnaud <arnaud@status.im>

---------

Signed-off-by: Arnaud <arnaud@status.im>
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
2024-08-14 21:08:04 +00:00
Giuliano Mega 2771ca6319
Add MIT/Apache licenses (#861)
* Add MIT/Apache licenses

* Center "Apache License"

Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>

* remove wrong legal entity; rename apache license file

---------

Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
2024-08-13 15:38:17 +00:00
Slava b7934fc686
Downgrade to gcc 13 on Windows (#874)
* Downgrade to gcc 13 on Windows

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

* Increase build job timeout to 90 minutes

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

---------

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-08-10 09:50:05 +00:00
Giuliano Mega 4f56f2af26
Chronos v4 Update (v3 Compat Mode) (#814)
* add changes to use chronos v4 in compat mode

* switch chronos to compat fix branch

* use nimbus-build-system with configurable Nim repo

* add missing imports

* add missing await

* bump compat

* pin nim version in Makefile

* add await instead of asyncSpawn to advertisement queue loop

* bump DHT to v0.5.0

* allow error state of `onBatch` to propagate upwards in test code

* pin Nim compiler commit to avoid fetching stale branch

* make CI build against branch head instead of merge

* fix handling of return values in testslotqueue
2024-07-18 21:04:33 +00:00
Giuliano Mega fbce240e3d
Fixes prover behavior with singleton proof trees (#859)
* add logs and test

* add Merkle proof checks

* factor out Circom input normalization, fix proof input serialization

* add test and update existing ones

* update circuit assets

* add back trace message

* switch contracts to fix branch

* update codex-contracts-eth to latest

* do not expose prove with prenormalized inputs
2024-07-18 13:25:06 +00:00
Slava 8f740b42e6
Update Release workflow (#858)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2024-07-10 18:14:28 +00:00
Tomasz Bekas 3ae73197c2
Bandaid for failing erasure coding (#855) 2024-07-03 14:44:00 +00:00
Giuliano Mega fbf1b51f57
Prover workshop band-aid (#853)
* add prover bandaid

* Improve error message text

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>

---------

Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-07-03 11:27:56 +00:00
Eric 67facb4b2a
feat(rest): adds erasure coding constraints when requesting storage (#848)
* Rest API: add erasure coding constraints when requesting storage

* clean up

* Make error message for "dataset too small" more informative.

* fix API integration test

---------

Co-authored-by: gmega <giuliano.mega@gmail.com>
2024-06-27 21:26:19 +00:00
Giuliano Mega b004ca75f6
Bump Nim to 1.6.21 (#851)
* bump Nim to 1.6.21 (range type reset fixes)

* remove incompatible versions from compiler matrix
2024-06-27 19:58:27 +00:00
Tomasz Bekas 11eaebfd77
Fix verifiable manifest constructor (#844)
* Fix verifiable manifest constructor

* Add integration test for verifiable manifest download

Add integration test for testing download of verifiable dataset after creating request for storage

* add missing import

* add testecbug to integration suite

* Remove hardhat instance from integration test

* change description, drop echo

---------

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: gmega <giuliano.mega@gmail.com>
2024-06-26 20:02:39 +00:00
Eric 5b131611ac
bump nim-leopard to handle no parity data (#845) 2024-06-26 15:41:06 +00:00
Eric a79bbd459a
prevent node crashing with `not val.isNil` (#843) 2024-06-26 14:22:44 +00:00
Giuliano Mega a55b676a42
provisional fix so EC errors do not crash the node on download (#841) 2024-06-25 21:38:53 +00:00
Slava ad53066131
ci: add verify_circuit=true to the releases (#840) 2024-06-25 18:31:50 +00:00
Giuliano Mega 8138ef5afd
Fix verifiable manifest initialization (#839)
* fix verifiable manifest initialization

* fix linearstrategy, use verifiableStrategy to select blocks for slots

* check for both strategies in attribute inheritance test
2024-06-21 22:50:56 +00:00
Giuliano Mega 4619260dc1
Fix StoreStream so it doesn't return parity bytes (#838)
* fix storestream so it doesn\'t return parity bits for protected/verifiable manifests

* use Cid.example instead of creating a mock manually
2024-06-21 11:09:59 +00:00
Tomasz Bekas ec7faa21b5
Block deletion with ref count & repostore refactor (#631) 2024-06-20 22:46:06 +00:00
Adam Uhlíř 1a57341b7d
fix: createReservation lock (#825)
* fix: createReservation lock

* fix: additional locking places

* fix: acquire lock

* chore: feedback

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>

* feat: withLock template and fixed tests

* fix: use proc for MockReservations constructor

* chore: feedback

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>

* chore: feedback implementation

---------

Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: markspanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-06-20 10:28:01 +00:00
Giuliano Mega 6e9bdf1d7e
Allow compiler version to be pinned consistently (#832)
* allow compiler version to be pinned consistently

* set pin to v1.6.14 to make it equivalent to current master
2024-06-17 18:59:25 +00:00
Giuliano Mega e124035772
remove stale and misleading dependency version metadata (#833) 2024-06-17 14:31:10 +00:00
Eric e422c9065f
config flag to enable CORS response headers for data downloads (#834) 2024-06-17 11:33:21 +00:00
Giuliano Mega d524252f1f
make CI build against branch head instead of merge (#831) 2024-06-13 01:54:50 +00:00
Tomasz Bekas f51ef528b1
Rework AsyncIter (#811)
* Rework AsyncIter

* Add tests for finishing iter on error

* Improved error handling for  and additional tests

* Use new style of constructors

* Handle future cancellation

* Docs for constructors
2024-06-10 22:47:29 +00:00
Ben Bierens fe9d9705f1
Fix/spec intsizes (#829)
* Sets int64 formats for byte-counting integers in openapi spec

* uints instead of ints

* uint64 not supported by every openAPI client
2024-06-06 09:44:58 +00:00
Ben Bierens e8f4658ea5
Sets int64 formats for byte-counting integers in openapi spec (#828)
* Sets int64 formats for byte-counting integers in openapi spec

* uints instead of ints
2024-06-06 06:20:46 +00:00
Slava a4a50fd714
Add Release workflow (#827)
* Install specific Rust version and add Linux arm64 support (#749)

* Add release workflow (#749)

* Update Rust requirements in the Building guide (#749)

* Fix rust_version input usage (#749)

* Update comments (#749)

* Use ARCH_OVERRIDE only for i386 (#749)

* Add an alternative method to install Rust (#749)

* Do not upload builds to the Codex network (#749)
2024-06-05 12:56:41 +00:00
Ben Bierens 3246c43174
Updates rust in docker. Pins circom-compat to pinned FFI branch (#818)
* Updates rust in docker. Pins circom-compat to pinned FFI branch

* Extracts builder image from codex.Dockerfile

* Updates readme

* Use single Dockerfile

* Revert readme back

---------

Co-authored-by: Veaceslav Doina <20563034+veaceslavdoina@users.noreply.github.com>
2024-05-31 08:23:11 +00:00