* fix: createReservation lock (#825)
* fix: createReservation lock
* fix: additional locking places
* fix: acquire lock
* chore: feedback
Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
* feat: withLock template and fixed tests
* fix: use proc for MockReservations constructor
* chore: feedback
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
* chore: feedback implementation
---------
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: markspanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* Block deletion with ref count & repostore refactor (#631)
* Fix StoreStream so it doesn't return parity bytes (#838)
* fix storestream so it doesn\'t return parity bits for protected/verifiable manifests
* use Cid.example instead of creating a mock manually
* Fix verifiable manifest initialization (#839)
* fix verifiable manifest initialization
* fix linearstrategy, use verifiableStrategy to select blocks for slots
* check for both strategies in attribute inheritance test
* ci: add verify_circuit=true to the releases (#840)
* provisional fix so EC errors do not crash the node on download (#841)
* prevent node crashing with `not val.isNil` (#843)
* bump nim-leopard to handle no parity data (#845)
* Fix verifiable manifest constructor (#844)
* Fix verifiable manifest constructor
* Add integration test for verifiable manifest download
Add integration test for testing download of verifiable dataset after creating request for storage
* add missing import
* add testecbug to integration suite
* Remove hardhat instance from integration test
* change description, drop echo
---------
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: gmega <giuliano.mega@gmail.com>
* Bump Nim to 1.6.21 (#851)
* bump Nim to 1.6.21 (range type reset fixes)
* remove incompatible versions from compiler matrix
* feat(rest): adds erasure coding constraints when requesting storage (#848)
* Rest API: add erasure coding constraints when requesting storage
* clean up
* Make error message for "dataset too small" more informative.
* fix API integration test
---------
Co-authored-by: gmega <giuliano.mega@gmail.com>
* Prover workshop band-aid (#853)
* add prover bandaid
* Improve error message text
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
---------
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* Bandaid for failing erasure coding (#855)
* Update Release workflow (#858)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Fixes prover behavior with singleton proof trees (#859)
* add logs and test
* add Merkle proof checks
* factor out Circom input normalization, fix proof input serialization
* add test and update existing ones
* update circuit assets
* add back trace message
* switch contracts to fix branch
* update codex-contracts-eth to latest
* do not expose prove with prenormalized inputs
* Chronos v4 Update (v3 Compat Mode) (#814)
* add changes to use chronos v4 in compat mode
* switch chronos to compat fix branch
* use nimbus-build-system with configurable Nim repo
* add missing imports
* add missing await
* bump compat
* pin nim version in Makefile
* add await instead of asyncSpawn to advertisement queue loop
* bump DHT to v0.5.0
* allow error state of `onBatch` to propagate upwards in test code
* pin Nim compiler commit to avoid fetching stale branch
* make CI build against branch head instead of merge
* fix handling of return values in testslotqueue
* Downgrade to gcc 13 on Windows (#874)
* Downgrade to gcc 13 on Windows
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Increase build job timeout to 90 minutes
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
---------
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Add MIT/Apache licenses (#861)
* Add MIT/Apache licenses
* Center "Apache License"
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
* remove wrong legal entity; rename apache license file
---------
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
* Add OPTIONS endpoint to allow the content-type header for the upload endpoint (#869)
* Add OPTIONS endpoint to allow the content-type header
exec git commit --amend --no-edit -S
* Remove useless header "Access-Control-Headers" and add cache
Signed-off-by: Arnaud <arnaud@status.im>
---------
Signed-off-by: Arnaud <arnaud@status.im>
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
* chore: add `downtimeProduct` config parameter (#867)
* chore: add `downtimeProduct` config parameter
* bump codex-contracts-eth to master
* Support CORS preflight requests when the storage request api returns an error (#878)
* Add CORS headers when the REST API is returning an error
* Use the allowedOrigin instead of the wilcard when setting the origin
Signed-off-by: Arnaud <arnaud@status.im>
---------
Signed-off-by: Arnaud <arnaud@status.im>
* refactor(marketplace): generic querying of historical marketplace events (#872)
* refactor(marketplace): move marketplace events to the Market abstraction
Move marketplace contract events to the Market abstraction so the types can be shared across all modules that call the Market abstraction.
* Remove unneeded conversion
* Switch to generic implementation of event querying
* change parent type to MarketplaceEvent
* Remove extra license file (#876)
* remove extra license
* center "apache license"
* Update advertising (#862)
* Setting up advertiser
* Wires up advertiser
* cleanup
* test compiles
* tests pass
* setting up test for advertiser
* Finishes advertiser tests
* fixes commonstore tests
* Review comments by Giuliano
* Race condition found by Giuliano
* Review comment by Dmitriy
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
* fixes tests
---------
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* feat: add `--payout-address` (#870)
* feat: add `--payout-address`
Allows SPs to be paid out to a separate address, keeping their profits secure.
Supports https://github.com/codex-storage/codex-contracts-eth/pull/144 in the nim-codex client.
* Remove optional payoutAddress
Change --payout-address so that it is no longer optional. There is no longer an overload in `Marketplace.sol` for `fillSlot` accepting no `payoutAddress`.
* Update integration tests to include --payout-address
* move payoutAddress from fillSlot to freeSlot
* Update integration tests to use required payoutAddress
- to make payoutAddress required, the integration tests needed to avoid building the cli params until just before starting the node, otherwise if cli params were added ad-hoc, there would be an error after a non-required parameter was added before a required parameter.
* support client payout address
- withdrawFunds requires a withdrawAddress parameter, directs payouts for withdrawing of client funds (for a cancelled request) to go to that address.
* fix integration test
adds --payout-address to validators
* refactor: support withdrawFunds and freeSlot optional parameters
- withdrawFunds has an optional parameter for withdrawRecipient
- freeSlot has optional parameters for rewardRecipient and collateralRecipient
- change --payout-address to --reward-recipient to match contract signature naming
* Revert "Update integration tests to include --payout-address"
This reverts commit 8f9535cf35.
There are some valid improvements to the integration tests, but they can be handled in a separate PR.
* small fix
* bump contracts to fix marketplace spec
* bump codex-contracts-eth, now rebased on master
* bump codex-contracts-eth
now that feat/reward-address has been merged to master
* clean up, comments
* Rework circuit downloader (#882)
* Introduces a start method to prover
* Moves backend creation into start method
* sets up three paths for backend initialization
* Extracts backend initialization to backend-factory
* Implements loading backend from cli files or previously downloaded local files
* Wires up downloading and unzipping
* functional implementation
* Fixes testprover.nim
* Sets up tests for backendfactory
* includes libzip-dev
* pulls in updated contracts
* removes integration cli tests for r1cs, wasm, and zkey file arguments.
* Fixes issue where inner-scope values are lost before returning
* sets local proof verification for dist-test images
* Adds two traces and bumps nim-ethers
* Adds separate path for circuit files
* Create circuit dir if not exists
* fix: make sure requestStorage is mined
* fix: correct place to plug confirm
* test: fixing contracts tests
* Restores gitmodules
* restores nim-datastore reference
* Sets up downloader exe
* sets up tool skeleton
* implements getting of circuit hash
* Implements downloader tool
* sets up test skeleton
* Implements test for cirdl
* includes testTools in testAll
* Cleanup building.md
* cleans up previous downloader implementation
* cleans up testbackendfactory
* moves start of prover into node.nim
* Fills in arguments in example command
* Initializes backend in prover constructor
* Restores tests
* Restores tests for cli instructions
* Review comments by Dmitriy, part 1
* Quotes path in download instruction.
* replaces curl with chronos http session
* Moves cirdl build output to 'build' folder.
* Fixes chronicles log output
* Add cirdl support to the codex Dockerfile
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Add cirdl support to the docker entrypoint
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Add cirdl support to the release workflow
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Disable verify_circuit flag for releases
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Removes backendFactory placeholder type
* wip
* Replaces zip library with status-im/zippy library (which supports zip and tar)
* Updates cirdl to not change circuitdir folder
* Switches from zip to tar.gz
* Review comments by Dmitriy
* updates codex-contracts-eth
* Adds testTools to CI
* Adds check for access to config.circuitdir
* Update fixture circuit zkey
* Update matrix to run tools tests on Windows
* Adds 'deps' dependency for cirdl
* Adjust docker-entrypoint.sh to use CODEX_CIRCUIT_DIR env var
* Review comments by Giuliano
---------
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Veaceslav Doina <20563034+veaceslavdoina@users.noreply.github.com>
* Support CORS for POST and PATCH availability endpoints (#897)
* Adds testnet marketplace address to known deployments (#911)
* API tweaks for OpenAPI, errors and endpoints (#886)
* All sort of tweaks
* docs: availability's minPrice doc
* Revert changes to the two node test example
* Change default EC params in REST API
Change default EC params in REST API to 3 nodes and 1 tolerance.
Adjust integration tests to honour these settings.
---------
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* remove erasure and por parameters from openapi spec (#915)
* Move Building Codex guide to the main docs site (#893)
* updates Marketplace tutorial documentation (#888)
* updates Marketplace tutorial documentation
* Applies review comments to marketplace-tutorial
* Final formatting touches
* moved `Prerequisites` around
* Fixes indentation in one JSON snippet
* Use CLI args when passed for cirdl in Docker entrypoint (#927)
* Use CLI args when passed for cirdl in Docker entrypoint
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Increase CI timeout
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
---------
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Validator - support partitioning of the slot id space (#890)
* Adds validatorPartitionSize and validatorPartitionIndex config options
* adds partitioning options to the validation type
* adds partitioning logic to the validator
* ignores partitionIndex when partitionSize is either 0 or 1
* clips the partition index to <<partitionIndex mod partitionSize>>
* handles negative values for the validation partition index
* updates long description of the new validator cli options
* makes default partitionSize to be 0 for better backward compatibility
* Improving formatting on validator CLI
* reactors validation params into a separate type and simplifies validation of validation params
* removes suspected duplication
* fixes typo in validator CLI help
* updates README
* Applies review comments - using optionals and range types to handle validation params
* Adds initializer to the configFactory for validatorMaxSlots
* [Review] update validator CLI description and README
* [Review]: renaming validationParams to validationConfig (config)
* [Review]: move validationconfig.nim to a higher level (next to validation.nim)
* changes backing type of MaxSlots to be int and makes sure slots are validated without limit when maxSlots is set to 0
* adds more end-to-end test for the validator and the groups
* fixes typo in README and conf.nim
* makes `maxSlotsConstraintRespected` and `shouldValidateSlot` private + updates the tests
* fixes public address of the signer account in the marketplace tutorial
* applies review comments - removes two tests
* Remove moved docs (#930)
* Remove moved document
* Update main Readme and point links to the documentation site
* feat(slot-reservations): Support reserving slots (#907)
* feat(slot-reservations): Support reserving slots
Closes#898.
Wire up reserveSlot and canReserveSlot contract calls, but don't call them
* Remove return value from `reserveSlot`
* convert EthersError to MarketError
* Move convertEthersError to reserveSlot
* bump codex-contracts-eth after rebase
* change `canReserveSlot` and `reserveSlot` parameters
Parameters for `canReserveSlot` and `reserveSlot` were changed from `SlotId` to `RequestId` and `UInt256 slotIndex`.
* bump codex-contracts-eth after rebase
* bump codex-contracts-eth to master after codex-contracts-eth/pull/177 merged
* feat(slot-reservations): Add SaleSlotReserving state (#917)
* convert EthersError to MarketError
* change `canReserveSlot` and `reserveSlot` parameters
Parameters for `canReserveSlot` and `reserveSlot` were changed from `SlotId` to `RequestId` and `UInt256 slotIndex`.
* Add SaleSlotReserving
Adds a new state, SaleSlotReserving, that attempts to reserve a slot before downloading.
If the slot cannot be reserved, the state moves to SaleIgnored.
On error, the state moves to SaleErrored.
SaleIgnored is also updated to pass in `reprocessSlot` and `returnBytes`, controlling the behaviour in the Sales module after the slot is ignored. This is because previously it was assumed that SaleIgnored was only reached when there was no Availability. This is no longer the case, since SaleIgnored can now be reached when a slot cannot be reserved.
* Update SalePreparing
Specify `reprocessSlot` and `returnBytes` when moving to `SaleIgnored` from `SalePreparing`.
Update tests to include test for a raised CatchableError.
* Fix unit test
* Modify `canReserveSlot` and `reverseSlot` params after rebase
* Update MockMarket with new `canReserveSlot` and `reserveSlot` params
* fix after rebase
also bump codex-contracts-eth to master
* Use Ubuntu 20.04 for Linux amd64 releases (#939)
* Use Ubuntu 20.04 for Linux amd64 releases (#932)
* Accept branches with the slash in the name for release workflow (#932)
* Increase artifacts retention-days for release workflow (#932)
* feat(slot-reservations): support SlotReservationsFull event (#926)
* Remove moved docs (#935)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Fix: null-ref in networkPeer (#937)
* Fixes nullref in networkPeer
* Removes inflight semaphore
* Revert "Removes inflight semaphore"
This reverts commit 26ec15c6f7.
---------
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
Signed-off-by: Arnaud <arnaud@status.im>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: markspanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: Tomasz Bekas <tomasz.bekas@gmail.com>
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
Co-authored-by: Arnaud <arno.deville@gmail.com>
Co-authored-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Arnaud <arnaud@status.im>
Co-authored-by: Marcin Czenko <marcin.czenko@pm.me>
* fix: createReservation lock (#825)
* fix: createReservation lock
* fix: additional locking places
* fix: acquire lock
* chore: feedback
Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
* feat: withLock template and fixed tests
* fix: use proc for MockReservations constructor
* chore: feedback
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
* chore: feedback implementation
---------
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: markspanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* Block deletion with ref count & repostore refactor (#631)
* Fix StoreStream so it doesn't return parity bytes (#838)
* fix storestream so it doesn\'t return parity bits for protected/verifiable manifests
* use Cid.example instead of creating a mock manually
* Fix verifiable manifest initialization (#839)
* fix verifiable manifest initialization
* fix linearstrategy, use verifiableStrategy to select blocks for slots
* check for both strategies in attribute inheritance test
* ci: add verify_circuit=true to the releases (#840)
* provisional fix so EC errors do not crash the node on download (#841)
* prevent node crashing with `not val.isNil` (#843)
* bump nim-leopard to handle no parity data (#845)
* Fix verifiable manifest constructor (#844)
* Fix verifiable manifest constructor
* Add integration test for verifiable manifest download
Add integration test for testing download of verifiable dataset after creating request for storage
* add missing import
* add testecbug to integration suite
* Remove hardhat instance from integration test
* change description, drop echo
---------
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: gmega <giuliano.mega@gmail.com>
* Bump Nim to 1.6.21 (#851)
* bump Nim to 1.6.21 (range type reset fixes)
* remove incompatible versions from compiler matrix
* feat(rest): adds erasure coding constraints when requesting storage (#848)
* Rest API: add erasure coding constraints when requesting storage
* clean up
* Make error message for "dataset too small" more informative.
* fix API integration test
---------
Co-authored-by: gmega <giuliano.mega@gmail.com>
* Prover workshop band-aid (#853)
* add prover bandaid
* Improve error message text
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
---------
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* Bandaid for failing erasure coding (#855)
* Update Release workflow (#858)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Fixes prover behavior with singleton proof trees (#859)
* add logs and test
* add Merkle proof checks
* factor out Circom input normalization, fix proof input serialization
* add test and update existing ones
* update circuit assets
* add back trace message
* switch contracts to fix branch
* update codex-contracts-eth to latest
* do not expose prove with prenormalized inputs
* Chronos v4 Update (v3 Compat Mode) (#814)
* add changes to use chronos v4 in compat mode
* switch chronos to compat fix branch
* use nimbus-build-system with configurable Nim repo
* add missing imports
* add missing await
* bump compat
* pin nim version in Makefile
* add await instead of asyncSpawn to advertisement queue loop
* bump DHT to v0.5.0
* allow error state of `onBatch` to propagate upwards in test code
* pin Nim compiler commit to avoid fetching stale branch
* make CI build against branch head instead of merge
* fix handling of return values in testslotqueue
* Downgrade to gcc 13 on Windows (#874)
* Downgrade to gcc 13 on Windows
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Increase build job timeout to 90 minutes
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
---------
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Add MIT/Apache licenses (#861)
* Add MIT/Apache licenses
* Center "Apache License"
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
* remove wrong legal entity; rename apache license file
---------
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
* Add OPTIONS endpoint to allow the content-type header for the upload endpoint (#869)
* Add OPTIONS endpoint to allow the content-type header
exec git commit --amend --no-edit -S
* Remove useless header "Access-Control-Headers" and add cache
Signed-off-by: Arnaud <arnaud@status.im>
---------
Signed-off-by: Arnaud <arnaud@status.im>
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
* chore: add `downtimeProduct` config parameter (#867)
* chore: add `downtimeProduct` config parameter
* bump codex-contracts-eth to master
* Support CORS preflight requests when the storage request api returns an error (#878)
* Add CORS headers when the REST API is returning an error
* Use the allowedOrigin instead of the wilcard when setting the origin
Signed-off-by: Arnaud <arnaud@status.im>
---------
Signed-off-by: Arnaud <arnaud@status.im>
* refactor(marketplace): generic querying of historical marketplace events (#872)
* refactor(marketplace): move marketplace events to the Market abstraction
Move marketplace contract events to the Market abstraction so the types can be shared across all modules that call the Market abstraction.
* Remove unneeded conversion
* Switch to generic implementation of event querying
* change parent type to MarketplaceEvent
* Remove extra license file (#876)
* remove extra license
* center "apache license"
* Update advertising (#862)
* Setting up advertiser
* Wires up advertiser
* cleanup
* test compiles
* tests pass
* setting up test for advertiser
* Finishes advertiser tests
* fixes commonstore tests
* Review comments by Giuliano
* Race condition found by Giuliano
* Review comment by Dmitriy
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
* fixes tests
---------
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* feat: add `--payout-address` (#870)
* feat: add `--payout-address`
Allows SPs to be paid out to a separate address, keeping their profits secure.
Supports https://github.com/codex-storage/codex-contracts-eth/pull/144 in the nim-codex client.
* Remove optional payoutAddress
Change --payout-address so that it is no longer optional. There is no longer an overload in `Marketplace.sol` for `fillSlot` accepting no `payoutAddress`.
* Update integration tests to include --payout-address
* move payoutAddress from fillSlot to freeSlot
* Update integration tests to use required payoutAddress
- to make payoutAddress required, the integration tests needed to avoid building the cli params until just before starting the node, otherwise if cli params were added ad-hoc, there would be an error after a non-required parameter was added before a required parameter.
* support client payout address
- withdrawFunds requires a withdrawAddress parameter, directs payouts for withdrawing of client funds (for a cancelled request) to go to that address.
* fix integration test
adds --payout-address to validators
* refactor: support withdrawFunds and freeSlot optional parameters
- withdrawFunds has an optional parameter for withdrawRecipient
- freeSlot has optional parameters for rewardRecipient and collateralRecipient
- change --payout-address to --reward-recipient to match contract signature naming
* Revert "Update integration tests to include --payout-address"
This reverts commit 8f9535cf35.
There are some valid improvements to the integration tests, but they can be handled in a separate PR.
* small fix
* bump contracts to fix marketplace spec
* bump codex-contracts-eth, now rebased on master
* bump codex-contracts-eth
now that feat/reward-address has been merged to master
* clean up, comments
* Rework circuit downloader (#882)
* Introduces a start method to prover
* Moves backend creation into start method
* sets up three paths for backend initialization
* Extracts backend initialization to backend-factory
* Implements loading backend from cli files or previously downloaded local files
* Wires up downloading and unzipping
* functional implementation
* Fixes testprover.nim
* Sets up tests for backendfactory
* includes libzip-dev
* pulls in updated contracts
* removes integration cli tests for r1cs, wasm, and zkey file arguments.
* Fixes issue where inner-scope values are lost before returning
* sets local proof verification for dist-test images
* Adds two traces and bumps nim-ethers
* Adds separate path for circuit files
* Create circuit dir if not exists
* fix: make sure requestStorage is mined
* fix: correct place to plug confirm
* test: fixing contracts tests
* Restores gitmodules
* restores nim-datastore reference
* Sets up downloader exe
* sets up tool skeleton
* implements getting of circuit hash
* Implements downloader tool
* sets up test skeleton
* Implements test for cirdl
* includes testTools in testAll
* Cleanup building.md
* cleans up previous downloader implementation
* cleans up testbackendfactory
* moves start of prover into node.nim
* Fills in arguments in example command
* Initializes backend in prover constructor
* Restores tests
* Restores tests for cli instructions
* Review comments by Dmitriy, part 1
* Quotes path in download instruction.
* replaces curl with chronos http session
* Moves cirdl build output to 'build' folder.
* Fixes chronicles log output
* Add cirdl support to the codex Dockerfile
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Add cirdl support to the docker entrypoint
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Add cirdl support to the release workflow
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Disable verify_circuit flag for releases
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Removes backendFactory placeholder type
* wip
* Replaces zip library with status-im/zippy library (which supports zip and tar)
* Updates cirdl to not change circuitdir folder
* Switches from zip to tar.gz
* Review comments by Dmitriy
* updates codex-contracts-eth
* Adds testTools to CI
* Adds check for access to config.circuitdir
* Update fixture circuit zkey
* Update matrix to run tools tests on Windows
* Adds 'deps' dependency for cirdl
* Adjust docker-entrypoint.sh to use CODEX_CIRCUIT_DIR env var
* Review comments by Giuliano
---------
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Veaceslav Doina <20563034+veaceslavdoina@users.noreply.github.com>
* Support CORS for POST and PATCH availability endpoints (#897)
* Adds testnet marketplace address to known deployments (#911)
* API tweaks for OpenAPI, errors and endpoints (#886)
* All sort of tweaks
* docs: availability's minPrice doc
* Revert changes to the two node test example
* Change default EC params in REST API
Change default EC params in REST API to 3 nodes and 1 tolerance.
Adjust integration tests to honour these settings.
---------
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
---------
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
Signed-off-by: Arnaud <arnaud@status.im>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: markspanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: Tomasz Bekas <tomasz.bekas@gmail.com>
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
Co-authored-by: Arnaud <arno.deville@gmail.com>
Co-authored-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Arnaud <arnaud@status.im>
* fix: createReservation lock (#825)
* fix: createReservation lock
* fix: additional locking places
* fix: acquire lock
* chore: feedback
Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
* feat: withLock template and fixed tests
* fix: use proc for MockReservations constructor
* chore: feedback
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
* chore: feedback implementation
---------
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: markspanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* Block deletion with ref count & repostore refactor (#631)
* Fix StoreStream so it doesn't return parity bytes (#838)
* fix storestream so it doesn\'t return parity bits for protected/verifiable manifests
* use Cid.example instead of creating a mock manually
* Fix verifiable manifest initialization (#839)
* fix verifiable manifest initialization
* fix linearstrategy, use verifiableStrategy to select blocks for slots
* check for both strategies in attribute inheritance test
* ci: add verify_circuit=true to the releases (#840)
* provisional fix so EC errors do not crash the node on download (#841)
* prevent node crashing with `not val.isNil` (#843)
* bump nim-leopard to handle no parity data (#845)
* Fix verifiable manifest constructor (#844)
* Fix verifiable manifest constructor
* Add integration test for verifiable manifest download
Add integration test for testing download of verifiable dataset after creating request for storage
* add missing import
* add testecbug to integration suite
* Remove hardhat instance from integration test
* change description, drop echo
---------
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: gmega <giuliano.mega@gmail.com>
---------
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: markspanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: Tomasz Bekas <tomasz.bekas@gmail.com>
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
* Rework AsyncIter
* Add tests for finishing iter on error
* Improved error handling for and additional tests
* Use new style of constructors
* Handle future cancellation
* Docs for constructors
* pulls in datastore-leveldb update
* bump
* Applies LevelDb as metadata store. Adds option for repostore.
* Sets submodule to main branch
* I can do syntax, me
* Removes wildcard from metadata query key
* Applies leveldb instead of sqlite-in-memory for tests
* Restores query key wildcard.
* Pins nim-datastore to latest master
* bumps leveldb to 0.1.4
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* json > nim-serde bump
Should wait until serde is integrated into nim-ethers before making these changes as there will be less import exceptions required.
* bump nim-serde
* change func to proc due to chronicles side effects
* import serde into utils/json, use as proxy
import nim-serde into utils/json and use utils/json as a proxy for serde functions, including overloading `%` and `fromJson` for application types.
* update tests to use serde
* bump serde to latest
* remove testjson -- no longer needed
* bump serde in nimble
* updates to reconcile rebase with master
* Update codex-contracts-eth
* contracts: update G2Point definition
* integration: disable automatic advancing of time
reason: makes reasoning about timing in tests harder,
because the period is set to 60 seconds in the
marketplace configuration, but this code switches to
a new period every 500 milliseconds
* integration: fix parameters of marketplace payouts test
* integration: update test settings
* integration: fix typo
* integration: workaround for hardhat issue
Subscriptions expire after 5 minutes when using
websockets. Use http and polling instead.
* integration: remove origDatasetSizeInBlocks
* integration: fix proof parameters for test
* integration: do not log output by default
* integration: fix failure rate in test
* integration: fix warning
* integration: include clock in logs
* integration: allow for more periods
5 periods was cutting it close, if we get too much
pointer downtime, then the test would fail
Fixes validator, which uses transaction simulation to
check whether a proof can be marked as missing.
Fixes gas estimations, so they are done on the pending
block, instead of the latest block.
* rework backend to instantiate key at initialization
* add groth16 convertes for solidity
* prover taks num samples on construction
* add zkey file
* rework helpers
* rename types
* update tests
* reworked test helpers
* rename types
* rework test
* test all slots artifacts
* bump to latest version
* wire in circom backend
* should contain leafs
* adding circom compad and circuits deps
* update windows build
* fix windows build
* improve test names
* move proving defaults to codextypes
* remove unnedded inmports and move defaults to codextypes
* capture error code on backend failure
* Smart contracts update: Groth16Proof instead of bytes
* Use dummy verifier for now, until we can create ZK proofs
* Fix tests: submit proof only when slot is filled
* Submit dummy proofs for now
* More detailed log when proof submission failed
* Use dummy verifier for integration tests
For now at least
* Fix mistake in blanket renaming to ethProvider
* Update to latest codex-contracts-eth
* feat: zkey-hash from chain
* Fix zkeyHash
---------
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
* implement a logging proxy
The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks
* Rename logging -> logutils to avoid ambiguity with common names
* clean up
* add setProperty for JsonRecord, remove nim-json-serialization conflict
* Allow specifying textlines and json format separately
Not specifying a LogFormat will apply the formatting to both textlines and json sinks.
Specifying a LogFormat will apply the formatting to only that sink.
* remove unneeded usages of std/json
We only need to import utils/json instead of std/json
* move serialization from rest/json to utils/json so it can be shared
* fix NoColors ambiguity
Was causing unit tests to fail on Windows.
* Remove nre usage to fix Windows error
Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)
* Add logutils module doc
* Shorten logutils.formatIt for `NBytes`
Both json and textlines formatIt were not needed, and could be combined into one formatIt
* remove debug integration test config
debug output and logformat of json for integration test logs
* Use ## module doc to support docgen
* bump nim-poseidon2 to export fromBytes
Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening.
* fixes to get compiling after rebasing master
* Add support for Result types being logged using formatIt
* rework merkle tree support
* rename merkletree -> codexmerkletree
* treed and proof encoding/decoding
* style
* adding codex merkle and coders tests
* use default hash codec
* proof size changed
* add from nodes test
* shorte file names
* wip poseidon tree
* shorten file names
* root returns a result
* import poseidon tests
* fix merge issues and cleanup a few warnings
* setting up slot builder
* Getting cids in slot
* ensures blocks are devisable by number of slots
* wip
* Implements indexing strategies
* Swaps in indexing strategy into erasure.
* wires slot and indexing tests up
* Fixes issue where indexing strategy stepped gives wrong values for smallest of ranges
* debugs indexing strategies
* Can select slot blocks
* finding number of pad cells
* Implements building slot tree
* finishes implementing slot builder
* Adds check that block size is a multiple of cell size
* Cleanup slotbuilder
* Review comments by Tomasz
* Fixes issue where ecK was used as numberOfSlots.
* rework merkle tree support
* deps
* rename merkletree -> codexmerkletree
* treed and proof encoding/decoding
* style
* adding codex merkle and coders tests
* remove new codecs for now
* proof size changed
* add from nodes test
* shorte file names
* wip poseidon tree
* shorten file names
* fix bad `elements` iter
* bump
* bump
* wip
* reworking slotbuilder
* move out of manifest
* expose getCidAndProof
* import index strat...
* remove getMHash
* remove unused artifacts
* alias zero
* add digest for multihash
* merge issues
* remove unused hashes
* add option to result converter
* misc
* fix tests
* add helper to derive EC block count
* rename method
* misc
* bump
* extract slot root building into own proc
* revert to manifest to accessor
---------
Co-authored-by: benbierens <thatbenbierens@gmail.com>
* Workaround for Hardhat timestamp bug
Likely due to a Hardhat bug in which the callbacks for subscription events are called and awaited before updating its local understanding of the last block time, Hardhat will report a block time in the `newHeads` event that is generally 1 second before the time reported from `getLatestBlock.timestamp`. This was causing issues with the OnChainClock's offset and therefore the `now()` used by the `OnChainClock` would sometimes be off by a second (or more), causing tests to fail.
This commit introduce a `codex_use_hardhat` compilation flag, that when set, will always get the latest block timestamp from Hardhat via the `getLatestBlock.timestamp` RPC call for `OnChainClock.now` calls. Otherwise, the last block timestamp reported in the `newHeads` event will be used.
Update the docker dist tests compilation flag for simulated proof failures (it was not correct), and explicitly add the `codex_use_hardhat=false` for clarity.
* enable simulated proof failures for coverage
* comment out failing test on linux -- will be replaced
* bump codex contracts eth
* add back clock offset for non-hardhat cases
* bump codex-contracts-eth
increases pointer by 67 blocks each period increase
* Add `codex_use_hardhat` flag to coverage tests
* Adds test for encoding/decoding protected manifest
* Setting up verifiable manifest
* mysterious mysteries
* Successful encoding test for verifiable manifests
* extracts toF out of users of manifest code
* Update codex/manifest/coders.nim
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
* Review comments by Dmitriy
* Adds missing verifiable print to $ method.
* Replace poseidon2 F type with int as temporary stand-in for verification hashes
* Replaces verification hash placeholder with CID
---------
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* [build] add nim-poseidon2
* [build] Workaround incompatibility constantine and secp256k1
Constantine sets compiler flag that is incompatible
with the assembly code in secp256k1:
5f7ba18f2e/constantine/platforms/isa/macro_assembler_x86.nim (L19)
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* [docs] fix two client scenario: add missing collateral
* [integration] separate step to wait for node to be started
* [cli] add option to specify ethereum private key
* Remove unused imports
* Fix warnings
* [integration] move type definitions to correct place
* [integration] wait a bit longer for a node to start in debug mode
When e.g. running against Taiko test net rpc, the node start
takes longer
* [integration] simplify handling of codex node and client
* [integration] add Taiko integration test
* [contracts] await token approval confirmation before next tx
* [contracts] deployment address of marketplace on Taiko
* [cli] --eth-private-key now takes a file name
Instead of supplying the private key on the command line,
expect the private key to be in a file with the correct
permissions.
* [utils] Fixes undeclared `activeChroniclesStream` on Windows
* [build] update nim-ethers to include PR #52
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
* [cli] Better error messages when reading eth private key
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
* [integration] simplify reading of cmd line arguments
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
* [build] update to latest version of nim-ethers
* [contracts] updated contract address for Taiko L2
* [build] update codex contracts to latest version
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
* extra utilities and tweaks
* add atlas lock
* update ignores
* break build into it's own script
* update url rules
* base off codexdht's
* compile fixes for Nim 1.6.14
* update submodules
* convert mapFailure to procs to work around type resolution issues
* add toml parser for multiaddress
* change error type on keyutils
* bump nimbus build to use 1.6.14
* update gitignore
* adding new deps submodules
* bump nim ci version
* even more fixes
* more libp2p changes
* update keys
* fix eventually function
* adding coverage test file
* move coverage to build.nims
* use nimcache/coverage
* move libp2p import for tests into helper.nim
* remove named bin
* bug fixes for networkpeers (from Dmitriy)
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
## Slot queue
Adds a slot queue, as per the [slot queue design](https://github.com/codex-storage/codex-research/blob/master/design/sales.md#slot-queue).
Any time storage is requested, all slots from that request are immediately added to the queue. Finished, Canclled, Failed requests remove all slots with that request id from the queue. SlotFreed events add a new slot to the queue and SlotFilled events remove the slot from the queue. This allows popping of a slot each time one is processed, making things much simpler.
When an entire request of slots is added to the queue, the slot indices are shuffled randomly to hopefully prevent nodes that pick up the same storage requested event from clashing on the first processed slot index. This allowed removal of assigning a random slot index in the SalePreparing state and it also ensured that all SalesAgents will have a slot index assigned to them at the start thus the removal of the optional slotIndex.
Remove slotId from SlotFreed event as it was not being used. RequestId and slotIndex were added to the SlotFreed event earlier and those are now being used
The slot queue invariant that prioritises queue items added to the queue relies on a scoring mechanism to sort them based on the [sort order in the design document](https://github.com/codex-storage/codex-research/blob/master/design/sales.md#sort-order).
When a storage request is handled by the sales module, a slot index was randomly assigned and then the slot was filled. Now, a random slot index is only assigned when adding an entire request to the slot queue. Additionally, the slot is checked that its state is `SlotState.Free` before continuing with the download process.
SlotQueue should always ensure the underlying AsyncHeapQueue has one less than the maximum items, ensuring the SlotQueue can always have space to add an additional item regardless if it’s full or not.
Constructing `SlotQueue.workers` in `SlotQueue.new` calls `newAsyncQueue` which causes side effects, so the construction call had to be moved to `SlotQueue.start`.
Prevent loading request from contract (network request) if there is an existing item in queue for that request.
Check availability before adding request to queue.
Add ability to query market contract for past events. When new availabilities are added, the `onReservationAdded` callback is triggered in which past `StorageRequested` events are queried, and those slots are added to the queue (filtered by availability on `push` and filtered by state in `SalePreparing`).
#### Request Workers
Limit the concurrent requests being processed in the queue by using a limited pool of workers (default = 3). Workers are in a data structure of type `AsyncQueue[SlotQueueWorker]`. This allows us to await a `popFirst` for available workers inside of the main SlotQueue event loop
Add an `onCleanUp` that stops the agents and removes them from the sales module agent list. `onCleanUp` is called from sales end states (eg ignored, cancelled, finished, failed, errored).
Add a `doneProcessing` future to `SlotQueueWorker` to be completed in the `OnProcessSlot` callback. Each `doneProcessing` future created is cancelled and awaited in `SlotQueue.stop` (thanks to `TrackableFuturees`), which forced `stop` to become async.
- Cancel dispatched workers and the `onProcessSlot` callbacks, prevents zombie callbacks
#### Add TrackableFutures
Allow tracking of futures in a module so they can be cancelled at a later time. Useful for asyncSpawned futures, but works for any future.
### Sales module
The sales module needed to subscribe to request events to ensure that the request queue was managed correctly on each event. In the process of doing this, the sales agents were updated to avoid subscribing to events in each agent, and instead dispatch received events from the sales module to all created sales agents. This would prevent memory leaks on having too many eventemitters subscribed to.
- prevent removal of agents from sales module while stopping, otherwise the agents seq len is modified while iterating
An additional sales agent state was added, `SalePreparing`, that handles all state machine setup, such as retrieving the request and subscribing to events that were previously in the `SaleDownloading` state.
Once agents have parked in an end state (eg ignored, cancelled, finished, failed, errored), they were not getting cleaned up and the sales module was keeping a handle on their reference. An `onCleanUp` callback was created to be called after the state machine enters an end state, which could prevent a memory leak if the number of requests coming in is high.
Move the SalesAgent callback raises pragmas from the Sales module to the proc definition in SalesAgent. This avoids having to catch `Exception`.
- remove unneeded error handling as pragmas were moved
Move sales.subscriptions from an object containing named subscriptions to a `seq[Subscription]` directly on the sales object.
Sales tests: shut down repo after sales stop, to fix SIGABRT in CI
### Add async Promise API
- modelled after JavaScript Promise API
- alternative to `asyncSpawn` that allows handling of async calls in a synchronous context (including access to the synchronous closure) with less additional procs to be declared
- Write less code, catch errors that would otherwise defect in asyncspawn, and execute a callback after completion
- Add cancellation callbacks to utils/then, ensuring cancellations are handled properly
## Dependencies
- bump codex-contracts-eth to support slot queue (https://github.com/codex-storage/codex-contracts-eth/pull/61)
- bump nim-ethers to 0.5.0
- Bump nim-json-rpc submodule to 0bf2bcb
---------
Co-authored-by: Jaremy Creechley <creechley@gmail.com>
* Support for building docker images with local modifications for the purpose of testing and debugging
* exposes peer information via debug/info
* api-names slightly kinder to json serializers
* Moves debug image-building
* fixes misalignment of debug peer info array
* Changes switchPeers source from KeyBook to AddressBook (filed ticket in libp2p, discussed with Tanguy)
* Limited success with dist-test peer discovery tests
* Removes unnecessary random-timer
* bumps dht repo. Adds peerId to formatNode
* Removes unused prints
* bumps libp2p-dht
* Exposes node address on debug api
* Adds traces
* review comments by me
* Hides debug/peers api behind compile flag
* Waiting for nim-libp2p-dht PR merge
* bumps libp2p-dht back to main after PRs were merged there.
* Cleanup
* [contracts] Add SlotFreed event
* [integration] allow test node to be stopped twice
* [cli] add --validator option
* [contracts] remove dead code
* [contracts] instantiate OnChainMarket and OnChainClock only once
* [contracts] add Validation
* [sales] remove duplicate import
* [market] add missing import
* [market] subscribe to all SlotFilled events
* [market] add freeSlot()
* [sales] fix warnings
* [market] subscribe to SlotFreed events
* [contracts] fix warning
* [validator] keep track of filled slots
* [validation] remove slots that have ended
* [proving] absorb Proofs into Market
Both Proofs and Market are abstractions around
the Marketplace contract, having them separately
is more trouble than it's worth at the moment.
* [market] add markProofAsMissing()
* [clock] speed up waiting for clock in tests
* [validator] mark proofs as missing
* [timer] fix error on node shutdown
* [cli] handle --persistence and --validator separately
* [market] allow retrieval of proof timeout value
* [validator] do not subscribe to SlotFreed events
Freed slots are already handled in
removeSlotsThatHaveEnded(), and onSlotsFreed()
interfered with its iterator.
* [validator] Start validation at the start of a new period
To decrease the likelihood that we hit the validation timeout.
* [validator] do not mark proofs as missing after timeout
* [market] check whether proof can be marked as missing
* [validator] simplify validation
Simulate a transaction to mark proof as missing, instead
of trying to keep track of all the conditions that may
lead to a proof being marked as missing.
* [build] use nim-ethers PR #40
Uses "pending" blocktag instead of "latest" blocktag
for better simulation of transactions before sending
them.
https://github.com/status-im/nim-ethers/pull/40
* [integration] integration test for validator
* [validator] monitor a maximum number of slots
Adds cli parameter --validator-max-slots.
* [market] fix missing collateral argument
After rebasing, add the new argument to fillSlot calls.
* [build] update to nim-ethers 0.2.5
* [validator] use Set instead of Table to keep track of slots
* [validator] add logging
* [validator] add test for slot failure
* [market] use "pending" blocktag to use more up to date block time
* [contracts] remove unused import
* [validator] fix: wait until after period ends
The smart contract checks that 'end < block.timestamp',
so we need to wait until the block timestamp is greater
than the period end.
Slava
Giuliano Mega
Tomasz Bekas
Ben Bierens
Adam Uhlíř
Eric
markspanbroek
Dmitriy Ryajov
Jaremy Creechley