* Add availability enabled parameter
* Return bytes to availability when finished
* Add until parameter
* Remove debug message
* Clean up and fix tests
* Update documentations and cleanup
* Avoid swallowing CancelledError
* Move until validation to reservations module
* Call onAvailabilityAdded callabck when the availability is enabled in sales
* Remove until validation in restapi when creating an availability
* Add openapi documentation
* Use results instead of stew/results (#1112)
* feat: request duration limit (#1057)
* feat: request duration limit
* Fix tests and duration type
* Add custom error
* Remove merge issue
* Update codex contracts eth
* Update market config and fix test
* Fix SlotReservationsConfig syntax
* Update dependencies
* test: remove doubled test
* chore: update contracts repo
---------
Co-authored-by: Arnaud <arnaud@status.im>
* fix(statemachine): do not raise from state.run (#1115)
* fix(statemachine): do not raise from state.run
* fix rebase
* fix exception handling in SaleProvingSimulated.prove
- re-raise CancelledError
- don't return State on CatchableError
- expect the Proofs_InvalidProof custom error instead of checking a string
* asyncSpawn salesagent.onCancelled
This was swallowing a KeyError in one of the tests (fixed in the previous commit)
* remove error handling states in asyncstatemachine
* revert unneeded changes
* formatting
* PR feedback, logging updates
* chore(integration): simplify block expiration integration test (#1100)
* chore(integration): simplify block expiration integration test
* clean up
* fix after rebase
* perf: contract storage optimizations (#1094)
* perf: contract storage optimizations
* Apply optimization changes
* Apply optimizing parameters sizing
* Update codex-contracts-eth
* bump latest changes in contracts branch
* Change requestDurationLimit to uint64
* fix tests
* fix tests
---------
Co-authored-by: Arnaud <arnaud@status.im>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* bump contracts to master (#1122)
* Add availability enabled parameter
* Return bytes to availability when finished
* Add until parameter
* Clean up and fix tests
* Move until validation to reservations module
* Apply suggestion changes: return the reservation module error
* Apply suggestion changes for until dates
* Apply suggestion changes: reorganize tests
* Fix indent
* Remove test related to timing issue
* Add raises errors to async pragram and remove useless try except
* Update open api documentation
* Fix wording
* Remove the httpClient restart statements
* Use market.getRequestEnd to set validUntil
* Remove returnBytes
* Use clock.now in testing
* Move the api validation file to the right file
---------
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* Use 422 http code when there is a validation error
* Update the open api description
* Fix typo
* Add more tests for total size
* Catch CancelledError because TrackedFuture raise no error
* Split rest api validation test to a new file
* Change the way of testing negative numbers
* Rename client variable and fix test status code
* Try to reduce the number of requests in CI when asserting in tests
* Fix rebase and remove safeEventually
* Add request validations
* Define expiry as required field in storage request params and fix tests
* Fix error messages
* Enable logs to figure out the issue with recurring failing test on macos
* Add custom errors raised by contract
* Remove custom error non existing anymore
* Update asynctest module
* Update timer tests after updating asynctest
* Add missing custom errors
* Separate mock state errors
* Remove the Option in the error setters
* Wrap the contract errors in MarketError
* Remove async raises (needs to address it in another PR)
* Wrap contract errors into specific error types
* Rename SlotNotFreeError to SlotStateMismatchError
* WiP: migrating CodexClient to chronos http client
* fix(api): fixes#1163
* feat: fully working API integration tests
* convert most of the tests in testupdownload
* feat: working updownload tests on async client
* feat: make testsales work with async codexclient
* feat: make testpurchasing work with async codexclient
* feat: make testblockexpiration work with async codexclient
* feat: make marketplacesuite work with async codexclient
* make testproofs work with async codexclient
* chore: refactor client to express higher level in terms of lower level operations
* fix: set correct content-length for erasure-coded datasets
* feat: make testecbug work with async client
* feat: make testvalidator work with async client
* refactor: simplify request aliases, add close operation
* wire back client.close at node shutdown
* refactor: remove unused exception
* fix: use await instead of waitFor on async call sites
* cleanup imports and logs
* add BlockHandle type
* revert deps
* refactor: async error handling and future tracking improvements
- Update async procedures to use explicit raises annotation
- Modify TrackedFutures to handle futures with no raised exceptions
- Replace `asyncSpawn` with explicit future tracking
- Update test suites to use `unittest2`
- Standardize error handling across network and async components
- Remove deprecated error handling patterns
This commit introduces a more robust approach to async error handling and future management, improving type safety and reducing potential runtime errors.
* bump nim-serde
* remove asyncSpawn
* rework background downloads and prefetch
* imporove logging
* refactor: enhance async procedures with error handling and raise annotations
* misc cleanup
* misc
* refactor: implement allFinishedFailed to aggregate future results with success and failure tracking
* refactor: update error handling in reader procedures to raise ChunkerError and CancelledError
* refactor: improve error handling in wantListHandler and accountHandler procedures
* refactor: simplify LPStreamReadError creation by consolidating parameters
* refactor: enhance error handling in AsyncStreamWrapper to catch unexpected errors
* refactor: enhance error handling in advertiser and discovery loops to improve resilience
* misc
* refactor: improve code structure and readability
* remove cancellation from addSlotToQueue
* refactor: add assertion for unexpected errors in local store checks
* refactor: prevent tracking of finished futures and improve test assertions
* refactor: improve error handling in local store checks
* remove usage of msgDetail
* feat: add initial implementation of discovery engine and related components
* refactor: improve task scheduling logic by removing unnecessary break statement
* break after scheduling a task
* make taskHandler cancelable
* refactor: update async handlers to raise CancelledError
* refactor(advertiser): streamline error handling and improve task flow in advertise loops
* fix: correct spelling of "divisible" in error messages and comments
* refactor(discovery): simplify discovery task loop and improve error handling
* refactor(engine): filter peers before processing in cancelBlocks procedure
* Call onAvailabilityAdded when freeSize, duration or minPricePerBytePerSecond is increased
* Rename onAvailabilityAdded to onAvailabilitySaved
* Rename OnAvailabilitySaved to OnAvailabilityUpserted
* Go back to OnAvailabilitySaved
* Add headers to support on progress on download
* Replace http session by http client in downloadBytes
* Use int instead of int64 for datasetSize
* Rename variable to avoid shallowing client
* Indicate that slot is being repaired when trying to download
* Fix tests
* Apply nph
* Calculate the repair collateral when adding the item into the queue
* Add slotCollateral calculation with getRequest cache and remove populationItem function
* Update with pricePerByte
* Simplify StorageAsk parameter
* Minor fixes
* Move cache request to another PR
* Rename SlotQueueItem collateral and required in init
* Use override func to optimise calls when the slot state is known
* Remove unused code
* Cosmetic change
* Use raiseMarketError helper
* Add exceptions to async pragma
* Cosmetic change
* Use raiseMarketError helper
* Let slotCollateral determines the slot sate
* Use configSync to avoid async pragma in onStorageRequested
* Add loadConfig function
* Add CatchableError to async pragma
* Add missing pragma raises errors
* Move loadConfig
* Avoid swallow CancelledError
* Avoid swallowing CancelledError
* Avoid swallowing CancelledError
* Update error messages
* Except MarketError instead of CatchableError
* Fix merge issue
* Log fatal when configuration cannot be loaded
* Propagate MarketError in slotCollateral
* Remove useless configSync
* Use result with explicit error
* Fix syntax
---------
Signed-off-by: Arnaud <arnaud@status.im>
* adding basic retry functionality
* avoid duplicate requests and batch them
* fix cancelling blocks
* properly resolve blocks
* minor cleanup - use `self`
* avoid useless asyncSpawn
* track retries
* limit max inflight and set libp2p maxIncomingStreams
* cleanup
* add basic yield in readLoop
* use tuple instead of object
* cleanup imports and logs
* increase defaults
* wip
* fix prefetch batching
* cleanup
* decrease timeouts to speedup tests
* remove outdated test
* add retry tests
* should track retries
* remove useless test
* use correct block address (index was off by 1)
* remove duplicate noop proc
* add BlockHandle type
* Use BlockHandle type
* add fetchLocal to control batching from local store
* add format target
* revert deps
* adjust quotaMaxBytes
* cleanup imports and logs
* revert deps
* cleanup blocks on cancelled
* terminate erasure and prefetch jobs on stream end
* split storing and retrieving data into separate tests
* track `b.discoveryLoop` future
* misc
* remove useless check
* fix(statemachine): do not raise from state.run
* fix rebase
* fix exception handling in SaleProvingSimulated.prove
- re-raise CancelledError
- don't return State on CatchableError
- expect the Proofs_InvalidProof custom error instead of checking a string
* asyncSpawn salesagent.onCancelled
This was swallowing a KeyError in one of the tests (fixed in the previous commit)
* remove error handling states in asyncstatemachine
* revert unneeded changes
* formatting
* PR feedback, logging updates
* chore(deps): bump ethers to propagate cancellations
Ethers was swallowing canellations and turning them into EthersErrors, which was causing the sales statemachine to error when it should have been simply cancelling the current state's run. Hopefully fixes the intermittently failing marketplace integration test.
* Add missing errors in async raises pragma
* bump to version of ethers that supports cancellations
---------
Co-authored-by: Arnaud <arnaud@status.im>
* feat(codex-node): add dataset deletion API to Codex node
* feat(api): add deletion of local datasets to API
* fix: logging, remove garbage, drop some CORS headers from DELETE request
* fix: change empty response return code to 204 instead of 200
* fix: add time-based idling to avoid locking up the node during deletes, fix API status code
* fix: uncomment commented tests committed by accident
* fix: return correct code when missing CID is a Manifest CID; add back CORS headers
* fix: remove lingering echo
* fix(build): compilation on macos when including nim-nat-traversal
- removes the `VERSION` rename to `VERSION_temp` in the Makefile
- instead, relies on `-iqoute` to include the `nim-nat-traversal/vendor/libnatpmp-upstream` directory in the search paths. `-iquote` will match the `vendor/libnatpmp-upstream/VERSION` file for `#include "version"` and not `#include <version>`, the latter being what is included by the macos sdk and was causing issues with `-I`. The [gcc 14.2 docs](https://gcc.gnu.org/onlinedocs/gcc-14.2.0/cpp/Invocation.html#index-I) describe how `-iquote` alleviates this issue:
> Directories specified with -iquote apply only to the quote form of the directive, #include "file". Directories specified with -I, -isystem, or -idirafter apply to lookup for both the #include "file" and #include <file> directives.
For more info, please see https://github.com/status-im/nim-nat-traversal/pull/34.
* bump nim-nat-traversal
Now that https://github.com/status-im/nim-nat-traversal/pull/34 has been merged, change back to master commit
* add ccache and sccache to speed up CI
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
* include testname and nim version in cache separation
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
* Make sure ccache has precedence over custom clang/llvm
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
* enable ccache for windows
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
* ccache: evict old files
Make sure old unused cache files are not lingering around for long
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
---------
Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
* fixes datasetSize and slotSize helpers (and also RandomChunker.example)
* adds overload for <<upload>> for seq[byte]
* changes RandomChunker.example to return seq[byte]
* fixes restapi tests after correcting RandomChunker.example
* review: use string.fromBytes from nim-stew to convert seq[byte] to string
* openAPI: StorageRequestCreation: reward => pricePerByte, collateral => collateralPerByte
* purchasing: reward => pricePerByte, collateral => collateralPerByte
* Updates availabilities and reservations to use totalCollateral, minPricePerByte, and maxCollateralPerByte
* Uses correct div operator when operating on UInt256
* proposal updating totalCollateral in availability
* makes sure that reading currentCollateral happens before freeing slot
* Updates naming
* fixes tests: unit and contracts
* uses feat/price-per-byte branch for codex-contracts-eth
* temporarily disables integration tests on CI
* introduces high level <<totalCollateral>> property for a cleaner external interface
* updates integration tests
* Applies review comments
* Updates description of totalCollateral in SalesAvailability
* updates codex-contracts-eth (price-per-byte)
