* fix: createReservation lock (#825)
* fix: createReservation lock
* fix: additional locking places
* fix: acquire lock
* chore: feedback
Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
* feat: withLock template and fixed tests
* fix: use proc for MockReservations constructor
* chore: feedback
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
* chore: feedback implementation
---------
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: markspanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* Block deletion with ref count & repostore refactor (#631)
* Fix StoreStream so it doesn't return parity bytes (#838)
* fix storestream so it doesn\'t return parity bits for protected/verifiable manifests
* use Cid.example instead of creating a mock manually
* Fix verifiable manifest initialization (#839)
* fix verifiable manifest initialization
* fix linearstrategy, use verifiableStrategy to select blocks for slots
* check for both strategies in attribute inheritance test
* ci: add verify_circuit=true to the releases (#840)
* provisional fix so EC errors do not crash the node on download (#841)
* prevent node crashing with `not val.isNil` (#843)
* bump nim-leopard to handle no parity data (#845)
* Fix verifiable manifest constructor (#844)
* Fix verifiable manifest constructor
* Add integration test for verifiable manifest download
Add integration test for testing download of verifiable dataset after creating request for storage
* add missing import
* add testecbug to integration suite
* Remove hardhat instance from integration test
* change description, drop echo
---------
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: gmega <giuliano.mega@gmail.com>
* Bump Nim to 1.6.21 (#851)
* bump Nim to 1.6.21 (range type reset fixes)
* remove incompatible versions from compiler matrix
* feat(rest): adds erasure coding constraints when requesting storage (#848)
* Rest API: add erasure coding constraints when requesting storage
* clean up
* Make error message for "dataset too small" more informative.
* fix API integration test
---------
Co-authored-by: gmega <giuliano.mega@gmail.com>
* Prover workshop band-aid (#853)
* add prover bandaid
* Improve error message text
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
---------
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* Bandaid for failing erasure coding (#855)
* Update Release workflow (#858)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Fixes prover behavior with singleton proof trees (#859)
* add logs and test
* add Merkle proof checks
* factor out Circom input normalization, fix proof input serialization
* add test and update existing ones
* update circuit assets
* add back trace message
* switch contracts to fix branch
* update codex-contracts-eth to latest
* do not expose prove with prenormalized inputs
* Chronos v4 Update (v3 Compat Mode) (#814)
* add changes to use chronos v4 in compat mode
* switch chronos to compat fix branch
* use nimbus-build-system with configurable Nim repo
* add missing imports
* add missing await
* bump compat
* pin nim version in Makefile
* add await instead of asyncSpawn to advertisement queue loop
* bump DHT to v0.5.0
* allow error state of `onBatch` to propagate upwards in test code
* pin Nim compiler commit to avoid fetching stale branch
* make CI build against branch head instead of merge
* fix handling of return values in testslotqueue
* Downgrade to gcc 13 on Windows (#874)
* Downgrade to gcc 13 on Windows
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Increase build job timeout to 90 minutes
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
---------
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Add MIT/Apache licenses (#861)
* Add MIT/Apache licenses
* Center "Apache License"
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
* remove wrong legal entity; rename apache license file
---------
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
* Add OPTIONS endpoint to allow the content-type header for the upload endpoint (#869)
* Add OPTIONS endpoint to allow the content-type header
exec git commit --amend --no-edit -S
* Remove useless header "Access-Control-Headers" and add cache
Signed-off-by: Arnaud <arnaud@status.im>
---------
Signed-off-by: Arnaud <arnaud@status.im>
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
* chore: add `downtimeProduct` config parameter (#867)
* chore: add `downtimeProduct` config parameter
* bump codex-contracts-eth to master
* Support CORS preflight requests when the storage request api returns an error (#878)
* Add CORS headers when the REST API is returning an error
* Use the allowedOrigin instead of the wilcard when setting the origin
Signed-off-by: Arnaud <arnaud@status.im>
---------
Signed-off-by: Arnaud <arnaud@status.im>
* refactor(marketplace): generic querying of historical marketplace events (#872)
* refactor(marketplace): move marketplace events to the Market abstraction
Move marketplace contract events to the Market abstraction so the types can be shared across all modules that call the Market abstraction.
* Remove unneeded conversion
* Switch to generic implementation of event querying
* change parent type to MarketplaceEvent
* Remove extra license file (#876)
* remove extra license
* center "apache license"
* Update advertising (#862)
* Setting up advertiser
* Wires up advertiser
* cleanup
* test compiles
* tests pass
* setting up test for advertiser
* Finishes advertiser tests
* fixes commonstore tests
* Review comments by Giuliano
* Race condition found by Giuliano
* Review comment by Dmitriy
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
* fixes tests
---------
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* feat: add `--payout-address` (#870)
* feat: add `--payout-address`
Allows SPs to be paid out to a separate address, keeping their profits secure.
Supports https://github.com/codex-storage/codex-contracts-eth/pull/144 in the nim-codex client.
* Remove optional payoutAddress
Change --payout-address so that it is no longer optional. There is no longer an overload in `Marketplace.sol` for `fillSlot` accepting no `payoutAddress`.
* Update integration tests to include --payout-address
* move payoutAddress from fillSlot to freeSlot
* Update integration tests to use required payoutAddress
- to make payoutAddress required, the integration tests needed to avoid building the cli params until just before starting the node, otherwise if cli params were added ad-hoc, there would be an error after a non-required parameter was added before a required parameter.
* support client payout address
- withdrawFunds requires a withdrawAddress parameter, directs payouts for withdrawing of client funds (for a cancelled request) to go to that address.
* fix integration test
adds --payout-address to validators
* refactor: support withdrawFunds and freeSlot optional parameters
- withdrawFunds has an optional parameter for withdrawRecipient
- freeSlot has optional parameters for rewardRecipient and collateralRecipient
- change --payout-address to --reward-recipient to match contract signature naming
* Revert "Update integration tests to include --payout-address"
This reverts commit 8f9535cf35.
There are some valid improvements to the integration tests, but they can be handled in a separate PR.
* small fix
* bump contracts to fix marketplace spec
* bump codex-contracts-eth, now rebased on master
* bump codex-contracts-eth
now that feat/reward-address has been merged to master
* clean up, comments
* Rework circuit downloader (#882)
* Introduces a start method to prover
* Moves backend creation into start method
* sets up three paths for backend initialization
* Extracts backend initialization to backend-factory
* Implements loading backend from cli files or previously downloaded local files
* Wires up downloading and unzipping
* functional implementation
* Fixes testprover.nim
* Sets up tests for backendfactory
* includes libzip-dev
* pulls in updated contracts
* removes integration cli tests for r1cs, wasm, and zkey file arguments.
* Fixes issue where inner-scope values are lost before returning
* sets local proof verification for dist-test images
* Adds two traces and bumps nim-ethers
* Adds separate path for circuit files
* Create circuit dir if not exists
* fix: make sure requestStorage is mined
* fix: correct place to plug confirm
* test: fixing contracts tests
* Restores gitmodules
* restores nim-datastore reference
* Sets up downloader exe
* sets up tool skeleton
* implements getting of circuit hash
* Implements downloader tool
* sets up test skeleton
* Implements test for cirdl
* includes testTools in testAll
* Cleanup building.md
* cleans up previous downloader implementation
* cleans up testbackendfactory
* moves start of prover into node.nim
* Fills in arguments in example command
* Initializes backend in prover constructor
* Restores tests
* Restores tests for cli instructions
* Review comments by Dmitriy, part 1
* Quotes path in download instruction.
* replaces curl with chronos http session
* Moves cirdl build output to 'build' folder.
* Fixes chronicles log output
* Add cirdl support to the codex Dockerfile
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Add cirdl support to the docker entrypoint
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Add cirdl support to the release workflow
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Disable verify_circuit flag for releases
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Removes backendFactory placeholder type
* wip
* Replaces zip library with status-im/zippy library (which supports zip and tar)
* Updates cirdl to not change circuitdir folder
* Switches from zip to tar.gz
* Review comments by Dmitriy
* updates codex-contracts-eth
* Adds testTools to CI
* Adds check for access to config.circuitdir
* Update fixture circuit zkey
* Update matrix to run tools tests on Windows
* Adds 'deps' dependency for cirdl
* Adjust docker-entrypoint.sh to use CODEX_CIRCUIT_DIR env var
* Review comments by Giuliano
---------
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Veaceslav Doina <20563034+veaceslavdoina@users.noreply.github.com>
* Support CORS for POST and PATCH availability endpoints (#897)
* Adds testnet marketplace address to known deployments (#911)
* API tweaks for OpenAPI, errors and endpoints (#886)
* All sort of tweaks
* docs: availability's minPrice doc
* Revert changes to the two node test example
* Change default EC params in REST API
Change default EC params in REST API to 3 nodes and 1 tolerance.
Adjust integration tests to honour these settings.
---------
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
---------
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
Signed-off-by: Giuliano Mega <giuliano.mega@gmail.com>
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
Signed-off-by: Arnaud <arnaud@status.im>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: markspanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: Tomasz Bekas <tomasz.bekas@gmail.com>
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
Co-authored-by: Arnaud <arno.deville@gmail.com>
Co-authored-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Arnaud <arnaud@status.im>
* fix: createReservation lock (#825)
* fix: createReservation lock
* fix: additional locking places
* fix: acquire lock
* chore: feedback
Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
* feat: withLock template and fixed tests
* fix: use proc for MockReservations constructor
* chore: feedback
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
* chore: feedback implementation
---------
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: markspanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* Block deletion with ref count & repostore refactor (#631)
* Fix StoreStream so it doesn't return parity bytes (#838)
* fix storestream so it doesn\'t return parity bits for protected/verifiable manifests
* use Cid.example instead of creating a mock manually
* Fix verifiable manifest initialization (#839)
* fix verifiable manifest initialization
* fix linearstrategy, use verifiableStrategy to select blocks for slots
* check for both strategies in attribute inheritance test
* ci: add verify_circuit=true to the releases (#840)
* provisional fix so EC errors do not crash the node on download (#841)
---------
Signed-off-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: markspanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: Tomasz Bekas <tomasz.bekas@gmail.com>
Co-authored-by: Giuliano Mega <giuliano.mega@gmail.com>
* pulls in datastore-leveldb update
* bump
* Applies LevelDb as metadata store. Adds option for repostore.
* Sets submodule to main branch
* I can do syntax, me
* Removes wildcard from metadata query key
* Applies leveldb instead of sqlite-in-memory for tests
* Restores query key wildcard.
* Pins nim-datastore to latest master
* bumps leveldb to 0.1.4
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* add seen flag
* Add MockSlotQueueItem and better prioritisation tests
* Update seen priority, and include in SlotQueueItem.init
* Re-add processed slots to queue
Re-add processed slots to queue if the sale was ignored or errored
* add pausing of queue
- when processing slots in queue, pause queue if item was marked seen
- if availability size is increased, trigger onAvailabilityAdded callback
- in sales, on availability added, clear 'seen' flags, then unpause the queue
- when items pushed to the queue, unpause the queue
* remove unused NoMatchingAvailabilityError from slotqueue
The slot queue should also have nothing to do with availabilities
* when all availabilities are empty, pause the queue
An empty availability is defined as size < DefaultBlockSize as this means even the smallest possible request could not be served. However, this is up for discussion.
* remove availability from onAvailabilitiesEmptied callback
* refactor onAvailabilityAdded and onAvailabilitiesEmptied
onAvailabilityAdded and onAvailabilitiesEmptied are now only called from reservations.update (and eventually reservations.delete once implemented).
- Add empty routine for Availability and Reservation
- Add allEmpty routine for Availability and Reservation, which returns true when all all Availability or Reservation objects in the datastore are empty.
* SlotQueue test support updates
* Sales module test support updates
* Reservations module tests for queue pausing
* Sales module tests for queue pausing
Includes tests for sales states cancelled, errored, ignored to ensure onCleanUp is called with correct parameters
* SlotQueue module tests for queue pausing
* fix existing sales test
* PR feedback
- indent `self.unpause`
- update comment for `clearSeenFlags`
* reprocessSlot in SaleErrored only when coming from downloading
* remove pausing of queue when availabilities are "emptied"
Queue pausing when all availiabilies are "emptied" is not necessary, given that the node would not be able to service slots once all its availabilities' freeSize are too small for the slots in the queue, and would then be paused anyway.
Add test that asserts the queue is paused once the freeSpace of availabilities drops too low to fill slots in the queue.
* Update clearing of seen flags
The asyncheapqueue update overload would need to check index bounds and ultimately a different solution was found using the mitems iterator.
* fix test
request.id was different before updating request.ask.slots, and that id was used to set the state in mockmarket.
* Change filled/cleanup future to nil, so no await is needed
* add wait to allow items to be added to queue
* do not unpause queue when seen items are pushed
* re-add seen item back to queue once paused
Previously, when a seen item was processed, it was first popped off the queue, then the queue was paused waiting to process that item once the queue was unpaused. Now, when a seen item is processed, it is popped off the queue, the queue is paused, then the item is re-added to the queue and the queue will wait until unpaused before it will continue popping items off the queue. If the item was not re-added to the queue, it would have been processed immediately once unpaused, however there may have been other items with higher priority pushed to the queue in the meantime. The queue would not be unpaused if those added items were already seen. In particular, this may happen when ignored items due to lack of availability are re-added to a paused queue. Those ignored items will likely have a higher priority than the item that was just seen (due to it having been processed first), causing the queue to the be paused.
* address PR comments
* json > nim-serde bump
Should wait until serde is integrated into nim-ethers before making these changes as there will be less import exceptions required.
* bump nim-serde
* change func to proc due to chronicles side effects
* import serde into utils/json, use as proxy
import nim-serde into utils/json and use utils/json as a proxy for serde functions, including overloading `%` and `fromJson` for application types.
* update tests to use serde
* bump serde to latest
* remove testjson -- no longer needed
* bump serde in nimble
* updates to reconcile rebase with master
* wire prover into node
* stricter case object checks
* return correct proof
* misc renames
* adding usefull traces
* fix nodes and tolerance to match expected params
* format challenges in logs
* add circom compat to solidity groth16 convertion
* update
* bump time to give nodes time to load with all circom artifacts
* misc
* misc
* use correct dataset geometry in erasure
* make errors more searchable
* use parens around `=? (await...)` calls
* styling
* styling
* use push raises
* fix to match constructor arguments
* merge master
* merge master
* integration: fix proof parameters for a test
Increased times due to ZK proof generation.
Increased storage requirement because we're now hosting
5 slots instead of 1.
* sales: calculate initial proof at start of period
reason: this ensures that the period (and therefore
the challenge) doesn't change while we're calculating
the proof
* integration: fix proof parameters for tests
Increased times due to waiting on next period.
Fixed data to be of right size.
Updated expected payout due to hosting 5 slots.
* sales: wait for stable proof challenge
When the block pointer is nearing the
wrap-around point, we wait another period
before calculating a proof.
* fix merge conflict
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* clock: add 1 second leeway before acting on timeouts
* sales: do not raise in proving loop when slot is cancelled
Allow the onCancelled callback to handle cancellation, and
the onFailed callback to handle failed requests.
* sales: cleanup proving tests
* sales: fix sales agent tests
* sales: stop cancellation loop when request started, finished or failed
* sales: fix flaky test
* sales: fix another flaky test
* clock: add comment explaining the + 1 second
Co-Authored-By: benbierens <thatbenbierens@gmail.com>
---------
Co-authored-by: benbierens <thatbenbierens@gmail.com>
* Smart contracts update: Groth16Proof instead of bytes
* Use dummy verifier for now, until we can create ZK proofs
* Fix tests: submit proof only when slot is filled
* Submit dummy proofs for now
* More detailed log when proof submission failed
* Use dummy verifier for integration tests
For now at least
* Fix mistake in blanket renaming to ethProvider
* Update to latest codex-contracts-eth
* feat: zkey-hash from chain
* Fix zkeyHash
---------
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
* implement a logging proxy
The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks
* Rename logging -> logutils to avoid ambiguity with common names
* clean up
* add setProperty for JsonRecord, remove nim-json-serialization conflict
* Allow specifying textlines and json format separately
Not specifying a LogFormat will apply the formatting to both textlines and json sinks.
Specifying a LogFormat will apply the formatting to only that sink.
* remove unneeded usages of std/json
We only need to import utils/json instead of std/json
* move serialization from rest/json to utils/json so it can be shared
* fix NoColors ambiguity
Was causing unit tests to fail on Windows.
* Remove nre usage to fix Windows error
Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)
* Add logutils module doc
* Shorten logutils.formatIt for `NBytes`
Both json and textlines formatIt were not needed, and could be combined into one formatIt
* remove debug integration test config
debug output and logformat of json for integration test logs
* Use ## module doc to support docgen
* bump nim-poseidon2 to export fromBytes
Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening.
* fixes to get compiling after rebasing master
* Add support for Result types being logged using formatIt
* Setting up testfixture for proof datasampler
* Sets up calculating number of cells in a slot
* Sets up tests for bitwise modulo
* Implements cell index collection
* setting up slot blocks module
* Implements getting treeCID from slot
* implements getting slot blocks by index
* Implements out-of-range check for slot index
* cleanup
* Sets up getting sample from block
* Implements selecting a cell sample from a block
* Implements building a minitree for block cells
* Adds method to get dataset block index from slot block index
* It's running
* splits up indexing
* almost there
* Fixes test. Implementation is now functional
* Refactoring to object-oriented
* Cleanup
* Lining up output type with updated reference code.
* setting up
* Updates expected samples
* Updates proof checking test to match new format
* move builder to own dir
* move sampler to own dir
* fix paths
* various changes to add support for the sampler
* wip sampler implementation
* don't use upraises
* wip sampler integration
* misc
* move tests around
* Various fixes to select correct slot and block index
* removing old tests
* cleanup
* misc
fix tests that work with correct cell indices
* remove unused file
* fixup logging
* add logscope
* truncate entropy to 31 bytes, otherwise it might be > than mod
* forwar getCidAndProof to local store
* misc
* Adds missing test for initial-proving state
* reverting back to correct slot/block indexing
* fix tests for revert
* misc
* misc
---------
Co-authored-by: benbierens <thatbenbierens@gmail.com>
* implement a logging proxy
The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks
* Rename logging -> logutils to avoid ambiguity with common names
* clean up
* add setProperty for JsonRecord, remove nim-json-serialization conflict
* Allow specifying textlines and json format separately
Not specifying a LogFormat will apply the formatting to both textlines and json sinks.
Specifying a LogFormat will apply the formatting to only that sink.
* remove unneeded usages of std/json
We only need to import utils/json instead of std/json
* move serialization from rest/json to utils/json so it can be shared
* fix NoColors ambiguity
Was causing unit tests to fail on Windows.
* Remove nre usage to fix Windows error
Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)
* Add logutils module doc
* Shorten logutils.formatIt for `NBytes`
Both json and textlines formatIt were not needed, and could be combined into one formatIt
* remove debug integration test config
debug output and logformat of json for integration test logs
* Use ## module doc to support docgen
## Problem
When Availabilities are created, the amount of bytes in the Availability are reserved in the repo, so those bytes on disk cannot be written to otherwise. When a request for storage is received by a node, if a previously created Availability is matched, an attempt will be made to fill a slot in the request (more accurately, the request's slots are added to the SlotQueue, and eventually those slots will be processed). During download, bytes that were reserved for the Availability were released (as they were written to disk). To prevent more bytes from being released than were reserved in the Availability, the Availability was marked as used during the download, so that no other requests would match the Availability, and therefore no new downloads (and byte releases) would begin. The unfortunate downside to this, is that the number of Availabilities a node has determines the download concurrency capacity. If, for example, a node creates a single Availability that covers all available disk space the operator is willing to use, that single Availability would mean that only one download could occur at a time, meaning the node could potentially miss out on storage opportunities.
## Solution
To alleviate the concurrency issue, each time a slot is processed, a Reservation is created, which takes size (aka reserved bytes) away from the Availability and stores them in the Reservation object. This can be done as many times as needed as long as there are enough bytes remaining in the Availability. Therefore, concurrent downloads are no longer limited by the number of Availabilities. Instead, they would more likely be limited to the SlotQueue's `maxWorkers`.
From a database design perspective, an Availability has zero or more Reservations.
Reservations are persisted in the RepoStore's metadata, along with Availabilities. The metadata store key path for Reservations is ` meta / sales / reservations / <availabilityId> / <reservationId>`, while Availabilities are stored one level up, eg `meta / sales / reservations / <availabilityId> `, allowing all Reservations for an Availability to be queried (this is not currently needed, but may be useful when work to restore Availability size is implemented, more on this later).
### Lifecycle
When a reservation is created, its size is deducted from the Availability, and when a reservation is deleted, any remaining size (bytes not written to disk) is returned to the Availability. If the request finishes, is cancelled (expired), or an error occurs, the Reservation is deleted (and any undownloaded bytes returned to the Availability). In addition, when the Sales module starts, any Reservations that are not actively being used in a filled slot, are deleted.
Having a Reservation persisted until after a storage request is completed, will allow for the originally set Availability size to be reclaimed once a request contract has been completed. This is a feature that is yet to be implemented, however the work in this PR is a step in the direction towards enabling this.
### Unknowns
Reservation size is determined by the `StorageAsk.slotSize`. If during download, more bytes than `slotSize` are attempted to be downloaded than this, then the Reservation update will fail, and the state machine will move to a `SaleErrored` state, deleting the Reservation. This will likely prevent the slot from being filled.
### Notes
Based on #514
* [sales] remove availability check before adding to slot queue
* [sales] add missing return statement
* [tests] remove 'eventuallyCheck' helper
* [sales] remove reservations from slot queue
* [tests] rename module `eventually` -> `always`
* [sales] increase slot queue size
Because it will now also hold items for which we haven't
checked availability yet.
* extra utilities and tweaks
* add atlas lock
* update ignores
* break build into it's own script
* update url rules
* base off codexdht's
* compile fixes for Nim 1.6.14
* update submodules
* convert mapFailure to procs to work around type resolution issues
* add toml parser for multiaddress
* change error type on keyutils
* bump nimbus build to use 1.6.14
* update gitignore
* adding new deps submodules
* bump nim ci version
* even more fixes
* more libp2p changes
* update keys
* fix eventually function
* adding coverage test file
* move coverage to build.nims
* use nimcache/coverage
* move libp2p import for tests into helper.nim
* remove named bin
* bug fixes for networkpeers (from Dmitriy)
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* Simplify `.then` (promise api) and tests
* Remove tracked future when cancelled. Add tracked future tests
* Track and cancel statemachine futures
The futures created in each asyncstatemachine instance are tracked, and each future is cancelled and waited in `stop`.
Change `asyncstatemachine.stop` to be async so `machine.trackedFutures.cancelAndWait` could be called.
Add a constructor for `asyncstatemachine` that initialises the `trackedFutures` instance, and call the constructor from derived class constructors.
## Slot queue
Adds a slot queue, as per the [slot queue design](https://github.com/codex-storage/codex-research/blob/master/design/sales.md#slot-queue).
Any time storage is requested, all slots from that request are immediately added to the queue. Finished, Canclled, Failed requests remove all slots with that request id from the queue. SlotFreed events add a new slot to the queue and SlotFilled events remove the slot from the queue. This allows popping of a slot each time one is processed, making things much simpler.
When an entire request of slots is added to the queue, the slot indices are shuffled randomly to hopefully prevent nodes that pick up the same storage requested event from clashing on the first processed slot index. This allowed removal of assigning a random slot index in the SalePreparing state and it also ensured that all SalesAgents will have a slot index assigned to them at the start thus the removal of the optional slotIndex.
Remove slotId from SlotFreed event as it was not being used. RequestId and slotIndex were added to the SlotFreed event earlier and those are now being used
The slot queue invariant that prioritises queue items added to the queue relies on a scoring mechanism to sort them based on the [sort order in the design document](https://github.com/codex-storage/codex-research/blob/master/design/sales.md#sort-order).
When a storage request is handled by the sales module, a slot index was randomly assigned and then the slot was filled. Now, a random slot index is only assigned when adding an entire request to the slot queue. Additionally, the slot is checked that its state is `SlotState.Free` before continuing with the download process.
SlotQueue should always ensure the underlying AsyncHeapQueue has one less than the maximum items, ensuring the SlotQueue can always have space to add an additional item regardless if it’s full or not.
Constructing `SlotQueue.workers` in `SlotQueue.new` calls `newAsyncQueue` which causes side effects, so the construction call had to be moved to `SlotQueue.start`.
Prevent loading request from contract (network request) if there is an existing item in queue for that request.
Check availability before adding request to queue.
Add ability to query market contract for past events. When new availabilities are added, the `onReservationAdded` callback is triggered in which past `StorageRequested` events are queried, and those slots are added to the queue (filtered by availability on `push` and filtered by state in `SalePreparing`).
#### Request Workers
Limit the concurrent requests being processed in the queue by using a limited pool of workers (default = 3). Workers are in a data structure of type `AsyncQueue[SlotQueueWorker]`. This allows us to await a `popFirst` for available workers inside of the main SlotQueue event loop
Add an `onCleanUp` that stops the agents and removes them from the sales module agent list. `onCleanUp` is called from sales end states (eg ignored, cancelled, finished, failed, errored).
Add a `doneProcessing` future to `SlotQueueWorker` to be completed in the `OnProcessSlot` callback. Each `doneProcessing` future created is cancelled and awaited in `SlotQueue.stop` (thanks to `TrackableFuturees`), which forced `stop` to become async.
- Cancel dispatched workers and the `onProcessSlot` callbacks, prevents zombie callbacks
#### Add TrackableFutures
Allow tracking of futures in a module so they can be cancelled at a later time. Useful for asyncSpawned futures, but works for any future.
### Sales module
The sales module needed to subscribe to request events to ensure that the request queue was managed correctly on each event. In the process of doing this, the sales agents were updated to avoid subscribing to events in each agent, and instead dispatch received events from the sales module to all created sales agents. This would prevent memory leaks on having too many eventemitters subscribed to.
- prevent removal of agents from sales module while stopping, otherwise the agents seq len is modified while iterating
An additional sales agent state was added, `SalePreparing`, that handles all state machine setup, such as retrieving the request and subscribing to events that were previously in the `SaleDownloading` state.
Once agents have parked in an end state (eg ignored, cancelled, finished, failed, errored), they were not getting cleaned up and the sales module was keeping a handle on their reference. An `onCleanUp` callback was created to be called after the state machine enters an end state, which could prevent a memory leak if the number of requests coming in is high.
Move the SalesAgent callback raises pragmas from the Sales module to the proc definition in SalesAgent. This avoids having to catch `Exception`.
- remove unneeded error handling as pragmas were moved
Move sales.subscriptions from an object containing named subscriptions to a `seq[Subscription]` directly on the sales object.
Sales tests: shut down repo after sales stop, to fix SIGABRT in CI
### Add async Promise API
- modelled after JavaScript Promise API
- alternative to `asyncSpawn` that allows handling of async calls in a synchronous context (including access to the synchronous closure) with less additional procs to be declared
- Write less code, catch errors that would otherwise defect in asyncspawn, and execute a callback after completion
- Add cancellation callbacks to utils/then, ensuring cancellations are handled properly
## Dependencies
- bump codex-contracts-eth to support slot queue (https://github.com/codex-storage/codex-contracts-eth/pull/61)
- bump nim-ethers to 0.5.0
- Bump nim-json-rpc submodule to 0bf2bcb
---------
Co-authored-by: Jaremy Creechley <creechley@gmail.com>
* [contracts] Add SlotFreed event
* [integration] allow test node to be stopped twice
* [cli] add --validator option
* [contracts] remove dead code
* [contracts] instantiate OnChainMarket and OnChainClock only once
* [contracts] add Validation
* [sales] remove duplicate import
* [market] add missing import
* [market] subscribe to all SlotFilled events
* [market] add freeSlot()
* [sales] fix warnings
* [market] subscribe to SlotFreed events
* [contracts] fix warning
* [validator] keep track of filled slots
* [validation] remove slots that have ended
* [proving] absorb Proofs into Market
Both Proofs and Market are abstractions around
the Marketplace contract, having them separately
is more trouble than it's worth at the moment.
* [market] add markProofAsMissing()
* [clock] speed up waiting for clock in tests
* [validator] mark proofs as missing
* [timer] fix error on node shutdown
* [cli] handle --persistence and --validator separately
* [market] allow retrieval of proof timeout value
* [validator] do not subscribe to SlotFreed events
Freed slots are already handled in
removeSlotsThatHaveEnded(), and onSlotsFreed()
interfered with its iterator.
* [validator] Start validation at the start of a new period
To decrease the likelihood that we hit the validation timeout.
* [validator] do not mark proofs as missing after timeout
* [market] check whether proof can be marked as missing
* [validator] simplify validation
Simulate a transaction to mark proof as missing, instead
of trying to keep track of all the conditions that may
lead to a proof being marked as missing.
* [build] use nim-ethers PR #40
Uses "pending" blocktag instead of "latest" blocktag
for better simulation of transactions before sending
them.
https://github.com/status-im/nim-ethers/pull/40
* [integration] integration test for validator
* [validator] monitor a maximum number of slots
Adds cli parameter --validator-max-slots.
* [market] fix missing collateral argument
After rebasing, add the new argument to fillSlot calls.
* [build] update to nim-ethers 0.2.5
* [validator] use Set instead of Table to keep track of slots
* [validator] add logging
* [validator] add test for slot failure
* [market] use "pending" blocktag to use more up to date block time
* [contracts] remove unused import
* [validator] fix: wait until after period ends
The smart contract checks that 'end < block.timestamp',
so we need to wait until the block timestamp is greater
than the period end.
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
Slava
Ben Bierens
Eric
Adam Uhlíř
markspanbroek
Giuliano Mega
Dmitriy Ryajov
Jaremy Creechley
Eric Mastro