* fix(sales): handle cancellation of slot cleanup
Ensures that processing slots from the slot queue
continues even when cleanup of a slot is cancelled.
Co-Authored-By: Eric <5089238+emizzle@users.noreply.github.com>
* chore(reservations): add more `raises` annotations
* Fix cleanup cancellation
* Add remove-agent to trackedfutures instead of the cleanup function
* Increase the timeout to match the request expiry
* Enable logs to debug on CI
* Remove useless except and do not return when add item back to slot queue fails
* Reduce poll interval to detect sale cancelled state
* Avoid cancelling cleanup routine
* Do not cancel creating reservation in order to avoid inconsistent state
* Remove useless try except
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* checked exceptions in stores
* makes asynciter as much exception safe as it gets
* introduce "SafeAsyncIter" that uses Results and limits exceptions to cancellations
* adds {.push raises: [].} to errors
* uses SafeAsyncIter in "listBlocks" and in "getBlockExpirations"
* simplifies safeasynciter (magic of auto)
* gets rid of ugly casts
* tiny fix in hte way we create raising futures in tests of safeasynciter
* Removes two more casts caused by using checked exceptions
* adds an extended explanation of one more complex SafeAsyncIter test
* adds missing "finishOnErr" param in slice constructor of SafeAsyncIter
* better fix for "Error: Exception can raise an unlisted exception: Exception" error.
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* Catch Marketplace_SlotIsFree and continue the cancelled process
* Add log message when the slot if free during failed state
* Reduce log level to debug for slot free error
* Separate slot mock errors
* Initialize variable in setyp
* Improve tests
* Remove non-meaningful checks and rename test
* Remove the Option in the error setters
* Return collateral when the state is cancelled only if the slot is filled by the host
* Do not propagate AsyncLockError
* Wrap contract error into specific error type
* Remove debug message
* Catch only SlotStateMismatchError in cancelled
* Fix error
* Remove returnBytesWas
* Use MarketError after raises pragma were defined
* Fix typo
* Fix lint
* Add availability enabled parameter
* Return bytes to availability when finished
* Add until parameter
* Remove debug message
* Clean up and fix tests
* Update documentations and cleanup
* Avoid swallowing CancelledError
* Move until validation to reservations module
* Call onAvailabilityAdded callabck when the availability is enabled in sales
* Remove until validation in restapi when creating an availability
* Add openapi documentation
* Use results instead of stew/results (#1112)
* feat: request duration limit (#1057)
* feat: request duration limit
* Fix tests and duration type
* Add custom error
* Remove merge issue
* Update codex contracts eth
* Update market config and fix test
* Fix SlotReservationsConfig syntax
* Update dependencies
* test: remove doubled test
* chore: update contracts repo
---------
Co-authored-by: Arnaud <arnaud@status.im>
* fix(statemachine): do not raise from state.run (#1115)
* fix(statemachine): do not raise from state.run
* fix rebase
* fix exception handling in SaleProvingSimulated.prove
- re-raise CancelledError
- don't return State on CatchableError
- expect the Proofs_InvalidProof custom error instead of checking a string
* asyncSpawn salesagent.onCancelled
This was swallowing a KeyError in one of the tests (fixed in the previous commit)
* remove error handling states in asyncstatemachine
* revert unneeded changes
* formatting
* PR feedback, logging updates
* chore(integration): simplify block expiration integration test (#1100)
* chore(integration): simplify block expiration integration test
* clean up
* fix after rebase
* perf: contract storage optimizations (#1094)
* perf: contract storage optimizations
* Apply optimization changes
* Apply optimizing parameters sizing
* Update codex-contracts-eth
* bump latest changes in contracts branch
* Change requestDurationLimit to uint64
* fix tests
* fix tests
---------
Co-authored-by: Arnaud <arnaud@status.im>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* bump contracts to master (#1122)
* Add availability enabled parameter
* Return bytes to availability when finished
* Add until parameter
* Clean up and fix tests
* Move until validation to reservations module
* Apply suggestion changes: return the reservation module error
* Apply suggestion changes for until dates
* Apply suggestion changes: reorganize tests
* Fix indent
* Remove test related to timing issue
* Add raises errors to async pragram and remove useless try except
* Update open api documentation
* Fix wording
* Remove the httpClient restart statements
* Use market.getRequestEnd to set validUntil
* Remove returnBytes
* Use clock.now in testing
* Move the api validation file to the right file
---------
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* Add missing custom errors
* Separate mock state errors
* Remove the Option in the error setters
* Wrap the contract errors in MarketError
* Remove async raises (needs to address it in another PR)
* Wrap contract errors into specific error types
* Rename SlotNotFreeError to SlotStateMismatchError
* cleanup imports and logs
* add BlockHandle type
* revert deps
* refactor: async error handling and future tracking improvements
- Update async procedures to use explicit raises annotation
- Modify TrackedFutures to handle futures with no raised exceptions
- Replace `asyncSpawn` with explicit future tracking
- Update test suites to use `unittest2`
- Standardize error handling across network and async components
- Remove deprecated error handling patterns
This commit introduces a more robust approach to async error handling and future management, improving type safety and reducing potential runtime errors.
* bump nim-serde
* remove asyncSpawn
* rework background downloads and prefetch
* imporove logging
* refactor: enhance async procedures with error handling and raise annotations
* misc cleanup
* misc
* refactor: implement allFinishedFailed to aggregate future results with success and failure tracking
* refactor: update error handling in reader procedures to raise ChunkerError and CancelledError
* refactor: improve error handling in wantListHandler and accountHandler procedures
* refactor: simplify LPStreamReadError creation by consolidating parameters
* refactor: enhance error handling in AsyncStreamWrapper to catch unexpected errors
* refactor: enhance error handling in advertiser and discovery loops to improve resilience
* misc
* refactor: improve code structure and readability
* remove cancellation from addSlotToQueue
* refactor: add assertion for unexpected errors in local store checks
* refactor: prevent tracking of finished futures and improve test assertions
* refactor: improve error handling in local store checks
* remove usage of msgDetail
* feat: add initial implementation of discovery engine and related components
* refactor: improve task scheduling logic by removing unnecessary break statement
* break after scheduling a task
* make taskHandler cancelable
* refactor: update async handlers to raise CancelledError
* refactor(advertiser): streamline error handling and improve task flow in advertise loops
* fix: correct spelling of "divisible" in error messages and comments
* refactor(discovery): simplify discovery task loop and improve error handling
* refactor(engine): filter peers before processing in cancelBlocks procedure
* Indicate that slot is being repaired when trying to download
* Fix tests
* Apply nph
* Calculate the repair collateral when adding the item into the queue
* Add slotCollateral calculation with getRequest cache and remove populationItem function
* Update with pricePerByte
* Simplify StorageAsk parameter
* Minor fixes
* Move cache request to another PR
* Rename SlotQueueItem collateral and required in init
* Use override func to optimise calls when the slot state is known
* Remove unused code
* Cosmetic change
* Use raiseMarketError helper
* Add exceptions to async pragma
* Cosmetic change
* Use raiseMarketError helper
* Let slotCollateral determines the slot sate
* Use configSync to avoid async pragma in onStorageRequested
* Add loadConfig function
* Add CatchableError to async pragma
* Add missing pragma raises errors
* Move loadConfig
* Avoid swallow CancelledError
* Avoid swallowing CancelledError
* Avoid swallowing CancelledError
* Update error messages
* Except MarketError instead of CatchableError
* Fix merge issue
* Log fatal when configuration cannot be loaded
* Propagate MarketError in slotCollateral
* Remove useless configSync
* Use result with explicit error
* Fix syntax
---------
Signed-off-by: Arnaud <arnaud@status.im>
* openAPI: StorageRequestCreation: reward => pricePerByte, collateral => collateralPerByte
* purchasing: reward => pricePerByte, collateral => collateralPerByte
* Updates availabilities and reservations to use totalCollateral, minPricePerByte, and maxCollateralPerByte
* Uses correct div operator when operating on UInt256
* proposal updating totalCollateral in availability
* makes sure that reading currentCollateral happens before freeing slot
* Updates naming
* fixes tests: unit and contracts
* uses feat/price-per-byte branch for codex-contracts-eth
* temporarily disables integration tests on CI
* introduces high level <<totalCollateral>> property for a cleaner external interface
* updates integration tests
* Applies review comments
* Updates description of totalCollateral in SalesAvailability
* updates codex-contracts-eth (price-per-byte)
* Move to version 2.0.6
* Update nim-confutils submodule to latest version
* Update dependencies
* Update Nim version to 2.0.12
* Add gcsafe pragma
* Add missing import
* Update specific conf for Nim 2.x
* Fix method signatures
* Revert erasure coding attempt to fix bug
* More gcsafe pragma
* Duplicate code from libp2p because it is not exported anymore
* Fix camelcase function names
* Use alreadySeen because need is not a bool anymore
* newLPStreamReadError does not exist anymore so use another error
* Replace ValidIpAddress by IpAddress
* Add gcsafe pragma
* Restore maintenance parameter deleted by mistake when removing esasure coding fix attempt code
* Update method signatures
* Copy LPStreamReadError code from libp2p which was removed
* Fix camel case
* Fix enums in tests
* Fix camel case
* Extract node components to a variable to make Nim 2 happy
* Update the tests using ValidIpAddress to IpAddress
* Fix cast for value which is already an option
* Set nim version to 2.0.x for CI
* Set nim version to 2.0.x for CI
* Move to miniupnp version 2.2.4 to avoid symlink error
* Set core.symlinks to false for Windows for miniupnp >= 2.2.5 support
* Update to Nim 2.0.14
* Update CI nim versions to 2.0.14
* Try with GCC 14
* Replace apt-fast by apt-get
* Update ubuntu runner to latest
* Use Ubuntu 20.04 for coverage
* Disable CI cache for coverage
* Add coverage property description
* Remove commented test
* Check the node value of seen instead of using alreadySeen
* Fix the merge. The taskpool work was reverted.
* Update nim-ethers submodule
* Remove deprecated ValidIpAddress. Fix missing case and imports.
* Fix a weird issue where nim-confutils cannot find NatAny
* Fix tests and remove useless static keyword
* adds a new overload of queryPastEvents allowing to query past events based on timestamp in the past
* adds state restoration to validator
* refactors a bit to get the tests back to work
* replaces deprecated generic methods from Market with methods for specific event types
* Refactors binary search
* adds market tests for querying past SlotFilled events and binary search
* Takes into account that <<earliest>> block available is not necessarily the genesis block
* Adds more logging and makes testing earliest block boundary more reliable
* adds validation tests for historical state restoration
* adds mockprovider to simplify and improve testing of the edge conditions
* adds slot reservation to the new tests after rebasing
* adds validation groups and group index in logs of validator
* adds integration test with two validators
* adds comment on how to enable logging in integration test executable itself
* testIntegration: makes list is running nodes injected and available in the body of the test
* validation: adds integration test for historical state
* adds more logging to validator
* integration test: validator only looks 30 days back for historical state
* adds logging of the slotState when removing slots during validation
* review and refactor validator integration tests
* adds validation to the set of integration tests
* Fixes mistyped name of the mock provider module in testMarket
* Fixes a typo in the name of the validation suite in integration tests
* Makes validation unit test a bit easier to follow
* better use of logScopes to reduce duplication
* improves timing and clarifies the test conditions
* uses http as default RPC provider for nodes running in integration tests as a workaround for dropped subscriptions
* simplifies the validation integration tests by waiting for failed request instead of tracking slots
* adds config option allowing selectively to set different provider url
* Brings back the default settings for RPC provider in integration tests
* use http RPC provider for clients in validation integration tests
* fine-tune the tests
* Makes validator integration test more robust - adds extra tracking
* brings tracking of marketplace event back to validator integration test
* refactors integration tests
* deletes tmp file
* adds <<return>> after forcing integration test to fail preliminarily
* re-enables all integration tests and matrix
* stops debug output in CI
* allows to choose a different RPC provider for a given integration test suite
* fixes signature of <<getBlock>> method in mockProvider
* adds missing import which seem to be braking integration tests on windows
* makes sure that clients, SPs, and validators use the same provider url
* makes validator integration tests using http at 127.0.0.1:8545
* testvalidator: stop resubscribing as we are now using http polling as rpc provider
* applying review comments
* groups queryPastStorage overrides together (review comment)
* groups the historical validation tests into a sub suite
* removes the temporary extensions in marketplacesuite and multinodesuite allowing to specify provider url
* simplifies validation integration tests
* Removes debug logs when waiting for request to fail
* Renaming waitForRequestFailed => waitForRequestToFail
* renames blockNumberForBlocksAgo to pastBlockTag and makes it private
* removes redundant debugging logs
* refines logging in validation
* removes dev logging from mockmarket
* improves exception handling in provider helper procs and prepares for extraction to a separate module
* Uses chronos instead of std/times for Duration
* extracts provider and binary search helpers to a separate module
* removes redundant log entry params from validator
* unifies the notation to consistently use method call syntax
* reuses ProviderError from nim-ethers in the provider extension
* clarifies the comment in multinodesuite
* uses == operator to check the predefined tags and raises exception when `BlockTag.pending` is requested.
* when waiting for request to fail, we break on any request state that is not Started
* removes tests that were moved to testProvider from testMarket
* extracts tests that use MockProvider to a separate async suite
* improves performance of the historical state restoration
* removing redundant log messages in validator (groupIndex and groups)
* adds testProvider to testContracts group
* removes unused import in testMarket
* feat: repair is rewarded
* chore: update contracts repo
* feat: proving loop handles repair case
* test: assert repair state
* chore: update contracts repo
* fix: upon unknown state of repair go to error
* convert EthersError to MarketError
* change `canReserveSlot` and `reserveSlot` parameters
Parameters for `canReserveSlot` and `reserveSlot` were changed from `SlotId` to `RequestId` and `UInt256 slotIndex`.
* Add SaleSlotReserving
Adds a new state, SaleSlotReserving, that attempts to reserve a slot before downloading.
If the slot cannot be reserved, the state moves to SaleIgnored.
On error, the state moves to SaleErrored.
SaleIgnored is also updated to pass in `reprocessSlot` and `returnBytes`, controlling the behaviour in the Sales module after the slot is ignored. This is because previously it was assumed that SaleIgnored was only reached when there was no Availability. This is no longer the case, since SaleIgnored can now be reached when a slot cannot be reserved.
* Update SalePreparing
Specify `reprocessSlot` and `returnBytes` when moving to `SaleIgnored` from `SalePreparing`.
Update tests to include test for a raised CatchableError.
* Fix unit test
* Modify `canReserveSlot` and `reverseSlot` params after rebase
* Update MockMarket with new `canReserveSlot` and `reserveSlot` params
* fix after rebase
also bump codex-contracts-eth to master
* refactor(marketplace): move marketplace events to the Market abstraction
Move marketplace contract events to the Market abstraction so the types can be shared across all modules that call the Market abstraction.
* Remove unneeded conversion
* Switch to generic implementation of event querying
* change parent type to MarketplaceEvent
* add seen flag
* Add MockSlotQueueItem and better prioritisation tests
* Update seen priority, and include in SlotQueueItem.init
* Re-add processed slots to queue
Re-add processed slots to queue if the sale was ignored or errored
* add pausing of queue
- when processing slots in queue, pause queue if item was marked seen
- if availability size is increased, trigger onAvailabilityAdded callback
- in sales, on availability added, clear 'seen' flags, then unpause the queue
- when items pushed to the queue, unpause the queue
* remove unused NoMatchingAvailabilityError from slotqueue
The slot queue should also have nothing to do with availabilities
* when all availabilities are empty, pause the queue
An empty availability is defined as size < DefaultBlockSize as this means even the smallest possible request could not be served. However, this is up for discussion.
* remove availability from onAvailabilitiesEmptied callback
* refactor onAvailabilityAdded and onAvailabilitiesEmptied
onAvailabilityAdded and onAvailabilitiesEmptied are now only called from reservations.update (and eventually reservations.delete once implemented).
- Add empty routine for Availability and Reservation
- Add allEmpty routine for Availability and Reservation, which returns true when all all Availability or Reservation objects in the datastore are empty.
* SlotQueue test support updates
* Sales module test support updates
* Reservations module tests for queue pausing
* Sales module tests for queue pausing
Includes tests for sales states cancelled, errored, ignored to ensure onCleanUp is called with correct parameters
* SlotQueue module tests for queue pausing
* fix existing sales test
* PR feedback
- indent `self.unpause`
- update comment for `clearSeenFlags`
* reprocessSlot in SaleErrored only when coming from downloading
* remove pausing of queue when availabilities are "emptied"
Queue pausing when all availiabilies are "emptied" is not necessary, given that the node would not be able to service slots once all its availabilities' freeSize are too small for the slots in the queue, and would then be paused anyway.
Add test that asserts the queue is paused once the freeSpace of availabilities drops too low to fill slots in the queue.
* Update clearing of seen flags
The asyncheapqueue update overload would need to check index bounds and ultimately a different solution was found using the mitems iterator.
* fix test
request.id was different before updating request.ask.slots, and that id was used to set the state in mockmarket.
* Change filled/cleanup future to nil, so no await is needed
* add wait to allow items to be added to queue
* do not unpause queue when seen items are pushed
* re-add seen item back to queue once paused
Previously, when a seen item was processed, it was first popped off the queue, then the queue was paused waiting to process that item once the queue was unpaused. Now, when a seen item is processed, it is popped off the queue, the queue is paused, then the item is re-added to the queue and the queue will wait until unpaused before it will continue popping items off the queue. If the item was not re-added to the queue, it would have been processed immediately once unpaused, however there may have been other items with higher priority pushed to the queue in the meantime. The queue would not be unpaused if those added items were already seen. In particular, this may happen when ignored items due to lack of availability are re-added to a paused queue. Those ignored items will likely have a higher priority than the item that was just seen (due to it having been processed first), causing the queue to the be paused.
* address PR comments
* wire prover into node
* stricter case object checks
* return correct proof
* misc renames
* adding usefull traces
* fix nodes and tolerance to match expected params
* format challenges in logs
* add circom compat to solidity groth16 convertion
* update
* bump time to give nodes time to load with all circom artifacts
* misc
* misc
* use correct dataset geometry in erasure
* make errors more searchable
* use parens around `=? (await...)` calls
* styling
* styling
* use push raises
* fix to match constructor arguments
* merge master
* merge master
* integration: fix proof parameters for a test
Increased times due to ZK proof generation.
Increased storage requirement because we're now hosting
5 slots instead of 1.
* sales: calculate initial proof at start of period
reason: this ensures that the period (and therefore
the challenge) doesn't change while we're calculating
the proof
* integration: fix proof parameters for tests
Increased times due to waiting on next period.
Fixed data to be of right size.
Updated expected payout due to hosting 5 slots.
* sales: wait for stable proof challenge
When the block pointer is nearing the
wrap-around point, we wait another period
before calculating a proof.
* fix merge conflict
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* add block cancellation support + tests
* tie issueCancellations into resolveBlocks for proper exception tracking, address comments
* pull cancellation as separate primitive in BlockExcNetwork
* use allFutures, rename issueBlockCancellations -> cancelBlocks
* use trc instead of wrn to register send error
* do not log peer IDs
* Smart contracts update: Groth16Proof instead of bytes
* Use dummy verifier for now, until we can create ZK proofs
* Fix tests: submit proof only when slot is filled
* Submit dummy proofs for now
* More detailed log when proof submission failed
* Use dummy verifier for integration tests
For now at least
* Fix mistake in blanket renaming to ethProvider
* Update to latest codex-contracts-eth
* feat: zkey-hash from chain
* Fix zkeyHash
---------
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
* Blockexchange uses merkle root and index to fetch blocks
* Links the network store getTree to the local store.
* Update codex/stores/repostore.nim
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>
* Rework erasure.nim to include recent cleanup
* Revert accidential changes to lib versions
* Addressing review comments
* Storing proofs instead of trees
* Fix a comment
* Fix broken tests
* Fix for broken testerasure.nim
* Addressing PR comments
---------
Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>
Co-authored-by: benbierens <thatbenbierens@gmail.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* [sales] remove availability check before adding to slot queue
* [sales] add missing return statement
* [tests] remove 'eventuallyCheck' helper
* [sales] remove reservations from slot queue
* [tests] rename module `eventually` -> `always`
* [sales] increase slot queue size
Because it will now also hold items for which we haven't
checked availability yet.
* extra utilities and tweaks
* add atlas lock
* update ignores
* break build into it's own script
* update url rules
* base off codexdht's
* compile fixes for Nim 1.6.14
* update submodules
* convert mapFailure to procs to work around type resolution issues
* add toml parser for multiaddress
* change error type on keyutils
* bump nimbus build to use 1.6.14
* update gitignore
* adding new deps submodules
* bump nim ci version
* even more fixes
* more libp2p changes
* update keys
* fix eventually function
* adding coverage test file
* move coverage to build.nims
* use nimcache/coverage
* move libp2p import for tests into helper.nim
* remove named bin
* bug fixes for networkpeers (from Dmitriy)
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
## Slot queue
Adds a slot queue, as per the [slot queue design](https://github.com/codex-storage/codex-research/blob/master/design/sales.md#slot-queue).
Any time storage is requested, all slots from that request are immediately added to the queue. Finished, Canclled, Failed requests remove all slots with that request id from the queue. SlotFreed events add a new slot to the queue and SlotFilled events remove the slot from the queue. This allows popping of a slot each time one is processed, making things much simpler.
When an entire request of slots is added to the queue, the slot indices are shuffled randomly to hopefully prevent nodes that pick up the same storage requested event from clashing on the first processed slot index. This allowed removal of assigning a random slot index in the SalePreparing state and it also ensured that all SalesAgents will have a slot index assigned to them at the start thus the removal of the optional slotIndex.
Remove slotId from SlotFreed event as it was not being used. RequestId and slotIndex were added to the SlotFreed event earlier and those are now being used
The slot queue invariant that prioritises queue items added to the queue relies on a scoring mechanism to sort them based on the [sort order in the design document](https://github.com/codex-storage/codex-research/blob/master/design/sales.md#sort-order).
When a storage request is handled by the sales module, a slot index was randomly assigned and then the slot was filled. Now, a random slot index is only assigned when adding an entire request to the slot queue. Additionally, the slot is checked that its state is `SlotState.Free` before continuing with the download process.
SlotQueue should always ensure the underlying AsyncHeapQueue has one less than the maximum items, ensuring the SlotQueue can always have space to add an additional item regardless if it’s full or not.
Constructing `SlotQueue.workers` in `SlotQueue.new` calls `newAsyncQueue` which causes side effects, so the construction call had to be moved to `SlotQueue.start`.
Prevent loading request from contract (network request) if there is an existing item in queue for that request.
Check availability before adding request to queue.
Add ability to query market contract for past events. When new availabilities are added, the `onReservationAdded` callback is triggered in which past `StorageRequested` events are queried, and those slots are added to the queue (filtered by availability on `push` and filtered by state in `SalePreparing`).
#### Request Workers
Limit the concurrent requests being processed in the queue by using a limited pool of workers (default = 3). Workers are in a data structure of type `AsyncQueue[SlotQueueWorker]`. This allows us to await a `popFirst` for available workers inside of the main SlotQueue event loop
Add an `onCleanUp` that stops the agents and removes them from the sales module agent list. `onCleanUp` is called from sales end states (eg ignored, cancelled, finished, failed, errored).
Add a `doneProcessing` future to `SlotQueueWorker` to be completed in the `OnProcessSlot` callback. Each `doneProcessing` future created is cancelled and awaited in `SlotQueue.stop` (thanks to `TrackableFuturees`), which forced `stop` to become async.
- Cancel dispatched workers and the `onProcessSlot` callbacks, prevents zombie callbacks
#### Add TrackableFutures
Allow tracking of futures in a module so they can be cancelled at a later time. Useful for asyncSpawned futures, but works for any future.
### Sales module
The sales module needed to subscribe to request events to ensure that the request queue was managed correctly on each event. In the process of doing this, the sales agents were updated to avoid subscribing to events in each agent, and instead dispatch received events from the sales module to all created sales agents. This would prevent memory leaks on having too many eventemitters subscribed to.
- prevent removal of agents from sales module while stopping, otherwise the agents seq len is modified while iterating
An additional sales agent state was added, `SalePreparing`, that handles all state machine setup, such as retrieving the request and subscribing to events that were previously in the `SaleDownloading` state.
Once agents have parked in an end state (eg ignored, cancelled, finished, failed, errored), they were not getting cleaned up and the sales module was keeping a handle on their reference. An `onCleanUp` callback was created to be called after the state machine enters an end state, which could prevent a memory leak if the number of requests coming in is high.
Move the SalesAgent callback raises pragmas from the Sales module to the proc definition in SalesAgent. This avoids having to catch `Exception`.
- remove unneeded error handling as pragmas were moved
Move sales.subscriptions from an object containing named subscriptions to a `seq[Subscription]` directly on the sales object.
Sales tests: shut down repo after sales stop, to fix SIGABRT in CI
### Add async Promise API
- modelled after JavaScript Promise API
- alternative to `asyncSpawn` that allows handling of async calls in a synchronous context (including access to the synchronous closure) with less additional procs to be declared
- Write less code, catch errors that would otherwise defect in asyncspawn, and execute a callback after completion
- Add cancellation callbacks to utils/then, ensuring cancellations are handled properly
## Dependencies
- bump codex-contracts-eth to support slot queue (https://github.com/codex-storage/codex-contracts-eth/pull/61)
- bump nim-ethers to 0.5.0
- Bump nim-json-rpc submodule to 0bf2bcb
---------
Co-authored-by: Jaremy Creechley <creechley@gmail.com>
Goal is to provide documentation and to enable conf utils byte params.
* Create byte units NByte and plumb through in size locations.
* add json serde
* plumb parseDuration to conf
* Fixes/workarounds for nimsuggest failures in codex.nim.
* remove rng prefix - it appears to work now
* format new's to be more consistent
* making proc formatting a bit more consistent
* [contracts] Add SlotFreed event
* [integration] allow test node to be stopped twice
* [cli] add --validator option
* [contracts] remove dead code
* [contracts] instantiate OnChainMarket and OnChainClock only once
* [contracts] add Validation
* [sales] remove duplicate import
* [market] add missing import
* [market] subscribe to all SlotFilled events
* [market] add freeSlot()
* [sales] fix warnings
* [market] subscribe to SlotFreed events
* [contracts] fix warning
* [validator] keep track of filled slots
* [validation] remove slots that have ended
* [proving] absorb Proofs into Market
Both Proofs and Market are abstractions around
the Marketplace contract, having them separately
is more trouble than it's worth at the moment.
* [market] add markProofAsMissing()
* [clock] speed up waiting for clock in tests
* [validator] mark proofs as missing
* [timer] fix error on node shutdown
* [cli] handle --persistence and --validator separately
* [market] allow retrieval of proof timeout value
* [validator] do not subscribe to SlotFreed events
Freed slots are already handled in
removeSlotsThatHaveEnded(), and onSlotsFreed()
interfered with its iterator.
* [validator] Start validation at the start of a new period
To decrease the likelihood that we hit the validation timeout.
* [validator] do not mark proofs as missing after timeout
* [market] check whether proof can be marked as missing
* [validator] simplify validation
Simulate a transaction to mark proof as missing, instead
of trying to keep track of all the conditions that may
lead to a proof being marked as missing.
* [build] use nim-ethers PR #40
Uses "pending" blocktag instead of "latest" blocktag
for better simulation of transactions before sending
them.
https://github.com/status-im/nim-ethers/pull/40
* [integration] integration test for validator
* [validator] monitor a maximum number of slots
Adds cli parameter --validator-max-slots.
* [market] fix missing collateral argument
After rebasing, add the new argument to fillSlot calls.
* [build] update to nim-ethers 0.2.5
* [validator] use Set instead of Table to keep track of slots
* [validator] add logging
* [validator] add test for slot failure
* [market] use "pending" blocktag to use more up to date block time
* [contracts] remove unused import
* [validator] fix: wait until after period ends
The smart contract checks that 'end < block.timestamp',
so we need to wait until the block timestamp is greater
than the period end.
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
* applying styleCheck
* stuck on vendor folder
* Applies style check
* Turns styleCheck back off
* switches to stylecheck:usages
* Fixes empty template casing
* rolls up nim-blscurve, nim-datastore, nim-ethers, nim-leopard, and nim-taskpools.
* bumps nim-confutils and removes unused import from fileutils.nim
* Unused using in fileutils.nim is required by CI
* Reverts bump of nim-confutils module
