[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
import std/sets
|
|
|
|
import std/sequtils
|
|
|
|
import std/sugar
|
|
|
|
import std/times
|
|
|
|
import pkg/asynctest
|
|
|
|
import pkg/chronos
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
import pkg/datastore
|
|
|
|
import pkg/questionable
|
|
|
|
import pkg/questionable/results
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
import pkg/codex/sales
|
|
|
|
import pkg/codex/sales/salesdata
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
import pkg/codex/sales/salescontext
|
|
|
|
import pkg/codex/sales/reservations
|
|
|
|
import pkg/codex/stores/repostore
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
import pkg/codex/proving
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
import pkg/codex/blocktype as bt
|
|
|
|
import pkg/codex/node
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
import ../helpers/mockmarket
|
|
|
|
import ../helpers/mockclock
|
|
|
|
import ../helpers/eventually
|
|
|
|
import ../examples
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
import ./helpers
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
|
|
|
|
suite "Sales":
|
|
|
|
|
|
|
|
let proof = exampleProof()
|
|
|
|
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
var availability: Availability
|
|
|
|
var request: StorageRequest
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
var sales: Sales
|
|
|
|
var market: MockMarket
|
|
|
|
var clock: MockClock
|
|
|
|
var proving: Proving
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
var reservations: Reservations
|
|
|
|
var repo: RepoStore
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
|
|
|
|
setup:
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
availability = Availability.init(
|
|
|
|
size=100.u256,
|
|
|
|
duration=60.u256,
|
2023-04-14 09:04:17 +00:00
|
|
|
minPrice=600.u256,
|
|
|
|
maxCollateral=400.u256
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
)
|
|
|
|
request = StorageRequest(
|
|
|
|
ask: StorageAsk(
|
|
|
|
slots: 4,
|
|
|
|
slotSize: 100.u256,
|
|
|
|
duration: 60.u256,
|
|
|
|
reward: 10.u256,
|
2023-04-14 09:04:17 +00:00
|
|
|
collateral: 200.u256,
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
),
|
|
|
|
content: StorageContent(
|
|
|
|
cid: "some cid"
|
|
|
|
),
|
|
|
|
expiry: (getTime() + initDuration(hours=1)).toUnix.u256
|
|
|
|
)
|
|
|
|
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
market = MockMarket.new()
|
|
|
|
clock = MockClock.new()
|
|
|
|
proving = Proving.new()
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
let repoDs = SQLiteDatastore.new(Memory).tryGet()
|
|
|
|
let metaDs = SQLiteDatastore.new(Memory).tryGet()
|
|
|
|
repo = RepoStore.new(repoDs, metaDs)
|
|
|
|
await repo.start()
|
|
|
|
sales = Sales.new(market, clock, proving, repo)
|
|
|
|
reservations = sales.context.reservations
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
sales.onStore = proc(request: StorageRequest,
|
|
|
|
slot: UInt256,
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
onBatch: BatchProc): Future[?!void] {.async.} =
|
|
|
|
return success()
|
2023-03-27 13:47:25 +00:00
|
|
|
proving.onProve = proc(slot: Slot): Future[seq[byte]] {.async.} =
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
return proof
|
|
|
|
await sales.start()
|
|
|
|
request.expiry = (clock.now() + 42).u256
|
|
|
|
|
|
|
|
teardown:
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
await repo.stop()
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
await sales.stop()
|
|
|
|
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
proc getAvailability: ?!Availability =
|
|
|
|
waitFor reservations.get(availability.id)
|
|
|
|
|
|
|
|
proc wasIgnored: Future[bool] {.async.} =
|
|
|
|
return
|
|
|
|
eventually sales.agents.len == 1 and # agent created at first
|
|
|
|
eventually sales.agents.len == 0 # then removed once ignored
|
|
|
|
|
|
|
|
test "makes storage unavailable when downloading a matched request":
|
|
|
|
var used = false
|
|
|
|
sales.onStore = proc(request: StorageRequest,
|
|
|
|
slot: UInt256,
|
|
|
|
onBatch: BatchProc): Future[?!void] {.async.} =
|
|
|
|
without avail =? await reservations.get(availability.id):
|
|
|
|
fail()
|
|
|
|
used = avail.used
|
|
|
|
return success()
|
|
|
|
|
|
|
|
check isOk await reservations.reserve(availability)
|
|
|
|
await market.requestStorage(request)
|
|
|
|
check eventually used
|
|
|
|
|
|
|
|
test "reduces remaining availability size after download":
|
|
|
|
let blk = bt.Block.example
|
|
|
|
request.ask.slotSize = blk.data.len.u256
|
|
|
|
availability.size = request.ask.slotSize + 1
|
|
|
|
sales.onStore = proc(request: StorageRequest,
|
|
|
|
slot: UInt256,
|
|
|
|
onBatch: BatchProc): Future[?!void] {.async.} =
|
|
|
|
await onBatch(@[blk])
|
|
|
|
return success()
|
|
|
|
check isOk await reservations.reserve(availability)
|
|
|
|
await market.requestStorage(request)
|
|
|
|
check eventually getAvailability().?size == success 1.u256
|
|
|
|
|
|
|
|
test "ignores download when duration not long enough":
|
|
|
|
availability.duration = request.ask.duration - 1
|
|
|
|
check isOk await reservations.reserve(availability)
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
await market.requestStorage(request)
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
check await wasIgnored()
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
test "ignores request when slot size is too small":
|
|
|
|
availability.size = request.ask.slotSize - 1
|
|
|
|
check isOk await reservations.reserve(availability)
|
|
|
|
await market.requestStorage(request)
|
|
|
|
check await wasIgnored()
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
|
|
|
|
test "ignores request when reward is too low":
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
availability.minPrice = request.ask.pricePerSlot + 1
|
|
|
|
check isOk await reservations.reserve(availability)
|
|
|
|
await market.requestStorage(request)
|
|
|
|
check await wasIgnored()
|
|
|
|
|
|
|
|
test "availability remains unused when request is ignored":
|
|
|
|
availability.minPrice = request.ask.pricePerSlot + 1
|
|
|
|
check isOk await reservations.reserve(availability)
|
|
|
|
await market.requestStorage(request)
|
|
|
|
check getAvailability().?used == success false
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
|
2023-04-14 09:04:17 +00:00
|
|
|
test "ignores request when asked collateral is too high":
|
|
|
|
var tooBigCollateral = request
|
|
|
|
tooBigCollateral.ask.collateral = availability.maxCollateral + 1
|
|
|
|
check isOk await reservations.reserve(availability)
|
|
|
|
await market.requestStorage(tooBigCollateral)
|
|
|
|
check await wasIgnored()
|
|
|
|
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
test "retrieves and stores data locally":
|
|
|
|
var storingRequest: StorageRequest
|
|
|
|
var storingSlot: UInt256
|
|
|
|
sales.onStore = proc(request: StorageRequest,
|
|
|
|
slot: UInt256,
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
onBatch: BatchProc): Future[?!void] {.async.} =
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
storingRequest = request
|
|
|
|
storingSlot = slot
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
return success()
|
|
|
|
check isOk await reservations.reserve(availability)
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
await market.requestStorage(request)
|
|
|
|
check eventually storingRequest == request
|
|
|
|
check storingSlot < request.ask.slots.u256
|
|
|
|
|
|
|
|
test "handles errors during state run":
|
|
|
|
var saleFailed = false
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
proving.onProve = proc(slot: Slot): Future[seq[byte]] {.async.} =
|
|
|
|
# raise exception so machine.onError is called
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
raise newException(ValueError, "some error")
|
|
|
|
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
# onClear is called in SaleErrored.run
|
|
|
|
sales.onClear = proc(request: StorageRequest,
|
|
|
|
idx: UInt256) =
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
saleFailed = true
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
check isOk await reservations.reserve(availability)
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
await market.requestStorage(request)
|
|
|
|
check eventually saleFailed
|
|
|
|
|
|
|
|
test "makes storage available again when data retrieval fails":
|
|
|
|
let error = newException(IOError, "data retrieval failed")
|
|
|
|
sales.onStore = proc(request: StorageRequest,
|
|
|
|
slot: UInt256,
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
onBatch: BatchProc): Future[?!void] {.async.} =
|
|
|
|
return failure(error)
|
|
|
|
check isOk await reservations.reserve(availability)
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
await market.requestStorage(request)
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
check eventually getAvailability().?used == success false
|
|
|
|
check getAvailability().?size == success availability.size
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
|
|
|
|
test "generates proof of storage":
|
|
|
|
var provingRequest: StorageRequest
|
|
|
|
var provingSlot: UInt256
|
2023-03-27 13:47:25 +00:00
|
|
|
proving.onProve = proc(slot: Slot): Future[seq[byte]] {.async.} =
|
|
|
|
provingRequest = slot.request
|
|
|
|
provingSlot = slot.slotIndex
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
check isOk await reservations.reserve(availability)
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
await market.requestStorage(request)
|
|
|
|
check eventually provingRequest == request
|
|
|
|
check provingSlot < request.ask.slots.u256
|
|
|
|
|
|
|
|
test "fills a slot":
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
check isOk await reservations.reserve(availability)
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
await market.requestStorage(request)
|
|
|
|
check eventually market.filled.len == 1
|
|
|
|
check market.filled[0].requestId == request.id
|
|
|
|
check market.filled[0].slotIndex < request.ask.slots.u256
|
|
|
|
check market.filled[0].proof == proof
|
|
|
|
check market.filled[0].host == await market.getSigner()
|
|
|
|
|
|
|
|
test "calls onSale when slot is filled":
|
|
|
|
var soldAvailability: Availability
|
|
|
|
var soldRequest: StorageRequest
|
|
|
|
var soldSlotIndex: UInt256
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
sales.onSale = proc(request: StorageRequest,
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
slotIndex: UInt256) =
|
|
|
|
if a =? availability:
|
|
|
|
soldAvailability = a
|
|
|
|
soldRequest = request
|
|
|
|
soldSlotIndex = slotIndex
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
check isOk await reservations.reserve(availability)
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
await market.requestStorage(request)
|
|
|
|
check eventually soldAvailability == availability
|
|
|
|
check soldRequest == request
|
|
|
|
check soldSlotIndex < request.ask.slots.u256
|
|
|
|
|
|
|
|
test "calls onClear when storage becomes available again":
|
|
|
|
# fail the proof intentionally to trigger `agent.finish(success=false)`,
|
|
|
|
# which then calls the onClear callback
|
2023-03-27 13:47:25 +00:00
|
|
|
proving.onProve = proc(slot: Slot): Future[seq[byte]] {.async.} =
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
raise newException(IOError, "proof failed")
|
|
|
|
var clearedRequest: StorageRequest
|
|
|
|
var clearedSlotIndex: UInt256
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
sales.onClear = proc(request: StorageRequest,
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
slotIndex: UInt256) =
|
|
|
|
clearedRequest = request
|
|
|
|
clearedSlotIndex = slotIndex
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
check isOk await reservations.reserve(availability)
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
await market.requestStorage(request)
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
check eventually clearedRequest == request
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
check clearedSlotIndex < request.ask.slots.u256
|
|
|
|
|
|
|
|
test "makes storage available again when other host fills the slot":
|
|
|
|
let otherHost = Address.example
|
|
|
|
sales.onStore = proc(request: StorageRequest,
|
|
|
|
slot: UInt256,
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
onBatch: BatchProc): Future[?!void] {.async.} =
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
await sleepAsync(chronos.hours(1))
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
return success()
|
|
|
|
check isOk await reservations.reserve(availability)
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
await market.requestStorage(request)
|
|
|
|
for slotIndex in 0..<request.ask.slots:
|
|
|
|
market.fillSlot(request.id, slotIndex.u256, proof, otherHost)
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
check eventually (await reservations.allAvailabilities) == @[availability]
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
|
|
|
|
test "makes storage available again when request expires":
|
|
|
|
sales.onStore = proc(request: StorageRequest,
|
|
|
|
slot: UInt256,
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
onBatch: BatchProc): Future[?!void] {.async.} =
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
await sleepAsync(chronos.hours(1))
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
return success()
|
|
|
|
check isOk await reservations.reserve(availability)
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
await market.requestStorage(request)
|
|
|
|
clock.set(request.expiry.truncate(int64))
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
check eventually (await reservations.allAvailabilities) == @[availability]
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
|
|
|
|
test "adds proving for slot when slot is filled":
|
|
|
|
var soldSlotIndex: UInt256
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
sales.onSale = proc(request: StorageRequest,
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
slotIndex: UInt256) =
|
|
|
|
soldSlotIndex = slotIndex
|
|
|
|
check proving.slots.len == 0
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
check isOk await reservations.reserve(availability)
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
await market.requestStorage(request)
|
|
|
|
check eventually proving.slots.len == 1
|
2023-03-27 13:47:25 +00:00
|
|
|
check proving.slots.contains(Slot(request: request, slotIndex: soldSlotIndex))
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
|
|
|
|
test "loads active slots from market":
|
|
|
|
let me = await market.getSigner()
|
|
|
|
|
|
|
|
request.ask.slots = 2
|
|
|
|
market.requested = @[request]
|
|
|
|
market.requestState[request.id] = RequestState.New
|
|
|
|
|
|
|
|
proc fillSlot(slotIdx: UInt256 = 0.u256) {.async.} =
|
|
|
|
let address = await market.getSigner()
|
|
|
|
let slot = MockSlot(requestId: request.id,
|
|
|
|
slotIndex: slotIdx,
|
|
|
|
proof: proof,
|
|
|
|
host: address)
|
|
|
|
market.filled.add slot
|
|
|
|
market.slotState[slotId(request.id, slotIdx)] = SlotState.Filled
|
|
|
|
|
|
|
|
let slot0 = MockSlot(requestId: request.id,
|
|
|
|
slotIndex: 0.u256,
|
|
|
|
proof: proof,
|
|
|
|
host: me)
|
|
|
|
await fillSlot(slot0.slotIndex)
|
|
|
|
|
|
|
|
let slot1 = MockSlot(requestId: request.id,
|
|
|
|
slotIndex: 1.u256,
|
|
|
|
proof: proof,
|
|
|
|
host: me)
|
|
|
|
await fillSlot(slot1.slotIndex)
|
|
|
|
market.activeSlots[me] = @[request.slotId(0.u256), request.slotId(1.u256)]
|
|
|
|
market.requested = @[request]
|
|
|
|
market.activeRequests[me] = @[request.id]
|
|
|
|
|
|
|
|
await sales.load()
|
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module
- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
- ClientInteractions (with purchasing)
- HostInteractions (with sales and proving)
- compilation fix for nim 1.2
[repostore] fix started flag, add tests
[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.
* Revert repostore changes
In favour of separate PR https://github.com/status-im/nim-codex/pull/374.
* remove warnings
* clean up
* tests: stop repostore during teardown
* change constructor type identifier
Change Contracts constructor to accept Contracts type instead of ContractInteractions.
* change constructor return type to Result instead of Option
* fix and split interactions tests
* clean up, fix tests
* find availability by slot id
* remove duplication in host/client interactions
* add test for finding availability by slotId
* log instead of raiseAssert when failed to mark availability as unused
* move to SaleErrored state instead of raiseAssert
* remove unneeded reverse
It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.
* update open api spec for potential rest endpoint errors
* move functions about available bytes to repostore
* WIP: reserve and release availabilities as needed
WIP: not tested yet
Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.
As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.
During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.
Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.
* delete availability when all bytes released
* fix tests + cleanup
* remove availability from SalesContext callbacks
Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.
Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.
* test clean up
* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
- there was a bug fixed that crashed the node due to a missing `return success` in onStore
- the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging
* fixes after rebase
1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.
* swap contracts branch to not support slot collateral
Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.
* modify Interactions and Deployment constructors
- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`
* Move `batchProc` declaration
`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.
* [reservations] rename `available` to `hasAvailable`
* [reservations] default error message to inner error msg
* add SaleIngored state
When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 07:05:16 +00:00
|
|
|
let expected = SalesData(requestId: request.id, request: some request)
|
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain
# Conflicts:
# codex/contracts/market.nim
* [marketplace] make on chain event callbacks async
# Conflicts:
# tests/codex/helpers/mockmarket.nim
* [marketplace] make availability optional for node restart
# Conflicts:
# tests/codex/testsales.nim
* [marketplace] add async state machine
Allows for `enterAsync` to be cancelled.
* [marketplace] move sale process to async state machine
* [marketplace] sales state machine tests
* bump dagger-contracts
* [marketplace] fix ci issue with chronicles output
* PR comments
- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-
* [marketplace] handle slot filled by other host
Handle the case when in the downloading, proving, or filling states, that another host fills the slot.
* [marketplace] use requestId for mySlots
* [marketplace] infer slot index from slotid
prevents reassigning a random slot index when restoring state from chain
* [marketplace] update to work with latest contracts
* [marketplace] clean up
* [marketplace] align with contract changes
- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState
Tests for changes
* [marketplace] add missing file
* [marketplace] bump codex-contracts-eth
* [config] remove unused imports
* [sales] cleanup
* [sales] fix: do not crash when fetching state fails
* [sales] make slotIndex non-optional
* Rebase and update NBS commit
Rebase on top of main and update NBS commit to the CI fix.
* [marketplace] use async subscription event handlers
* [marketplace] support slotIndex no longer optional
Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.
* [marketplace] sales state machine: use slotState
Use `slotState` instead of `requestState` for sales state machine.
* [marketplace] clean up
* [statemachine] adds a statemachine for async workflows
Allows events to be scheduled synchronously.
See https://github.com/status-im/nim-codex/pull/344
Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>
* [market] make market callbacks synchronous
* [statemachine] export Event
* [statemachine] ensure that no errors are raised
* [statemachine] add machine parameter to run method
* [statemachine] initialize queue on start
* [statemachine] check futures before cancelling them
* [sales] use new async state machine
- states use new run() method and event mechanism
- StartState starts subscriptions and loads request
* [statemachine] fix unsusbscribe before subscribe
* [sales] replace old state transition tests
* [sales] separate state machine from sales data
* [sales] remove reference from SalesData to Sales
* [sales] separate sales context from sales
* [sales] move decoupled types into their own modules
* [sales] move retrieveRequest to SalesData
* [sales] move subscription logic into SalesAgent
* [sales] unsubscribe when finished or errored
* [build] revert back to released version of nim-ethers
* [sales] remove SaleStart state
* [sales] add missing base method
* [sales] move asyncSpawn helper to utils
* [sales] fix imports
* [sales] remove unused variables
* [sales statemachine] add async state machine error handling (#349)
* [statemachine] add error handling to asyncstatemachine
- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.
* [sales] rename onSaleFailed to onSaleErrored
Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.
* [statemachine] forward error to state directly
without going through a machine method first
* [statemachine] remove unnecessary error handling
AsyncQueueFullError is already handled in schedule()
* [statemachine] test that cancellation ignores onError
* [sales] simplify error handling in states
Rely on the state machine error handling instead
of catching errors in the state run method
---------
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
* [statemachine] prevent memory leaks
prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started
* [salesagent] add salesagent-specific tests
1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 13:34:26 +00:00
|
|
|
# because sales.load() calls agent.start, we won't know the slotIndex
|
|
|
|
# randomly selected for the agent, and we also won't know the value of
|
|
|
|
# `failed`/`fulfilled`/`cancelled` futures, so we need to compare
|
|
|
|
# the properties we know
|
|
|
|
# TODO: when calling sales.load(), slot index should be restored and not
|
|
|
|
# randomly re-assigned, so this may no longer be needed
|
|
|
|
proc `==` (data0, data1: SalesData): bool =
|
|
|
|
return data0.requestId == data1.requestId and
|
|
|
|
data0.request == data1.request
|
|
|
|
|
|
|
|
check eventually sales.agents.len == 2
|
|
|
|
check sales.agents.all(agent => agent.data == expected)
|