codex-storage
/
nim-codex
mirror of https://github.com/codex-storage/nim-codex.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

[marketplace] Load sales state from chain (#306) 

* [marketplace] get active slots from chain

# Conflicts:
#	codex/contracts/market.nim

* [marketplace] make on chain event callbacks async

# Conflicts:
#	tests/codex/helpers/mockmarket.nim

* [marketplace] make availability optional for node restart

# Conflicts:
#	tests/codex/testsales.nim

* [marketplace] add async state machine

Allows for `enterAsync` to be cancelled.

* [marketplace] move sale process to async state machine

* [marketplace] sales state machine tests

* bump dagger-contracts

* [marketplace] fix ci issue with chronicles output

* PR comments

- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-

* [marketplace] handle slot filled by other host

Handle the case when in the downloading, proving, or filling states, that another host fills the slot.

* [marketplace] use requestId for mySlots

* [marketplace] infer slot index from slotid

prevents reassigning a random slot index when restoring state from chain

* [marketplace] update to work with latest contracts

* [marketplace] clean up

* [marketplace] align with contract changes

- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState

Tests for changes

* [marketplace] add missing file

* [marketplace] bump codex-contracts-eth

* [config] remove unused imports

* [sales] cleanup

* [sales] fix: do not crash when fetching state fails

* [sales] make slotIndex non-optional

* Rebase and update NBS commit

Rebase on top of main and update NBS commit to the CI fix.

* [marketplace] use async subscription event handlers

* [marketplace] support slotIndex no longer optional

Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.

* [marketplace] sales state machine: use slotState

Use `slotState` instead of `requestState` for sales state machine.

* [marketplace] clean up

* [statemachine] adds a statemachine for async workflows

Allows events to be scheduled synchronously.

See https://github.com/status-im/nim-codex/pull/344

Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>

* [market] make market callbacks synchronous

* [statemachine] export Event

* [statemachine] ensure that no errors are raised

* [statemachine] add machine parameter to run method

* [statemachine] initialize queue on start

* [statemachine] check futures before cancelling them

* [sales] use new async state machine

- states use new run() method and event mechanism
- StartState starts subscriptions and loads request

* [statemachine] fix unsusbscribe before subscribe

* [sales] replace old state transition tests

* [sales] separate state machine from sales data

* [sales] remove reference from SalesData to Sales

* [sales] separate sales context from sales

* [sales] move decoupled types into their own modules

* [sales] move retrieveRequest to SalesData

* [sales] move subscription logic into SalesAgent

* [sales] unsubscribe when finished or errored

* [build] revert back to released version of nim-ethers

* [sales] remove SaleStart state

* [sales] add missing base method

* [sales] move asyncSpawn helper to utils

* [sales] fix imports

* [sales] remove unused variables

* [sales statemachine] add async state machine error handling (#349)

* [statemachine] add error handling to asyncstatemachine

- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.

* [sales] rename onSaleFailed to onSaleErrored

Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.

* [statemachine] forward error to state directly

without going through a machine method first

* [statemachine] remove unnecessary error handling

AsyncQueueFullError is already handled in schedule()

* [statemachine] test that cancellation ignores onError

* [sales] simplify error handling in states

Rely on the state machine error handling instead
of catching errors in the state run method

---------

Co-authored-by: Mark Spanbroek <mark@spanbroek.net>

* [statemachine] prevent memory leaks

prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started

* [salesagent] add salesagent-specific tests

1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.

---------

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
This commit is contained in:
Eric Mastro 2023-03-09 00:34:26 +11:00 committed by GitHub
parent d263ca0735
commit 25f68c1e4c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
53 changed files with 1830 additions and 448 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
codex/contracts/Readme.md
View File

@ -39,7 +39,8 @@ Hosts need to put up collateral before participating in storage contracts.
A host can learn about the amount of collateral that is required:
```nim
let collateral = await marketplace.collateral()
let config = await marketplace.config()
let collateral = config.collateral.initialAmount
```
The host then needs to prepare a payment to the smart contract by calling the

71
codex/contracts/config.nim Normal file
View File

@ -0,0 +1,71 @@
import pkg/contractabi
import pkg/ethers/fields
import pkg/questionable/results
export contractabi
type
MarketplaceConfig* = object
collateral*: CollateralConfig
proofs*: ProofConfig
CollateralConfig* = object
initialAmount*: UInt256 # amount of collateral necessary to fill a slot
minimumAmount*: UInt256 # frees slot when collateral drops below this minimum
slashCriterion*: UInt256 # amount of proofs missed that lead to slashing
slashPercentage*: UInt256 # percentage of the collateral that is slashed
ProofConfig* = object
period*: UInt256 # proofs requirements are calculated per period (in seconds)
timeout*: UInt256 # mark proofs as missing before the timeout (in seconds)
downtime*: uint8 # ignore this much recent blocks for proof requirements
func fromTuple(_: type ProofConfig, tupl: tuple): ProofConfig =
ProofConfig(
period: tupl[0],
timeout: tupl[1],
downtime: tupl[2]
)
func fromTuple(_: type CollateralConfig, tupl: tuple): CollateralConfig =
CollateralConfig(
initialAmount: tupl[0],
minimumAmount: tupl[1],
slashCriterion: tupl[2],
slashPercentage: tupl[3]
)
func fromTuple(_: type MarketplaceConfig, tupl: tuple): MarketplaceConfig =
MarketplaceConfig(
collateral: tupl[0],
proofs: tupl[1]
)
func solidityType*(_: type ProofConfig): string =
solidityType(ProofConfig.fieldTypes)
func solidityType*(_: type CollateralConfig): string =
solidityType(CollateralConfig.fieldTypes)
func solidityType*(_: type MarketplaceConfig): string =
solidityType(CollateralConfig.fieldTypes)
func encode*(encoder: var AbiEncoder, slot: ProofConfig) =
encoder.write(slot.fieldValues)
func encode*(encoder: var AbiEncoder, slot: CollateralConfig) =
encoder.write(slot.fieldValues)
func encode*(encoder: var AbiEncoder, slot: MarketplaceConfig) =
encoder.write(slot.fieldValues)
func decode*(decoder: var AbiDecoder, T: type ProofConfig): ?!T =
let tupl = ?decoder.read(ProofConfig.fieldTypes)
success ProofConfig.fromTuple(tupl)
func decode*(decoder: var AbiDecoder, T: type CollateralConfig): ?!T =
let tupl = ?decoder.read(CollateralConfig.fieldTypes)
success CollateralConfig.fromTuple(tupl)
func decode*(decoder: var AbiDecoder, T: type MarketplaceConfig): ?!T =
let tupl = ?decoder.read(MarketplaceConfig.fieldTypes)
success MarketplaceConfig.fromTuple(tupl)

22
codex/contracts/market.nim
View File

@ -31,6 +31,9 @@ method getSigner*(market: OnChainMarket): Future[Address] {.async.} =
method myRequests*(market: OnChainMarket): Future[seq[RequestId]] {.async.} =
return await market.contract.myRequests
method mySlots*(market: OnChainMarket): Future[seq[SlotId]] {.async.} =
return await market.contract.mySlots()
method requestStorage(market: OnChainMarket, request: StorageRequest){.async.} =
await market.contract.requestStorage(request)
@ -43,15 +46,19 @@ method getRequest(market: OnChainMarket,
return none StorageRequest
raise e
method getState*(market: OnChainMarket,
method requestState*(market: OnChainMarket,
requestId: RequestId): Future[?RequestState] {.async.} =
try:
return some await market.contract.state(requestId)
return some await market.contract.requestState(requestId)
except ProviderError as e:
if e.revertReason.contains("Unknown request"):
return none RequestState
raise e
method slotState*(market: OnChainMarket,
slotId: SlotId): Future[SlotState] {.async.} =
return await market.contract.slotState(slotId)
method getRequestEnd*(market: OnChainMarket,
id: RequestId): Future[SecondsSince1970] {.async.} =
return await market.contract.requestEnd(id)
@ -66,6 +73,15 @@ method getHost(market: OnChainMarket,
else:
return none Address
method getRequestFromSlotId*(market: OnChainMarket,
slotId: SlotId): Future[?StorageRequest] {.async.} =
try:
return some await market.contract.getRequestFromSlotId(slotId)
except ProviderError as e:
if e.revertReason.contains("Slot is free"):
return none StorageRequest
raise e
method fillSlot(market: OnChainMarket,
requestId: RequestId,
slotIndex: UInt256,
@ -119,7 +135,7 @@ method subscribeRequestFailed*(market: OnChainMarket,
requestId: RequestId,
callback: OnRequestFailed):
Future[MarketSubscription] {.async.} =
proc onEvent(event: RequestFailed) {.upraises:[].} =
proc onEvent(event: RequestFailed) {.upraises:[]} =
if event.requestId == requestId:
callback(event.requestId)
let subscription = await market.contract.subscribe(RequestFailed, onEvent)

10
codex/contracts/marketplace.nim
View File

@ -4,9 +4,11 @@ import pkg/stint
import pkg/chronos
import ../clock
import ./requests
import ./config
export stint
export ethers
export config
type
Marketplace* = ref object of Contract
@ -28,7 +30,7 @@ type
proof*: seq[byte]
proc collateral*(marketplace: Marketplace): UInt256 {.contract, view.}
proc config*(marketplace: Marketplace): MarketplaceConfig {.contract, view.}
proc slashMisses*(marketplace: Marketplace): UInt256 {.contract, view.}
proc slashPercentage*(marketplace: Marketplace): UInt256 {.contract, view.}
proc minCollateralThreshold*(marketplace: Marketplace): UInt256 {.contract, view.}
@ -43,12 +45,14 @@ proc withdrawFunds*(marketplace: Marketplace, requestId: RequestId) {.contract.}
proc freeSlot*(marketplace: Marketplace, id: SlotId) {.contract.}
proc getRequest*(marketplace: Marketplace, id: RequestId): StorageRequest {.contract, view.}
proc getHost*(marketplace: Marketplace, id: SlotId): Address {.contract, view.}
proc getRequestFromSlotId*(marketplace: Marketplace, id: SlotId): StorageRequest {.contract, view.}
proc myRequests*(marketplace: Marketplace): seq[RequestId] {.contract, view.}
proc state*(marketplace: Marketplace, requestId: RequestId): RequestState {.contract, view.}
proc mySlots*(marketplace: Marketplace): seq[SlotId] {.contract, view.}
proc requestState*(marketplace: Marketplace, requestId: RequestId): RequestState {.contract, view.}
proc slotState*(marketplace: Marketplace, slotId: SlotId): SlotState {.contract, view.}
proc requestEnd*(marketplace: Marketplace, requestId: RequestId): SecondsSince1970 {.contract, view.}
proc proofPeriod*(marketplace: Marketplace): UInt256 {.contract, view.}
proc proofTimeout*(marketplace: Marketplace): UInt256 {.contract, view.}
proc proofEnd*(marketplace: Marketplace, id: SlotId): UInt256 {.contract, view.}

18
codex/contracts/proofs.nim
View File

@ -21,7 +21,8 @@ proc new*(_: type OnChainProofs, marketplace: Marketplace): OnChainProofs =
OnChainProofs(marketplace: marketplace, pollInterval: DefaultPollInterval)
method periodicity*(proofs: OnChainProofs): Future[Periodicity] {.async.} =
let period = await proofs.marketplace.proofPeriod()
let config = await proofs.marketplace.config()
let period = config.proofs.period
return Periodicity(seconds: period)
method isProofRequired*(proofs: OnChainProofs,
@ -29,7 +30,7 @@ method isProofRequired*(proofs: OnChainProofs,
try:
return await proofs.marketplace.isProofRequired(id)
except ProviderError as e:
if e.revertReason.contains("Slot empty"):
if e.revertReason.contains("Slot is free"):
return false
raise e
@ -38,18 +39,13 @@ method willProofBeRequired*(proofs: OnChainProofs,
try:
return await proofs.marketplace.willProofBeRequired(id)
except ProviderError as e:
if e.revertReason.contains("Slot empty"):
if e.revertReason.contains("Slot is free"):
return false
raise e
method getProofEnd*(proofs: OnChainProofs,
id: SlotId): Future[UInt256] {.async.} =
try:
return await proofs.marketplace.proofEnd(id)
except ProviderError as e:
if e.revertReason.contains("Slot empty"):
return 0.u256
raise e
method slotState*(proofs: OnChainProofs,
id: SlotId): Future[SlotState] {.async.} =
return await proofs.marketplace.slotState(id)
method submitProof*(proofs: OnChainProofs,
id: SlotId,

6
codex/contracts/requests.nim
View File

@ -39,6 +39,12 @@ type
Cancelled
Finished
Failed
SlotState* {.pure.} = enum
Free
Filled
Finished
Failed
Paid
proc `==`*(x, y: Nonce): bool {.borrow.}
proc `==`*(x, y: RequestId): bool {.borrow.}

13
codex/market.nim
View File

@ -28,15 +28,22 @@ method requestStorage*(market: Market,
method myRequests*(market: Market): Future[seq[RequestId]] {.base, async.} =
raiseAssert("not implemented")
method mySlots*(market: Market): Future[seq[SlotId]] {.base, async.} =
raiseAssert("not implemented")
method getRequest*(market: Market,
id: RequestId):
Future[?StorageRequest] {.base, async.} =
raiseAssert("not implemented")
method getState*(market: Market,
method requestState*(market: Market,
requestId: RequestId): Future[?RequestState] {.base, async.} =
raiseAssert("not implemented")
method slotState*(market: Market,
slotId: SlotId): Future[SlotState] {.base, async.} =
raiseAssert("not implemented")
method getRequestEnd*(market: Market,
id: RequestId): Future[SecondsSince1970] {.base, async.} =
raiseAssert("not implemented")
@ -46,6 +53,10 @@ method getHost*(market: Market,
slotIndex: UInt256): Future[?Address] {.base, async.} =
raiseAssert("not implemented")
method getRequestFromSlotId*(market: Market,
slotId: SlotId): Future[?StorageRequest] {.base, async.} =
raiseAssert("not implemented")
method fillSlot*(market: Market,
requestId: RequestId,
slotIndex: UInt256,

4
codex/node.nim
View File

@ -333,7 +333,7 @@ proc start*(node: CodexNodeRef) {.async.} =
# TODO: remove Sales callbacks, pass BlockStore and StorageProofs instead
contracts.sales.onStore = proc(request: StorageRequest,
slot: UInt256,
availability: Availability) {.async.} =
availability: ?Availability) {.async.} =
## store data in local storage
##
@ -354,7 +354,7 @@ proc start*(node: CodexNodeRef) {.async.} =
if fetchRes.isErr:
raise newException(CodexError, "Unable to retrieve blocks")
contracts.sales.onClear = proc(availability: Availability,
contracts.sales.onClear = proc(availability: ?Availability,
request: StorageRequest,
slotIndex: UInt256) =
# TODO: remove data from local storage

3
codex/proving.nim
View File

@ -38,7 +38,8 @@ proc removeEndedContracts(proving: Proving) {.async.} =
let now = proving.clock.now().u256
var ended: HashSet[SlotId]
for id in proving.slots:
if now >= (await proving.proofs.getProofEnd(id)):
let state = await proving.proofs.slotState(id)
if state != SlotState.Filled:
ended.incl(id)
proving.slots.excl(ended)

2
codex/purchasing/states/started.nim
View File

@ -21,8 +21,10 @@ method enterAsync*(state: PurchaseStarted) {.async.} =
try:
let fut = await one(ended, failed)
if fut.id == failed.id:
ended.cancel()
state.switch(PurchaseFailed())
else:
failed.cancel()
state.switch(PurchaseFinished())
await subscription.unsubscribe()
except CatchableError as error:

2
codex/purchasing/states/unknown.nim
View File

@ -14,7 +14,7 @@ method enterAsync(state: PurchaseUnknown) {.async.} =
try:
if (request =? await purchase.market.getRequest(purchase.requestId)) and
(requestState =? await purchase.market.getState(purchase.requestId)):
(requestState =? await purchase.market.requestState(purchase.requestId)):
purchase.request = some request

267
codex/sales.nim
View File

@ -9,6 +9,12 @@ import ./market
import ./clock
import ./proving
import ./contracts/requests
import ./sales/salescontext
import ./sales/salesagent
import ./sales/availability
import ./sales/statemachine
import ./sales/states/downloading
import ./sales/states/unknown
## Sales holds a list of available storage that it may sell.
##
@ -29,55 +35,36 @@ import ./contracts/requests
## | | ---- storage proof ---> |
export stint
export availability
type
Sales* = ref object
market: Market
clock: Clock
subscription: ?market.Subscription
available*: seq[Availability]
onStore: ?OnStore
onProve: ?OnProve
onClear: ?OnClear
onSale: ?OnSale
proving: Proving
Availability* = object
id*: array[32, byte]
size*: UInt256
duration*: UInt256
minPrice*: UInt256
SalesAgent = ref object
sales: Sales
requestId: RequestId
ask: StorageAsk
availability: Availability
request: ?StorageRequest
slotIndex: ?UInt256
subscription: ?market.Subscription
running: ?Future[void]
waiting: ?Future[void]
finished: bool
OnStore = proc(request: StorageRequest,
slot: UInt256,
availability: Availability): Future[void] {.gcsafe, upraises: [].}
OnProve = proc(request: StorageRequest,
slot: UInt256): Future[seq[byte]] {.gcsafe, upraises: [].}
OnClear = proc(availability: Availability,
request: StorageRequest,
slotIndex: UInt256) {.gcsafe, upraises: [].}
OnSale = proc(availability: Availability,
request: StorageRequest,
slotIndex: UInt256) {.gcsafe, upraises: [].}
context*: SalesContext
subscription*: ?market.Subscription
available: seq[Availability]
agents*: seq[SalesAgent]
func new*(_: type Sales,
market: Market,
clock: Clock,
proving: Proving): Sales =
Sales(
market: market,
clock: clock,
proving: proving
)
proc `onStore=`*(sales: Sales, onStore: OnStore) =
sales.context.onStore = some onStore
proc `onProve=`*(sales: Sales, onProve: OnProve) =
sales.context.onProve = some onProve
proc `onClear=`*(sales: Sales, onClear: OnClear) =
sales.context.onClear = some onClear
proc `onSale=`*(sales: Sales, callback: OnSale) =
sales.context.onSale = some callback
proc onStore*(sales: Sales): ?OnStore = sales.context.onStore
proc onProve*(sales: Sales): ?OnProve = sales.context.onProve
proc onClear*(sales: Sales): ?OnClear = sales.context.onClear
proc onSale*(sales: Sales): ?OnSale = sales.context.onSale
proc available*(sales: Sales): seq[Availability] = sales.available
proc init*(_: type Availability,
size: UInt256,
@ -87,140 +74,95 @@ proc init*(_: type Availability,
doAssert randomBytes(id) == 32
Availability(id: id, size: size, duration: duration, minPrice: minPrice)
proc `onStore=`*(sales: Sales, onStore: OnStore) =
sales.onStore = some onStore
proc `onProve=`*(sales: Sales, onProve: OnProve) =
sales.onProve = some onProve
proc `onClear=`*(sales: Sales, onClear: OnClear) =
sales.onClear = some onClear
proc `onSale=`*(sales: Sales, callback: OnSale) =
sales.onSale = some callback
func add*(sales: Sales, availability: Availability) =
sales.available.add(availability)
if not sales.available.contains(availability):
sales.available.add(availability)
# TODO: add to disk (persist), serialise to json.
func remove*(sales: Sales, availability: Availability) =
sales.available.keepItIf(it != availability)
# TODO: remove from disk availability, mark as in use by assigning
# a slotId, so that it can be used for restoration (node restart)
func findAvailability(sales: Sales, ask: StorageAsk): ?Availability =
func new*(_: type Sales,
market: Market,
clock: Clock,
proving: Proving): Sales =
let sales = Sales(context: SalesContext(
market: market,
clock: clock,
proving: proving
))
proc onSaleErrored(availability: Availability) =
sales.add(availability)
sales.context.onSaleErrored = some onSaleErrored
sales
func findAvailability*(sales: Sales, ask: StorageAsk): ?Availability =
for availability in sales.available:
if ask.slotSize <= availability.size and
ask.duration <= availability.duration and
ask.pricePerSlot >= availability.minPrice:
return some availability
proc finish(agent: SalesAgent, success: bool) =
if agent.finished:
return
agent.finished = true
if subscription =? agent.subscription:
asyncSpawn subscription.unsubscribe()
if running =? agent.running:
running.cancel()
if waiting =? agent.waiting:
waiting.cancel()
if success:
if request =? agent.request and
slotIndex =? agent.slotIndex:
agent.sales.proving.add(request.slotId(slotIndex))
if onSale =? agent.sales.onSale:
onSale(agent.availability, request, slotIndex)
else:
if onClear =? agent.sales.onClear and
request =? agent.request and
slotIndex =? agent.slotIndex:
onClear(agent.availability, request, slotIndex)
agent.sales.add(agent.availability)
proc selectSlot(agent: SalesAgent) =
proc randomSlotIndex(numSlots: uint64): UInt256 =
let rng = Rng.instance
let slotIndex = rng.rand(agent.ask.slots - 1)
agent.slotIndex = some slotIndex.u256
let slotIndex = rng.rand(numSlots - 1)
return slotIndex.u256
proc onSlotFilled(agent: SalesAgent,
requestId: RequestId,
slotIndex: UInt256) {.async.} =
try:
let market = agent.sales.market
let host = await market.getHost(requestId, slotIndex)
let me = await market.getSigner()
agent.finish(success = (host == me.some))
except CatchableError:
agent.finish(success = false)
proc findSlotIndex(numSlots: uint64,
requestId: RequestId,
slotId: SlotId): ?UInt256 =
for i in 0..<numSlots:
if slotId(requestId, i.u256) == slotId:
return some i.u256
proc subscribeSlotFilled(agent: SalesAgent, slotIndex: UInt256) {.async.} =
proc onSlotFilled(requestId: RequestId,
slotIndex: UInt256) {.gcsafe, upraises:[].} =
asyncSpawn agent.onSlotFilled(requestId, slotIndex)
let market = agent.sales.market
let subscription = await market.subscribeSlotFilled(agent.requestId,
slotIndex,
onSlotFilled)
agent.subscription = some subscription
return none UInt256
proc waitForExpiry(agent: SalesAgent) {.async.} =
without request =? agent.request:
return
await agent.sales.clock.waitUntil(request.expiry.truncate(int64))
agent.finish(success = false)
proc start(agent: SalesAgent) {.async.} =
try:
let sales = agent.sales
let market = sales.market
let availability = agent.availability
without onStore =? sales.onStore:
raiseAssert "onStore callback not set"
without onProve =? sales.onProve:
raiseAssert "onProve callback not set"
sales.remove(availability)
agent.selectSlot()
without slotIndex =? agent.slotIndex:
raiseAssert "no slot selected"
await agent.subscribeSlotFilled(slotIndex)
agent.request = await market.getRequest(agent.requestId)
without request =? agent.request:
agent.finish(success = false)
return
agent.waiting = some agent.waitForExpiry()
await onStore(request, slotIndex, availability)
let proof = await onProve(request, slotIndex)
await market.fillSlot(request.id, slotIndex, proof)
except CancelledError:
raise
except CatchableError as e:
error "SalesAgent failed", msg = e.msg
agent.finish(success = false)
proc handleRequest(sales: Sales, requestId: RequestId, ask: StorageAsk) =
proc handleRequest(sales: Sales,
requestId: RequestId,
ask: StorageAsk) =
without availability =? sales.findAvailability(ask):
return
let agent = SalesAgent(
sales: sales,
requestId: requestId,
ask: ask,
availability: availability
sales.remove(availability)
# TODO: check if random slot is actually available (not already filled)
let slotIndex = randomSlotIndex(ask.slots)
let agent = newSalesAgent(
sales.context,
requestId,
slotIndex,
some availability,
none StorageRequest
)
agent.start(SaleDownloading())
sales.agents.add agent
agent.running = some agent.start()
proc load*(sales: Sales) {.async.} =
let market = sales.context.market
# TODO: restore availability from disk
let slotIds = await market.mySlots()
for slotId in slotIds:
# TODO: this needs to be optimised
if request =? await market.getRequestFromSlotId(slotId):
let availability = sales.findAvailability(request.ask)
without slotIndex =? findSlotIndex(request.ask.slots,
request.id,
slotId):
raiseAssert "could not find slot index"
let agent = newSalesAgent(
sales.context,
request.id,
slotIndex,
availability,
some request)
agent.start(SaleUnknown())
sales.agents.add agent
proc start*(sales: Sales) {.async.} =
doAssert sales.subscription.isNone, "Sales already started"
@ -229,7 +171,7 @@ proc start*(sales: Sales) {.async.} =
sales.handleRequest(requestId, ask)
try:
sales.subscription = some await sales.market.subscribeRequests(onRequest)
sales.subscription = some await sales.context.market.subscribeRequests(onRequest)
except CatchableError as e:
error "Unable to start sales", msg = e.msg
@ -240,3 +182,6 @@ proc stop*(sales: Sales) {.async.} =
await subscription.unsubscribe()
except CatchableError as e:
warn "Unsubscribe failed", msg = e.msg
for agent in sales.agents:
await agent.stop()

8
codex/sales/availability.nim Normal file
View File

@ -0,0 +1,8 @@
import pkg/stint
type
Availability* = object
id*: array[32, byte]
size*: UInt256
duration*: UInt256
minPrice*: UInt256

123
codex/sales/salesagent.nim Normal file
View File

@ -0,0 +1,123 @@
import pkg/chronos
import pkg/upraises
import pkg/stint
import ../contracts/requests
import ../utils/asyncspawn
import ./statemachine
import ./salescontext
import ./salesdata
import ./availability
type SalesAgent* = ref object of Machine
context*: SalesContext
data*: SalesData
subscribed: bool
proc newSalesAgent*(context: SalesContext,
requestId: RequestId,
slotIndex: UInt256,
availability: ?Availability,
request: ?StorageRequest): SalesAgent =
SalesAgent(
context: context,
data: SalesData(
requestId: requestId,
availability: availability,
slotIndex: slotIndex,
request: request))
proc retrieveRequest*(agent: SalesAgent) {.async.} =
let data = agent.data
let market = agent.context.market
if data.request.isNone:
data.request = await market.getRequest(data.requestId)
proc subscribeCancellation(agent: SalesAgent) {.async.} =
let data = agent.data
let market = agent.context.market
let clock = agent.context.clock
proc onCancelled() {.async.} =
without request =? data.request:
return
await clock.waitUntil(request.expiry.truncate(int64))
if not data.fulfilled.isNil:
asyncSpawn data.fulfilled.unsubscribe(), ignore = CatchableError
agent.schedule(cancelledEvent(request))
data.cancelled = onCancelled()
proc onFulfilled(_: RequestId) =
data.cancelled.cancel()
data.fulfilled =
await market.subscribeFulfillment(data.requestId, onFulfilled)
proc subscribeFailure(agent: SalesAgent) {.async.} =
let data = agent.data
let market = agent.context.market
proc onFailed(_: RequestId) =
without request =? data.request:
return
asyncSpawn data.failed.unsubscribe(), ignore = CatchableError
agent.schedule(failedEvent(request))
data.failed =
await market.subscribeRequestFailed(data.requestId, onFailed)
proc subscribeSlotFilled(agent: SalesAgent) {.async.} =
let data = agent.data
let market = agent.context.market
proc onSlotFilled(requestId: RequestId, slotIndex: UInt256) =
asyncSpawn data.slotFilled.unsubscribe(), ignore = CatchableError
agent.schedule(slotFilledEvent(requestId, data.slotIndex))
data.slotFilled =
await market.subscribeSlotFilled(data.requestId,
data.slotIndex,
onSlotFilled)
proc subscribe*(agent: SalesAgent) {.async.} =
if agent.subscribed:
return
await agent.subscribeCancellation()
await agent.subscribeFailure()
await agent.subscribeSlotFilled()
agent.subscribed = true
proc unsubscribe*(agent: SalesAgent) {.async.} =
if not agent.subscribed:
return
let data = agent.data
try:
if not data.fulfilled.isNil:
await data.fulfilled.unsubscribe()
data.fulfilled = nil
except CatchableError:
discard
try:
if not data.failed.isNil:
await data.failed.unsubscribe()
data.failed = nil
except CatchableError:
discard
try:
if not data.slotFilled.isNil:
await data.slotFilled.unsubscribe()
data.slotFilled = nil
except CatchableError:
discard
if not data.cancelled.isNil:
await data.cancelled.cancelAndWait()
data.cancelled = nil
agent.subscribed = false
proc stop*(agent: SalesAgent) {.async.} =
procCall Machine(agent).stop()
await agent.unsubscribe()

28
codex/sales/salescontext.nim Normal file
View File

@ -0,0 +1,28 @@
import pkg/upraises
import ../market
import ../clock
import ../proving
import ./availability
type
SalesContext* = ref object
market*: Market
clock*: Clock
onStore*: ?OnStore
onProve*: ?OnProve
onClear*: ?OnClear
onSale*: ?OnSale
onSaleErrored*: ?OnSaleErrored
proving*: Proving
OnStore* = proc(request: StorageRequest,
slot: UInt256,
availability: ?Availability): Future[void] {.gcsafe, upraises: [].}
OnProve* = proc(request: StorageRequest,
slot: UInt256): Future[seq[byte]] {.gcsafe, upraises: [].}
OnClear* = proc(availability: ?Availability,# TODO: when availability changes introduced, make availability non-optional (if we need to keep it at all)
request: StorageRequest,
slotIndex: UInt256) {.gcsafe, upraises: [].}
OnSale* = proc(availability: ?Availability, # TODO: when availability changes introduced, make availability non-optional (if we need to keep it at all)
request: StorageRequest,
slotIndex: UInt256) {.gcsafe, upraises: [].}
OnSaleErrored* = proc(availability: Availability) {.gcsafe, upraises: [].}

16
codex/sales/salesdata.nim Normal file
View File

@ -0,0 +1,16 @@
import pkg/chronos
import ../contracts/requests
import ../market
import ./availability
type
SalesData* = ref object
requestId*: RequestId
ask*: StorageAsk
availability*: ?Availability # TODO: when availability persistence is added, change this to not optional
request*: ?StorageRequest
slotIndex*: UInt256
failed*: market.Subscription
fulfilled*: market.Subscription
slotFilled*: market.Subscription
cancelled*: Future[void]

42
codex/sales/statemachine.nim Normal file
View File

@ -0,0 +1,42 @@
import pkg/questionable
import pkg/upraises
import ../errors
import ../utils/asyncstatemachine
import ../market
import ../clock
import ../proving
import ../contracts/requests
export market
export clock
export asyncstatemachine
export proving
type
SaleState* = ref object of State
SaleError* = ref object of CodexError
method `$`*(state: SaleState): string {.base.} =
raiseAssert "not implemented"
method onCancelled*(state: SaleState, request: StorageRequest): ?State {.base, upraises:[].} =
discard
method onFailed*(state: SaleState, request: StorageRequest): ?State {.base, upraises:[].} =
discard
method onSlotFilled*(state: SaleState, requestId: RequestId,
slotIndex: UInt256): ?State {.base, upraises:[].} =
discard
proc cancelledEvent*(request: StorageRequest): Event =
return proc (state: State): ?State =
SaleState(state).onCancelled(request)
proc failedEvent*(request: StorageRequest): Event =
return proc (state: State): ?State =
SaleState(state).onFailed(request)
proc slotFilledEvent*(requestId: RequestId, slotIndex: UInt256): Event =
return proc (state: State): ?State =
SaleState(state).onSlotFilled(requestId, slotIndex)

14
codex/sales/states/cancelled.nim Normal file
View File

@ -0,0 +1,14 @@
import ../statemachine
import ./errorhandling
import ./errored
type
SaleCancelled* = ref object of ErrorHandlingState
SaleCancelledError* = object of CatchableError
SaleTimeoutError* = object of SaleCancelledError
method `$`*(state: SaleCancelled): string = "SaleCancelled"
method run*(state: SaleCancelled, machine: Machine): Future[?State] {.async.} =
let error = newException(SaleTimeoutError, "Sale cancelled due to timeout")
return some State(SaleErrored(error: error))

42
codex/sales/states/downloading.nim Normal file
View File

@ -0,0 +1,42 @@
import ../../market
import ../salesagent
import ../statemachine
import ./errorhandling
import ./cancelled
import ./failed
import ./filled
import ./proving
type
SaleDownloading* = ref object of ErrorHandlingState
failedSubscription: ?market.Subscription
hasCancelled: ?Future[void]
method `$`*(state: SaleDownloading): string = "SaleDownloading"
method onCancelled*(state: SaleDownloading, request: StorageRequest): ?State =
return some State(SaleCancelled())
method onFailed*(state: SaleDownloading, request: StorageRequest): ?State =
return some State(SaleFailed())
method onSlotFilled*(state: SaleDownloading, requestId: RequestId,
slotIndex: UInt256): ?State =
return some State(SaleFilled())
method run*(state: SaleDownloading, machine: Machine): Future[?State] {.async.} =
let agent = SalesAgent(machine)
let data = agent.data
let context = agent.context
await agent.retrieveRequest()
await agent.subscribe()
without onStore =? context.onStore:
raiseAssert "onStore callback not set"
without request =? data.request:
raiseAssert "no sale request"
await onStore(request, data.slotIndex, data.availability)
return some State(SaleProving())

34
codex/sales/states/errored.nim Normal file
View File

@ -0,0 +1,34 @@
import pkg/upraises
import pkg/chronicles
import ../statemachine
import ../salesagent
type SaleErrored* = ref object of SaleState
error*: ref CatchableError
method `$`*(state: SaleErrored): string = "SaleErrored"
method onError*(state: SaleState, err: ref CatchableError): ?State {.upraises:[].} =
error "error during SaleErrored run", error = err.msg
method run*(state: SaleErrored, machine: Machine): Future[?State] {.async.} =
let agent = SalesAgent(machine)
let data = agent.data
let context = agent.context
if onClear =? context.onClear and
request =? data.request and
slotIndex =? data.slotIndex:
onClear(data.availability, request, slotIndex)
# TODO: when availability persistence is added, change this to not optional
# NOTE: with this in place, restoring state for a restarted node will
# never free up availability once finished. Persisting availability
# on disk is required for this.
if onSaleErrored =? context.onSaleErrored and
availability =? data.availability:
onSaleErrored(availability)
await agent.unsubscribe()
error "Sale error", error=state.error.msg

9
codex/sales/states/errorhandling.nim Normal file
View File

@ -0,0 +1,9 @@
import pkg/questionable
import ../statemachine
import ./errored
type
ErrorHandlingState* = ref object of SaleState
method onError*(state: ErrorHandlingState, error: ref CatchableError): ?State =
some State(SaleErrored(error: error))

13
codex/sales/states/failed.nim Normal file
View File

@ -0,0 +1,13 @@
import ../statemachine
import ./errorhandling
import ./errored
type
SaleFailed* = ref object of ErrorHandlingState
SaleFailedError* = object of SaleError
method `$`*(state: SaleFailed): string = "SaleFailed"
method run*(state: SaleFailed, machine: Machine): Future[?State] {.async.} =
let error = newException(SaleFailedError, "Sale failed")
return some State(SaleErrored(error: error))

32
codex/sales/states/filled.nim Normal file
View File

@ -0,0 +1,32 @@
import pkg/questionable
import ../statemachine
import ../salesagent
import ./errorhandling
import ./errored
import ./finished
import ./cancelled
import ./failed
type
SaleFilled* = ref object of ErrorHandlingState
HostMismatchError* = object of CatchableError
method onCancelled*(state: SaleFilled, request: StorageRequest): ?State =
return some State(SaleCancelled())
method onFailed*(state: SaleFilled, request: StorageRequest): ?State =
return some State(SaleFailed())
method `$`*(state: SaleFilled): string = "SaleFilled"
method run*(state: SaleFilled, machine: Machine): Future[?State] {.async.} =
let data = SalesAgent(machine).data
let market = SalesAgent(machine).context.market
let host = await market.getHost(data.requestId, data.slotIndex)
let me = await market.getSigner()
if host == me.some:
return some State(SaleFinished())
else:
let error = newException(HostMismatchError, "Slot filled by other host")
return some State(SaleErrored(error: error))

30
codex/sales/states/filling.nim Normal file
View File

@ -0,0 +1,30 @@
import ../../market
import ../statemachine
import ../salesagent
import ./errorhandling
import ./filled
import ./errored
import ./cancelled
import ./failed
type
SaleFilling* = ref object of ErrorHandlingState
proof*: seq[byte]
method `$`*(state: SaleFilling): string = "SaleFilling"
method onCancelled*(state: SaleFilling, request: StorageRequest): ?State =
return some State(SaleCancelled())
method onFailed*(state: SaleFilling, request: StorageRequest): ?State =
return some State(SaleFailed())
method onSlotFilled*(state: SaleFilling, requestId: RequestId,
slotIndex: UInt256): ?State =
return some State(SaleFilled())
method run(state: SaleFilling, machine: Machine): Future[?State] {.async.} =
let data = SalesAgent(machine).data
let market = SalesAgent(machine).context.market
await market.fillSlot(data.requestId, data.slotIndex, state.proof)

37
codex/sales/states/finished.nim Normal file
View File

@ -0,0 +1,37 @@
import pkg/chronos
import ../statemachine
import ../salesagent
import ./errorhandling
import ./cancelled
import ./failed
type
SaleFinished* = ref object of ErrorHandlingState
method `$`*(state: SaleFinished): string = "SaleFinished"
method onCancelled*(state: SaleFinished, request: StorageRequest): ?State =
return some State(SaleCancelled())
method onFailed*(state: SaleFinished, request: StorageRequest): ?State =
return some State(SaleFailed())
method run*(state: SaleFinished, machine: Machine): Future[?State] {.async.} =
let agent = SalesAgent(machine)
let data = agent.data
let context = agent.context
if request =? data.request and
slotIndex =? data.slotIndex:
context.proving.add(request.slotId(slotIndex))
if onSale =? context.onSale:
onSale(data.availability, request, slotIndex)
# TODO: Keep track of contract completion using local clock. When contract
# has finished, we need to add back availability to the sales module.
# This will change when the state machine is updated to include the entire
# sales process, as well as when availability is persisted, so leaving it
# as a TODO for now.
await agent.unsubscribe()

35
codex/sales/states/proving.nim Normal file
View File

@ -0,0 +1,35 @@
import ../statemachine
import ../salesagent
import ./errorhandling
import ./filling
import ./cancelled
import ./failed
import ./filled
type
SaleProving* = ref object of ErrorHandlingState
method `$`*(state: SaleProving): string = "SaleProving"
method onCancelled*(state: SaleProving, request: StorageRequest): ?State =
return some State(SaleCancelled())
method onFailed*(state: SaleProving, request: StorageRequest): ?State =
return some State(SaleFailed())
method onSlotFilled*(state: SaleProving, requestId: RequestId,
slotIndex: UInt256): ?State =
return some State(SaleFilled())
method run*(state: SaleProving, machine: Machine): Future[?State] {.async.} =
let data = SalesAgent(machine).data
let context = SalesAgent(machine).context
without request =? data.request:
raiseAssert "no sale request"
without onProve =? context.onProve:
raiseAssert "onProve callback not set"
let proof = await onProve(request, data.slotIndex)
return some State(SaleFilling(proof: proof))

46
codex/sales/states/unknown.nim Normal file
View File

@ -0,0 +1,46 @@
import ../statemachine
import ../salesagent
import ./filled
import ./finished
import ./failed
import ./errored
import ./cancelled
type
SaleUnknown* = ref object of SaleState
SaleUnknownError* = object of CatchableError
UnexpectedSlotError* = object of SaleUnknownError
method `$`*(state: SaleUnknown): string = "SaleUnknown"
method onCancelled*(state: SaleUnknown, request: StorageRequest): ?State =
return some State(SaleCancelled())
method onFailed*(state: SaleUnknown, request: StorageRequest): ?State =
return some State(SaleFailed())
method run*(state: SaleUnknown, machine: Machine): Future[?State] {.async.} =
let agent = SalesAgent(machine)
let data = agent.data
let market = agent.context.market
await agent.retrieveRequest()
await agent.subscribe()
let slotId = slotId(data.requestId, data.slotIndex)
without slotState =? await market.slotState(slotId):
let error = newException(SaleUnknownError, "cannot retrieve slot state")
return some State(SaleErrored(error: error))
case slotState
of SlotState.Free:
let error = newException(UnexpectedSlotError,
"slot state on chain should not be 'free'")
return some State(SaleErrored(error: error))
of SlotState.Filled:
return some State(SaleFilled())
of SlotState.Finished, SlotState.Paid:
return some State(SaleFinished())
of SlotState.Failed:
return some State(SaleFailed())

5
codex/storageproofs/timing/proofs.nim
View File

@ -1,5 +1,6 @@
import pkg/chronos
import pkg/stint
import pkg/questionable
import pkg/upraises
import ./periods
import ../../contracts/requests
@ -26,8 +27,8 @@ method willProofBeRequired*(proofs: Proofs,
id: SlotId): Future[bool] {.base, async.} =
raiseAssert("not implemented")
method getProofEnd*(proofs: Proofs,
id: SlotId): Future[UInt256] {.base, async.} =
method slotState*(proofs: Proofs,
id: SlotId): Future[SlotState] {.base, async.} =
raiseAssert("not implemented")
method submitProof*(proofs: Proofs,

10
codex/utils/asyncspawn.nim Normal file
View File

@ -0,0 +1,10 @@
import pkg/chronos
proc asyncSpawn*(future: Future[void], ignore: type CatchableError) =
proc ignoringError {.async.} =
try:
await future
except ignore:
discard
asyncSpawn ignoringError()

86
codex/utils/asyncstatemachine.nim Normal file
View File

@ -0,0 +1,86 @@
import pkg/questionable
import pkg/chronos
import pkg/chronicles
import pkg/upraises
push: {.upraises:[].}
type
Machine* = ref object of RootObj
state: State
running: Future[void]
scheduled: AsyncQueue[Event]
scheduling: Future[void]
started: bool
State* = ref object of RootObj
Event* = proc(state: State): ?State {.gcsafe, upraises:[].}
proc transition(_: type Event, previous, next: State): Event =
return proc (state: State): ?State =
if state == previous:
return some next
proc schedule*(machine: Machine, event: Event) =
if not machine.started:
return
try:
machine.scheduled.putNoWait(event)
except AsyncQueueFullError:
raiseAssert "unlimited queue is full?!"
method run*(state: State, machine: Machine): Future[?State] {.base, async.} =
discard
method onError*(state: State, error: ref CatchableError): ?State {.base.} =
raise (ref Defect)(msg: "error in state machine: " & error.msg, parent: error)
proc onError(machine: Machine, error: ref CatchableError): Event =
return proc (state: State): ?State =
state.onError(error)
proc run(machine: Machine, state: State) {.async.} =
try:
if next =? await state.run(machine):
machine.schedule(Event.transition(state, next))
except CancelledError:
discard
proc scheduler(machine: Machine) {.async.} =
proc onRunComplete(udata: pointer) {.gcsafe.} =
var fut = cast[FutureBase](udata)
if fut.failed():
machine.schedule(machine.onError(fut.error))
try:
while true:
let event = await machine.scheduled.get()
if next =? event(machine.state):
if not machine.running.isNil:
await machine.running.cancelAndWait()
machine.state = next
machine.running = machine.run(machine.state)
machine.running.addCallback(onRunComplete)
except CancelledError:
discard
proc start*(machine: Machine, initialState: State) =
if machine.started:
return
if machine.scheduled.isNil:
machine.scheduled = newAsyncQueue[Event]()
machine.scheduling = machine.scheduler()
machine.started = true
machine.schedule(Event.transition(machine.state, initialState))
proc stop*(machine: Machine) =
if not machine.started:
return
if not machine.scheduling.isNil:
machine.scheduling.cancel()
if not machine.running.isNil:
machine.running.cancel()
machine.started = false

28
codex/utils/statemachine.nim
View File

@ -1,3 +1,5 @@
import std/typetraits
import pkg/chronicles
import pkg/questionable
import pkg/chronos
import ./optionalcast
@ -62,6 +64,9 @@ type
State* = ref object of RootObj
context: ?StateMachine
method `$`*(state: State): string {.base.} =
(typeof state).name
method enter(state: State) {.base.} =
discard
@ -88,6 +93,8 @@ proc switch*(oldState, newState: State) =
type
AsyncState* = ref object of State
activeTransition: ?Future[void]
StateMachineAsync* = ref object of StateMachine
method enterAsync(state: AsyncState) {.base, async.} =
discard
@ -100,3 +107,24 @@ method enter(state: AsyncState) =
method exit(state: AsyncState) =
asyncSpawn state.exitAsync()
proc switchAsync*(machine: StateMachineAsync, newState: AsyncState) {.async.} =
if state =? (machine.state as AsyncState):
trace "Switching sales state", `from` = $state, to = $newState
if activeTransition =? state.activeTransition and
not activeTransition.completed:
await activeTransition.cancelAndWait()
# should wait for exit before switch. could add a transition option during
# switch if we don't need to wait
await state.exitAsync()
state.context = none StateMachine
else:
trace "Switching state", `from` = "no state", to = $newState
machine.state = some State(newState)
newState.context = some StateMachine(machine)
newState.activeTransition = some newState.enterAsync()
proc switchAsync*(oldState, newState: AsyncState) {.async.} =
if context =? oldState.context:
await StateMachineAsync(context).switchAsync(newState)

54
tests/codex/helpers/mockmarket.nim
View File

@ -2,6 +2,8 @@ import std/sequtils
import std/tables
import std/hashes
import pkg/codex/market
import pkg/codex/contracts/requests
import pkg/codex/contracts/config
export market
export tables
@ -9,23 +11,26 @@ export tables
type
MockMarket* = ref object of Market
activeRequests*: Table[Address, seq[RequestId]]
activeSlots*: Table[Address, seq[SlotId]]
requested*: seq[StorageRequest]
requestEnds*: Table[RequestId, SecondsSince1970]
state*: Table[RequestId, RequestState]
requestState*: Table[RequestId, RequestState]
slotState*: Table[SlotId, SlotState]
fulfilled*: seq[Fulfillment]
filled*: seq[Slot]
filled*: seq[MockSlot]
withdrawn*: seq[RequestId]
signer: Address
subscriptions: Subscriptions
config: MarketplaceConfig
Fulfillment* = object
requestId*: RequestId
proof*: seq[byte]
host*: Address
Slot* = object
MockSlot* = object
requestId*: RequestId
host*: Address
slotIndex*: UInt256
proof*: seq[byte]
host*: Address
Subscriptions = object
onRequest: seq[RequestSubscription]
onFulfillment: seq[FulfillmentSubscription]
@ -60,7 +65,20 @@ proc hash*(requestId: RequestId): Hash =
hash(requestId.toArray)
proc new*(_: type MockMarket): MockMarket =
MockMarket(signer: Address.example)
let config = MarketplaceConfig(
collateral: CollateralConfig(
initialAmount: 100.u256,
minimumAmount: 40.u256,
slashCriterion: 3.u256,
slashPercentage: 10.u256
),
proofs: ProofConfig(
period: 10.u256,
timeout: 5.u256,
downtime: 64.uint8
)
)
MockMarket(signer: Address.example, config: config)
method getSigner*(market: MockMarket): Future[Address] {.async.} =
return market.signer
@ -74,6 +92,9 @@ method requestStorage*(market: MockMarket, request: StorageRequest) {.async.} =
method myRequests*(market: MockMarket): Future[seq[RequestId]] {.async.} =
return market.activeRequests[market.signer]
method mySlots*(market: MockMarket): Future[seq[SlotId]] {.async.} =
return market.activeSlots[market.signer]
method getRequest(market: MockMarket,
id: RequestId): Future[?StorageRequest] {.async.} =
for request in market.requested:
@ -81,15 +102,30 @@ method getRequest(market: MockMarket,
return some request
return none StorageRequest
method getState*(market: MockMarket,
method getRequestFromSlotId*(market: MockMarket,
slotId: SlotId): Future[?StorageRequest] {.async.} =
for slot in market.filled:
if slotId(slot.requestId, slot.slotIndex) == slotId:
return await market.getRequest(slot.requestId)
return none StorageRequest
method requestState*(market: MockMarket,
requestId: RequestId): Future[?RequestState] {.async.} =
return market.state.?[requestId]
return market.requestState.?[requestId]
method slotState*(market: MockMarket,
slotId: SlotId): Future[SlotState] {.async.} =
if market.slotState.hasKey(slotId):
return market.slotState[slotId]
else:
return SlotState.Free
method getRequestEnd*(market: MockMarket,
id: RequestId): Future[SecondsSince1970] {.async.} =
return market.requestEnds[id]
method getHost(market: MockMarket,
method getHost*(market: MockMarket,
requestId: RequestId,
slotIndex: UInt256): Future[?Address] {.async.} =
for slot in market.filled:
@ -130,7 +166,7 @@ proc fillSlot*(market: MockMarket,
slotIndex: UInt256,
proof: seq[byte],
host: Address) =
let slot = Slot(
let slot = MockSlot(
requestId: requestId,
slotIndex: slotIndex,
proof: proof,

15
tests/codex/helpers/mockproofs.nim
View File

@ -11,6 +11,7 @@ type
proofsToBeRequired: HashSet[SlotId]
proofEnds: Table[SlotId, UInt256]
subscriptions: seq[MockSubscription]
slotStates: Table[SlotId, SlotState]
MockSubscription* = ref object of Subscription
proofs: MockProofs
callback: OnProofSubmitted
@ -49,13 +50,6 @@ method willProofBeRequired*(mock: MockProofs,
proc setProofEnd*(mock: MockProofs, id: SlotId, proofEnd: UInt256) =
mock.proofEnds[id] = proofEnd
method getProofEnd*(mock: MockProofs,
id: SlotId): Future[UInt256] {.async.} =
if mock.proofEnds.hasKey(id):
return mock.proofEnds[id]
else:
return UInt256.high
method submitProof*(mock: MockProofs,
id: SlotId,
proof: seq[byte]) {.async.} =
@ -71,3 +65,10 @@ method subscribeProofSubmission*(mock: MockProofs,
method unsubscribe*(subscription: MockSubscription) {.async, upraises:[].} =
subscription.proofs.subscriptions.keepItIf(it != subscription)
method slotState*(mock: MockProofs,
slotId: SlotId): Future[SlotState] {.async.} =
return mock.slotStates[slotId]
proc setSlotState*(mock: MockProofs, slotId: SlotId, state: SlotState) =
mock.slotStates[slotId] = state

29
tests/codex/sales/states/testdownloading.nim Normal file
View File

@ -0,0 +1,29 @@
import std/unittest
import pkg/questionable
import pkg/codex/contracts/requests
import pkg/codex/sales/states/downloading
import pkg/codex/sales/states/cancelled
import pkg/codex/sales/states/failed
import pkg/codex/sales/states/filled
import ../../examples
suite "sales state 'downloading'":
let request = StorageRequest.example
let slotIndex = (request.ask.slots div 2).u256
var state: SaleDownloading
setup:
state = SaleDownloading.new()
test "switches to cancelled state when request expires":
let next = state.onCancelled(request)
check !next of SaleCancelled
test "switches to failed state when request fails":
let next = state.onFailed(request)
check !next of SaleFailed
test "switches to filled state when slot is filled":
let next = state.onSlotFilled(request.id, slotIndex)
check !next of SaleFilled

46
tests/codex/sales/states/testfilled.nim Normal file
View File

@ -0,0 +1,46 @@
import pkg/asynctest
import pkg/codex/contracts/requests
import pkg/codex/sales
import pkg/codex/sales/salesagent
import pkg/codex/sales/salescontext
import pkg/codex/sales/states/filled
import pkg/codex/sales/states/errored
import pkg/codex/sales/states/finished
import ../../helpers/mockmarket
import ../../examples
suite "sales state 'filled'":
let request = StorageRequest.example
let slotIndex = (request.ask.slots div 2).u256
var market: MockMarket
var slot: MockSlot
var agent: SalesAgent
var state: SaleFilled
setup:
market = MockMarket.new()
slot = MockSlot(requestId: request.id,
host: Address.example,
slotIndex: slotIndex,
proof: @[])
let context = SalesContext(market: market)
agent = newSalesAgent(context,
request.id,
slotIndex,
Availability.none,
StorageRequest.none)
state = SaleFilled.new()
test "switches to finished state when slot is filled by me":
slot.host = await market.getSigner()
market.filled = @[slot]
let next = await state.run(agent)
check !next of SaleFinished
test "switches to error state when slot is filled by another host":
slot.host = Address.example
market.filled = @[slot]
let next = await state.run(agent)
check !next of SaleErrored

29
tests/codex/sales/states/testfilling.nim Normal file
View File

@ -0,0 +1,29 @@
import std/unittest
import pkg/questionable
import pkg/codex/contracts/requests
import pkg/codex/sales/states/filling
import pkg/codex/sales/states/cancelled
import pkg/codex/sales/states/failed
import pkg/codex/sales/states/filled
import ../../examples
suite "sales state 'filling'":
let request = StorageRequest.example
let slotIndex = (request.ask.slots div 2).u256
var state: SaleFilling
setup:
state = SaleFilling.new()
test "switches to cancelled state when request expires":
let next = state.onCancelled(request)
check !next of SaleCancelled
test "switches to failed state when request fails":
let next = state.onFailed(request)
check !next of SaleFailed
test "switches to filled state when slot is filled":
let next = state.onSlotFilled(request.id, slotIndex)
check !next of SaleFilled

23
tests/codex/sales/states/testfinished.nim Normal file
View File

@ -0,0 +1,23 @@
import std/unittest
import pkg/questionable
import pkg/codex/contracts/requests
import pkg/codex/sales/states/finished
import pkg/codex/sales/states/cancelled
import pkg/codex/sales/states/failed
import ../../examples
suite "sales state 'finished'":
let request = StorageRequest.example
var state: SaleFinished
setup:
state = SaleFinished.new()
test "switches to cancelled state when request expires":
let next = state.onCancelled(request)
check !next of SaleCancelled
test "switches to failed state when request fails":
let next = state.onFailed(request)
check !next of SaleFailed

30
tests/codex/sales/states/testproving.nim Normal file
View File

@ -0,0 +1,30 @@
import std/unittest
import pkg/questionable
import pkg/codex/contracts/requests
import pkg/codex/sales/states/proving
import pkg/codex/sales/states/cancelled
import pkg/codex/sales/states/failed
import pkg/codex/sales/states/filled
import ../../examples
suite "sales state 'proving'":
let request = StorageRequest.example
let slotIndex = (request.ask.slots div 2).u256
var state: SaleProving
setup:
state = SaleProving.new()
test "switches to cancelled state when request expires":
let next = state.onCancelled(request)
check !next of SaleCancelled
test "switches to failed state when request fails":
let next = state.onFailed(request)
check !next of SaleFailed
test "switches to filled state when slot is filled":
let next = state.onSlotFilled(request.id, slotIndex)
check !next of SaleFilled

61
tests/codex/sales/states/testunknown.nim Normal file
View File

@ -0,0 +1,61 @@
import pkg/asynctest
import pkg/codex/contracts/requests
import pkg/codex/sales
import pkg/codex/sales/salesagent
import pkg/codex/sales/salescontext
import pkg/codex/sales/states/unknown
import pkg/codex/sales/states/errored
import pkg/codex/sales/states/filled
import pkg/codex/sales/states/finished
import pkg/codex/sales/states/failed
import ../../helpers/mockmarket
import ../../examples
suite "sales state 'unknown'":
let request = StorageRequest.example
let slotIndex = (request.ask.slots div 2).u256
let slotId = slotId(request.id, slotIndex)
var market: MockMarket
var agent: SalesAgent
var state: SaleUnknown
setup:
market = MockMarket.new()
let context = SalesContext(market: market)
agent = newSalesAgent(context,
request.id,
slotIndex,
Availability.none,
StorageRequest.none)
state = SaleUnknown.new()
test "switches to error state when on chain state cannot be fetched":
let next = await state.run(agent)
check !next of SaleErrored
test "switches to error state when on chain state is 'free'":
market.slotState[slotId] = SlotState.Free
let next = await state.run(agent)
check !next of SaleErrored
test "switches to filled state when on chain state is 'filled'":
market.slotState[slotId] = SlotState.Filled
let next = await state.run(agent)
check !next of SaleFilled
test "switches to finished state when on chain state is 'finished'":
market.slotState[slotId] = SlotState.Finished
let next = await state.run(agent)
check !next of SaleFinished
test "switches to finished state when on chain state is 'paid'":
market.slotState[slotId] = SlotState.Paid
let next = await state.run(agent)
check !next of SaleFinished
test "switches to failed state when on chain state is 'failed'":
market.slotState[slotId] = SlotState.Failed
let next = await state.run(agent)
check !next of SaleFailed

290
tests/codex/sales/testsales.nim Normal file
View File

@ -0,0 +1,290 @@
import std/sets
import std/sequtils
import std/sugar
import std/times
import pkg/asynctest
import pkg/chronos
import pkg/codex/sales
import pkg/codex/sales/salesdata
import pkg/codex/proving
import ../helpers/mockmarket
import ../helpers/mockclock
import ../helpers/eventually
import ../examples
suite "Sales":
let availability = Availability.init(
size=100.u256,
duration=60.u256,
minPrice=600.u256
)
var request = StorageRequest(
ask: StorageAsk(
slots: 4,
slotSize: 100.u256,
duration: 60.u256,
reward: 10.u256,
),
content: StorageContent(
cid: "some cid"
),
expiry: (getTime() + initDuration(hours=1)).toUnix.u256
)
let proof = exampleProof()
var sales: Sales
var market: MockMarket
var clock: MockClock
var proving: Proving
setup:
market = MockMarket.new()
clock = MockClock.new()
proving = Proving.new()
sales = Sales.new(market, clock, proving)
sales.onStore = proc(request: StorageRequest,
slot: UInt256,
availability: ?Availability) {.async.} =
discard
sales.onProve = proc(request: StorageRequest,
slot: UInt256): Future[seq[byte]] {.async.} =
return proof
await sales.start()
request.expiry = (clock.now() + 42).u256
teardown:
await sales.stop()
test "has no availability initially":
check sales.available.len == 0
test "can add available storage":
let availability1 = Availability.example
let availability2 = Availability.example
sales.add(availability1)
check sales.available.contains(availability1)
sales.add(availability2)
check sales.available.contains(availability1)
check sales.available.contains(availability2)
test "can remove available storage":
sales.add(availability)
sales.remove(availability)
check sales.available.len == 0
test "generates unique ids for storage availability":
let availability1 = Availability.init(1.u256, 2.u256, 3.u256)
let availability2 = Availability.init(1.u256, 2.u256, 3.u256)
check availability1.id != availability2.id
test "makes storage unavailable when matching request comes in":
sales.add(availability)
await market.requestStorage(request)
check eventually sales.available.len == 0
test "ignores request when no matching storage is available":
sales.add(availability)
var tooBig = request
tooBig.ask.slotSize = request.ask.slotSize + 1
await market.requestStorage(tooBig)
await sleepAsync(1.millis)
check eventually sales.available == @[availability]
test "ignores request when reward is too low":
sales.add(availability)
var tooCheap = request
tooCheap.ask.reward = request.ask.reward - 1
await market.requestStorage(tooCheap)
await sleepAsync(1.millis)
check eventually sales.available == @[availability]
test "retrieves and stores data locally":
var storingRequest: StorageRequest
var storingSlot: UInt256
var storingAvailability: Availability
sales.onStore = proc(request: StorageRequest,
slot: UInt256,
availability: ?Availability) {.async.} =
storingRequest = request
storingSlot = slot
check availability.isSome
storingAvailability = !availability
sales.add(availability)
await market.requestStorage(request)
check eventually storingRequest == request
check storingSlot < request.ask.slots.u256
check storingAvailability == availability
test "handles errors during state run":
var saleFailed = false
sales.onStore = proc(request: StorageRequest,
slot: UInt256,
availability: ?Availability) {.async.} =
# raise an exception so machine.onError is called
raise newException(ValueError, "some error")
# onSaleErrored is called in SaleErrored.run
proc onSaleErrored(availability: Availability) =
saleFailed = true
sales.context.onSaleErrored = some onSaleErrored
sales.add(availability)
await market.requestStorage(request)
check eventually saleFailed
test "makes storage available again when data retrieval fails":
let error = newException(IOError, "data retrieval failed")
sales.onStore = proc(request: StorageRequest,
slot: UInt256,
availability: ?Availability) {.async.} =
raise error
sales.add(availability)
await market.requestStorage(request)
await sleepAsync(1.millis)
check eventually sales.available == @[availability]
test "generates proof of storage":
var provingRequest: StorageRequest
var provingSlot: UInt256
sales.onProve = proc(request: StorageRequest,
slot: UInt256): Future[seq[byte]] {.async.} =
provingRequest = request
provingSlot = slot
sales.add(availability)
await market.requestStorage(request)
check eventually provingRequest == request
check provingSlot < request.ask.slots.u256
test "fills a slot":
sales.add(availability)
await market.requestStorage(request)
check eventually market.filled.len == 1
check market.filled[0].requestId == request.id
check market.filled[0].slotIndex < request.ask.slots.u256
check market.filled[0].proof == proof
check market.filled[0].host == await market.getSigner()
test "calls onSale when slot is filled":
var soldAvailability: Availability
var soldRequest: StorageRequest
var soldSlotIndex: UInt256
sales.onSale = proc(availability: ?Availability,
request: StorageRequest,
slotIndex: UInt256) =
if a =? availability:
soldAvailability = a
soldRequest = request
soldSlotIndex = slotIndex
sales.add(availability)
await market.requestStorage(request)
check eventually soldAvailability == availability
check soldRequest == request
check soldSlotIndex < request.ask.slots.u256
test "calls onClear when storage becomes available again":
# fail the proof intentionally to trigger `agent.finish(success=false)`,
# which then calls the onClear callback
sales.onProve = proc(request: StorageRequest,
slot: UInt256): Future[seq[byte]] {.async.} =
raise newException(IOError, "proof failed")
var clearedAvailability: Availability
var clearedRequest: StorageRequest
var clearedSlotIndex: UInt256
sales.onClear = proc(availability: ?Availability,
request: StorageRequest,
slotIndex: UInt256) =
if a =? availability:
clearedAvailability = a
clearedRequest = request
clearedSlotIndex = slotIndex
sales.add(availability)
await market.requestStorage(request)
check eventually clearedAvailability == availability
check clearedRequest == request
check clearedSlotIndex < request.ask.slots.u256
test "makes storage available again when other host fills the slot":
let otherHost = Address.example
sales.onStore = proc(request: StorageRequest,
slot: UInt256,
availability: ?Availability) {.async.} =
await sleepAsync(chronos.hours(1))
sales.add(availability)
await market.requestStorage(request)
await sleepAsync(1.millis)
for slotIndex in 0..<request.ask.slots:
market.fillSlot(request.id, slotIndex.u256, proof, otherHost)
await sleepAsync(chronos.seconds(2))
check sales.available == @[availability]
test "makes storage available again when request expires":
sales.onStore = proc(request: StorageRequest,
slot: UInt256,
availability: ?Availability) {.async.} =
await sleepAsync(chronos.hours(1))
sales.add(availability)
await market.requestStorage(request)
await sleepAsync(1.millis)
clock.set(request.expiry.truncate(int64))
check eventually (sales.available == @[availability])
test "adds proving for slot when slot is filled":
var soldSlotIndex: UInt256
sales.onSale = proc(availability: ?Availability,
request: StorageRequest,
slotIndex: UInt256) =
soldSlotIndex = slotIndex
check proving.slots.len == 0
sales.add(availability)
await market.requestStorage(request)
check eventually proving.slots.len == 1
check proving.slots.contains(request.slotId(soldSlotIndex))
test "loads active slots from market":
let me = await market.getSigner()
request.ask.slots = 2
market.requested = @[request]
market.requestState[request.id] = RequestState.New
proc fillSlot(slotIdx: UInt256 = 0.u256) {.async.} =
let address = await market.getSigner()
let slot = MockSlot(requestId: request.id,
slotIndex: slotIdx,
proof: proof,
host: address)
market.filled.add slot
market.slotState[slotId(request.id, slotIdx)] = SlotState.Filled
let slot0 = MockSlot(requestId: request.id,
slotIndex: 0.u256,
proof: proof,
host: me)
await fillSlot(slot0.slotIndex)
let slot1 = MockSlot(requestId: request.id,
slotIndex: 1.u256,
proof: proof,
host: me)
await fillSlot(slot1.slotIndex)
market.activeSlots[me] = @[request.slotId(0.u256), request.slotId(1.u256)]
market.requested = @[request]
market.activeRequests[me] = @[request.id]
await sales.load()
let expected = SalesData(requestId: request.id,
availability: none Availability,
request: some request)
# because sales.load() calls agent.start, we won't know the slotIndex
# randomly selected for the agent, and we also won't know the value of
# `failed`/`fulfilled`/`cancelled` futures, so we need to compare
# the properties we know
# TODO: when calling sales.load(), slot index should be restored and not
# randomly re-assigned, so this may no longer be needed
proc `==` (data0, data1: SalesData): bool =
return data0.requestId == data1.requestId and
data0.availability == data1.availability and
data0.request == data1.request
check eventually sales.agents.len == 2
check sales.agents.all(agent => agent.data == expected)

165
tests/codex/sales/testsalesagent.nim Normal file
View File

@ -0,0 +1,165 @@
import std/sets
import std/sequtils
import std/sugar
import std/times
import pkg/asynctest
import pkg/chronos
import pkg/codex/sales
import pkg/codex/sales/salesagent
import pkg/codex/sales/salescontext
import pkg/codex/sales/statemachine
import pkg/codex/sales/states/errorhandling
import pkg/codex/proving
import ../helpers/mockmarket
import ../helpers/mockclock
import ../helpers/eventually
import ../examples
var onCancelCalled = false
var onFailedCalled = false
var onSlotFilledCalled = false
var onErrorCalled = false
type
MockState = ref object of SaleState
MockErrorState = ref object of ErrorHandlingState
method `$`*(state: MockState): string = "MockState"
method onCancelled*(state: MockState, request: StorageRequest): ?State =
onCancelCalled = true
method onFailed*(state: MockState, request: StorageRequest): ?State =
onFailedCalled = true
method onSlotFilled*(state: MockState, requestId: RequestId,
slotIndex: UInt256): ?State =
onSlotFilledCalled = true
method onError*(state: MockErrorState, err: ref CatchableError): ?State =
onErrorCalled = true
method run*(state: MockErrorState, machine: Machine): Future[?State] {.async.} =
raise newException(ValueError, "failure")
suite "Sales agent":
let availability = Availability.init(
size=100.u256,
duration=60.u256,
minPrice=600.u256
)
var request = StorageRequest(
ask: StorageAsk(
slots: 4,
slotSize: 100.u256,
duration: 60.u256,
reward: 10.u256,
),
content: StorageContent(
cid: "some cid"
),
expiry: (getTime() + initDuration(hours=1)).toUnix.u256
)
var agent: SalesAgent
var context: SalesContext
var slotIndex: UInt256
var market: MockMarket
var clock: MockClock
setup:
market = MockMarket.new()
clock = MockClock.new()
context = SalesContext(market: market, clock: clock)
slotIndex = 0.u256
onCancelCalled = false
onFailedCalled = false
onSlotFilledCalled = false
agent = newSalesAgent(context,
request.id,
slotIndex,
some availability,
some request)
request.expiry = (getTime() + initDuration(hours=1)).toUnix.u256
teardown:
await agent.stop()
test "can retrieve request":
agent = newSalesAgent(context,
request.id,
slotIndex,
some availability,
none StorageRequest)
market.requested = @[request]
await agent.retrieveRequest()
check agent.data.request == some request
test "subscribe assigns subscriptions/futures":
await agent.subscribe()
check not agent.data.cancelled.isNil
check not agent.data.failed.isNil
check not agent.data.fulfilled.isNil
check not agent.data.slotFilled.isNil
test "unsubscribe deassigns subscriptions/futures":
await agent.subscribe()
await agent.unsubscribe()
check agent.data.cancelled.isNil
check agent.data.failed.isNil
check agent.data.fulfilled.isNil
check agent.data.slotFilled.isNil
test "subscribe can be called multiple times, without overwriting subscriptions/futures":
await agent.subscribe()
let cancelled = agent.data.cancelled
let failed = agent.data.failed
let fulfilled = agent.data.fulfilled
let slotFilled = agent.data.slotFilled
await agent.subscribe()
check cancelled == agent.data.cancelled
check failed == agent.data.failed
check fulfilled == agent.data.fulfilled
check slotFilled == agent.data.slotFilled
test "unsubscribe can be called multiple times":
await agent.subscribe()
await agent.unsubscribe()
await agent.unsubscribe()
test "subscribe can be called when request expiry has lapsed":
# succeeds when agent.data.fulfilled.isNil
request.expiry = (getTime() - initDuration(seconds=1)).toUnix.u256
agent.data.request = some request
check agent.data.fulfilled.isNil
await agent.subscribe()
test "current state onCancelled called when cancel emitted":
let state = MockState.new()
agent.start(state)
await agent.subscribe()
clock.set(request.expiry.truncate(int64))
check eventually onCancelCalled
test "cancelled future is finished (cancelled) when fulfillment emitted":
agent.start(MockState.new())
await agent.subscribe()
market.emitRequestFulfilled(request.id)
check eventually agent.data.cancelled.cancelled()
test "current state onFailed called when failed emitted":
agent.start(MockState.new())
await agent.subscribe()
market.emitRequestFailed(request.id)
check eventually onFailedCalled
test "current state onSlotFilled called when slot filled emitted":
agent.start(MockState.new())
await agent.subscribe()
market.emitSlotFilled(request.id, slotIndex)
check eventually onSlotFilledCalled
test "ErrorHandlingState.onError can be overridden at the state level":
agent.start(MockErrorState.new())
check eventually onErrorCalled

8
tests/codex/sales/teststates.nim Normal file
View File

@ -0,0 +1,8 @@
import ./states/testunknown
import ./states/testdownloading
import ./states/testfilling
import ./states/testfinished
import ./states/testproving
import ./states/testfilled
{.warning[UnusedImport]: off.}

5
tests/codex/testproving.nim
View File

@ -47,6 +47,7 @@ suite "Proving":
proc onProofRequired(id: SlotId) =
called = true
proving.onProofRequired = onProofRequired
proofs.setSlotState(id, SlotState.Filled)
proofs.setProofRequired(id, true)
await proofs.advanceToNextPeriod()
check eventually called
@ -59,6 +60,8 @@ suite "Proving":
proc onProofRequired(id: SlotId) =
callbackIds.add(id)
proving.onProofRequired = onProofRequired
proofs.setSlotState(id1, SlotState.Filled)
proofs.setSlotState(id2, SlotState.Filled)
proofs.setProofRequired(id1, true)
await proofs.advanceToNextPeriod()
check eventually callbackIds == @[id1]
@ -76,6 +79,7 @@ suite "Proving":
proving.onProofRequired = onProofRequired
proofs.setProofRequired(id, false)
proofs.setProofToBeRequired(id, true)
proofs.setSlotState(id, SlotState.Filled)
await proofs.advanceToNextPeriod()
check eventually called
@ -90,6 +94,7 @@ suite "Proving":
proving.onProofRequired = onProofRequired
proofs.setProofRequired(id, true)
await proofs.advanceToNextPeriod()
proofs.setSlotState(id, SlotState.Finished)
check eventually (not proving.slots.contains(id))
check not called

28
tests/codex/testpurchasing.nim
View File

@ -141,11 +141,11 @@ suite "Purchasing state machine":
let request1, request2, request3, request4, request5 = StorageRequest.example
market.requested = @[request1, request2, request3, request4, request5]
market.activeRequests[me] = @[request1.id, request2.id, request3.id, request4.id, request5.id]
market.state[request1.id] = RequestState.New
market.state[request2.id] = RequestState.Started
market.state[request3.id] = RequestState.Cancelled
market.state[request4.id] = RequestState.Finished
market.state[request5.id] = RequestState.Failed
market.requestState[request1.id] = RequestState.New
market.requestState[request2.id] = RequestState.Started
market.requestState[request3.id] = RequestState.Cancelled
market.requestState[request4.id] = RequestState.Finished
market.requestState[request5.id] = RequestState.Failed
# ensure the started state doesn't error, giving a false positive test result
market.requestEnds[request2.id] = clock.now() - 1
@ -162,7 +162,7 @@ suite "Purchasing state machine":
let request = StorageRequest.example
let purchase = Purchase.new(request, market, clock)
market.requested = @[request]
market.state[request.id] = RequestState.New
market.requestState[request.id] = RequestState.New
purchase.switch(PurchaseUnknown())
check (purchase.state as PurchaseSubmitted).isSome
@ -171,7 +171,7 @@ suite "Purchasing state machine":
let purchase = Purchase.new(request, market, clock)
market.requestEnds[request.id] = clock.now() + request.ask.duration.truncate(int64)
market.requested = @[request]
market.state[request.id] = RequestState.Started
market.requestState[request.id] = RequestState.Started
purchase.switch(PurchaseUnknown())
check (purchase.state as PurchaseStarted).isSome
@ -179,7 +179,7 @@ suite "Purchasing state machine":
let request = StorageRequest.example
let purchase = Purchase.new(request, market, clock)
market.requested = @[request]
market.state[request.id] = RequestState.Cancelled
market.requestState[request.id] = RequestState.Cancelled
purchase.switch(PurchaseUnknown())
check (purchase.state as PurchaseErrored).isSome
check purchase.error.?msg == "Purchase cancelled due to timeout".some
@ -188,7 +188,7 @@ suite "Purchasing state machine":
let request = StorageRequest.example
let purchase = Purchase.new(request, market, clock)
market.requested = @[request]
market.state[request.id] = RequestState.Finished
market.requestState[request.id] = RequestState.Finished
purchase.switch(PurchaseUnknown())
check (purchase.state as PurchaseFinished).isSome
@ -196,7 +196,7 @@ suite "Purchasing state machine":
let request = StorageRequest.example
let purchase = Purchase.new(request, market, clock)
market.requested = @[request]
market.state[request.id] = RequestState.Failed
market.requestState[request.id] = RequestState.Failed
purchase.switch(PurchaseUnknown())
check (purchase.state as PurchaseErrored).isSome
check purchase.error.?msg == "Purchase failed".some
@ -206,7 +206,7 @@ suite "Purchasing state machine":
let request = StorageRequest.example
market.requested = @[request]
market.activeRequests[me] = @[request.id]
market.state[request.id] = RequestState.Started
market.requestState[request.id] = RequestState.Started
market.requestEnds[request.id] = clock.now() + request.ask.duration.truncate(int64)
await purchasing.load()
@ -226,7 +226,7 @@ suite "Purchasing state machine":
let request = StorageRequest.example
market.requested = @[request]
market.activeRequests[me] = @[request.id]
market.state[request.id] = RequestState.Started
market.requestState[request.id] = RequestState.Started
market.requestEnds[request.id] = clock.now() + request.ask.duration.truncate(int64)
await purchasing.load()
@ -234,7 +234,7 @@ suite "Purchasing state machine":
clock.advance(request.ask.duration.truncate(int64))
# now check the result
proc getState: ?PurchaseState =
proc requestState: ?PurchaseState =
purchasing.getPurchase(PurchaseId(request.id)).?state as PurchaseState
check eventually (getState() as PurchaseFinished).isSome
check eventually (requestState() as PurchaseFinished).isSome

214
tests/codex/testsales.nim
View File

@ -1,212 +1,4 @@
import std/sets
import pkg/asynctest
import pkg/chronos
import pkg/codex/contracts/requests
import pkg/codex/proving
import pkg/codex/sales
import ./helpers/mockmarket
import ./helpers/mockclock
import ./helpers/eventually
import ./examples
import ./sales/testsales
import ./sales/teststates
suite "Sales":
let availability = Availability.init(
size=100.u256,
duration=60.u256,
minPrice=600.u256
)
var request = StorageRequest(
ask: StorageAsk(
slots: 4,
slotSize: 100.u256,
duration: 60.u256,
reward: 10.u256,
),
content: StorageContent(
cid: "some cid"
)
)
let proof = exampleProof()
var sales: Sales
var market: MockMarket
var clock: MockClock
var proving: Proving
setup:
market = MockMarket.new()
clock = MockClock.new()
proving = Proving.new()
sales = Sales.new(market, clock, proving)
sales.onStore = proc(request: StorageRequest,
slot: UInt256,
availability: Availability) {.async.} =
discard
sales.onProve = proc(request: StorageRequest,
slot: UInt256): Future[seq[byte]] {.async.} =
return proof
await sales.start()
request.expiry = (clock.now() + 42).u256
teardown:
await sales.stop()
test "has no availability initially":
check sales.available.len == 0
test "can add available storage":
let availability1 = Availability.example
let availability2 = Availability.example
sales.add(availability1)
check sales.available.contains(availability1)
sales.add(availability2)
check sales.available.contains(availability1)
check sales.available.contains(availability2)
test "can remove available storage":
sales.add(availability)
sales.remove(availability)
check sales.available.len == 0
test "generates unique ids for storage availability":
let availability1 = Availability.init(1.u256, 2.u256, 3.u256)
let availability2 = Availability.init(1.u256, 2.u256, 3.u256)
check availability1.id != availability2.id
test "makes storage unavailable when matching request comes in":
sales.add(availability)
await market.requestStorage(request)
check sales.available.len == 0
test "ignores request when no matching storage is available":
sales.add(availability)
var tooBig = request
tooBig.ask.slotSize = request.ask.slotSize + 1
await market.requestStorage(tooBig)
check sales.available == @[availability]
test "ignores request when reward is too low":
sales.add(availability)
var tooCheap = request
tooCheap.ask.reward = request.ask.reward - 1
await market.requestStorage(tooCheap)
check sales.available == @[availability]
test "retrieves and stores data locally":
var storingRequest: StorageRequest
var storingSlot: UInt256
var storingAvailability: Availability
sales.onStore = proc(request: StorageRequest,
slot: UInt256,
availability: Availability) {.async.} =
storingRequest = request
storingSlot = slot
storingAvailability = availability
sales.add(availability)
await market.requestStorage(request)
check storingRequest == request
check storingSlot < request.ask.slots.u256
check storingAvailability == availability
test "makes storage available again when data retrieval fails":
let error = newException(IOError, "data retrieval failed")
sales.onStore = proc(request: StorageRequest,
slot: UInt256,
availability: Availability) {.async.} =
raise error
sales.add(availability)
await market.requestStorage(request)
check sales.available == @[availability]
test "generates proof of storage":
var provingRequest: StorageRequest
var provingSlot: UInt256
sales.onProve = proc(request: StorageRequest,
slot: UInt256): Future[seq[byte]] {.async.} =
provingRequest = request
provingSlot = slot
sales.add(availability)
await market.requestStorage(request)
check provingRequest == request
check provingSlot < request.ask.slots.u256
test "fills a slot":
sales.add(availability)
await market.requestStorage(request)
check market.filled.len == 1
check market.filled[0].requestId == request.id
check market.filled[0].slotIndex < request.ask.slots.u256
check market.filled[0].proof == proof
check market.filled[0].host == await market.getSigner()
test "calls onSale when slot is filled":
var soldAvailability: Availability
var soldRequest: StorageRequest
var soldSlotIndex: UInt256
sales.onSale = proc(availability: Availability,
request: StorageRequest,
slotIndex: UInt256) =
soldAvailability = availability
soldRequest = request
soldSlotIndex = slotIndex
sales.add(availability)
await market.requestStorage(request)
check soldAvailability == availability
check soldRequest == request
check soldSlotIndex < request.ask.slots.u256
test "calls onClear when storage becomes available again":
# fail the proof intentionally to trigger `agent.finish(success=false)`,
# which then calls the onClear callback
sales.onProve = proc(request: StorageRequest,
slot: UInt256): Future[seq[byte]] {.async.} =
raise newException(IOError, "proof failed")
var clearedAvailability: Availability
var clearedRequest: StorageRequest
var clearedSlotIndex: UInt256
sales.onClear = proc(availability: Availability,
request: StorageRequest,
slotIndex: UInt256) =
clearedAvailability = availability
clearedRequest = request
clearedSlotIndex = slotIndex
sales.add(availability)
await market.requestStorage(request)
check clearedAvailability == availability
check clearedRequest == request
check clearedSlotIndex < request.ask.slots.u256
test "makes storage available again when other host fills the slot":
let otherHost = Address.example
sales.onStore = proc(request: StorageRequest,
slot: UInt256,
availability: Availability) {.async.} =
await sleepAsync(1.hours)
sales.add(availability)
await market.requestStorage(request)
for slotIndex in 0..<request.ask.slots:
market.fillSlot(request.id, slotIndex.u256, proof, otherHost)
check sales.available == @[availability]
test "makes storage available again when request expires":
sales.onStore = proc(request: StorageRequest,
slot: UInt256,
availability: Availability) {.async.} =
await sleepAsync(1.hours)
sales.add(availability)
await market.requestStorage(request)
clock.set(request.expiry.truncate(int64))
check eventually (sales.available == @[availability])
test "adds proving for slot when slot is filled":
var soldSlotIndex: UInt256
sales.onSale = proc(availability: Availability,
request: StorageRequest,
slotIndex: UInt256) =
soldSlotIndex = slotIndex
check proving.slots.len == 0
sales.add(availability)
await market.requestStorage(request)
check proving.slots.len == 1
check proving.slots.contains(request.slotId(soldSlotIndex))
{.warning[UnusedImport]: off.}

2
tests/codex/testutils.nim
View File

@ -1,5 +1,7 @@
import ./utils/teststatemachine
import ./utils/teststatemachineasync
import ./utils/testoptionalcast
import ./utils/testkeyutils
import ./utils/testasyncstatemachine
{.warning[UnusedImport]: off.}

110
tests/codex/utils/testasyncstatemachine.nim Normal file
View File

@ -0,0 +1,110 @@
import pkg/asynctest
import pkg/questionable
import pkg/chronos
import pkg/upraises
import codex/utils/asyncstatemachine
import ../helpers/eventually
type
State1 = ref object of State
State2 = ref object of State
State3 = ref object of State
State4 = ref object of State
var runs, cancellations, errors = [0, 0, 0, 0]
method run(state: State1, machine: Machine): Future[?State] {.async.} =
inc runs[0]
return some State(State2.new())
method run(state: State2, machine: Machine): Future[?State] {.async.} =
inc runs[1]
try:
await sleepAsync(1.hours)
except CancelledError:
inc cancellations[1]
raise
method run(state: State3, machine: Machine): Future[?State] {.async.} =
inc runs[2]
method run(state: State4, machine: Machine): Future[?State] {.async.} =
inc runs[3]
raise newException(ValueError, "failed")
method onMoveToNextStateEvent*(state: State): ?State {.base, upraises:[].} =
discard
method onMoveToNextStateEvent(state: State2): ?State =
some State(State3.new())
method onMoveToNextStateEvent(state: State3): ?State =
some State(State1.new())
method onError(state: State1, error: ref CatchableError): ?State =
inc errors[0]
method onError(state: State2, error: ref CatchableError): ?State =
inc errors[1]
method onError(state: State3, error: ref CatchableError): ?State =
inc errors[2]
method onError(state: State4, error: ref CatchableError): ?State =
inc errors[3]
some State(State2.new())
suite "async state machines":
var machine: Machine
proc moveToNextStateEvent(state: State): ?State =
state.onMoveToNextStateEvent()
setup:
runs = [0, 0, 0, 0]
cancellations = [0, 0, 0, 0]
errors = [0, 0, 0, 0]
machine = Machine.new()
test "should call run on start state":
machine.start(State1.new())
check eventually runs[0] == 1
test "moves to next state when run completes":
machine.start(State1.new())
check eventually runs == [1, 1, 0, 0]
test "state2 moves to state3 on event":
machine.start(State2.new())
machine.schedule(moveToNextStateEvent)
check eventually runs == [0, 1, 1, 0]
test "state transition will cancel the running state":
machine.start(State2.new())
machine.schedule(moveToNextStateEvent)
check eventually cancellations == [0, 1, 0, 0]
test "scheduled events are handled one after the other":
machine.start(State2.new())
machine.schedule(moveToNextStateEvent)
machine.schedule(moveToNextStateEvent)
check eventually runs == [1, 2, 1, 0]
test "stops scheduling and current state":
machine.start(State2.new())
await sleepAsync(1.millis)
machine.stop()
machine.schedule(moveToNextStateEvent)
await sleepAsync(1.millis)
check runs == [0, 1, 0, 0]
check cancellations == [0, 1, 0, 0]
test "forwards errors to error handler":
machine.start(State4.new())
check eventually errors == [0, 0, 0, 1] and runs == [0, 1, 0, 1]
test "error handler ignores CancelledError":
machine.start(State2.new())
machine.schedule(moveToNextStateEvent)
check eventually cancellations == [0, 1, 0, 0]
check errors == [0, 0, 0, 0]

30
tests/codex/utils/teststatemachineasync.nim Normal file
View File

@ -0,0 +1,30 @@
import pkg/asynctest
import pkg/chronos
import pkg/questionable
import codex/utils/statemachine
type
AsyncMachine = ref object of StateMachineAsync
LongRunningStart = ref object of AsyncState
LongRunningFinish = ref object of AsyncState
LongRunningError = ref object of AsyncState
Callback = proc(): Future[void] {.gcsafe.}
proc triggerIn(time: Duration, cb: Callback) {.async.} =
await sleepAsync(time)
await cb()
method enterAsync(state: LongRunningStart) {.async.} =
proc cb() {.async.} =
await state.switchAsync(LongRunningFinish())
asyncSpawn triggerIn(500.milliseconds, cb)
await sleepAsync(1.seconds)
await state.switchAsync(LongRunningError())
suite "async state machines":
test "can cancel a state":
let am = AsyncMachine()
await am.switchAsync(LongRunningStart())
await sleepAsync(2.seconds)
check (am.state as LongRunningFinish).isSome

5
tests/contracts/testContracts.nim
View File

@ -34,8 +34,9 @@ ethersuite "Marketplace contracts":
await token.mint(await client.getAddress(), 1_000_000_000.u256)
await token.mint(await host.getAddress(), 1000_000_000.u256)
collateral = await marketplace.collateral()
periodicity = Periodicity(seconds: await marketplace.proofPeriod())
let config = await marketplace.config()
collateral = config.collateral.initialAmount
periodicity = Periodicity(seconds: config.proofs.period)
request = StorageRequest.example
request.client = await client.getAddress()

59
tests/contracts/testMarket.nim
View File

@ -1,6 +1,7 @@
import std/options
import pkg/chronos
import pkg/ethers/testing
import pkg/stew/byteutils
import codex/contracts
import codex/contracts/testtoken
import codex/storageproofs
@ -24,12 +25,13 @@ ethersuite "On-Chain Market":
token = TestToken.new(!deployment.address(TestToken), provider.getSigner())
await token.mint(accounts[0], 1_000_000_000.u256)
let collateral = await marketplace.collateral()
let config = await marketplace.config()
let collateral = config.collateral.initialAmount
await token.approve(marketplace.address, collateral)
await marketplace.deposit(collateral)
market = OnChainMarket.new(marketplace)
periodicity = Periodicity(seconds: await marketplace.proofPeriod())
periodicity = Periodicity(seconds: config.proofs.period)
request = StorageRequest.example
request.client = accounts[0]
@ -187,14 +189,13 @@ ethersuite "On-Chain Market":
for slotIndex in 0..request.ask.maxSlotLoss:
let slotId = request.slotId(slotIndex.u256)
while true:
try:
await waitUntilProofRequired(slotId)
let missingPeriod = periodicity.periodOf(await provider.currentTime())
await provider.advanceTime(periodicity.seconds)
await marketplace.markProofAsMissing(slotId, missingPeriod)
except ProviderError as e:
if e.revertReason == "Slot empty":
break
let slotState = await marketplace.slotState(slotId)
if slotState == SlotState.Free:
break
await waitUntilProofRequired(slotId)
let missingPeriod = periodicity.periodOf(await provider.currentTime())
await provider.advanceTime(periodicity.seconds)
await marketplace.markProofAsMissing(slotId, missingPeriod)
check receivedIds == @[request.id]
await subscription.unsubscribe()
@ -230,9 +231,45 @@ ethersuite "On-Chain Market":
await market.requestStorage(request2)
check (await market.myRequests()) == @[request.id, request2.id]
test "retrieves correct request state when request is unknown":
check (await market.requestState(request.id)) == none RequestState
test "can retrieve request state":
await token.approve(marketplace.address, request.price)
await market.requestStorage(request)
for slotIndex in 0..<request.ask.slots:
await market.fillSlot(request.id, slotIndex.u256, proof)
check (await market.getState(request.id)) == some RequestState.Started
check (await market.requestState(request.id)) == some RequestState.Started
test "can retrieve active slots":
await token.approve(marketplace.address, request.price)
await market.requestStorage(request)
await market.fillSlot(request.id, slotIndex - 1, proof)
await market.fillSlot(request.id, slotIndex, proof)
let slotId1 = request.slotId(slotIndex - 1)
let slotId2 = request.slotId(slotIndex)
check (await market.mySlots()) == @[slotId1, slotId2]
test "returns none when slot is empty":
await token.approve(marketplace.address, request.price)
await market.requestStorage(request)
let slotId = request.slotId(slotIndex)
check (await market.getRequestFromSlotId(slotId)) == none StorageRequest
test "can retrieve request details from slot id":
await token.approve(marketplace.address, request.price)
await market.requestStorage(request)
await market.fillSlot(request.id, slotIndex, proof)
let slotId = request.slotId(slotIndex)
check (await market.getRequestFromSlotId(slotId)) == some request
test "retrieves correct slot state when request is unknown":
let slotId = request.slotId(slotIndex)
check (await market.slotState(slotId)) == SlotState.Free
test "retrieves correct slot state once filled":
await token.approve(marketplace.address, request.price)
await market.requestStorage(request)
await market.fillSlot(request.id, slotIndex, proof)
let slotId = request.slotId(slotIndex)
check (await market.slotState(slotId)) == SlotState.Filled

11
tests/contracts/testProofs.nim
View File

@ -1,4 +1,5 @@
import codex/contracts
import codex/contracts/testtoken
import ../ethertest
import ./examples
import ./time
@ -18,7 +19,8 @@ ethersuite "On-Chain Proofs":
test "can retrieve proof periodicity":
let periodicity = await proofs.periodicity()
let periodLength = await marketplace.proofPeriod()
let config = await marketplace.config()
let periodLength = config.proofs.period
check periodicity.seconds == periodLength
test "supports checking whether proof is required now":
@ -27,8 +29,8 @@ ethersuite "On-Chain Proofs":
test "supports checking whether proof is required soon":
check (await proofs.willProofBeRequired(contractId)) == false
test "retrieves proof end time":
check (await proofs.getProofEnd(contractId)) == 0.u256
test "retrieves correct slot state when request is unknown":
check (await proofs.slotState(SlotId.example)) == SlotState.Free
test "submits proofs":
await proofs.submitProof(contractId, proof)
@ -55,6 +57,3 @@ ethersuite "On-Chain Proofs":
test "proof will not be required when slot is empty":
check not await proofs.willProofBeRequired(contractId)
test "proof end is zero when slot is empty":
check (await proofs.getProofEnd(contractId)) == 0.u256

3
tests/integration/tokens.nim
View File

@ -12,4 +12,5 @@ proc mint*(signer: Signer, amount = 1_000_000.u256) {.async.} =
proc deposit*(signer: Signer) {.async.} =
## Deposits sufficient collateral into the Marketplace contract.
let marketplace = Marketplace.new(!deployment().address(Marketplace), signer)
await marketplace.deposit(await marketplace.collateral())
let config = await marketplace.config()
await marketplace.deposit(config.collateral.initialAmount)

2
vendor/codex-contracts-eth vendored

@ -1 +1 @@
Subproject commit ce335d0568b4d056def6570f7d5d5c0d6a840083
Subproject commit cde543626236bd48188354d842cbe1513052c560