nim-codex/codex/validation.nim

112 lines
3.6 KiB
Nim
Raw Normal View History

Validator (#387) * [contracts] Add SlotFreed event * [integration] allow test node to be stopped twice * [cli] add --validator option * [contracts] remove dead code * [contracts] instantiate OnChainMarket and OnChainClock only once * [contracts] add Validation * [sales] remove duplicate import * [market] add missing import * [market] subscribe to all SlotFilled events * [market] add freeSlot() * [sales] fix warnings * [market] subscribe to SlotFreed events * [contracts] fix warning * [validator] keep track of filled slots * [validation] remove slots that have ended * [proving] absorb Proofs into Market Both Proofs and Market are abstractions around the Marketplace contract, having them separately is more trouble than it's worth at the moment. * [market] add markProofAsMissing() * [clock] speed up waiting for clock in tests * [validator] mark proofs as missing * [timer] fix error on node shutdown * [cli] handle --persistence and --validator separately * [market] allow retrieval of proof timeout value * [validator] do not subscribe to SlotFreed events Freed slots are already handled in removeSlotsThatHaveEnded(), and onSlotsFreed() interfered with its iterator. * [validator] Start validation at the start of a new period To decrease the likelihood that we hit the validation timeout. * [validator] do not mark proofs as missing after timeout * [market] check whether proof can be marked as missing * [validator] simplify validation Simulate a transaction to mark proof as missing, instead of trying to keep track of all the conditions that may lead to a proof being marked as missing. * [build] use nim-ethers PR #40 Uses "pending" blocktag instead of "latest" blocktag for better simulation of transactions before sending them. https://github.com/status-im/nim-ethers/pull/40 * [integration] integration test for validator * [validator] monitor a maximum number of slots Adds cli parameter --validator-max-slots. * [market] fix missing collateral argument After rebasing, add the new argument to fillSlot calls. * [build] update to nim-ethers 0.2.5 * [validator] use Set instead of Table to keep track of slots * [validator] add logging * [validator] add test for slot failure * [market] use "pending" blocktag to use more up to date block time * [contracts] remove unused import * [validator] fix: wait until after period ends The smart contract checks that 'end < block.timestamp', so we need to wait until the block timestamp is greater than the period end.
2023-04-19 13:06:00 +00:00
import std/sets
import std/sequtils
import pkg/chronos
import pkg/chronicles
Validator (#387) * [contracts] Add SlotFreed event * [integration] allow test node to be stopped twice * [cli] add --validator option * [contracts] remove dead code * [contracts] instantiate OnChainMarket and OnChainClock only once * [contracts] add Validation * [sales] remove duplicate import * [market] add missing import * [market] subscribe to all SlotFilled events * [market] add freeSlot() * [sales] fix warnings * [market] subscribe to SlotFreed events * [contracts] fix warning * [validator] keep track of filled slots * [validation] remove slots that have ended * [proving] absorb Proofs into Market Both Proofs and Market are abstractions around the Marketplace contract, having them separately is more trouble than it's worth at the moment. * [market] add markProofAsMissing() * [clock] speed up waiting for clock in tests * [validator] mark proofs as missing * [timer] fix error on node shutdown * [cli] handle --persistence and --validator separately * [market] allow retrieval of proof timeout value * [validator] do not subscribe to SlotFreed events Freed slots are already handled in removeSlotsThatHaveEnded(), and onSlotsFreed() interfered with its iterator. * [validator] Start validation at the start of a new period To decrease the likelihood that we hit the validation timeout. * [validator] do not mark proofs as missing after timeout * [market] check whether proof can be marked as missing * [validator] simplify validation Simulate a transaction to mark proof as missing, instead of trying to keep track of all the conditions that may lead to a proof being marked as missing. * [build] use nim-ethers PR #40 Uses "pending" blocktag instead of "latest" blocktag for better simulation of transactions before sending them. https://github.com/status-im/nim-ethers/pull/40 * [integration] integration test for validator * [validator] monitor a maximum number of slots Adds cli parameter --validator-max-slots. * [market] fix missing collateral argument After rebasing, add the new argument to fillSlot calls. * [build] update to nim-ethers 0.2.5 * [validator] use Set instead of Table to keep track of slots * [validator] add logging * [validator] add test for slot failure * [market] use "pending" blocktag to use more up to date block time * [contracts] remove unused import * [validator] fix: wait until after period ends The smart contract checks that 'end < block.timestamp', so we need to wait until the block timestamp is greater than the period end.
2023-04-19 13:06:00 +00:00
import ./market
import ./clock
export market
export sets
type
Validation* = ref object
slots: HashSet[SlotId]
maxSlots: int
clock: Clock
market: Market
subscriptions: seq[Subscription]
running: Future[void]
periodicity: Periodicity
proofTimeout: UInt256
logScope:
topics = "codex validator"
proc new*(
_: type Validation,
clock: Clock,
market: Market,
maxSlots: int
): Validation =
## Create a new Validation instance
Validator (#387) * [contracts] Add SlotFreed event * [integration] allow test node to be stopped twice * [cli] add --validator option * [contracts] remove dead code * [contracts] instantiate OnChainMarket and OnChainClock only once * [contracts] add Validation * [sales] remove duplicate import * [market] add missing import * [market] subscribe to all SlotFilled events * [market] add freeSlot() * [sales] fix warnings * [market] subscribe to SlotFreed events * [contracts] fix warning * [validator] keep track of filled slots * [validation] remove slots that have ended * [proving] absorb Proofs into Market Both Proofs and Market are abstractions around the Marketplace contract, having them separately is more trouble than it's worth at the moment. * [market] add markProofAsMissing() * [clock] speed up waiting for clock in tests * [validator] mark proofs as missing * [timer] fix error on node shutdown * [cli] handle --persistence and --validator separately * [market] allow retrieval of proof timeout value * [validator] do not subscribe to SlotFreed events Freed slots are already handled in removeSlotsThatHaveEnded(), and onSlotsFreed() interfered with its iterator. * [validator] Start validation at the start of a new period To decrease the likelihood that we hit the validation timeout. * [validator] do not mark proofs as missing after timeout * [market] check whether proof can be marked as missing * [validator] simplify validation Simulate a transaction to mark proof as missing, instead of trying to keep track of all the conditions that may lead to a proof being marked as missing. * [build] use nim-ethers PR #40 Uses "pending" blocktag instead of "latest" blocktag for better simulation of transactions before sending them. https://github.com/status-im/nim-ethers/pull/40 * [integration] integration test for validator * [validator] monitor a maximum number of slots Adds cli parameter --validator-max-slots. * [market] fix missing collateral argument After rebasing, add the new argument to fillSlot calls. * [build] update to nim-ethers 0.2.5 * [validator] use Set instead of Table to keep track of slots * [validator] add logging * [validator] add test for slot failure * [market] use "pending" blocktag to use more up to date block time * [contracts] remove unused import * [validator] fix: wait until after period ends The smart contract checks that 'end < block.timestamp', so we need to wait until the block timestamp is greater than the period end.
2023-04-19 13:06:00 +00:00
Validation(clock: clock, market: market, maxSlots: maxSlots)
proc slots*(validation: Validation): seq[SlotId] =
validation.slots.toSeq
proc getCurrentPeriod(validation: Validation): UInt256 =
return validation.periodicity.periodOf(validation.clock.now().u256)
proc waitUntilNextPeriod(validation: Validation) {.async.} =
let period = validation.getCurrentPeriod()
let periodEnd = validation.periodicity.periodEnd(period)
trace "Waiting until next period", currentPeriod = period
await validation.clock.waitUntil(periodEnd.truncate(int64) + 1)
proc subscribeSlotFilled(validation: Validation) {.async.} =
proc onSlotFilled(requestId: RequestId, slotIndex: UInt256) =
let slotId = slotId(requestId, slotIndex)
if slotId notin validation.slots:
if validation.slots.len < validation.maxSlots:
trace "Adding slot", slotId = $slotId
Validator (#387) * [contracts] Add SlotFreed event * [integration] allow test node to be stopped twice * [cli] add --validator option * [contracts] remove dead code * [contracts] instantiate OnChainMarket and OnChainClock only once * [contracts] add Validation * [sales] remove duplicate import * [market] add missing import * [market] subscribe to all SlotFilled events * [market] add freeSlot() * [sales] fix warnings * [market] subscribe to SlotFreed events * [contracts] fix warning * [validator] keep track of filled slots * [validation] remove slots that have ended * [proving] absorb Proofs into Market Both Proofs and Market are abstractions around the Marketplace contract, having them separately is more trouble than it's worth at the moment. * [market] add markProofAsMissing() * [clock] speed up waiting for clock in tests * [validator] mark proofs as missing * [timer] fix error on node shutdown * [cli] handle --persistence and --validator separately * [market] allow retrieval of proof timeout value * [validator] do not subscribe to SlotFreed events Freed slots are already handled in removeSlotsThatHaveEnded(), and onSlotsFreed() interfered with its iterator. * [validator] Start validation at the start of a new period To decrease the likelihood that we hit the validation timeout. * [validator] do not mark proofs as missing after timeout * [market] check whether proof can be marked as missing * [validator] simplify validation Simulate a transaction to mark proof as missing, instead of trying to keep track of all the conditions that may lead to a proof being marked as missing. * [build] use nim-ethers PR #40 Uses "pending" blocktag instead of "latest" blocktag for better simulation of transactions before sending them. https://github.com/status-im/nim-ethers/pull/40 * [integration] integration test for validator * [validator] monitor a maximum number of slots Adds cli parameter --validator-max-slots. * [market] fix missing collateral argument After rebasing, add the new argument to fillSlot calls. * [build] update to nim-ethers 0.2.5 * [validator] use Set instead of Table to keep track of slots * [validator] add logging * [validator] add test for slot failure * [market] use "pending" blocktag to use more up to date block time * [contracts] remove unused import * [validator] fix: wait until after period ends The smart contract checks that 'end < block.timestamp', so we need to wait until the block timestamp is greater than the period end.
2023-04-19 13:06:00 +00:00
validation.slots.incl(slotId)
let subscription = await validation.market.subscribeSlotFilled(onSlotFilled)
validation.subscriptions.add(subscription)
proc removeSlotsThatHaveEnded(validation: Validation) {.async.} =
var ended: HashSet[SlotId]
for slotId in validation.slots:
let state = await validation.market.slotState(slotId)
if state != SlotState.Filled:
trace "Removing slot", slot = $slotId
Validator (#387) * [contracts] Add SlotFreed event * [integration] allow test node to be stopped twice * [cli] add --validator option * [contracts] remove dead code * [contracts] instantiate OnChainMarket and OnChainClock only once * [contracts] add Validation * [sales] remove duplicate import * [market] add missing import * [market] subscribe to all SlotFilled events * [market] add freeSlot() * [sales] fix warnings * [market] subscribe to SlotFreed events * [contracts] fix warning * [validator] keep track of filled slots * [validation] remove slots that have ended * [proving] absorb Proofs into Market Both Proofs and Market are abstractions around the Marketplace contract, having them separately is more trouble than it's worth at the moment. * [market] add markProofAsMissing() * [clock] speed up waiting for clock in tests * [validator] mark proofs as missing * [timer] fix error on node shutdown * [cli] handle --persistence and --validator separately * [market] allow retrieval of proof timeout value * [validator] do not subscribe to SlotFreed events Freed slots are already handled in removeSlotsThatHaveEnded(), and onSlotsFreed() interfered with its iterator. * [validator] Start validation at the start of a new period To decrease the likelihood that we hit the validation timeout. * [validator] do not mark proofs as missing after timeout * [market] check whether proof can be marked as missing * [validator] simplify validation Simulate a transaction to mark proof as missing, instead of trying to keep track of all the conditions that may lead to a proof being marked as missing. * [build] use nim-ethers PR #40 Uses "pending" blocktag instead of "latest" blocktag for better simulation of transactions before sending them. https://github.com/status-im/nim-ethers/pull/40 * [integration] integration test for validator * [validator] monitor a maximum number of slots Adds cli parameter --validator-max-slots. * [market] fix missing collateral argument After rebasing, add the new argument to fillSlot calls. * [build] update to nim-ethers 0.2.5 * [validator] use Set instead of Table to keep track of slots * [validator] add logging * [validator] add test for slot failure * [market] use "pending" blocktag to use more up to date block time * [contracts] remove unused import * [validator] fix: wait until after period ends The smart contract checks that 'end < block.timestamp', so we need to wait until the block timestamp is greater than the period end.
2023-04-19 13:06:00 +00:00
ended.incl(slotId)
validation.slots.excl(ended)
proc markProofAsMissing(validation: Validation,
slotId: SlotId,
period: Period) {.async.} =
logScope:
currentPeriod = validation.getCurrentPeriod()
Validator (#387) * [contracts] Add SlotFreed event * [integration] allow test node to be stopped twice * [cli] add --validator option * [contracts] remove dead code * [contracts] instantiate OnChainMarket and OnChainClock only once * [contracts] add Validation * [sales] remove duplicate import * [market] add missing import * [market] subscribe to all SlotFilled events * [market] add freeSlot() * [sales] fix warnings * [market] subscribe to SlotFreed events * [contracts] fix warning * [validator] keep track of filled slots * [validation] remove slots that have ended * [proving] absorb Proofs into Market Both Proofs and Market are abstractions around the Marketplace contract, having them separately is more trouble than it's worth at the moment. * [market] add markProofAsMissing() * [clock] speed up waiting for clock in tests * [validator] mark proofs as missing * [timer] fix error on node shutdown * [cli] handle --persistence and --validator separately * [market] allow retrieval of proof timeout value * [validator] do not subscribe to SlotFreed events Freed slots are already handled in removeSlotsThatHaveEnded(), and onSlotsFreed() interfered with its iterator. * [validator] Start validation at the start of a new period To decrease the likelihood that we hit the validation timeout. * [validator] do not mark proofs as missing after timeout * [market] check whether proof can be marked as missing * [validator] simplify validation Simulate a transaction to mark proof as missing, instead of trying to keep track of all the conditions that may lead to a proof being marked as missing. * [build] use nim-ethers PR #40 Uses "pending" blocktag instead of "latest" blocktag for better simulation of transactions before sending them. https://github.com/status-im/nim-ethers/pull/40 * [integration] integration test for validator * [validator] monitor a maximum number of slots Adds cli parameter --validator-max-slots. * [market] fix missing collateral argument After rebasing, add the new argument to fillSlot calls. * [build] update to nim-ethers 0.2.5 * [validator] use Set instead of Table to keep track of slots * [validator] add logging * [validator] add test for slot failure * [market] use "pending" blocktag to use more up to date block time * [contracts] remove unused import * [validator] fix: wait until after period ends The smart contract checks that 'end < block.timestamp', so we need to wait until the block timestamp is greater than the period end.
2023-04-19 13:06:00 +00:00
try:
if await validation.market.canProofBeMarkedAsMissing(slotId, period):
trace "Marking proof as missing", slotId = $slotId, periodProofMissed = period
Validator (#387) * [contracts] Add SlotFreed event * [integration] allow test node to be stopped twice * [cli] add --validator option * [contracts] remove dead code * [contracts] instantiate OnChainMarket and OnChainClock only once * [contracts] add Validation * [sales] remove duplicate import * [market] add missing import * [market] subscribe to all SlotFilled events * [market] add freeSlot() * [sales] fix warnings * [market] subscribe to SlotFreed events * [contracts] fix warning * [validator] keep track of filled slots * [validation] remove slots that have ended * [proving] absorb Proofs into Market Both Proofs and Market are abstractions around the Marketplace contract, having them separately is more trouble than it's worth at the moment. * [market] add markProofAsMissing() * [clock] speed up waiting for clock in tests * [validator] mark proofs as missing * [timer] fix error on node shutdown * [cli] handle --persistence and --validator separately * [market] allow retrieval of proof timeout value * [validator] do not subscribe to SlotFreed events Freed slots are already handled in removeSlotsThatHaveEnded(), and onSlotsFreed() interfered with its iterator. * [validator] Start validation at the start of a new period To decrease the likelihood that we hit the validation timeout. * [validator] do not mark proofs as missing after timeout * [market] check whether proof can be marked as missing * [validator] simplify validation Simulate a transaction to mark proof as missing, instead of trying to keep track of all the conditions that may lead to a proof being marked as missing. * [build] use nim-ethers PR #40 Uses "pending" blocktag instead of "latest" blocktag for better simulation of transactions before sending them. https://github.com/status-im/nim-ethers/pull/40 * [integration] integration test for validator * [validator] monitor a maximum number of slots Adds cli parameter --validator-max-slots. * [market] fix missing collateral argument After rebasing, add the new argument to fillSlot calls. * [build] update to nim-ethers 0.2.5 * [validator] use Set instead of Table to keep track of slots * [validator] add logging * [validator] add test for slot failure * [market] use "pending" blocktag to use more up to date block time * [contracts] remove unused import * [validator] fix: wait until after period ends The smart contract checks that 'end < block.timestamp', so we need to wait until the block timestamp is greater than the period end.
2023-04-19 13:06:00 +00:00
await validation.market.markProofAsMissing(slotId, period)
else:
let inDowntime {.used.} = await validation.market.inDowntime(slotId)
trace "Proof not missing", checkedPeriod = period, inDowntime
Validator (#387) * [contracts] Add SlotFreed event * [integration] allow test node to be stopped twice * [cli] add --validator option * [contracts] remove dead code * [contracts] instantiate OnChainMarket and OnChainClock only once * [contracts] add Validation * [sales] remove duplicate import * [market] add missing import * [market] subscribe to all SlotFilled events * [market] add freeSlot() * [sales] fix warnings * [market] subscribe to SlotFreed events * [contracts] fix warning * [validator] keep track of filled slots * [validation] remove slots that have ended * [proving] absorb Proofs into Market Both Proofs and Market are abstractions around the Marketplace contract, having them separately is more trouble than it's worth at the moment. * [market] add markProofAsMissing() * [clock] speed up waiting for clock in tests * [validator] mark proofs as missing * [timer] fix error on node shutdown * [cli] handle --persistence and --validator separately * [market] allow retrieval of proof timeout value * [validator] do not subscribe to SlotFreed events Freed slots are already handled in removeSlotsThatHaveEnded(), and onSlotsFreed() interfered with its iterator. * [validator] Start validation at the start of a new period To decrease the likelihood that we hit the validation timeout. * [validator] do not mark proofs as missing after timeout * [market] check whether proof can be marked as missing * [validator] simplify validation Simulate a transaction to mark proof as missing, instead of trying to keep track of all the conditions that may lead to a proof being marked as missing. * [build] use nim-ethers PR #40 Uses "pending" blocktag instead of "latest" blocktag for better simulation of transactions before sending them. https://github.com/status-im/nim-ethers/pull/40 * [integration] integration test for validator * [validator] monitor a maximum number of slots Adds cli parameter --validator-max-slots. * [market] fix missing collateral argument After rebasing, add the new argument to fillSlot calls. * [build] update to nim-ethers 0.2.5 * [validator] use Set instead of Table to keep track of slots * [validator] add logging * [validator] add test for slot failure * [market] use "pending" blocktag to use more up to date block time * [contracts] remove unused import * [validator] fix: wait until after period ends The smart contract checks that 'end < block.timestamp', so we need to wait until the block timestamp is greater than the period end.
2023-04-19 13:06:00 +00:00
except CancelledError:
raise
except CatchableError as e:
error "Marking proof as missing failed", msg = e.msg
Validator (#387) * [contracts] Add SlotFreed event * [integration] allow test node to be stopped twice * [cli] add --validator option * [contracts] remove dead code * [contracts] instantiate OnChainMarket and OnChainClock only once * [contracts] add Validation * [sales] remove duplicate import * [market] add missing import * [market] subscribe to all SlotFilled events * [market] add freeSlot() * [sales] fix warnings * [market] subscribe to SlotFreed events * [contracts] fix warning * [validator] keep track of filled slots * [validation] remove slots that have ended * [proving] absorb Proofs into Market Both Proofs and Market are abstractions around the Marketplace contract, having them separately is more trouble than it's worth at the moment. * [market] add markProofAsMissing() * [clock] speed up waiting for clock in tests * [validator] mark proofs as missing * [timer] fix error on node shutdown * [cli] handle --persistence and --validator separately * [market] allow retrieval of proof timeout value * [validator] do not subscribe to SlotFreed events Freed slots are already handled in removeSlotsThatHaveEnded(), and onSlotsFreed() interfered with its iterator. * [validator] Start validation at the start of a new period To decrease the likelihood that we hit the validation timeout. * [validator] do not mark proofs as missing after timeout * [market] check whether proof can be marked as missing * [validator] simplify validation Simulate a transaction to mark proof as missing, instead of trying to keep track of all the conditions that may lead to a proof being marked as missing. * [build] use nim-ethers PR #40 Uses "pending" blocktag instead of "latest" blocktag for better simulation of transactions before sending them. https://github.com/status-im/nim-ethers/pull/40 * [integration] integration test for validator * [validator] monitor a maximum number of slots Adds cli parameter --validator-max-slots. * [market] fix missing collateral argument After rebasing, add the new argument to fillSlot calls. * [build] update to nim-ethers 0.2.5 * [validator] use Set instead of Table to keep track of slots * [validator] add logging * [validator] add test for slot failure * [market] use "pending" blocktag to use more up to date block time * [contracts] remove unused import * [validator] fix: wait until after period ends The smart contract checks that 'end < block.timestamp', so we need to wait until the block timestamp is greater than the period end.
2023-04-19 13:06:00 +00:00
proc markProofsAsMissing(validation: Validation) {.async.} =
for slotId in validation.slots:
let previousPeriod = validation.getCurrentPeriod() - 1
await validation.markProofAsMissing(slotId, previousPeriod)
proc run(validation: Validation) {.async.} =
trace "Validation started"
try:
while true:
await validation.waitUntilNextPeriod()
await validation.removeSlotsThatHaveEnded()
await validation.markProofsAsMissing()
except CancelledError:
trace "Validation stopped"
discard
except CatchableError as e:
error "Validation failed", msg = e.msg
proc start*(validation: Validation) {.async.} =
validation.periodicity = await validation.market.periodicity()
validation.proofTimeout = await validation.market.proofTimeout()
await validation.subscribeSlotFilled()
validation.running = validation.run()
proc stop*(validation: Validation) {.async.} =
await validation.running.cancelAndWait()
while validation.subscriptions.len > 0:
let subscription = validation.subscriptions.pop()
await subscription.unsubscribe()