Commit Graph

80 Commits

Author SHA1 Message Date
Eric e8e9820d5b
feat: add `--payout-address` (#870)
* feat: add `--payout-address`

Allows SPs to be paid out to a separate address, keeping their profits secure.
Supports https://github.com/codex-storage/codex-contracts-eth/pull/144 in the nim-codex client.

* Remove optional payoutAddress

Change --payout-address so that it is no longer optional. There is no longer an overload in `Marketplace.sol` for `fillSlot` accepting no `payoutAddress`.

* Update integration tests to include --payout-address

* move payoutAddress from fillSlot to freeSlot

* Update integration tests to use required payoutAddress

- to make payoutAddress required, the integration tests needed to avoid building the cli params until just before starting the node, otherwise if cli params were added ad-hoc, there would be an error after a non-required parameter was added before a required parameter.

* support client payout address

- withdrawFunds requires a withdrawAddress parameter, directs payouts for withdrawing of client funds (for a cancelled request) to go to that address.

* fix integration test

adds --payout-address to validators

* refactor: support withdrawFunds and freeSlot optional parameters

- withdrawFunds has an optional parameter for withdrawRecipient
- freeSlot has optional parameters for rewardRecipient and collateralRecipient
- change --payout-address to --reward-recipient to match contract signature naming

* Revert "Update integration tests to include --payout-address"

This reverts commit 8f9535cf35b0f2b183ac4013a7ed11b246486964.
There are some valid improvements to the integration tests, but they can be handled in a separate PR.

* small fix

* bump contracts to fix marketplace spec

* bump codex-contracts-eth, now rebased on master

* bump codex-contracts-eth

now that feat/reward-address has been merged to master

* clean up, comments
2024-09-17 04:18:15 +00:00
Eric 63ee4ca0f9
refactor(marketplace): generic querying of historical marketplace events (#872)
* refactor(marketplace): move marketplace events to the Market abstraction

Move marketplace contract events to the Market abstraction so the types can be shared across all modules that call the Market abstraction.

* Remove unneeded conversion

* Switch to generic implementation of event querying

* change parent type to MarketplaceEvent
2024-08-16 02:55:25 +00:00
Eric eeb048e386
chore: add `downtimeProduct` config parameter (#867)
* chore: add `downtimeProduct` config parameter

* bump codex-contracts-eth to master
2024-08-15 02:31:02 +00:00
markspanbroek 3046b7636c
Cleanup integration tests (#757)
* integration: move REST API tests into their own module

* integration: move upload and download tests into their own module

* integration: move purchasing tests into their own module

* integration: move marketplace tests to the right module

* integration: mine a block *after* starting nodes

To ensure that tests involving multiple nodes do
not start with out-of-sync clocks

* Fix: do not swallow CancellationErrors

* integration: avoid underflow in UInt256

* network: remove unnecessary error handling

No Exceptions can occur, only Defects, because everything
is asyncSpawned.

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>

* network: do not raise in asyncSpawned proc

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2024-05-23 15:29:30 +00:00
Adam Uhlíř e208d0b13a
feat: one confirmation for all transactions (#802)
* fix: make sure requestStorage is mined

* fix: correct place to plug confirm

* test: fixing contracts tests

* feat: one confirmation for all transactions

* fix: don't wait for confirmations only mined block
2024-05-14 15:40:29 +00:00
Adam Uhlíř 1a0d2d424e
feat: expiry specified with number of seconds (#793) 2024-05-06 17:35:46 +02:00
Eric 43f0bf1c1c
refactor: add MarketError (#670)
* Add MarketError

Add MarketError and convert all EthersErrors (ProviderError, SignerError) to MarketError

* Include token contract call in conversion of ethers error
2024-03-21 00:30:04 +00:00
markspanbroek 4d069599c9
Use real verifier (#737)
* use a real verifying contract address

* contracts: cleanup

* marketplacesuite: set correct circuit files, interval mining

* Proofs tests updates

Contains changes to get the proving tests working reliably.

* integration: use correct circom artifacts for creating proofs

* integration: cleanup

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2024-03-12 14:20:42 +00:00
markspanbroek 8589e63d34
Update contract (#734)
* Update codex-contracts-eth

* contracts: update G2Point definition

* integration: disable automatic advancing of time

reason: makes reasoning about timing in tests harder,
because the period is set to 60 seconds in the
marketplace configuration, but this code switches to
a new period every 500 milliseconds

* integration: fix parameters of marketplace payouts test

* integration: update test settings

* integration: fix typo

* integration: workaround for hardhat issue

Subscriptions expire after 5 minutes when using
websockets. Use http and polling instead.

* integration: remove origDatasetSizeInBlocks

* integration: fix proof parameters for test

* integration: do not log output by default

* integration: fix failure rate in test

* integration: fix warning

* integration: include clock in logs

* integration: allow for more periods

5 periods was cutting it close, if we get too much
pointer downtime, then the test would fail
2024-03-12 08:18:25 +00:00
markspanbroek d1658d7b77
Fix on-chain clock (#732)
* market: use `pending` blocktag when querying onchain state

* clock: use wall clock in integration tests

reason: we'll need to wait for the next period in
integration tests, and we can't do that if the
time doesn't advance

* clock: remove unused field

* integration: use pending block time to get current time

* clock: fix on-chain clock for hardhat

Only use 'latest' block for updates
Only update the first time you see a block

* integration: do not start tests with a very outdated block

* integration: allow for longer expiry period
2024-03-11 16:57:20 +00:00
Eric 61ec4275c8
Revert "remove call to `waitFor` from clock.now and make it async (#691)" (#707)
This reverts commit 591be9446a.
2024-02-16 22:12:16 +00:00
markspanbroek 2cf892c467
Smart contracts update: Groth16Proof instead of bytes (#683)
* Smart contracts update: Groth16Proof instead of bytes

* Use dummy verifier for now, until we can create ZK proofs

* Fix tests: submit proof only when slot is filled

* Submit dummy proofs for now

* More detailed log when proof submission failed

* Use dummy verifier for integration tests

For now at least

* Fix mistake in blanket renaming to ethProvider

* Update to latest codex-contracts-eth

* feat: zkey-hash from chain

* Fix zkeyHash

---------

Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
2024-02-07 07:50:35 +01:00
Giuliano Mega 591be9446a
remove call to `waitFor` from clock.now and make it async (#691) 2024-02-05 08:36:28 -03:00
Eric de88fd2c53
feat: create logging proxy (#663)
* implement a logging proxy

The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks

* Rename logging -> logutils to avoid ambiguity with common names

* clean up

* add setProperty for JsonRecord, remove nim-json-serialization conflict

* Allow specifying textlines and json format separately

Not specifying a LogFormat will apply the formatting to both textlines and json sinks.

Specifying a LogFormat will apply the formatting to only that sink.

* remove unneeded usages of std/json

We only need to import utils/json instead of std/json

* move serialization from rest/json to utils/json so it can be shared

* fix NoColors ambiguity

Was causing unit tests to fail on Windows.

* Remove nre usage to fix Windows error

Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)

* Add logutils module doc

* Shorten logutils.formatIt for `NBytes`

Both json and textlines formatIt were not needed, and could be combined into one formatIt

* remove debug integration test config

debug output and logformat of json for integration test logs

* Use ## module doc to support docgen

* bump nim-poseidon2 to export fromBytes

Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening.

* fixes to get compiling after rebasing master

* Add support for Result types being logged using formatIt
2024-01-22 23:35:03 -08:00
Eric b186afbc4f
bump ethers, remove upraises from callbacks (#664) 2024-01-09 15:51:34 +11:00
Eric f293082ae9
Reverts logging-proxy, commit 27f585eb6f (#660) 2023-12-20 13:24:40 +11:00
Eric 27f585eb6f
feat: create logging proxy (#653)
* implement a logging proxy

The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks

* Rename logging -> logutils to avoid ambiguity with common names

* clean up

* add setProperty for JsonRecord, remove nim-json-serialization conflict

* Allow specifying textlines and json format separately

Not specifying a LogFormat will apply the formatting to both textlines and json sinks.

Specifying a LogFormat will apply the formatting to only that sink.

* remove unneeded usages of std/json

We only need to import utils/json instead of std/json

* move serialization from rest/json to utils/json so it can be shared

* fix NoColors ambiguity

Was causing unit tests to fail on Windows.

* Remove nre usage to fix Windows error

Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)

* Add logutils module doc

* Shorten logutils.formatIt for `NBytes`

Both json and textlines formatIt were not needed, and could be combined into one formatIt

* remove debug integration test config

debug output and logformat of json for integration test logs

* Use ## module doc to support docgen
2023-12-19 22:12:47 +00:00
Eric df6b9c6760
Workaround for Hardhat last block timestamp bug (#644)
* Workaround for Hardhat timestamp bug

Likely due to a Hardhat bug in which the callbacks for subscription events are called and awaited before updating its local understanding of the last block time, Hardhat will report a block time in the `newHeads` event that is generally 1 second before the time reported from `getLatestBlock.timestamp`. This was causing issues with the OnChainClock's offset and therefore the `now()` used by the `OnChainClock` would sometimes be off by a second (or more), causing tests to fail.

This commit introduce a `codex_use_hardhat` compilation flag, that when set, will always get the latest block timestamp from Hardhat via the `getLatestBlock.timestamp` RPC call for `OnChainClock.now` calls. Otherwise, the last block timestamp reported in the `newHeads` event will be used.

Update the docker dist tests compilation flag for simulated proof failures (it was not correct), and explicitly add the `codex_use_hardhat=false` for clarity.

* enable simulated proof failures for coverage

* comment out failing test on linux -- will be replaced

* bump codex contracts eth

* add back clock offset for non-hardhat cases

* bump codex-contracts-eth

increases pointer by 67 blocks each period increase

* Add `codex_use_hardhat` flag to coverage tests
2023-12-20 08:06:24 +11:00
Eric 2c621e0421
chore: improve log information (#656)
* Add proof downtime to validator logs

* Add previous state to state machine transition log
2023-12-19 04:29:18 +00:00
Ben Bierens a9f8090bb4
Add challenge to prove callback (#649)
* Sets up passing proof challenge to onProve callback

* Implements tests for proof challenge

* Removes unused import
2023-12-11 11:29:15 +01:00
Eric 3907ca4095
Add get active slot /slots/{slotId} to REST api, use utils/json (#645)
* Add get active slot /slots/{slotId} to REST api, use utils/json

- Add endpoint /slots/{slotId} to get an active SalesAgent from the Sales module. Used in integration tests to test when a sale has reached a certain state. Those integration test changes will be included in a larger PR, coming later.
- Add OpenAPI changes for new endpoint and associated components
- Use utils/json instead of nim-json-serialization. Required exemption of imports from several packages that export nim-json-serialization by default.

* Only except `toJson` from import/export of chronicles
2023-12-07 12:16:36 +11:00
markspanbroek b62ccf5a8a
Update Request: remove PoR, add merkle root (#590)
* [build] update codex-contracts-eth

* [contracts] update request parameters

- Remove PoR parameters and totalChunks
- Add merkleRoot
2023-11-27 12:25:01 +00:00
Adam Uhlíř 79fce39dbf
fix: require expiry from api (#629)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-11-22 11:35:26 +00:00
Adam Uhlíř 8681a40ee7
feat: update expiry when data downloaded and slot filled (#619)
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-11-22 10:09:12 +00:00
Adam Uhlíř 2fc71cf81b
feat: partial payouts for cancelled requests (#561) 2023-10-24 10:12:54 +00:00
Adam Uhlíř c28627d16f
chore(logging): purchases logging (#567) 2023-10-19 10:12:49 +02:00
Adam Uhlíř 5f4218ba1c
fix: use increaseAllowance for token approving (#565) 2023-10-16 13:09:54 +02:00
markspanbroek 71cd35112b
Taiko L2 (#483)
* [docs] fix two client scenario: add missing collateral

* [integration] separate step to wait for node to be started

* [cli] add option to specify ethereum private key

* Remove unused imports

* Fix warnings

* [integration] move type definitions to correct place

* [integration] wait a bit longer for a node to start in debug mode

When e.g. running against Taiko test net rpc, the node start
takes longer

* [integration] simplify handling of codex node and client

* [integration] add Taiko integration test

* [contracts] await token approval confirmation before next tx

* [contracts] deployment address of marketplace on Taiko

* [cli] --eth-private-key now takes a file name

Instead of supplying the private key on the command line,
expect the private key to be in a file with the correct
permissions.

* [utils] Fixes undeclared `activeChroniclesStream` on Windows

* [build] update nim-ethers to include PR #52

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [cli] Better error messages when reading eth private key

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [integration] simplify reading of cmd line arguments

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [build] update to latest version of nim-ethers

* [contracts] updated contract address for Taiko L2

* [build] update codex contracts to latest version

---------

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
2023-09-13 16:17:56 +02:00
Eric 37b3d99c3d
Improve integration testing client (CodexClient) and json serialization (#514)
* Improve integration testing client (CodexClient) and json serialization

The current client used for integration testing against the REST endpoints for Codex accepts and passes primitive types. This caused a hard to diagnose bug where a `uint` was not being deserialized correctly.

In addition, the json de/serializing done between the CodexClient and REST client was not easy to read and was not tested.

These changes bring non-primitive types to most of the CodexClient functions, allowing us to lean on the compiler to ensure we're providing correct typings. More importantly, a json de/serialization util was created as a drop-in replacement for the std/json lib, with the main two differences being that field serialization is opt-in (instead of opt-out as in the case of json_serialization) and serialization errors are captured and logged, making debugging serialization issues much easier.

* Update integration test to use nodes=2 and tolerance=1

* clean up
2023-09-01 15:44:41 +10:00
Adam Uhlíř f459a2c6f6
refactor: merging proving module into sales (#469)
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2023-08-21 12:26:43 +02:00
Eric 1d161d383e
Slot queue (#455)
## Slot queue
Adds a slot queue, as per the [slot queue design](https://github.com/codex-storage/codex-research/blob/master/design/sales.md#slot-queue).

Any time storage is requested, all slots from that request are immediately added to the queue. Finished, Canclled, Failed requests remove all slots with that request id from the queue. SlotFreed events add a new slot to the queue and SlotFilled events remove the slot from the queue. This allows popping of a slot each time one is processed, making things much simpler.

When an entire request of slots is added to the queue, the slot indices are shuffled randomly to hopefully prevent nodes that pick up the same storage requested event from clashing on the first processed slot index. This allowed removal of assigning a random slot index in the SalePreparing state and it also ensured that all SalesAgents will have a slot index assigned to them at the start thus the removal of the optional slotIndex.

Remove slotId from SlotFreed event as it was not being used. RequestId and slotIndex were added to the SlotFreed event earlier and those are now being used

The slot queue invariant that prioritises queue items added to the queue relies on a scoring mechanism to sort them based on the [sort order in the design document](https://github.com/codex-storage/codex-research/blob/master/design/sales.md#sort-order).

When a storage request is handled by the sales module, a slot index was randomly assigned and then the slot was filled. Now, a random slot index is only assigned when adding an entire request to the slot queue. Additionally, the slot is checked that its state is `SlotState.Free` before continuing with the download process.

SlotQueue should always ensure the underlying AsyncHeapQueue has one less than the maximum items, ensuring the SlotQueue can always have space to add an additional item regardless if it’s full or not.

Constructing `SlotQueue.workers` in `SlotQueue.new` calls `newAsyncQueue` which causes side effects, so the construction call had to be moved to `SlotQueue.start`.

Prevent loading request from contract (network request) if there is an existing item in queue for that request.

Check availability before adding request to queue.

Add ability to query market contract for past events. When new availabilities are added, the `onReservationAdded` callback is triggered in which past `StorageRequested` events are queried, and those slots are added to the queue (filtered by availability on `push` and filtered by state in `SalePreparing`).

#### Request Workers
Limit the concurrent requests being processed in the queue by using a limited pool of workers (default = 3). Workers are in a data structure of type `AsyncQueue[SlotQueueWorker]`. This allows us to await a `popFirst` for available workers inside of the main SlotQueue event loop

Add an `onCleanUp` that stops the agents and removes them from the sales module agent list. `onCleanUp` is called from sales end states (eg ignored, cancelled, finished, failed, errored).

Add a `doneProcessing` future to `SlotQueueWorker` to be completed in the `OnProcessSlot` callback. Each `doneProcessing` future created is cancelled and awaited in `SlotQueue.stop` (thanks to `TrackableFuturees`), which forced `stop` to become async.
  - Cancel dispatched workers and the `onProcessSlot` callbacks, prevents zombie callbacks

#### Add TrackableFutures
Allow tracking of futures in a module so they can be cancelled at a later time. Useful for asyncSpawned futures, but works for any future.

### Sales module
The sales module needed to subscribe to request events to ensure that the request queue was managed correctly on each event. In the process of doing this, the sales agents were updated to avoid subscribing to events in each agent, and instead dispatch received events from the sales module to all created sales agents. This would prevent memory leaks on having too many eventemitters subscribed to.
  - prevent removal of agents from sales module while stopping, otherwise the agents seq len is modified while iterating

An additional sales agent state was added, `SalePreparing`, that handles all state machine setup, such as retrieving the request and subscribing to events that were previously in the `SaleDownloading` state.

Once agents have parked in an end state (eg ignored, cancelled, finished, failed, errored), they were not getting cleaned up and the sales module was keeping a handle on their reference. An `onCleanUp` callback was created to be called after the state machine enters an end state, which could prevent a memory leak if the number of requests coming in is high.

Move the SalesAgent callback raises pragmas from the Sales module to the proc definition in SalesAgent. This avoids having to catch `Exception`.
  - remove unneeded error handling as pragmas were moved

Move sales.subscriptions from an object containing named subscriptions to a `seq[Subscription]` directly on the sales object.

Sales tests: shut down repo after sales stop, to fix SIGABRT in CI

### Add async Promise API
  - modelled after JavaScript Promise API
  - alternative to `asyncSpawn` that allows handling of async calls in a synchronous context (including access to the synchronous closure) with less additional procs to be declared
  - Write less code, catch errors that would otherwise defect in asyncspawn, and execute a callback after completion
  - Add cancellation callbacks to utils/then, ensuring cancellations are handled properly

## Dependencies
- bump codex-contracts-eth to support slot queue (https://github.com/codex-storage/codex-contracts-eth/pull/61)
- bump nim-ethers to 0.5.0
- Bump nim-json-rpc submodule to 0bf2bcb

---------

Co-authored-by: Jaremy Creechley <creechley@gmail.com>
2023-07-25 12:50:30 +10:00
markspanbroek 3879ec8e3a
[clock] waitUntil() completes immediately when block arrives (#475)
Previously it could take up to one second to complete
the future. This messed with the timings in the
integration tests and made them less predictable.
2023-07-13 11:19:45 +02:00
Jaremy Creechley e47b38af11
Improving proc/func formatting consistency (#454)
* Fixes/workarounds for nimsuggest failures in codex.nim.

* remove rng prefix - it appears to work now

* format new's to be more consistent

* making proc formatting a bit more consistent
2023-06-22 08:11:18 -07:00
Eric Mastro 2a92dc9702
[marketplace] Simulate invalid proof submissions (#393) 2023-06-22 20:32:18 +10:00
Adam Uhlíř 4cd8dd2e05
feat: slots rest api (#443)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-06-20 14:52:15 +02:00
Adam Uhlíř c075890737
feat: storage host collects reward (#416)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-05-15 09:02:57 +02:00
Adam Uhlíř 4d028c6cb3
feat: contract address management (#405)
Co-authored-by: Eric Mastro <github@egonat.me>
2023-05-03 09:24:25 +02:00
markspanbroek d56eb6aee1
Validator (#387)
* [contracts] Add SlotFreed event

* [integration] allow test node to be stopped twice

* [cli] add --validator option

* [contracts] remove dead code

* [contracts] instantiate OnChainMarket and OnChainClock only once

* [contracts] add Validation

* [sales] remove duplicate import

* [market] add missing import

* [market] subscribe to all SlotFilled events

* [market] add freeSlot()

* [sales] fix warnings

* [market] subscribe to SlotFreed events

* [contracts] fix warning

* [validator] keep track of filled slots

* [validation] remove slots that have ended

* [proving] absorb Proofs into Market

Both Proofs and Market are abstractions around
the Marketplace contract, having them separately
is more trouble than it's worth at the moment.

* [market] add markProofAsMissing()

* [clock] speed up waiting for clock in tests

* [validator] mark proofs as missing

* [timer] fix error on node shutdown

* [cli] handle --persistence and --validator separately

* [market] allow retrieval of proof timeout value

* [validator] do not subscribe to SlotFreed events

Freed slots are already handled in
removeSlotsThatHaveEnded(), and onSlotsFreed()
interfered with its iterator.

* [validator] Start validation at the start of a new period

To decrease the likelihood that we hit the validation timeout.

* [validator] do not mark proofs as missing after timeout

* [market] check whether proof can be marked as missing

* [validator] simplify validation

Simulate a transaction to mark proof as missing, instead
of trying to keep track of all the conditions that may
lead to a proof being marked as missing.

* [build] use nim-ethers PR #40

Uses "pending" blocktag instead of "latest" blocktag
for better simulation of transactions before sending
them.

https://github.com/status-im/nim-ethers/pull/40

* [integration] integration test for validator

* [validator] monitor a maximum number of slots

Adds cli parameter --validator-max-slots.

* [market] fix missing collateral argument

After rebasing, add the new argument to fillSlot calls.

* [build] update to nim-ethers 0.2.5

* [validator] use Set instead of Table to keep track of slots

* [validator] add logging

* [validator] add test for slot failure

* [market] use "pending" blocktag to use more up to date block time

* [contracts] remove unused import

* [validator] fix: wait until after period ends

The smart contract checks that 'end < block.timestamp',
so we need to wait until the block timestamp is greater
than the period end.
2023-04-19 15:06:00 +02:00
Adam Uhlíř 131d003a0c
feat: collateral per slot (#390)
Co-authored-by: Eric Mastro <github@egonat.me>
2023-04-14 11:04:17 +02:00
Eric Mastro ccf349bd14
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module

- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
  - ClientInteractions (with purchasing)
  - HostInteractions (with sales and proving)
- compilation fix for nim 1.2

[repostore] fix started flag, add tests

[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.

* Revert repostore changes

In favour of separate PR https://github.com/status-im/nim-codex/pull/374.

* remove warnings

* clean up

* tests: stop repostore during teardown

* change constructor type identifier

Change Contracts constructor to accept Contracts type instead of ContractInteractions.

* change constructor return type to Result instead of Option

* fix and split interactions tests

* clean up, fix tests

* find availability by slot id

* remove duplication in host/client interactions

* add test for finding availability by slotId

* log instead of raiseAssert when failed to mark availability as unused

* move to SaleErrored state instead of raiseAssert

* remove unneeded reverse

It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.

* update open api spec for potential rest endpoint errors

* move functions about available bytes to repostore

* WIP: reserve and release availabilities as needed

WIP: not tested yet

Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.

As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.

During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.

Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.

* delete availability when all bytes released

* fix tests + cleanup

* remove availability from SalesContext callbacks

Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.

Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.

* test clean up

* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
  - there was a bug fixed that crashed the node due to a missing `return success` in onStore
  - the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging

* fixes after rebase

1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.

* swap contracts branch to not support slot collateral

Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.

* modify Interactions and Deployment constructors

- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`

* Move `batchProc` declaration

`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.

* [reservations] rename `available` to `hasAvailable`

* [reservations] default error message to inner error msg

* add SaleIngored state

When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 17:05:16 +10:00
Adam Uhlíř 3198db414d
fix: approving tokens transfer when creating request (#385) 2023-03-30 11:34:38 +02:00
markspanbroek 4ffe7b8e06
Generate proofs when required (#383)
* [maintenance] speedup integration test

* [rest api] add proofProbability parameter to storage requests

* [integration] negotiation test ends when contract starts

* [integration] reusable 2 node setup for tests

* [integration] introduce CodexClient for tests

* [node] submit storage proofs when required

* [contracts] Add Slot type

* [proving] replace onProofRequired & submitProof with onProve

Removes duplication between Sales.onProve() and
Proving.onProofRequired()
2023-03-27 15:47:25 +02:00
Eric Mastro 25f68c1e4c
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain

# Conflicts:
#	codex/contracts/market.nim

* [marketplace] make on chain event callbacks async

# Conflicts:
#	tests/codex/helpers/mockmarket.nim

* [marketplace] make availability optional for node restart

# Conflicts:
#	tests/codex/testsales.nim

* [marketplace] add async state machine

Allows for `enterAsync` to be cancelled.

* [marketplace] move sale process to async state machine

* [marketplace] sales state machine tests

* bump dagger-contracts

* [marketplace] fix ci issue with chronicles output

* PR comments

- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-

* [marketplace] handle slot filled by other host

Handle the case when in the downloading, proving, or filling states, that another host fills the slot.

* [marketplace] use requestId for mySlots

* [marketplace] infer slot index from slotid

prevents reassigning a random slot index when restoring state from chain

* [marketplace] update to work with latest contracts

* [marketplace] clean up

* [marketplace] align with contract changes

- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState

Tests for changes

* [marketplace] add missing file

* [marketplace] bump codex-contracts-eth

* [config] remove unused imports

* [sales] cleanup

* [sales] fix: do not crash when fetching state fails

* [sales] make slotIndex non-optional

* Rebase and update NBS commit

Rebase on top of main and update NBS commit to the CI fix.

* [marketplace] use async subscription event handlers

* [marketplace] support slotIndex no longer optional

Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.

* [marketplace] sales state machine: use slotState

Use `slotState` instead of `requestState` for sales state machine.

* [marketplace] clean up

* [statemachine] adds a statemachine for async workflows

Allows events to be scheduled synchronously.

See https://github.com/status-im/nim-codex/pull/344

Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>

* [market] make market callbacks synchronous

* [statemachine] export Event

* [statemachine] ensure that no errors are raised

* [statemachine] add machine parameter to run method

* [statemachine] initialize queue on start

* [statemachine] check futures before cancelling them

* [sales] use new async state machine

- states use new run() method and event mechanism
- StartState starts subscriptions and loads request

* [statemachine] fix unsusbscribe before subscribe

* [sales] replace old state transition tests

* [sales] separate state machine from sales data

* [sales] remove reference from SalesData to Sales

* [sales] separate sales context from sales

* [sales] move decoupled types into their own modules

* [sales] move retrieveRequest to SalesData

* [sales] move subscription logic into SalesAgent

* [sales] unsubscribe when finished or errored

* [build] revert back to released version of nim-ethers

* [sales] remove SaleStart state

* [sales] add missing base method

* [sales] move asyncSpawn helper to utils

* [sales] fix imports

* [sales] remove unused variables

* [sales statemachine] add async state machine error handling (#349)

* [statemachine] add error handling to asyncstatemachine

- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.

* [sales] rename onSaleFailed to onSaleErrored

Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.

* [statemachine] forward error to state directly

without going through a machine method first

* [statemachine] remove unnecessary error handling

AsyncQueueFullError is already handled in schedule()

* [statemachine] test that cancellation ignores onError

* [sales] simplify error handling in states

Rely on the state machine error handling instead
of catching errors in the state run method

---------

Co-authored-by: Mark Spanbroek <mark@spanbroek.net>

* [statemachine] prevent memory leaks

prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started

* [salesagent] add salesagent-specific tests

1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.

---------

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 14:34:26 +01:00
Eric Mastro df729be261
[marketplace] support latest contracts changes (#327)
* [marketplace] support removal of Storage contract

* [marketplace] change submod dep dagger-contracts to codex-contracts-eth
2023-01-19 16:58:04 +11:00
markspanbroek cdd175279c
Fix nimble requirements (#305)
* [build] Update nim-ethers to 0.2.4

* [build] set correct lrucache requirement in nimble file
2022-11-16 13:10:23 +01:00
markspanbroek 4175689745
Load purchase state from chain (#283)
* [purchasing] Simplify test

* [utils] Move StorageRequest.example up one level

* [purchasing] Load purchases from market

* [purchasing] load purchase states

* Implement myRequest() and getState() methods for OnChainMarket

* [proofs] Fix intermittently failing tests

Ensures that examples of proofs in tests are never of length 0;
these are considered invalid proofs by the smart contract logic.

* [contracts] Fix failing test

With the new solidity contracts update, a contract can only
be paid out after it started.

* [market] Add method to get request end time

* [purchasing] wait until purchase is finished

Purchase.wait() would previously wait until purchase
was started, now we wait until it is finished.

* [purchasing] Handle 'finished' and 'failed' states

* [marketplace] move to failed state once request fails

- Add support for subscribing to request failure events.
- Add supporting contract tests for subscribing to request failure events.
- Allow the PurchaseStarted state to move to PurchaseFailure once a request failure event is emitted
- Add supporting tests for moving from PurchaseStarted to PurchaseFailure
- Add state transition tests for PurchaseUnknown.

* [marketplace] Fix test with longer sleepAsync

* [integration] Add function to restart a codex node

* [purchasing] Set client address before requesting storage

To prevent the purchase id (which equals the request id)
from changing once it's been submitted.

* [contracts] Fix: OnChainMarket.getState()

Had the wrong method signature before

* [purchasing] Load purchases on node start

* [purchasing] Rename state 'PurchaseError' to 'PurchaseErrored'

Allows for an exception type called 'PurchaseError'

* [purchasing] Load purchases in background

No longer calls market.getRequest() for every purchase
on node start.

* [contracts] Add `$` for RequestId, SlotId and Nonce

To aid with debugging

* [purchasing] Add Purchasing.stop()

To ensure that all contract interactions have both a
start() and a stop() for

* [tests] Remove sleepAsync where possible

Use `eventually` loop instead, to make sure that we're
not waiting unnecessarily.

* [integration] Fix: handle non-json response in test

* [purchasing] Add purchase state to json

* [integration] Ensure that purchase is submitted before restart

Fixes test failure on slower CI

* [purchasing] re-implement `description` as method

Allows description to be set in the same module where the
state type is defined.

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [contracts] fix typo

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [market] Use more generic error type

Should we decide to change the provider type later

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
2022-11-08 08:10:17 +01:00
Eric Mastro 25df599c08 Update to new reverts API, add tests
1. use `ProviderError` to catch contract errors instaed of `JsonRpcProviderError`.
2. Add tests for `isProofRequired`, `willProofBeRequired`, and `getProofEnd` when slot is empty (and is call is reverted)
3. bump nim-ethers to branch that has new `reverts` API and `ProviderError` support.
4. Update to new `reverts` API.
2022-10-25 15:10:35 +11:00
Eric Mastro 5eb65336fd catch and check revert reasons
Ensure specific contract reverts are handled and returned as booleans or default values

# Conflicts:
#	codex/contracts/proofs.nim
2022-10-25 15:10:35 +11:00
Eric Mastro 10d4dcd70d bump nim-ethers, fix tests
# Conflicts:
#	vendor/nim-ethers
2022-10-25 15:10:35 +11:00
Eric Mastro dab65eb6bd catch contract errors to return bool
# Conflicts:
#	codex/contracts/proofs.nim
#	vendor/nim-ethers
2022-10-25 15:10:35 +11:00