diff --git a/design/sales flow charts/renewals_cleanup.jpg b/design/sales flow charts/renewals_cleanup.jpg new file mode 100644 index 0000000..a266b2a Binary files /dev/null and b/design/sales flow charts/renewals_cleanup.jpg differ diff --git a/design/sales2.md b/design/sales2.md new file mode 100644 index 0000000..d3ff71f --- /dev/null +++ b/design/sales2.md @@ -0,0 +1,370 @@ +# Sales module (add purchasing module) + +The sales module is responsible for selling a node's available storage in the +[marketplace](./marketplace.md). In order to do so, it needs to create an +Availability for the storage provider (SP) to establish under which +conditions it is willing to enter into a sale. + +```ascii +------------------------------------------------------------------ +| | +| Sales | +| | +| ^ | | +| | | updates ------------------ | +| | --------------> | | | +| | | SalesStorage | | +| ------------------- | | | +| queries ------------------ | +| ^ ^ | +| | | | +| | | Availability + SaleOrder | +| dedicated quota | | state | +| v v | +| ---------------- ----------------- | +| | SalesRepo | | MetadataStore | | +| ---------------- ----------------- | +------------------------------------------------------------------ +``` + +The `SalesStorage` module manages the SP's availability and snapshots of past +and present sales or `SalesOrders`, both of which are persisted in the `MetadataStore`. SPs can add +and update their availability, which is managed through the `SalesStorage` +module. As a `SalesOrder` traverses the sales state machine, it is created and +updated1 through the `SalesStorage` module. Queries for availability +and `SalesOrders` will also occur in the `SalesStorage` module. Datasets that +are downloaded and deleted as part of the sales process will be handled in the +`SalesRepo` module. + +1 Updates are only needed to support [tracking the latest state in + the `SalesOrder`](#tracking-latest-state-machine-state). + +## Query support + +The `SalesStorage` module will need to support querying the availability and sales +data so the caller can understand if a sale can be serviced and to support clean +up routines. The following queries will need to be supported: + +1. To know if there is enough space on disk for a new sale, the `SalesStorage` + module can be queried for the remaining sales quota in its dedicated + `SalesRepo` partition. In the future, this can be optimised to [prevent + unnecessary resource + consumption](#concurrent-workers-prevent-unnecessary-resource-consumption), + by additionally querying the slot size of `SalesOrders` that are in or past + the Downloading state. +2. Clean up routines will need to know the "active sales", or any `SalesOrders` + in the `/active` key namespace (those that have not been archived) through + the state machine or clean up routines. +3. Servicing a new slot will require sufficient "total collateral", which is the + remaining balance in the funding account. In the future, this can be + optimised to [prevent unnecessary resource + consumption](#concurrent-workers-prevent-unnecessary-resource-consumption), + by additionally querying the collateral of `SalesOrders` that are in or past + the Downloading state. + +## `SalesRepo` module + +The `SalesRepo` module is responsible for interacting with its underlying +`RepoStore`. This additional layer abstracts away some of the required +implementation routine needed for the `RepoStore`, while also allowing the +`RepoStore` to change independent of the sales module. It will expose functions +for storing and deleting datasets: + +```mermaid +--- +config: + look: neo + layout: dagre +--- +classDiagram +direction TB + class RepoStore { + +putBlock(BlockAddress) + +delBlock(BlockAddress) + } + class SalesRepo { + -RepoStore repo + +store(BlockAddress): Stores the manifest dataset. + +delete(BlockAddress): Deletes the manifest dataset in the RepoStore. + } + class SalesStorage { + -salesRepo: SalesRepo + } + SalesRepo --* RepoStore + SalesStorage <--* SalesRepo + class SalesRepo:::focusClass + classDef focusClass fill:#c4fdff,stroke:#333,stroke-width:4px,color:black +``` + +## Availability + +The SP's availability determines which sales it is willing to attempt to enter +into. In other words, it represents *future sales* that an SP is willing to take +on. It consists of parameters that will be matched to incoming storage +requests via the slot queue. + + +| Property | Description | +|----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| +| `id` | ID of the Availability. Note: this is only needed if there is support for [multiple availabilities](#multiple-availabilities). | +| `duration` | Maximum duration of a storage request the SP is willing to host new slots for. | +| `minPricePerBytePerSecond` | Minimum price per byte per second that the SP is willing to host new slots for. | +| `enabled` | If set to false, the availability will not accept new slots. Updates to this value will not impact any existing slots that are already being hosted. | +| `until` | Specifies the latest timestamp after which the availability will no longer host any slots. If set to 0, there will be no restrictions. | + +The availability of a SP consists of the maximum duration and the minimum price +per byte per second to sell storage for. + +### Funding account vs profit account + +SPs should control two accounts: a funding account, and a profits account. The +funds in the funding account represent the total collateral that a SP is willing +to risk in all of its sales combined. This account will need to have some funds +in it before slots can be hosted, assuming the storage request requires +collateral. If a SP has been partially or wholly slashed in one of their sales, +they may wish to top up this account to ensure there is sufficient collateral +for future sales. + +The profits account is the account for which proceeds from sales are paid into. +To minimise risk, this account should be stored in cold storage. + +It is recommended that the profit account is a separate account from the funding +account so that profits are not placed at risk by being used as collateral. If a +SP specifies the same account for funding and profits, and the SP is (partially +or wholly) slashed, future collateral deposits may use their profits from +previous sales. + +Note: having a separate profit account relies on the ability of the Vault +contract to support multiple accounts. + +### Total collateral + +The concept of "total collateral" means the total collateral the SP is willing +to risk at any one point in time. In other words, it is willing to risk "total +collateral" tokens for all of its active sales combined. Total collateral is +determined by the balance of funds in the SP's funding account. So, any funds in +the funding account are considered available to use as collateral for filling +slots. + +From the marketplace perspective, slots cannot be filled if there is an +insufficient balance in the funding account. + +### `Availability` lifecycle + +A user can add, update, or delete an `Availability` at any time. The +`Availability` will be stored in the MetadataStore. Only one `Availability` can +be created and once created, it will exist permanently in the MetadataStore +until it is deleted. The properties of a created `Availability` can be updated +at any time. + +Because availability(ies) represents *future* sales (and not active sales), and +because fields of the matching `Availability` are persisted in a `SalesOrder`, +availabilities are not tied to active sales and can be manipulated at any time. + +## `SalesOrder` object + +The `SalesOrder` object represents a snapshot of the sale parameters at the time +a slot is processed in the slot queue. It can be thought of as the market +conditions at the time of sale. It includes fields of the storage request, slot, +and the matching availability fields. + +| Property | Description | +|----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `requestId` | RequestId of the StorageRequest. Can be used to retrieve storage request details. | +| `slotIndex` | lot index of the slot being processed. | +| `duration` | `duration` from the matched Availability. | +| `minPricePerBytePerSecond` | `minPricePerByte` from the matched Availabilty. | +| `state` | Latest state in the sales state machine that was reached. Note: this is only needed when there support for [tracking the latest state in the `SalesOrder`](#tracking-latest-state-machine-state). | + +### `SalesOrder` lifecycle + +At the point a SP reaches the `SaleDownload` state, a `SalesOrder` is created +and it will live permanently in the MetadataStore. `SalesOrder` objects cannot +be deleted as they represent historical sales of the SP. + +When the `SalesOrder` object is first created, its key will be created in the +`/active` namespace. After data for the `SalesOrder` has been deleted (if there +is any) in a clean up procedure, the key will be moved from the `/active` +namespace to the `/archive` namespace. These key namespace manipulations +facilitate future lookups in active/passive clean up operations. + +If there's support for [tracking the latest state in the +`SalesOrder`](#tracking-latest-state-machine-state), `SalesOrder.state` +will be modified as the sale progresses through each state of the Sales state +machine. + +## Cleanup routines + +The responsibility of the cleanup routine is to ensure that any data associated +with a Sale is deleted from the `SalesRepo`. Once the data has been deleted, the +`SalesOrder` will reflect that it has been cleaned up by being archived. + +There are two types of cleanup routines that a SP node will take part in: active +and passive. Active cleanup routines are run as part of a state in the Sales +state machine. Passive cleanup routines are continuously run at a specified time +interval. Both perform a similar task, however the active cleanups operate on a +single `SalesOrder`, while passive cleanups operate over a set of `SalesOrders` +and have additional conditions for cleanup. + +### Active cleanup + +The active cleanup routine is typically run as part of the a final state in the +Sales state machine, ie `SaleFinished`. In this routine, active sales will be +retrieved from the Marketplace contract via `mySlots`. If the slot id associated +with the sale is not in the set of active sales, any data associated with the +slot will be deleted. Then, the `SalesOrder` will be archived, by moving its key +to the `/archive` namespace. + +### Passive cleanup + +At regular time intervals, active sales will be retrieved from the Marketplace +contract via `mySlots`. Then, all `SalesOrders` in the `/active` namespace will +be queried. Any `SalesOrders` with a slot id not in the set of active sales and +with a `StorageRequest` state that is "completed" (failed, cancelled, finished), +will have the data associated with the slot deleted, if there is any. Then, the +`SalesOrder` will be archived by moving its key to the `/archive` namespace. +`SalesOrders` with a `StorageRequest` state that is not yet completed should not +have their data deleted, as the SP may be in the process of starting to host a +slot, with the sale in an early state of the Sales state machine. + +### Node startup + +On node startup, the passive cleanup routine should be run. + +## Sale flow + +[Insert flow charts] + +## Optimisations and features + +### Multiple availabilities + +Multiple availabilities are useful to allow SPs to understand which Availability +parameters produce the most profit for them. Multiple availabilities can be +updated or deleted at any time. This is possible because there is no +availability ID stored in the `SalesOrder` object. + +Note that the total collateral across all availabilities that a SP is +willing to risk remains as the balance of funds in the funding account. + +### Concurrent workers support + +Concurrent workers allow a SP to reserve, download, generate an initial proof +for, and fill multiple slots simultaneously. This could prevent SPs from missing +sale opportunities that arise while they are reserving, downloading, generating, +and generating an initial proof for another sale. The trade off, however, is +that concurrent workers will require more system resources than a single worker. +In addition, concurrency is difficult to reason about, can introduce +difficult-to-debug bugs, and also opens up the possibility of unnecessary +reserving, downloading, and proof generation (discussed below). Therefore, it is +imperative this feature is implemented carefully. + +### Tracking latest state machine state + +Tracking the latest state machine state in locally persisted `SalesOrders` can +allow for historical sales listings (eg REST api or Codex app), sales +performance analysis (eg profit), and availability optimisations. + +After a `StorageRequest` is completed, it is removed from the contract's +`mySlots` storage, with a locally-persisted `SalesOrder` being the only +remaining information about the sale. Without having the latest state persisted, +`SalesOrders` will be archived, but the SP will not know what the final state of +a `SalesOrders` was when it was archived. For example, it will not be able to +distinguish between a sale that errored and a slot that was successfully +hosted. This information is useful for listing states of sales, but also for +optimisations. + +Active sale data is stored on chain in the Marketplace contract (`mySlots`). +However, these slots are slots that have already been filled by the SP. +When making a decision to service a new slot, the SP can optimise its decision +with information about sales that may be at an earlier stage in the sales +process, ie downloading, proof generating, or filling. To +facilitate this, `SalesOrder.state` would need to track the latest state of the +sale in the sales state machine. + +Tracking the latest state opens up the possibility for further optimisations, +see below. + +### Concurrent workers: prevent unnecessary resource consumption + +Depends on: Tracking latest state machine state
+Depends on: Concurrent workers + +To prevent unnecessary reserving, downloading, and proof generation when there +are concurrent workers, when checking to ensure there's enough collateral +available, instead of only checking the funding account's current balance, also +check collateral that will be used to fill slots in `SalesOrders` that are +`/active`. Without this check, SPs may reserve, download, and generate a proof +for a sale that would ultimately result in not having enough collateral. For +example, if funding account balance is 100, and the SP is currently downloading +two sales with 100 collateral each, then that would mean that the download that +finishes last will ultimately be wasted as the SP would not have enough +collateral to fill both slots. + +### Renewals: prevent dataset deletion + +During renewals, there could potentially be a new sale for the same dataset that +is already in an active sale. The `SlotId` (and `RequestId`) will differ, +however the CID will be the same. Renewals should occur well before the initial +sale finishes. However, if the new sale is close in time to the completion of +the first sale, then as the dataset for the first sale is being cleaned up, +it may delete the dataset that is needed by the new sale. The new sale may have +been in the process of being downloaded, or having proofs generated. + +This can be prevented by having an in-memory ref count of datasets. When a +dataset is stored, the ref count of the dataset (`hash(treeCid, slotIndex)`) is +incremented. When the dataset is deleted, the ref count is decremented. Only +when the ref count is 0 is the dataset actually deleted in the `RepoStore`. The +ref count does not require persistence because on startup, hosted slots will not +be deleted. + +Ref count handling can be managed in `SalesRepo` module. This module is +responsible for interacting with the underlying `RepoStore`, and managing the +internal ref count. It will expose functions for storing and deleting datasets. + +Note that any calls to ref count should be locked, as they may be read and +updated concurrently. + +This is how the `SalesRepo` module will interact with `RepoStore` and the +marketplace: + +```mermaid +--- +config: + look: neo + layout: dagre +--- +classDiagram +direction TB + class RepoStore { + +putBlock(BlockAddress) + +delBlock(BlockAddress) + } + class SalesRepo { + +Table~BlockAddress, uint~ refCount + -RepoStore repo + +store(BlockAddress): Stores the manifest dataset and increments the ref count of the manifest. + +delete(BlockAddress): Decrements the ref count of the manifest and deletes the manifest dataset in the RepoStore if the ref count is zero. + } + class SalesStorage { + -salesRepo: SalesRepo + } + SalesRepo --* RepoStore + SalesStorage <--* SalesRepo + class SalesRepo:::focusClass + classDef focusClass fill:#c4fdff,stroke:#333,stroke-width:4px,color:black +``` + +#### Alternative idea + +Preventing deletion of datasets that are being downloaded or proof generating +can also be achieved by first checking if the slot id exists in `/mySlots` (at this stage +the initial sale should no longer be in `/mySlots`). If it does not, then check +if there are more than one `/active` (reached downloading) `SalesOrders` with +the same `hash(treeCid, slotIndex)` that exist. If there are not, delete the +dataset. Finally, archive the `SalesOrder`. + +![Cleanup handling renewals]() + +## Purchasing +