status-im/dagger-research
2
0
Fork 0
mirror of https://github.com/status-im/dagger-research.git synced 2025-02-23 19:58:24 +00:00
Code Issues Projects Releases Wiki Activity

Update reservations design

Browse Source 
Incorporates review comments from
dryajov, emizzle and AuHau.
This commit is contained in:
Mark Spanbroek 2023-03-28 19:28:16 +02:00 committed by markspanbroek
parent f36eff730f
commit adb0158fb8
1 changed files with 117 additions and 46 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

163
design/sales.md
View File

@ -4,8 +4,7 @@ Sales module
The sales module is responsible for selling a node's available storage in the The sales module is responsible for selling a node's available storage in the
[marketplace](./marketplace.md). In order to do so it needs to know how much [marketplace](./marketplace.md). In order to do so it needs to know how much
storage is available. It also needs to be able to reserve parts of the storage, storage is available. It also needs to be able to reserve parts of the storage,
to make sure that it is not used for other purposes while we are filling a slot to make sure that it is not used for other purposes.
in a storage request, or while we're attempting to fill a slot.
--------------------------------------------------- ---------------------------------------------------
| | | |
@ -20,65 +19,115 @@ in a storage request, or while we're attempting to fill a slot.
| ^ ^ | | ^ ^ |
----------------------------|---------|----------- ----------------------------|---------|-----------
| | | |
availability | | state reserved space | | state
v v v v
---------------- ----------------- ---------------- -----------------
| Repo | | Datastore | | Repo | | Datastore |
---------------- ----------------- ---------------- -----------------
The Sales module keeps track of storage reservations in its internal The reservations module keeps track of storage that is available to be sold.
Reservations module. It keeps a record of which pieces of reserved storage Users are able to add availability to indicate how much storage they are willing
belong to which storage request slot. It queries and updates the amount of to sell and under which conditions.
available storage in the Repo. It uses a Datastore to persist its own state.
The Repo exposes the following functions that allow the Reservations module to Availability
query and update the amount of available storage: amount
maximum duration
minimum price
Repository API: Availabilities consist of an amount of storage, the maximum duration and minimum
function available(): amount price to sell it for. They represent storage that is for sale, but not yet sold.
function reserve(amount) This is information local to the node that can be altered without affecting
function release(amount) global state.
The Datastore is a generic key-value store that is used to persist the state of Adding availability
the Reservations module, so that it survives node restarts. -------------------
Datastore API: When a user adds availability, then the reservations module wil check whether
function put(key, value) there is enough space available in the Repo. If there is enough space, then it
function get(key): value will increase the amount of reserved space in the Repo. It persists the state of
all availabilities to the Datastore, to ensure that they can be restored when a
node is restarted.
Reserving storage space User Reservations Repo Datastore
----------------------- | | | |
| add availability | | |
| ---------------->| check free space | |
| |----------------->| |
| | reserve amount | |
| |----------------->| |
| | |
| | persist availability |
| |------------------------------>|
Selling storage
---------------
When a request for storage is submitted on chain, the sales module decides When a request for storage is submitted on chain, the sales module decides
whether or not it wants to act on it. If the requested duration and size match whether or not it wants to act on it. First, it tries to find an availability
what the node wants to sell and the reward is sufficient, then the sales module that matches the requested amount, duration, and price. The matching
will go through several steps to try and fill a slot in the request. availability will be removed from the reservations module, so that it can't be
sold twice. If an availability matches, but is larger than the requested
storage, then the Sales module may decide to split the availability into a part
that we can use for the request, and a remainder that can be sold separately.
First, it will select a slot from the request to fill. Then, it will reserve It then selects a slot from the request to fill, and starts downloading its
space in the Repo to ensure that it keeps enough space available to store the content chunk by chunk. For each chunk that is successfully downloaded, a bit of
content. It will mark the reserved space as belonging to the slot. Next, it will reserved space in the Repo is released. The content is stored in the Repo with a
download the content, calculate a storage proof, and submit the proof on chain. time-to-live value that ensures that the content remains in the Repo until the
If any of these later steps fail, then the node should release the storage that request expires.
it reserved earlier.
When a slot was filled successfully, then its storage should not be released Once the entire content is downloaded, the sales module will calculate a storage
until the request ends. proof, and submit the proof on chain. If these steps are all successful, then
this node has filled the slot. Once the other slots are filled by other nodes
the request will start. The time-to-live value of the content should then be
updated to match the duration of the storage request.
Releasing storage space Marketplace Sales Reservations Repo
----------------------- | | | |
| incoming request | | |
|------------------->| find reservation | |
| |-------------------->| |
| | remove reservation | |
| |-------------------->| |
| | | |
| | store content |
| |----------------------------------->|
| | set time-to-live |
| |----------------------------------->|
| | release reserved space |
| |----------------------------------->|
| submit proof | |
|<-------------------| |
| | |
. . .
. . .
| request started | |
|------------------->| update time-to-live |
| |----------------------------------->|
When a storage request ends, or an attempt to fill a slot fails, it is important Ending a request
to release the reserved space that belonged to the slot. To ensure that the ----------------
right amount of space is released, and that it is only released once, we keep
track of how much space is reserved in the Repo for a slot.
Releasing storage space goes in three steps. First, we look up the amount of When a storage request comes to an end, then the content can be removed from the
space that is reserved for the slot in the Datastore. Then we release that repo and the storage space can be made available for sale again. The same should
amount in the Repo, and remove the entry for the slot from the Datastore. happen when something went wrong in the process of selling storage.
The first step ensures that we release the correct amount of space from the The time-to-live value should be removed from the content in the Repo, reserved
Repo. The last step ensures that we do not release space from the Repo more than space in the Repo should be increased again, and the availability that was used
once. for the request can be re-added to the reservations module.
Sales Reservations Repo
| | |
| | |
| |
| remove time to live |
|----------------------------------->|
| increase reserved space |
|----------------------------------->|
| |
| re-add availability | |
|-------------------->| |
| | |
Persisting state Persisting state
---------------- ----------------
@ -88,6 +137,28 @@ this includes the slots that a host is filling and the state of each slot. This
ensures that a node's local view of slot states does not deviate from the ensures that a node's local view of slot states does not deviate from the
network view, even when the network changes while the node is down. The rest of network view, even when the network changes while the node is down. The rest of
the state is kept on local disk by the Repo and the Datastore. How much space is the state is kept on local disk by the Repo and the Datastore. How much space is
reserved by the sales module, and how much remains available to sell is reserved to be sold is persisted on disk by the Repo. The availabilities are
persisted on disk by the Repo. A mapping between amounts of reserved space and persisted on disk by the Datastore.
slots is persisted on disk by the Datastore.
Repo
----
The Repo exposes the following functions that allow the reservations module to
query the amount of available storage, to update the amount of reserved
space, and to store data for a guaranteed amount of time.
Repository API:
function available(): amount
function reserve(amount)
function release(amount)
function setTtl(cid, ttl)
Datastore
---------
The Datastore is a generic key-value store that is used to persist the state of
the Reservations module, so that it survives node restarts.
Datastore API:
function put(key, value)
function get(key): value