logos-storage/logos-storage-spec
1
0
Fork 0
mirror of https://github.com/logos-storage/logos-storage-spec.git synced 2026-01-07 16:03:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore: proof reading and tweaks

Browse Source
This commit is contained in:
Adam Uhlíř 2024-08-13 11:20:36 +02:00
parent 7445fb5014
commit c97b056f70
No known key found for this signature in database
GPG Key ID: 1D17A9E81F76155B
1 changed files with 58 additions and 50 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

108
specs/marketplace.md
View File

@ -40,31 +40,30 @@ The Codex network aims to create a peer-to-peer storage engine with strong data
data persistence guarantees, and node storage incentives. data persistence guarantees, and node storage incentives.
An important component of Codex network is a Marketplace. It is a place which mediates negotiations of all parties An important component of Codex network is a Marketplace. It is a place which mediates negotiations of all parties
in order to provide persistence in the network. It also provides ways to enforce agreements and **TODO** with repair. to provide persistence in the network. It also provides ways to enforce agreements and facilitate repair when a storage node drops out.
Marketplace is defined by smart-contract that is deployed to EVM-compatible blockchain. It has several flows which are Marketplace is defined by smart-contract deployed to EVM-compatible blockchain. It has several flows which are
linked with roles in the network and which the participating node takes upon itself. It can be one role or multiple at the same time. linked with roles in the network and which the participating node takes upon itself. It can be one role or multiple at the same time.
This specification describes these flows. This specification describes these flows.
The Marketplace handles storage requests, the storage slot state, The Marketplace handles storage requests, the storage slot state,
storage provider rewards, storage provider collaterals, and storage proof state. storage provider rewards, storage provider collaterals, and storage proof state.
If a node implementation wants to participate in the persistence layer of Codex it needs to choose which role(s) it wants If a node implementation wants to participate in the persistence layer of Codex, it needs to choose which role(s) it wants
to support and implement properer flows otherwise it won't be compatible with the rest of the Codex network. to support and implement proper flows. Otherwise, it won't be compatible with the rest of the Codex network.
### Roles ### Roles
There are 3 main roles in the network - client, storage provider (SP) and validator. There are three main roles in the network - client, storage provider (SP) and validator.
Client is a potentially short-lived node in the network, that interacts with it with the main purpose of persisting Client is a potentially short-lived node in the network that mainly interacts with the purpose of persisting
its data in the network. its data in the network.
Storage Provider is long-term participant in the network, that store other's data for profit. It needs to provide a proof Storage Provider is a long-term participant in the network that stores other data for profit. It needs to provide a proof
to the smart contract that it possesses the data from time to time. to the smart contract that it possesses the data from time to time.
Validator is a node that helps with enforcing the storage provider's duties to comply with storage requests that they Validator validates that Storage Providers correctly are fulfilling their duties and that they provide proofs of storage
accepted. When it detects that SP should have submitted a proof but non was submitted on-chain, it triggers on-chain on time.
function which will handle this case. Validator is rewarded for correct invocation of this function.
## Storage Request Lifecycle ## Storage Request Lifecycle
@ -97,11 +96,11 @@ function which will handle this case. Validator is rewarded for correct invocati
└────────┬─┬─────────────────────────────────────────────────┘ └────────┬─┬─────────────────────────────────────────────────┘
│ │ ▲ │ │ ▲
│ │ │ │ │ │
│ │ SP kicked out and Slot Freed ┌───────┴────────┐ │ │ SP kicked out and Slot freed ┌───────┴────────┐
All good │ ├─────────────────────────────►│ Repair process │ All good │ ├─────────────────────────────►│ Repair process │
Time ran out │ │ └────────────────┘ Time ran out │ │ └────────────────┘
│ │ │ │
│ │ Too much Slots Freed ┌────────┐ │ │ Too many Slots freed ┌────────┐
│ └─────────────────────────────►│ Failed │ │ └─────────────────────────────►│ Failed │
▼ └────────┘ ▼ └────────┘
┌──────────┐ ┌──────────┐
@ -113,27 +112,27 @@ Time ran out │ │ └─────────
## Client role ## Client role
Client role represent nodes that mediate persisting data inside Codex network. The Client role represents nodes that mediate persisting data inside the Codex network.
There are 2 parts for client role: There are two parts of a client role:
- Requesting storage from the network - creating storage request. - Requesting storage from the network - creating storage request.
- Withdrawing funds from storage requests. - Withdrawing funds from storage requests.
### Creating storage requests ### Creating storage requests
When the client node is prompted by the user to create a storage request, When the user prompts the client node to create a storage request,
it SHOULD receive the input parameters for the storage request from the user. it SHOULD receive the input parameters for the storage request from the user.
To create a request to persist a dataset on the Codex network, To create a request to persist a dataset on the Codex network,
client nodes MUST split the dataset into data chunks, $(c_1, c_2, c_3, \ldots, c_{n})$. client nodes MUST split the dataset into data chunks, $(c_1, c_2, c_3, \ldots, c_{n})$.
Using an erasure coding technique and input parameters, the data chunks are encoded and placed into separate slots. Using an erasure coding technique and input parameters, the data chunks are encoded and placed into separate slots.
The erasure coding technique MUST be the [Reed-Soloman algorithm](https://hackmd.io/FB58eZQoTNm-dnhu0Y1XnA). The erasure coding technique MUST be the [Reed-Soloman algorithm](https://hackmd.io/FB58eZQoTNm-dnhu0Y1XnA).
The final slot's roots and other metadata MUST be placed into Manifest. Manifest's CID is then used as the `cid` of the The final slot's roots and other metadata MUST be placed into Manifest (**TODO: Manifest RFC**). Manifest's CID MUST then
stored dataset. be used as the `cid` of the stored dataset.
After the dataset is prepared, it MUST submit a transaction with the desired request parameters which are represented After the dataset is prepared, node MUST submit a transaction with the desired request parameters which are represented
as `Request` object and its sub-objects. Bellow are described its properties: as `Request` object and its sub-objects to function `requestStorage(request)`. Bellow are described its properties:
```solidity ```solidity
struct Request { struct Request {
@ -187,35 +186,34 @@ struct Content {
``` ```
Notes about some of the parameters: Notes about some parameters:
`cid` `cid`
An identifier used to locate the Manifest representing the dataset. An identifier used to locate the Manifest representing the dataset.
- MUST be a [CIDv1](https://github.com/multiformats/cid#cidv1) with sha-256 based [multihash](https://github.com/multiformats/multihash). - MUST be a [CIDv1](https://github.com/multiformats/cid#cidv1) with sha-256 based [multihash](https://github.com/multiformats/multihash).
- Data it represents SHOULD be discoverable in the network, otherwise Request will get cancelled. - Data it represents SHOULD be discoverable in the network, otherwise the Request will get canceled.
`reward` `reward`
- It is an REQUIRED amount to be included in the transaction for a storage request. - It MUST be an amount of tokens offered per slot per second.
- It SHOULD be an amount of tokens offered per slot per second. - The Ethereum address that submits the `requestStorage()` transaction MUST have [approval](https://docs.openzeppelin.com/contracts/2.x/api/token/erc20#IERC20-approve-address-uint256-) for transfer of at least the same amount on the ERC20 based token, that the network uses.
- The Client address MUST have [approval](https://docs.openzeppelin.com/contracts/2.x/api/token/erc20#IERC20-approve-address-uint256-) for transfer of at least the same amount on the ERC20 based token, that the network utilizes.
`collateral` `collateral`
- Amount of tokens that the storage providers MUST submit when they fill slots. - Amount of tokens that the storage providers submit when they fill slots.
- Collateral is then slashed or forfeited if the storage providers fail to provide the service requested by the Request (more information bellow). - Collateral is then slashed or forfeited if the storage providers fail to provide the service requested by the Request (more information bellow).
`proofProbability` `proofProbability`
Determines the inverse probability that a proof is required in a period: $\frac{1}{proofProbability}$ Determines the inverse probability that a proof is required in a period: $\frac{1}{proofProbability}$
- Storage providers are REQUIRED to provide proofs of storage to the marketplace smart contract when they are prompted to by the smart contract. - Storage providers are required to provide proofs of storage to the marketplace smart contract when they are prompted to by the smart contract.
- The frequency is non-deterministic in order to prevent from pre-calculation attacks, but it is affected by this parameter. - The frequency is non-deterministic in order to prevent from pre-calculation attacks, but it is affected by this parameter.
`expiry` `expiry`
- Parameter is specified as duration in seconds, hence the final deadline timestamp is calculated at the moment when the transaction is mined. - The parameter is specified as duration in seconds, hence the final deadline timestamp is calculated at the moment when the transaction is mined.
`nonce` `nonce`
@ -223,15 +221,15 @@ Determines the inverse probability that a proof is required in a period: $\frac{
#### Renewal of Storage Request #### Renewal of Storage Request
It should be noted that Marketplace does not support extending Requests. It is REQUIRED that if user wants to It should be noted that Marketplace does not support extending Requests. It is REQUIRED that if the user wants to
extend the Request's duration, somebody submits a new Request transaction with the same CID **well before the original extend the Request's duration, somebody submits a new Request transaction with the same CID **well before the original
Request finishes**. In this way the data will be still persisted in the network at the time when new (or the current) storage providers Request finishes**. In this way, the data will be still persisted in the network at the time when new (or the current) storage providers
can retrieve the dataset in order to fill slots of the new Request. need to retrieve the dataset to fill slots of the new Request.
### Withdrawing funds ### Withdrawing funds
The client node SHOULD monitor the status of Requests that it created. The node can utilize on-chain state in order to The client node SHOULD monitor the status of Requests that it created. The node can utilize on-chain state to
fetch the list of the active Requests linked to the client node's blockchain address using function `myRequests()`, that fetch the list of the active Requests linked to the client node's ethereum address using function `myRequests()`, that
returns array of `RequestId`s. This list is kept up to date by the smart contract itself. returns array of `RequestId`s. This list is kept up to date by the smart contract itself.
When Request reaches states `Cancelled` (not all slots filled after `expiry` timeout) or `Failed` (too many slots gets freed and data is non-recoverable) When Request reaches states `Cancelled` (not all slots filled after `expiry` timeout) or `Failed` (too many slots gets freed and data is non-recoverable)
@ -239,7 +237,7 @@ the client node SHOULD initiate withdrawal of the remaining funds from the contr
- `Cancelled` state MAY be detected using timeout specified from function `requestExpiresAt(requestId)` **and** not detecting emitted `RequestFulfilled(requestId)` event. - `Cancelled` state MAY be detected using timeout specified from function `requestExpiresAt(requestId)` **and** not detecting emitted `RequestFulfilled(requestId)` event.
- `Failed` state MAY be detected using `RequestFailed(requestId)` event emitted from the smart contract. - `Failed` state MAY be detected using `RequestFailed(requestId)` event emitted from the smart contract.
- `Finished` state MAY be detected setting timeout specified from function `getRequestEnd(requestId)`. - `Finished` state MAY be detected by setting timeout specified from function `getRequestEnd(requestId)`.
## Storage Provider role ## Storage Provider role
@ -250,7 +248,7 @@ There are several parts to hosting a slot:
- Filling a slot - Filling a slot
- Proving - Proving
- Repairing slots - Repairing a slot
- Collecting Request's reward and collateral - Collecting Request's reward and collateral
### Filling slot ### Filling slot
@ -263,38 +261,43 @@ When new Request is created `StorageRequested(requestId, ask, expiry)` event is
It is then up to the Storage Provider node to decide based on the emitted parameters if it wants to participate in the It is then up to the Storage Provider node to decide based on the emitted parameters if it wants to participate in the
Request and try to fill its slot(s). This decision SHOULD be done based on parameters specified by the node operator. Request and try to fill its slot(s). This decision SHOULD be done based on parameters specified by the node operator.
If the node decide to ignore this Request, no action is need, otherwise the node HAVE TO follow the remaining steps. If the node decides to ignore this Request, no action is necessary. If the node wants to try to fill a a slot, then
it MUST follow the remaining steps.
Node MUST decide which Slot specified by slot's index it wants to try to fill in. Node MAY try filling multiple Node MUST decide which Slot specified by slot's index it wants to try to fill in. Node MAY try filling multiple
slots. In order to fill a slot, node first MUST download the slot's data using slot's root that can be retrieved slots. In order to fill a slot, node first MUST download the slot's data using slot's root that can be retrieved
from Manifest specified in `request.content.cid` (**TODO: Manifest RFC**). This object can be retrieved from the smart contract using `getRequest(requestId)`. from Manifest specified in `request.content.cid` (**TODO: Manifest RFC**).
Then node MUST generate proof over the downloaded data (**TODO: Proving RFC**). This Request object can be retrieved from the smart contract using `getRequest(requestId)`.
Then the node MUST generate proof over the downloaded data (**TODO: Proving RFC**).
When proof is ready it then MUST create transaction for smart contract call `fillSlot()` with following REQUIRED: When proof is ready it then MUST create transaction for smart contract call `fillSlot()` with following REQUIRED:
- Parameters: - Parameters:
- `requestId` - ID of the Request. - `requestId` - ID of the Request.
- `slotIndex` - Index that the node is trying to fill. - `slotIndex` - Index that the node is trying to fill.
- `proof` - `Groth16Proof` proof structure, generated over the dataset. - `proof` - `Groth16Proof` proof structure, generated over the slot's data.
- The Ethereum address of the node from which the transaction originates MUST have [approval](https://docs.openzeppelin.com/contracts/2.x/api/token/erc20#IERC20-approve-address-uint256-) for transfer of at least the amount required as collateral for the Request on the ERC20 based token, that the network utilizes. - The Ethereum address of the node from which the transaction originates MUST have [approval](https://docs.openzeppelin.com/contracts/2.x/api/token/erc20#IERC20-approve-address-uint256-) for transfer of at least the amount required as collateral for the Request on the ERC20 based token, that the network utilizes.
If the proof is invalid, or slot was already filled by other node then the transaction If the proof is invalid, or slot was already filled by another node, then the transaction
will revert, otherwise `SlotFilled(requestId, slotIndex)` event is emitted. If the transaction is successful then the will revert, otherwise `SlotFilled(requestId, slotIndex)` event is emitted. If the transaction is successful, then the
node SHOULD transition into __proving__ state as it will need to submit proof of data possession when prompted by the node SHOULD transition into __proving__ state as it will need to submit proof of data possession when prompted by the
contract. contract.
It should be noted that if the node see the `SlotFilled` emitted for slot that he is downloading the dataset or It should be noted that if the node sees the `SlotFilled` emitted for slot that he is downloading the dataset or
generating proof for, then node SHOULD stop and choose different non-filled slot to try to fill. generating proof for, then node SHOULD stop and choose different non-filled slot to try to fill as the chosen slot
was filled by other node.
### Proving ### Proving
Once node fills a slot it MUST periodically, yet non-deterministically provide proof to the smart contract that it Once a node fills a slot, it MUST periodically, yet non-deterministically provide proof to the smart contract that it
stores the data it should. Node MAY detect that proof is required using the `isProofRequired(slotId)` or that it will stores the data it should. Node SHOULD detect that proof is required using the `isProofRequired(slotId)` or that it will
be required using the `willProofBeRequired(slotId)` in case the node is in [downtime](https://github.com/codex-storage/codex-research/blob/41c4b4409d2092d0a5475aca0f28995034e58d14/design/storage-proof-timing.md). be required using the `willProofBeRequired(slotId)` in case the node is in [downtime](https://github.com/codex-storage/codex-research/blob/41c4b4409d2092d0a5475aca0f28995034e58d14/design/storage-proof-timing.md).
Once node knows it has to provide a proof it MUST obtain the proof challenge using `getChallenge(slotId)` which then Once node knows it has to provide a proof it MUST get the proof challenge using `getChallenge(slotId)` which then
NEEDS to be incorporated into the proof generation as described in Proving RFC (**TODO: Proving RFC**). NEEDS to be incorporated into the proof generation as described in Proving RFC (**TODO: Proving RFC**).
When proof is generated it MUST be submitted with a transaction calling `submitProof(slotId, proof)` function.
#### Slashing #### Slashing
There is a slashing scheme in place that is orchestrated by the smart contract to incentive correct behavior There is a slashing scheme in place that is orchestrated by the smart contract to incentive correct behavior
@ -304,7 +307,7 @@ the same for all the participants in the network. The concrete values of this sc
The slashing works in the following way: The slashing works in the following way:
- Node MAY miss at most `config.collateral.slashCriterion` proofs before it is slashed. - Node MAY miss at most `config.collateral.slashCriterion` proofs before it is slashed.
- It is then slashed `config.collateral.slashPercentage` percentage **of the originally asked collateral**. - It is then slashed `config.collateral.slashPercentage` percentage **of the originally asked collateral** (hence the slashing amount is always the same for the given request).
- If the number of times the node was slashed reaches above `config.collateral.maxNumberOfSlashes`, then the slot is freed, the remaining of node's collateral is burned and the slot is offered to other nodes for repair. Contract also emits the `SlotFreed(requestId, slotIndex)` event. - If the number of times the node was slashed reaches above `config.collateral.maxNumberOfSlashes`, then the slot is freed, the remaining of node's collateral is burned and the slot is offered to other nodes for repair. Contract also emits the `SlotFreed(requestId, slotIndex)` event.
If the number of concurrent freed slots reaches above the `request.ask.maxSlotLoss`, then the dataset is lost and the Request is failed. If the number of concurrent freed slots reaches above the `request.ask.maxSlotLoss`, then the dataset is lost and the Request is failed.
@ -314,11 +317,13 @@ The collateral of all the nodes that hosted Request's slots is burned and the ev
When slot is freed because of too many missed proofs, which SHOULD be detected by listening on the `SlotFreed(requestId, slotIndex)` event, then When slot is freed because of too many missed proofs, which SHOULD be detected by listening on the `SlotFreed(requestId, slotIndex)` event, then
storage provider node can decide if it wants to participate in the repairing of the slot. The node SHOULD, similarly like with slot's filling, storage provider node can decide if it wants to participate in the repairing of the slot. The node SHOULD, similarly like with slot's filling,
consider the node's operator configuration when making the decision. consider the node's operator configuration when making the decision. The storage provider node that originally hosted
the freed slot MAY also participate in the data repair, but my refilling the slot it **won't** recover its original collateral
and needs to submit new collateral with the `fillSlot()` call.
The repair process is the same as with the filling slots, with one difference that the node MUST use the erasure coding to The repair process is the same as with the filling slots, with one difference that the node MUST use the erasure coding to
reconstruct the original dataset. As this requires retrieving more data of the dataset from the network, the node that reconstruct the original dataset. As this requires retrieving more data of the dataset from the network, the node that
will successfully fill the repair node will be granted additional reward. (**TODO: Implementation**) will successfully repair slot by filling the freed slot will be granted additional reward. (**TODO: Implementation**)
The repair process is then as follows: The repair process is then as follows:
@ -330,13 +335,13 @@ The repair process is then as follows:
### Collecting funds ### Collecting funds
Storage Provider node SHOULD monitor Requests and slots it hosts. In case it needs to discover what slots it is hosting, Storage Provider node SHOULD monitor Requests and slots it hosts. In case it needs to discover what slots it is hosting,
for example, because the node had to restart, then it SHOULD use the contract call `mySlots()`, which returns slots IDs for example, because the node had to restart, then it SHOULD use the contract call `mySlots()`, which returns array of `SlotID`s
associated with the Ethereum address from which the contract call originates. This list is kept up to date by the smart contract itself. associated with the Ethereum address from which the contract call originates. This list is kept up to date by the smart contract itself.
When node slot's Requests reaches states `Cancelled`, `Finished` or `Failed` it SHOULD call the contract's `freeSlot(slotId)` function. When node slot's Requests reaches states `Cancelled`, `Finished` or `Failed` it SHOULD call the contract's `freeSlot(slotId)` function.
These states can be detected using: These states can be detected using:
- `Cancelled` state MAY be detected by setting timeout using `expiry` **and** not detecting `RequestFulfilled(requestId)` event. There is also `RequestCancelled` event emitted, yet that is not guaranteed to be emitted at the time of expiry. - `Cancelled` state MAY be detected by setting timeout using `expiry` **and** not detecting `RequestFulfilled(requestId)` event. There is also `RequestCancelled` event emitted, but the node SHOULD NOT use it for asserting expiry as it is not guaranteed to be emitted at the time of expiry.
- `Finished` state MAY be detected by setting timeout specified from function `getRequestEnd(requestId)`. - `Finished` state MAY be detected by setting timeout specified from function `getRequestEnd(requestId)`.
- `Failed` state MAY be detected by listening to the `RequestFailed(requestId)` event emitted. - `Failed` state MAY be detected by listening to the `RequestFailed(requestId)` event emitted.
@ -360,6 +365,9 @@ to add the slot to the watched slots. Then after the end of every period validat
it SHOULD submit transaction with call to the function `markProofAsMissing(slotId, period)` that validates the correctness it SHOULD submit transaction with call to the function `markProofAsMissing(slotId, period)` that validates the correctness
and if right, will reward the validator with a reward. and if right, will reward the validator with a reward.
Validator MAY decide to validate only part of the slot's space when it detects that it can't keep up validating slots
before the end of validation `timeout`.
## Copyright ## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).