mirror of
https://github.com/logos-storage/logos-storage-spec.git
synced 2026-01-03 22:13:08 +00:00
chore: proof reading and tweaks
This commit is contained in:
Adam Uhlíř
parent
7445fb5014
commit
c97b056f70
@ -40,31 +40,30 @@ The Codex network aims to create a peer-to-peer storage engine with strong data
|
||||
data persistence guarantees, and node storage incentives.
|
||||
|
||||
An important component of Codex network is a Marketplace. It is a place which mediates negotiations of all parties
|
||||
in order to provide persistence in the network. It also provides ways to enforce agreements and **TODO** with repair.
|
||||
to provide persistence in the network. It also provides ways to enforce agreements and facilitate repair when a storage node drops out.
|
||||
|
||||
Marketplace is defined by smart-contract that is deployed to EVM-compatible blockchain. It has several flows which are
|
||||
Marketplace is defined by smart-contract deployed to EVM-compatible blockchain. It has several flows which are
|
||||
linked with roles in the network and which the participating node takes upon itself. It can be one role or multiple at the same time.
|
||||
This specification describes these flows.
|
||||
|
||||
The Marketplace handles storage requests, the storage slot state,
|
||||
storage provider rewards, storage provider collaterals, and storage proof state.
|
||||
|
||||
If a node implementation wants to participate in the persistence layer of Codex it needs to choose which role(s) it wants
|
||||
to support and implement properer flows otherwise it won't be compatible with the rest of the Codex network.
|
||||
If a node implementation wants to participate in the persistence layer of Codex, it needs to choose which role(s) it wants
|
||||
to support and implement proper flows. Otherwise, it won't be compatible with the rest of the Codex network.
|
||||
|
||||
### Roles
|
||||
|
||||
There are 3 main roles in the network - client, storage provider (SP) and validator.
|
||||
There are three main roles in the network - client, storage provider (SP) and validator.
|
||||
|
||||
Client is a potentially short-lived node in the network, that interacts with it with the main purpose of persisting
|
||||
Client is a potentially short-lived node in the network that mainly interacts with the purpose of persisting
|
||||
its data in the network.
|
||||
|
||||
Storage Provider is long-term participant in the network, that store other's data for profit. It needs to provide a proof
|
||||
Storage Provider is a long-term participant in the network that stores other data for profit. It needs to provide a proof
|
||||
to the smart contract that it possesses the data from time to time.
|
||||
|
||||
Validator is a node that helps with enforcing the storage provider's duties to comply with storage requests that they
|
||||
accepted. When it detects that SP should have submitted a proof but non was submitted on-chain, it triggers on-chain
|
||||
function which will handle this case. Validator is rewarded for correct invocation of this function.
|
||||
Validator validates that Storage Providers correctly are fulfilling their duties and that they provide proofs of storage
|
||||
on time.
|
||||
|
||||
## Storage Request Lifecycle
|
||||
|
||||
@ -97,11 +96,11 @@ function which will handle this case. Validator is rewarded for correct invocati
|
||||
└────────┬─┬─────────────────────────────────────────────────┘
|
||||
│ │ ▲
|
||||
│ │ │
|
||||
│ │ SP kicked out and Slot Freed ┌───────┴────────┐
|
||||
│ │ SP kicked out and Slot freed ┌───────┴────────┐
|
||||
All good │ ├─────────────────────────────►│ Repair process │
|
||||
Time ran out │ │ └────────────────┘
|
||||
│ │
|
||||
│ │ Too much Slots Freed ┌────────┐
|
||||
│ │ Too many Slots freed ┌────────┐
|
||||
│ └─────────────────────────────►│ Failed │
|
||||
▼ └────────┘
|
||||
┌──────────┐
|
||||
@ -113,27 +112,27 @@ Time ran out │ │ └─────────
|
||||
|
||||
## Client role
|
||||
|
||||
Client role represent nodes that mediate persisting data inside Codex network.
|
||||
The Client role represents nodes that mediate persisting data inside the Codex network.
|
||||
|
||||
There are 2 parts for client role:
|
||||
There are two parts of a client role:
|
||||
|
||||
- Requesting storage from the network - creating storage request.
|
||||
- Withdrawing funds from storage requests.
|
||||
|
||||
### Creating storage requests
|
||||
|
||||
When the client node is prompted by the user to create a storage request,
|
||||
When the user prompts the client node to create a storage request,
|
||||
it SHOULD receive the input parameters for the storage request from the user.
|
||||
|
||||
To create a request to persist a dataset on the Codex network,
|
||||
client nodes MUST split the dataset into data chunks, $(c_1, c_2, c_3, \ldots, c_{n})$.
|
||||
Using an erasure coding technique and input parameters, the data chunks are encoded and placed into separate slots.
|
||||
The erasure coding technique MUST be the [Reed-Soloman algorithm](https://hackmd.io/FB58eZQoTNm-dnhu0Y1XnA).
|
||||
The final slot's roots and other metadata MUST be placed into Manifest. Manifest's CID is then used as the `cid` of the
|
||||
stored dataset.
|
||||
The final slot's roots and other metadata MUST be placed into Manifest (**TODO: Manifest RFC**). Manifest's CID MUST then
|
||||
be used as the `cid` of the stored dataset.
|
||||
|
||||
After the dataset is prepared, it MUST submit a transaction with the desired request parameters which are represented
|
||||
as `Request` object and its sub-objects. Bellow are described its properties:
|
||||
After the dataset is prepared, node MUST submit a transaction with the desired request parameters which are represented
|
||||
as `Request` object and its sub-objects to function `requestStorage(request)`. Bellow are described its properties:
|
||||
|
||||
```solidity
|
||||
struct Request {
|
||||
@ -187,35 +186,34 @@ struct Content {
|
||||
|
||||
```
|
||||
|
||||
Notes about some of the parameters:
|
||||
Notes about some parameters:
|
||||
|
||||
`cid`
|
||||
|
||||
An identifier used to locate the Manifest representing the dataset.
|
||||
- MUST be a [CIDv1](https://github.com/multiformats/cid#cidv1) with sha-256 based [multihash](https://github.com/multiformats/multihash).
|
||||
- Data it represents SHOULD be discoverable in the network, otherwise Request will get cancelled.
|
||||
- Data it represents SHOULD be discoverable in the network, otherwise the Request will get canceled.
|
||||
|
||||
`reward`
|
||||
|
||||
- It is an REQUIRED amount to be included in the transaction for a storage request.
|
||||
- It SHOULD be an amount of tokens offered per slot per second.
|
||||
- The Client address MUST have [approval](https://docs.openzeppelin.com/contracts/2.x/api/token/erc20#IERC20-approve-address-uint256-) for transfer of at least the same amount on the ERC20 based token, that the network utilizes.
|
||||
- It MUST be an amount of tokens offered per slot per second.
|
||||
- The Ethereum address that submits the `requestStorage()` transaction MUST have [approval](https://docs.openzeppelin.com/contracts/2.x/api/token/erc20#IERC20-approve-address-uint256-) for transfer of at least the same amount on the ERC20 based token, that the network uses.
|
||||
|
||||
`collateral`
|
||||
|
||||
- Amount of tokens that the storage providers MUST submit when they fill slots.
|
||||
- Amount of tokens that the storage providers submit when they fill slots.
|
||||
- Collateral is then slashed or forfeited if the storage providers fail to provide the service requested by the Request (more information bellow).
|
||||
|
||||
`proofProbability`
|
||||
|
||||
Determines the inverse probability that a proof is required in a period: $\frac{1}{proofProbability}$
|
||||
|
||||
- Storage providers are REQUIRED to provide proofs of storage to the marketplace smart contract when they are prompted to by the smart contract.
|
||||
- Storage providers are required to provide proofs of storage to the marketplace smart contract when they are prompted to by the smart contract.
|
||||
- The frequency is non-deterministic in order to prevent from pre-calculation attacks, but it is affected by this parameter.
|
||||
|
||||
`expiry`
|
||||
|
||||
- Parameter is specified as duration in seconds, hence the final deadline timestamp is calculated at the moment when the transaction is mined.
|
||||
- The parameter is specified as duration in seconds, hence the final deadline timestamp is calculated at the moment when the transaction is mined.
|
||||
|
||||
`nonce`
|
||||
|
||||
@ -223,15 +221,15 @@ Determines the inverse probability that a proof is required in a period: $\frac{
|
||||
|
||||
#### Renewal of Storage Request
|
||||
|
||||
It should be noted that Marketplace does not support extending Requests. It is REQUIRED that if user wants to
|
||||
It should be noted that Marketplace does not support extending Requests. It is REQUIRED that if the user wants to
|
||||
extend the Request's duration, somebody submits a new Request transaction with the same CID **well before the original
|
||||
Request finishes**. In this way the data will be still persisted in the network at the time when new (or the current) storage providers
|
||||
can retrieve the dataset in order to fill slots of the new Request.
|
||||
Request finishes**. In this way, the data will be still persisted in the network at the time when new (or the current) storage providers
|
||||
need to retrieve the dataset to fill slots of the new Request.
|
||||
|
||||
### Withdrawing funds
|
||||
|
||||
The client node SHOULD monitor the status of Requests that it created. The node can utilize on-chain state in order to
|
||||
fetch the list of the active Requests linked to the client node's blockchain address using function `myRequests()`, that
|
||||
The client node SHOULD monitor the status of Requests that it created. The node can utilize on-chain state to
|
||||
fetch the list of the active Requests linked to the client node's ethereum address using function `myRequests()`, that
|
||||
returns array of `RequestId`s. This list is kept up to date by the smart contract itself.
|
||||
|
||||
When Request reaches states `Cancelled` (not all slots filled after `expiry` timeout) or `Failed` (too many slots gets freed and data is non-recoverable)
|
||||
@ -239,7 +237,7 @@ the client node SHOULD initiate withdrawal of the remaining funds from the contr
|
||||
|
||||
- `Cancelled` state MAY be detected using timeout specified from function `requestExpiresAt(requestId)` **and** not detecting emitted `RequestFulfilled(requestId)` event.
|
||||
- `Failed` state MAY be detected using `RequestFailed(requestId)` event emitted from the smart contract.
|
||||
- `Finished` state MAY be detected setting timeout specified from function `getRequestEnd(requestId)`.
|
||||
- `Finished` state MAY be detected by setting timeout specified from function `getRequestEnd(requestId)`.
|
||||
|
||||
## Storage Provider role
|
||||
|
||||
@ -250,7 +248,7 @@ There are several parts to hosting a slot:
|
||||
|
||||
- Filling a slot
|
||||
- Proving
|
||||
- Repairing slots
|
||||
- Repairing a slot
|
||||
- Collecting Request's reward and collateral
|
||||
|
||||
### Filling slot
|
||||
@ -263,38 +261,43 @@ When new Request is created `StorageRequested(requestId, ask, expiry)` event is
|
||||
|
||||
It is then up to the Storage Provider node to decide based on the emitted parameters if it wants to participate in the
|
||||
Request and try to fill its slot(s). This decision SHOULD be done based on parameters specified by the node operator.
|
||||
If the node decide to ignore this Request, no action is need, otherwise the node HAVE TO follow the remaining steps.
|
||||
If the node decides to ignore this Request, no action is necessary. If the node wants to try to fill a a slot, then
|
||||
it MUST follow the remaining steps.
|
||||
|
||||
Node MUST decide which Slot specified by slot's index it wants to try to fill in. Node MAY try filling multiple
|
||||
slots. In order to fill a slot, node first MUST download the slot's data using slot's root that can be retrieved
|
||||
from Manifest specified in `request.content.cid` (**TODO: Manifest RFC**). This object can be retrieved from the smart contract using `getRequest(requestId)`.
|
||||
Then node MUST generate proof over the downloaded data (**TODO: Proving RFC**).
|
||||
from Manifest specified in `request.content.cid` (**TODO: Manifest RFC**).
|
||||
This Request object can be retrieved from the smart contract using `getRequest(requestId)`.
|
||||
Then the node MUST generate proof over the downloaded data (**TODO: Proving RFC**).
|
||||
|
||||
When proof is ready it then MUST create transaction for smart contract call `fillSlot()` with following REQUIRED:
|
||||
|
||||
- Parameters:
|
||||
- `requestId` - ID of the Request.
|
||||
- `slotIndex` - Index that the node is trying to fill.
|
||||
- `proof` - `Groth16Proof` proof structure, generated over the dataset.
|
||||
- `proof` - `Groth16Proof` proof structure, generated over the slot's data.
|
||||
- The Ethereum address of the node from which the transaction originates MUST have [approval](https://docs.openzeppelin.com/contracts/2.x/api/token/erc20#IERC20-approve-address-uint256-) for transfer of at least the amount required as collateral for the Request on the ERC20 based token, that the network utilizes.
|
||||
|
||||
If the proof is invalid, or slot was already filled by other node then the transaction
|
||||
will revert, otherwise `SlotFilled(requestId, slotIndex)` event is emitted. If the transaction is successful then the
|
||||
If the proof is invalid, or slot was already filled by another node, then the transaction
|
||||
will revert, otherwise `SlotFilled(requestId, slotIndex)` event is emitted. If the transaction is successful, then the
|
||||
node SHOULD transition into __proving__ state as it will need to submit proof of data possession when prompted by the
|
||||
contract.
|
||||
|
||||
It should be noted that if the node see the `SlotFilled` emitted for slot that he is downloading the dataset or
|
||||
generating proof for, then node SHOULD stop and choose different non-filled slot to try to fill.
|
||||
It should be noted that if the node sees the `SlotFilled` emitted for slot that he is downloading the dataset or
|
||||
generating proof for, then node SHOULD stop and choose different non-filled slot to try to fill as the chosen slot
|
||||
was filled by other node.
|
||||
|
||||
### Proving
|
||||
|
||||
Once node fills a slot it MUST periodically, yet non-deterministically provide proof to the smart contract that it
|
||||
stores the data it should. Node MAY detect that proof is required using the `isProofRequired(slotId)` or that it will
|
||||
Once a node fills a slot, it MUST periodically, yet non-deterministically provide proof to the smart contract that it
|
||||
stores the data it should. Node SHOULD detect that proof is required using the `isProofRequired(slotId)` or that it will
|
||||
be required using the `willProofBeRequired(slotId)` in case the node is in [downtime](https://github.com/codex-storage/codex-research/blob/41c4b4409d2092d0a5475aca0f28995034e58d14/design/storage-proof-timing.md).
|
||||
|
||||
Once node knows it has to provide a proof it MUST obtain the proof challenge using `getChallenge(slotId)` which then
|
||||
Once node knows it has to provide a proof it MUST get the proof challenge using `getChallenge(slotId)` which then
|
||||
NEEDS to be incorporated into the proof generation as described in Proving RFC (**TODO: Proving RFC**).
|
||||
|
||||
When proof is generated it MUST be submitted with a transaction calling `submitProof(slotId, proof)` function.
|
||||
|
||||
#### Slashing
|
||||
|
||||
There is a slashing scheme in place that is orchestrated by the smart contract to incentive correct behavior
|
||||
@ -304,7 +307,7 @@ the same for all the participants in the network. The concrete values of this sc
|
||||
The slashing works in the following way:
|
||||
|
||||
- Node MAY miss at most `config.collateral.slashCriterion` proofs before it is slashed.
|
||||
- It is then slashed `config.collateral.slashPercentage` percentage **of the originally asked collateral**.
|
||||
- It is then slashed `config.collateral.slashPercentage` percentage **of the originally asked collateral** (hence the slashing amount is always the same for the given request).
|
||||
- If the number of times the node was slashed reaches above `config.collateral.maxNumberOfSlashes`, then the slot is freed, the remaining of node's collateral is burned and the slot is offered to other nodes for repair. Contract also emits the `SlotFreed(requestId, slotIndex)` event.
|
||||
|
||||
If the number of concurrent freed slots reaches above the `request.ask.maxSlotLoss`, then the dataset is lost and the Request is failed.
|
||||
@ -314,11 +317,13 @@ The collateral of all the nodes that hosted Request's slots is burned and the ev
|
||||
|
||||
When slot is freed because of too many missed proofs, which SHOULD be detected by listening on the `SlotFreed(requestId, slotIndex)` event, then
|
||||
storage provider node can decide if it wants to participate in the repairing of the slot. The node SHOULD, similarly like with slot's filling,
|
||||
consider the node's operator configuration when making the decision.
|
||||
consider the node's operator configuration when making the decision. The storage provider node that originally hosted
|
||||
the freed slot MAY also participate in the data repair, but my refilling the slot it **won't** recover its original collateral
|
||||
and needs to submit new collateral with the `fillSlot()` call.
|
||||
|
||||
The repair process is the same as with the filling slots, with one difference that the node MUST use the erasure coding to
|
||||
reconstruct the original dataset. As this requires retrieving more data of the dataset from the network, the node that
|
||||
will successfully fill the repair node will be granted additional reward. (**TODO: Implementation**)
|
||||
will successfully repair slot by filling the freed slot will be granted additional reward. (**TODO: Implementation**)
|
||||
|
||||
The repair process is then as follows:
|
||||
|
||||
@ -330,13 +335,13 @@ The repair process is then as follows:
|
||||
### Collecting funds
|
||||
|
||||
Storage Provider node SHOULD monitor Requests and slots it hosts. In case it needs to discover what slots it is hosting,
|
||||
for example, because the node had to restart, then it SHOULD use the contract call `mySlots()`, which returns slots IDs
|
||||
for example, because the node had to restart, then it SHOULD use the contract call `mySlots()`, which returns array of `SlotID`s
|
||||
associated with the Ethereum address from which the contract call originates. This list is kept up to date by the smart contract itself.
|
||||
|
||||
When node slot's Requests reaches states `Cancelled`, `Finished` or `Failed` it SHOULD call the contract's `freeSlot(slotId)` function.
|
||||
These states can be detected using:
|
||||
|
||||
- `Cancelled` state MAY be detected by setting timeout using `expiry` **and** not detecting `RequestFulfilled(requestId)` event. There is also `RequestCancelled` event emitted, yet that is not guaranteed to be emitted at the time of expiry.
|
||||
- `Cancelled` state MAY be detected by setting timeout using `expiry` **and** not detecting `RequestFulfilled(requestId)` event. There is also `RequestCancelled` event emitted, but the node SHOULD NOT use it for asserting expiry as it is not guaranteed to be emitted at the time of expiry.
|
||||
- `Finished` state MAY be detected by setting timeout specified from function `getRequestEnd(requestId)`.
|
||||
- `Failed` state MAY be detected by listening to the `RequestFailed(requestId)` event emitted.
|
||||
|
||||
@ -360,6 +365,9 @@ to add the slot to the watched slots. Then after the end of every period validat
|
||||
it SHOULD submit transaction with call to the function `markProofAsMissing(slotId, period)` that validates the correctness
|
||||
and if right, will reward the validator with a reward.
|
||||
|
||||
Validator MAY decide to validate only part of the slot's space when it detects that it can't keep up validating slots
|
||||
before the end of validation `timeout`.
|
||||
|
||||
## Copyright
|
||||
|
||||
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user