From 264bfa17f54b7973432d000446a725a4b5a6134a Mon Sep 17 00:00:00 2001 From: Marcin Czenko Date: Sun, 29 Sep 2024 21:18:01 +0200 Subject: [PATCH] updates Marketplace tutorial documentation (#888) * updates Marketplace tutorial documentation * Applies review comments to marketplace-tutorial * Final formatting touches * moved `Prerequisites` around * Fixes indentation in one JSON snippet --- docs/Marketplace.md | 474 ++++++++++++++++++++++++++++++++------------ 1 file changed, 348 insertions(+), 126 deletions(-) diff --git a/docs/Marketplace.md b/docs/Marketplace.md index 53712b0c..0e69c295 100644 --- a/docs/Marketplace.md +++ b/docs/Marketplace.md @@ -1,13 +1,9 @@ # Running a Local Codex Network with Marketplace Support -This tutorial will teach you how to run a small Codex network with the _storage marketplace_ enabled; i.e., the functionality in Codex which allows participants to offer and buy storage in a market, ensuring that storage providers honor their part of the deal by means of cryptographic proofs. - -To complete this tutorial, you will need: - -* the [geth](https://github.com/ethereum/go-ethereum) Ethereum client; -* a Codex binary, which [you can compile from source](https://github.com/codex-storage/nim-codex?tab=readme-ov-file#build-and-run). - -We will also be using [bash](https://en.wikipedia.org/wiki/Bash_(Unix_shell)) syntax throughout. If you use a different shell, you may need to adapt things to your platform. +This tutorial will teach you how to run a small Codex network with the +_storage marketplace_ enabled; i.e., the functionality in Codex which +allows participants to offer and buy storage in a market, ensuring that +storage providers honor their part of the deal by means of cryptographic proofs. In this tutorial, you will: @@ -16,55 +12,91 @@ In this tutorial, you will: 3. [Run Codex](#3-run-codex); 4. [Buy and Sell Storage in the Marketplace](#4-buy-and-sell-storage-on-the-marketplace). -We strongly suggest you to create a folder (e.g. `marketplace-tutorial`), and switch into it before beginning. +## Prerequisites + +To complete this tutorial, you will need: + +* the [geth](https://github.com/ethereum/go-ethereum) Ethereum client; + You need version `1.13.x` of geth as newer versions no longer support + Proof of Authority (PoA). This tutorial was tested using geth version `1.13.15`. +* a Codex binary, which [you can compile from source](https://github.com/codex-storage/nim-codex?tab=readme-ov-file#build-and-run). + +We will also be using [bash](https://en.wikipedia.org/wiki/Bash_(Unix_shell)) +syntax throughout. If you use a different shell, you may need to adapt +things to your platform. + +To get started, create a new folder where we will keep the tutorial-related +files so that we can keep them separate from the codex repository. +We assume the name of the folder to be `marketplace-tutorial`. ## 1. Set Up a Geth PoA Network -For this tutorial, we will use a simple [Proof-of-Authority](https://github.com/ethereum/EIPs/issues/225) network with geth. The first step is creating a _signer account_: an account which will be used by geth to sign the blocks in the network. Any block signed by a signer is accepted as valid. +For this tutorial, we will use a simple +[Proof-of-Authority](https://github.com/ethereum/EIPs/issues/225) network +with geth. The first step is creating a _signer account_: an account which +will be used by geth to sign the blocks in the network. +Any block signed by a signer is accepted as valid. ### 1.1. Create a Signer Account -To create a signer account, run: +To create a signer account, from the `marketplace-tutorial` directory run: ```bash geth account new --datadir geth-data ``` -The account generator will ask you to input a password, which you can leave blank. It will then print some information, including the account's public address: +The account generator will ask you to input a password, which you can +leave blank. It will then print some information, +including the account's public address: ```bash -INFO [03-22|12:58:05.637] Maximum peer count ETH=50 total=50 -INFO [03-22|12:58:05.638] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory" +INFO [09-29|16:49:24.244] Maximum peer count ETH=50 total=50 Your new account is locked with a password. Please give a password. Do not forget this password. Password: Repeat password: Your new key was generated -Public address of the key: 0x93976895c4939d99837C8e0E1779787718EF8368 -... +Public address of the key: 0x33A904Ad57D0E2CB8ffe347D3C0E83C2e875E7dB +Path of the secret key file: geth-data/keystore/UTC--2024-09-29T14-49-31.655272000Z--33a904ad57d0e2cb8ffe347d3c0e83c2e875e7db + +- You can share your public address with anyone. Others need it to interact with you. +- You must NEVER share the secret key with anyone! The key controls access to your funds! +- You must BACKUP your key file! Without the key, it's impossible to access account funds! +- You must REMEMBER your password! Without the password, it's impossible to decrypt the key! ``` -In this example, the public address of the signer account is `0x93976895c4939d99837C8e0E1779787718EF8368`. Yours will print a different address. Save it for later usage. +In this example, the public address of the signer account is +`0x33A904Ad57D0E2CB8ffe347D3C0E83C2e875E7dB`. +Yours will print a different address. Save it for later usage. Next set an environment variable for later usage: -```sh +```bash export GETH_SIGNER_ADDR="0x0000000000000000000000000000000000000000" echo ${GETH_SIGNER_ADDR} > geth_signer_address.txt ``` +> Here make sure you replace `0x0000000000000000000000000000000000000000` +> with your public address of the signer account +> (`0x93976895c4939d99837C8e0E1779787718EF8368` in our example). + ### 1.2. Configure The Network and Create the Genesis Block -The next step is telling geth what kind of network you want to run. We will be running a [pre-merge](https://ethereum.org/en/roadmap/merge/) network with Proof-of-Authority consensus. To get that working, create a `network.json` file. +The next step is telling geth what kind of network you want to run. +We will be running a [pre-merge](https://ethereum.org/en/roadmap/merge/) +network with Proof-of-Authority consensus. +To get that working, create a `network.json` file. -If you set the GETH_SIGNER_ADDR variable above you can run to create the `network.json` file: +If you set the `GETH_SIGNER_ADDR` variable above you can run the following +command to create the `network.json` file: -```sh +```bash echo "{\"config\": { \"chainId\": 12345, \"homesteadBlock\": 0, \"eip150Block\": 0, \"eip155Block\": 0, \"eip158Block\": 0, \"byzantiumBlock\": 0, \"constantinopleBlock\": 0, \"petersburgBlock\": 0, \"istanbulBlock\": 0, \"berlinBlock\": 0, \"londonBlock\": 0, \"arrowGlacierBlock\": 0, \"grayGlacierBlock\": 0, \"clique\": { \"period\": 1, \"epoch\": 30000 } }, \"difficulty\": \"1\", \"gasLimit\": \"8000000\", \"extradata\": \"0x0000000000000000000000000000000000000000000000000000000000000000${GETH_SIGNER_ADDR:2}0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000\", \"alloc\": { \"${GETH_SIGNER_ADDR}\": { \"balance\": \"10000000000000000000000\"}}}" > network.json ``` -You can also manually create the file with the following content modified with your signer private key: +You can also manually create the file remembering update it with your +signer public address: ```json { @@ -89,9 +121,9 @@ You can also manually create the file with the following content modified with y }, "difficulty": "1", "gasLimit": "8000000", - "extradata": "0x000000000000000000000000000000000000000000000000000000000000000093976895c4939d99837C8e0E1779787718EF83680000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", + "extradata": "0x000000000000000000000000000000000000000000000000000000000000000033A904Ad57D0E2CB8ffe347D3C0E83C2e875E7dB0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", "alloc": { - "0x93976895c4939d99837C8e0E1779787718EF8368": { + "0x33A904Ad57D0E2CB8ffe347D3C0E83C2e875E7dB": { "balance": "10000000000000000000000" } } @@ -99,10 +131,11 @@ You can also manually create the file with the following content modified with y ``` Note that the signer account address is embedded in two different places: -* inside of the `"extradata"` string, surrounded by zeroes and stripped of its `0x` prefix; +* inside of the `"extradata"` string, surrounded by zeroes and stripped of + its `0x` prefix; * as an entry key in the `alloc` session. -Make sure to replace that ID with the account ID that you wrote down in Step 1.1. - + Make sure to replace that ID with the account ID that you wrote down in + [Step 1.1](#11-create-a-signer-account). Once `network.json` is created, you can initialize the network with: @@ -110,9 +143,36 @@ Once `network.json` is created, you can initialize the network with: geth init --datadir geth-data network.json ``` +The output of the above command you may include some warnings, like: + +```bash +WARN [08-21|14:48:12.305] Unknown config environment variable envvar=GETH_SIGNER_ADDR +``` + +or even errors when running the command for the first time: + +```bash +ERROR[08-21|14:48:12.399] Head block is not reachable +``` + +The important part is that at the end you should see something similar to: + +```bash +INFO [08-21|14:48:12.639] Successfully wrote genesis state database=lightchaindata hash=768bf1..42d06a +``` + ### 1.3. Start your PoA Node -We are now ready to start our $1$-node, private blockchain. To launch the signer node, open a separate terminal on the same working directory and run: +We are now ready to start our $1$-node, private blockchain. +To launch the signer node, open a separate terminal in the same working +directory and make sure you have the `GETH_SIGNER_ADDR` set. +For convenience use the `geth_signer_address.txt`: + +```bash +export GETH_SIGNER_ADDR=$(cat geth_signer_address.txt) +``` + +Having the `GETH_SIGNER_ADDR` variable set, run: ```bash geth\ @@ -127,57 +187,138 @@ geth\ --allow-insecure-unlock ``` -Note that, once again, the signer account created in Step 1.1 appears both in `--unlock` and `--allow-insecure-unlock`. Make sure you have the `GETH_SIGNER_ADDR` set. +Note that, once again, the signer account created in +[Step 1.1](#11-create-a-signer-account) appears both in +`--unlock` and `--allow-insecure-unlock`. -Geth will prompt you to insert the account's password as it starts up. Once you do that, it should be able to start up and begin "mining" blocks. +Geth will prompt you to insert the account's password as it starts up. +Once you do that, it should be able to start up and begin "mining" blocks. + +Also here, you may encounter errors like: + +```bash +ERROR[08-21|15:00:27.625] Bootstrap node filtered by netrestrict id=c845e51a5e470e44 ip=18.138.108.67 +ERROR[08-21|15:00:27.625] Bootstrap node filtered by netrestrict id=f23ac6da7c02f84a ip=3.209.45.79 +ERROR[08-21|15:00:27.625] Bootstrap node filtered by netrestrict id=ef2d7ab886910dc8 ip=65.108.70.101 +ERROR[08-21|15:00:27.625] Bootstrap node filtered by netrestrict id=6b36f791352f15eb ip=157.90.35.166 +``` + +You can safely ignore them. + +If the command above fails with: + +```bash +Fatal: Failed to register the Ethereum service: only PoS networks are supported, please transition old ones with Geth v1.13.x +``` + +make sure, you are running the correct Geth version +(see Section [Prerequisites](#prerequisites)) ## 2. Set Up The Marketplace -You will need to open new terminal for this section and geth needs to be running already. Setting up the Codex marketplace entails: +You will need to open new terminal for this section and geth needs to be +running already. Setting up the Codex marketplace entails: 1. Deploying the Codex Marketplace contracts to our private blockchain -2. Setup Ethereum accounts we will use to buy and sell storage in the Codex marketplace +2. Setup Ethereum accounts we will use to buy and sell storage in + the Codex marketplace 3. Provisioning those accounts with the required token balances ### 2.1. Deploy the Codex Marketplace Contracts -To deploy the contracts, start by cloning the Codex contracts repository locally and installing its dependencies: +Make sure you leave the `marketplace-tutorial` directory, and clone +the `codex-storage/nim-codex.git`: ```bash -git clone https://github.com/codex-storage/codex-contracts-eth -cd codex-contracts-eth +git clone https://github.com/codex-storage/nim-codex.git +``` + +> If you just want to clone the repo to run the tutorial, you can +> skip the history and just download the head of the master branch by using +> `--depth 1` option: `git clone --depth 1 https://github.com/codex-storage/nim-codex.git` + +Thus, our directory structure for the purpose of this tutorial looks like this: + +```bash +| +|-- nim-codex +└-- marketplace-tutorial +``` + +> You could clone the `codex-storage/nim-codex.git` to some other location. +> Just to keeps things nicely separated it is best to make sure that +> `nim-codex` is not under `marketplace-tutorial` directory. + +Now, from the `nim-codex` folder run: + +```bash +make update && make +``` + +> This may take a moment as it will also build the `nim` compiler. Be patient. + +Now, in order to start a local Ethereum network run: + +```bash +cd vendor/codex-contracts-eth npm install ``` -You now must **wait until $256$ blocks are mined in your PoA network**, or deploy will fail. This should take about $4$ minutes and $30$ seconds. You can check which block height you are currently at by running: + +> While writing the document we used `node` version `v20.17.0` and +> `npm` version `10.8.2`. + +Before continuing you now must **wait until $256$ blocks are mined** +**in your PoAnetwork**, or deploy will fail. This should take about +$4$ minutes and $30$ seconds. You can check which block height you are +currently at by running the following command +**from the `marketplace-tutorial` folder**: ```bash -geth attach --exec web3.eth.blockNumber ../geth-data/geth.ipc +geth attach --exec web3.eth.blockNumber ./geth-data/geth.ipc ``` -once that gets past $256$, you are ready to go. To deploy contracts, run: +once that gets past $256$, you are ready to go. + +To deploy contracts, from the `codex-contracts-eth` directory run: ```bash -export DISTTEST_NETWORK_URL=http://localhost:8545 # bootstrap node -npx hardhat --network codexdisttestnetwork deploy && cd ../ +export DISTTEST_NETWORK_URL=http://localhost:8545 +npx hardhat --network codexdisttestnetwork deploy ``` -If the command completes successfully, you are ready to prepare the accounts. +If the command completes successfully, you will see the output similar +to this one: + +```bash +Deployed Marketplace with Groth16 Verifier at: +0xCf0df6C52B02201F78E8490B6D6fFf5A82fC7BCd +``` +> of course your address will be different. + +You are now ready to prepare the accounts. ### 2.2. Generate the Required Accounts -We will run $2$ Codex nodes: a **storage provider**, which will sell storage on the network, and a **client**, which will buy and use such storage; we therefore need two valid Ethereum accounts. We could create random accounts by using one of the many tools available to that end but, since this is a tutorial running on a local private network, we will simply provide you with two pre-made accounts along with their private keys which you can copy and paste instead: +We will run $2$ Codex nodes: a **storage provider**, which will sell storage +on the network, and a **client**, which will buy and use such storage; +we therefore need two valid Ethereum accounts. We could create random +accounts by using one of the many tools available to that end but, since +this is a tutorial running on a local private network, we will simply +provide you with two pre-made accounts along with their private keys, +which you can copy and paste instead: -First make sure you're back in the `marketplace-tutorial` folder and not the `codex-contracts-eth` subfolder. Then set these variables: +First make sure you're back in the `marketplace-tutorial` folder and +not the `codex-contracts-eth` subfolder. Then set these variables: **Storage:** -```sh +```bash export ETH_STORAGE_ADDR=0x45BC5ca0fbdD9F920Edd12B90908448C30F32a37 export ETH_STORAGE_PK=0x06c7ac11d4ee1d0ccb53811b71802fa92d40a5a174afad9f2cb44f93498322c3 echo $ETH_STORAGE_PK > storage.pkey && chmod 0600 storage.pkey ``` **Client:** -```sh +```bash export ETH_CLIENT_ADDR=0x9F0C62Fe60b22301751d6cDe1175526b9280b965 export ETH_CLIENT_PK=0x5538ec03c956cb9d0bee02a25b600b0225f1347da4071d0fd70c521fdc63c2fc echo $ETH_CLIENT_PK > client.pkey && chmod 0600 client.pkey @@ -185,25 +326,34 @@ echo $ETH_CLIENT_PK > client.pkey && chmod 0600 client.pkey ### 2.3. Provision Accounts with Tokens -We now need to transfer some ETH to each of the accounts, as well as provide them with some Codex tokens for the storage node to use as collateral and for the client node to buy actual storage. +We now need to transfer some ETH to each of the accounts, as well as provide +them with some Codex tokens for the storage node to use as collateral and +for the client node to buy actual storage. -Although the process is not particularly complicated, I suggest you use [the script we prepared](https://github.com/gmega/local-codex-bare/blob/main/scripts/mint-tokens.js) for that. This script, essentially: +Although the process is not particularly complicated, I suggest you use +[the script we prepared](https://github.com/gmega/local-codex-bare/blob/main/scripts/mint-tokens.js) +for that. This script, essentially: 1. reads the Marketplace contract address and its ABI from the deployment data; -2. transfers $1$ ETH from the signer account to a target account if the target account has no ETH balance; +2. transfers $1$ ETH from the signer account to a target account if the target + account has no ETH balance; 3. mints $n$ Codex tokens and adds it into the target account's balance. -To use the script, just download it into a local file named `mint-tokens.js`, for instance using curl: +To use the script, just download it into a local file named `mint-tokens.js`, +for instance using `curl` (make sure you are in +the `marketplace-tutorial` directory): ```bash -# set the contract file location -export CONTRACT_DEPLOY_FULL="codex-contracts-eth/deployments/codexdisttestnetwork" -export GETH_SIGNER_ADDR=$(cat geth_signer_address.txt) # download script curl https://raw.githubusercontent.com/gmega/codex-local-bare/main/scripts/mint-tokens.js -o mint-tokens.js ``` +Then run: + ```bash +# set the contract file location (we assume you are in the marketplace-tutorial directory) +export CONTRACT_DEPLOY_FULL="../nim-codex/vendor/codex-contracts-eth/deployments/codexdisttestnetwork" +export GETH_SIGNER_ADDR=$(cat geth_signer_address.txt) # Installs Web3-js npm install web3 # Provides tokens to the storage account. @@ -212,7 +362,17 @@ node ./mint-tokens.js $CONTRACT_DEPLOY_FULL/TestToken.json $GETH_SIGNER_ADDR 0x4 node ./mint-tokens.js $CONTRACT_DEPLOY_FULL/TestToken.json $GETH_SIGNER_ADDR 0x9F0C62Fe60b22301751d6cDe1175526b9280b965 10000000000 ``` -If you get a message like `Usage: mint-tokens.js ` then you need to ensure you have +If you get a message like + +```bash +Usage: mint-tokens.js +``` + +then you need to ensure you provided all the required arguments. +In particular you need to ensure that the `GETH_SIGNER_ADDR` env variable +holds the signer address (we used +`export GETH_SIGNER_ADDR=$(cat geth_signer_address.txt)` above to +make sure it is set). ## 3. Run Codex @@ -220,46 +380,65 @@ With accounts and geth in place, we can now start the Codex nodes. ### 3.1. Storage Node -The storage node will be the one storing data and submitting the proofs of storage to the chain. To do that, it needs access to: +The storage node will be the one storing data and submitting the proofs of +storage to the chain. To do that, it needs access to: -1. the address of the Marketplace contract that has been deployed to the local geth node in [Step 2.1](#21-deploy-the-codex-marketplace-contracts); -2. the sample ceremony files which are shipped in the Codex contracts repo. +1. the address of the Marketplace contract that has been deployed to + the local geth node in [Step 2.1](#21-deploy-the-codex-marketplace-contracts); +2. the sample ceremony files which are shipped in the Codex contracts repo + (`nim-codex/vendor/codex-contracts-eth`). -Recall you have clone the `codex-contracts-eth` repository in Step 2.1. All of the required files are in there. - -**Address of the Marketplace Contract.** The contract address can be found inside of the file `codex-contracts-eth/deployments/codexdisttestnetwork/Marketplace.json`: +**Address of the Marketplace Contract.** The contract address can be found +inside of the file `nim-codex/vendor/codex-contracts-eth/deployments/codexdisttestnetwork/Marketplace.json`. +We captured that location above in `CONTRACT_DEPLOY_FULL` variable, thus, from +the `marketplace-tutorial` folder just run: ```bash grep '"address":' ${CONTRACT_DEPLOY_FULL}/Marketplace.json ``` which should print something like: -```sh - "address": "0x8891732D890f5A7B7181fBc70F7482DE28a7B60f", +```bash +"address": "0xCf0df6C52B02201F78E8490B6D6fFf5A82fC7BCd", ``` +> This address should match the address we got earlier when deploying +> the Marketplace contract above. + Then run the following with the correct market place address: -```sh +```bash export MARKETPLACE_ADDRESS="0x0000000000000000000000000000000000000000" echo ${MARKETPLACE_ADDRESS} > marketplace_address.txt ``` -**Prover ceremony files.** The ceremony files are under the `codex-contracts-eth/verifier/networks/codexdisttestnetwork` subdirectory. There are three of them: `proof_main.r1cs`, `proof_main.zkey`, and `prooof_main.wasm`. We will need all of them to start the Codex storage node. +where you replace `0x0000000000000000000000000000000000000000` with +the Marketplace contract above in +[Step 2.1](#21-deploy-the-codex-marketplace-contracts). + +**Prover ceremony files.** The ceremony files are under the +`nim-codex/vendor/codex-contracts-eth/verifier/networks/codexdisttestnetwork` +subdirectory. There are three of them: `proof_main.r1cs`, `proof_main.zkey`, +and `prooof_main.wasm`. We will need all of them to start the Codex storage node. **Starting the storage node.** Let: -* `PROVER_ASSETS` contain the directory where the prover ceremony files are located. **This must be an absolute path**; +* `PROVER_ASSETS` contain the directory where the prover ceremony files are + located. **This must be an absolute path**; * `CODEX_BINARY` contain the location of your Codex binary; -* `MARKETPLACE_ADDRESS` contain the address of the Marketplace contract (obtained above). +* `MARKETPLACE_ADDRESS` contain the address of the Marketplace contract + (we have already set it above). -Set these paths into environment variables (modify it with the correct paths if you changed them above): +Set these paths into environment variables (make sure you are in +the `marketplace-tutorial` directory): -```sh -export CONTRACT_DEPLOY_FULL=$(realpath "codex-contracts-eth/deployments/codexdisttestnetwork") -export PROVER_ASSETS=$(realpath "codex-contracts-eth/verifier/networks/codexdisttestnetwork/") -export CODEX_BINARY=$(realpath "../build/codex") +```bash +export CONTRACT_DEPLOY_FULL=$(realpath "../nim-codex/vendor/codex-contracts-eth/deployments/codexdisttestnetwork") +export PROVER_ASSETS=$(realpath "../nim-codex/vendor/codex-contracts-eth/verifier/networks/codexdisttestnetwork/") +export CODEX_BINARY=$(realpath "../nim-codex/build/codex") export MARKETPLACE_ADDRESS=$(cat marketplace_address.txt) ``` +> you may notice, that we have already set the `CONTRACT_DEPLOY_FULL` variable +> above. Here, we make sure it is an absolute path. To launch the storage node, run: @@ -288,23 +467,28 @@ The client node is started similarly except that: * we need to pass the SPR of the storage node so it can form a network with it; * since it does not run any proofs, it does not require any ceremony files. -We get the Signed Peer Record (SPR) of the storage node so we can bootstrap the client node with it. To get the SPR, issue the following call: +We get the Signed Peer Record (SPR) of the storage node so we can bootstrap +the client node with it. To get the SPR, issue the following call: ```bash -curl -H 'Accept: text/plain' 'http://localhost:8000/api/codex/v1/spr' +curl -H 'Accept: text/plain' 'http://localhost:8000/api/codex/v1/spr' --write-out '\n' ``` -You should get the SPR back starting with `spr:`. Next set these paths into environment variables: +You should get the SPR back starting with `spr:`. + +Before you proceed, open new terminal, and enter `marketplace-tutorial` directory. + +Next set these paths into environment variables: ```bash # set the SPR for the storage node export STORAGE_NODE_SPR=$(curl -H 'Accept: text/plain' 'http://localhost:8000/api/codex/v1/spr') # basic vars -export CONTRACT_DEPLOY_FULL=$(realpath "codex-contracts-eth/deployments/codexdisttestnetwork") -export PROVER_ASSETS=$(realpath "codex-contracts-eth/verifier/networks/codexdisttestnetwork/") -export CODEX_BINARY=$(realpath "../build/codex") +export CONTRACT_DEPLOY_FULL=$(realpath "../nim-codex/vendor/codex-contracts-eth/deployments/codexdisttestnetwork") +export CODEX_BINARY=$(realpath "../nim-codex/build/codex") export MARKETPLACE_ADDRESS=$(cat marketplace_address.txt) ``` +and then run: ```bash ${CODEX_BINARY}\ @@ -321,11 +505,16 @@ ${CODEX_BINARY}\ ## 4. Buy and Sell Storage on the Marketplace -Any storage negotiation has two sides: a buyer and a seller. Before we can actually request storage, therefore, we must first put some of it for sale. +Any storage negotiation has two sides: a buyer and a seller. +Therefore, before we can actually request storage, we must first offer +some of it for sale. ### 4.1 Sell Storage -The following request will cause the storage node to put out $50\text{MB}$ of storage for sale for $1$ hour, at a price of $1$ Codex token per byte per second, while expressing that it's willing to take at most a $1000$ Codex token penalty for not fulfilling its part of the contract.[^1] +The following request will cause the storage node to put out $50\text{MB}$ +of storage for sale for $1$ hour, at a price of $1$ Codex token +per slot per second, while expressing that it's willing to take at most +a $1000$ Codex token penalty for not fulfilling its part of the contract.[^1] ```bash curl 'http://localhost:8000/api/codex/v1/sales/availability' \ @@ -338,66 +527,90 @@ curl 'http://localhost:8000/api/codex/v1/sales/availability' \ }' ``` -This should return a response with an id a string (e.g. `"id": "0x552ef12a2ee64ca22b237335c7e1df884df36d22bfd6506b356936bc718565d4"`) which identifies this storage offer. To check the current storage offers for this node, you can issue: +This should return a JSON response containing an `id` (e.g. `"id": "0xb55b3bc7aac2563d5bf08ce8a177a38b5a40254bfa7ee8f9c52debbb176d44b0"`) +which identifies this storage offer. + +> To make JSON responses more readable, you can try +> [jq](https://jqlang.github.io/jq/) JSON formatting utility +> by just adding `| jq` after the command. +> On macOS you can install with `brew install jq`. + +To check the current storage offers for this node, you can issue: ```bash curl 'http://localhost:8000/api/codex/v1/sales/availability' ``` -This should print a list of offers, with the one you just created figuring among them. +or with `jq`: + +```bash +curl 'http://localhost:8000/api/codex/v1/sales/availability' | jq +``` + +This should print a list of offers, with the one you just created figuring +among them (for our tutorial, there will be only one offer returned +at this time). ## 4.2. Buy Storage -Before we can buy storage, we must have some actual data to request storage for. Start by uploading a small file to your client node. On Linux you could, for instance, use `dd` to generate a $100KB$ file: +Before we can buy storage, we must have some actual data to request +storage for. Start by uploading a small file to your client node. +On Linux (or macOS) you could, for instance, use `dd` to generate a $1M$ file: ```bash -dd if=/dev/urandom of=./data.bin bs=100K count=1 +dd if=/dev/urandom of=./data.bin bs=1M count=1 ``` -but any small file will do. Assuming your file is named `data.bin`, you can upload it with: +Assuming your file is named `data.bin`, you can upload it with: ```bash -curl "http://localhost:8001/api/codex/v1/data" --data-bin @data.bin +curl --request POST http://localhost:8001/api/codex/v1/data --header 'Content-Type: application/octet-stream' --write-out '\n' -T ./data.bin ``` -Once the upload completes, you should see a CID (e.g. `zDvZRwzm2mK7tvDzKScRLapqGdgNTLyyEBvx1TQY37J2CdWdS6Sj`) for the file printed to the terminal. Use that CID in the purchase request: +Once the upload completes, you should see a _Content Identifier_, +or _CID_ (e.g. `zDvZRwzm2mK7tvDzKScRLapqGdgNTLyyEBvx1TQY37J2CdWdS6Sj`) +for the uploaded file printed to the terminal. +Use that CID in the purchase request: ```bash +# make sure to replace the CID before with the CID you got in the previous step export CID=zDvZRwzm2mK7tvDzKScRLapqGdgNTLyyEBvx1TQY37J2CdWdS6Sj -export EXPIRY_TIME=$((1000 + $(date +%s))) # current time + 1000 seconds - # adjust expiry_time as desired, see below ``` ```bash curl "http://localhost:8001/api/codex/v1/storage/request/${CID}" \ - --header 'Content-Type: application/json' \ + --header 'Content-Type: application/octet-stream' \ --data "{ - \"duration\": \"1200\", + \"duration\": \"600\", \"reward\": \"1\", \"proofProbability\": \"3\", - \"expiry\": \"${EXPIRY_TIME}\", - \"nodes\": 1, - \"tolerance\": 0, + \"expiry\": \"500\", + \"nodes\": 3, + \"tolerance\": 1, \"collateral\": \"1000\" - }" + }" \ + --write-out '\n' ``` The parameters under `--data` say that: -1. we want to purchase storage for our file for $20$ minutes (`"duration": "1200"`); -2. we are willing to pay up to $1$ token per byte, per second (`"reward": "1"`); -3. our file will be split into four pieces (`"nodes": 3` and `"tolerance": 1`), so that we only need three pieces to rebuild the file; i.e., we can tolerate that at most one node stops storing our data; either due to failure or other reasons; -4. we demand `1000` tokens in collateral from storage providers for each piece. Since there are $4$ such pieces, there will be `4000` in total collateral committed by all of the storage providers taken together once our request is fulfilled. - -Finally, the `expiry` puts a cap on the block time at which our request expires. This has to be at most `current block time + duration`, which means this request can fail if you input the wrong number, which you likely will if you do not know what the current block time is. Fear not, however, as you can try an an arbitrary number (e.g. `1000`), and look at the failure message: - - `Expiry needs to be in future. Now: 1711995463` - -to compute a valid one. Just take the number in the error message and add the duration; i.e., `1711995463 + 1200 = 1711996663`, then use the resulting number (`1711996663`) as expiry and things should work. The request should return a purchase ID (e.g. `1d0ec5261e3364f8b9d1cf70324d70af21a9b5dccba380b24eb68b4762249185`), which you can use track the completion of your request in the marketplace. +1. we want to purchase storage for our file for $5$ minutes (`"duration": "600"`); +2. we are willing to pay up to $1$ token per slot per second (`"reward": "1"`) +3. our file will be split into three pieces (`"nodes": 3`). + Because we set `"tolerance": 1` we only need two (`nodes - tolerance`) + pieces to rebuild the file; i.e., we can tolerate that at most one node + stops storing our data; either due to failure or other reasons; +4. we demand `1000` tokens in collateral from storage providers for each piece. + Since there are $3$ such pieces, there will be `3000` in total collateral + committed by the storage provider(s) once our request is started. +5. finally, the `expiry` puts a time limit for filling all the slots by + the storage provider(s). If slot are not filled by the `expire` interval, + the request will timeout and fail. ## 4.3. Track your Storage Requests -POSTing a storage request will make it available in the storage market, and a storage node will eventually pick it up. +POSTing a storage request will make it available in the storage market, +and a storage node will eventually pick it up. You can poll the status of your request by means of: ```bash @@ -415,30 +628,39 @@ This returns a result like: ```json { - "requestId": "0x6c698cd0ad71c41982f83097d6fa75beb582924e08a658357a1cd4d7a2a6766d", - "request": { - "client": "0xed6c3c20358f0217919a30c98d72e29ceffedc33", - "ask": { - "slots": 3, - "slotSize": "262144", - "duration": "1000", - "proofProbability": "3", - "reward": "1", - "collateral": "1", - "maxSlotLoss": 1 - }, - "content": { - "cid": "zDvZRwzm3nnkekFLCACmWyKdkYixsX3j9gJhkvFtfYA5K9bpXQnC" - }, - "expiry": "1711992852", - "nonce": "0x9f5e651ecd3bf73c914f8ed0b1088869c64095c0d7bd50a38fc92ebf66ff5915", - "id": "0x6c698cd0ad71c41982f83097d6fa75beb582924e08a658357a1cd4d7a2a6766d" - }, - "state": "submitted", + "requestId": "0x86501e4677a728c6a8031971d09b921c3baa268af06b9f17f1b745e7dba5d330", + "request": { + "client": "0x9f0c62fe60b22301751d6cde1175526b9280b965", + "ask": { + "slots": 3, + "slotSize": "262144", + "duration": "1000", + "proofProbability": "3", + "reward": "1", + "collateral": "1", + "maxSlotLoss": 1 + }, + "content": { + "cid": "zDvZRwzkyw1E7ABaUSmgtNEDjC7opzhUoHo99Vpvc98cDWeCs47u" + }, + "expiry": "1711992852", + "nonce": "0x9f5e651ecd3bf73c914f8ed0b1088869c64095c0d7bd50a38fc92ebf66ff5915", + "id": "0x6c698cd0ad71c41982f83097d6fa75beb582924e08a658357a1cd4d7a2a6766d" + }, + "state": "submitted", "error": null } ``` -Shows that a request has been submitted but has not yet been filled. Your request will be successful once `"state"` shows `"started"`. Anything other than that means the request has not been completely processed yet, and an `"error"` state other than `null` means it failed. +Shows that a request has been submitted but has not yet been filled. +Your request will be successful once `"state"` shows `"started"`. +Anything other than that means the request has not been completely +processed yet, and an `"error"` state other than `null` means it failed. -[^1]: Codex files get partitioned into pieces called "slots" and distributed to various storage providers. The collateral refers to one such slot, and will be slowly eaten away as the storage provider fails to deliver timely proofs, but the actual logic is [more involved than that](https://github.com/codex-storage/codex-contracts-eth/blob/6c9f797f408608958714024b9055fcc330e3842f/contracts/Marketplace.sol#L209). +Well, it was quite a journey, wasn't it? You can congratulate yourself for +successfully finishing the codex marketplace tutorial! + +[^1]: Codex files get partitioned into pieces called "slots" and distributed +to various storage providers. The collateral refers to one such slot, +and will be slowly eaten away as the storage provider fails to deliver +timely proofs, but the actual logic is [more involved than that](https://github.com/codex-storage/codex-contracts-eth/blob/6c9f797f408608958714024b9055fcc330e3842f/contracts/Marketplace.sol#L209).