Update Run Codex using Docker (#30)

* Fix headers level and adjust page outline

* Update Run Codex guide
This commit is contained in:
Slava 2024-10-01 18:50:42 +03:00 committed by GitHub
parent 291d65a865
commit c20f9d2b17
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 76 additions and 23 deletions

View File

@ -1,3 +1,6 @@
---
outline: [2, 3]
---
# Running a Local Codex Network with Marketplace Support # Running a Local Codex Network with Marketplace Support
This tutorial will teach you how to run a small Codex network with the This tutorial will teach you how to run a small Codex network with the
@ -551,7 +554,7 @@ This should print a list of offers, with the one you just created figuring
among them (for our tutorial, there will be only one offer returned among them (for our tutorial, there will be only one offer returned
at this time). at this time).
## 4.2. Buy Storage ### 4.2. Buy Storage
Before we can buy storage, we must have some actual data to request Before we can buy storage, we must have some actual data to request
storage for. Start by uploading a small file to your client node. storage for. Start by uploading a small file to your client node.
@ -607,7 +610,7 @@ The parameters under `--data` say that:
the storage provider(s). If slot are not filled by the `expire` interval, the storage provider(s). If slot are not filled by the `expire` interval,
the request will timeout and fail. the request will timeout and fail.
## 4.3. Track your Storage Requests ### 4.3. Track your Storage Requests
POSTing a storage request will make it available in the storage market, POSTing a storage request will make it available in the storage market,
and a storage node will eventually pick it up. and a storage node will eventually pick it up.

View File

@ -184,7 +184,7 @@ But, it will use a default `data-dir` value and we can pass a custom one:
codex --data-dir=datadir codex --data-dir=datadir
``` ```
This will run Codex as an isolated instance, and if we would like to join an existing network, it is required to pass a [bootstrap node](architecture#network-architecture). We can pass multiple nodes as well: This will run Codex as an isolated instance, and if we would like to join an existing network, it is required to pass a [bootstrap node](#codex-bootstrap-node). We can pass multiple nodes as well:
```shell ```shell
codex \ codex \
--data-dir=datadir \ --data-dir=datadir \
@ -246,7 +246,7 @@ codex \
After node is up and running and port-forwarding configurations was done, we should be able to [Upload a file](/learn/using#upload-a-file)/[Download a file](/learn/using#download-a-file) in the network using [API](/developers/api). After node is up and running and port-forwarding configurations was done, we should be able to [Upload a file](/learn/using#upload-a-file)/[Download a file](/learn/using#download-a-file) in the network using [API](/developers/api).
We also can use [Codex Marketplace UI](https://marketplace.codex.storage) for files upload/download. You also can use [Codex Marketplace UI](https://marketplace.codex.storage) for files upload/download.
And to be able to purchase a storage, we should run [Codex node with marketplace support](#codex-node-with-marketplace-support). And to be able to purchase a storage, we should run [Codex node with marketplace support](#codex-node-with-marketplace-support).
@ -298,7 +298,7 @@ And to be able to purchase a storage, we should run [Codex node with marketplace
After node is up and running, and your address has founds, you should be able to [Purchase storage](/learn/using#purchase-storage) using [API](/developers/api). After node is up and running, and your address has founds, you should be able to [Purchase storage](/learn/using#purchase-storage) using [API](/developers/api).
We also can use [Codex Marketplace UI](https://marketplace.codex.storage) for storage purchase. You also can use [Codex Marketplace UI](https://marketplace.codex.storage) for storage purchase.
#### Codex storage node #### Codex storage node
@ -313,6 +313,26 @@ Every [network](/networks/networks) uses its own generated set of the files whic
To download circuit files and make them available to Codex app, we have a stand-alone utility - `cirdl`. It can be [compiled from the sources](/learn/build#circuit-download-tool) or downloaded from the [GitHub release page](https://github.com/codex-storage/nim-codex/releases). To download circuit files and make them available to Codex app, we have a stand-alone utility - `cirdl`. It can be [compiled from the sources](/learn/build#circuit-download-tool) or downloaded from the [GitHub release page](https://github.com/codex-storage/nim-codex/releases).
1. Create ethereum key 1. Create ethereum key
<details>
<summary>example</summary>
> [!CAUTION]
> Please use key generation service for demo purpose only.
```shell
response=$(curl -s https://key.codex.storage)
awk -F ': ' '/private/ {print $2}' <<<"${response}" > eth.key
awk -F ': ' '/address/ {print $2}' <<<"${response}" > eth.address
chmod 600 eth.key
```
Show your ethereum address:
```shell
cat eth.address
```
```
0x412665aFAb17768cd9aACE6E00537Cc6D5524Da9
```
</details>
2. To download circuit files, we should pass directory, RPC endpoint and marketplace address to the circuit downloader: 2. To download circuit files, we should pass directory, RPC endpoint and marketplace address to the circuit downloader:
```shell ```shell
@ -340,9 +360,7 @@ To download circuit files and make them available to Codex app, we have a stand-
--eth-private-key=eth.key \ --eth-private-key=eth.key \
--marketplace-address=0xCDef8d6884557be4F68dC265b6bB2E3e52a6C9d6 \ --marketplace-address=0xCDef8d6884557be4F68dC265b6bB2E3e52a6C9d6 \
prover \ prover \
--circom-r1cs=datadir/circuits/proof_main.r1cs \ --circuit-dir
--circom-wasm=datadir/circuits/proof_main.wasm \
--circom-zkey=datadir/circuits/proof_main.zkey
``` ```
> [!NOTE] > [!NOTE]
@ -351,7 +369,7 @@ To download circuit files and make them available to Codex app, we have a stand-
After node is up and running, and your address has founds, you should be able to [sell the storage](/learn/using#create-storage-availability) using [API](/developers/api). After node is up and running, and your address has founds, you should be able to [sell the storage](/learn/using#create-storage-availability) using [API](/developers/api).
We also can use [Codex Marketplace UI](https://marketplace.codex.storage) to sell the storage. You also can use [Codex Marketplace UI](https://marketplace.codex.storage) to sell the storage.
#### Codex bootstrap node #### Codex bootstrap node
@ -363,7 +381,7 @@ codex \
--disc-port=8090 --disc-port=8090
``` ```
And we can get bootstrap node SPR via [API](https://api.codex.storage/#tag/Debug/operation/getDebugInfo) call: To get bootstrap node SPR we can use [API](https://api.codex.storage/#tag/Debug/operation/getDebugInfo) call:
```shell ```shell
curl -s localhost:8080/api/codex/v1/debug/info | jq -r '.spr' curl -s localhost:8080/api/codex/v1/debug/info | jq -r '.spr'
``` ```
@ -398,7 +416,7 @@ We also ship Codex in Docker containers, which can be run on `amd64` and `arm64`
- `ENV_PATH` - path to the file, in form `env=value` which will be sourced and available for Codex at run. That is useful for Kubernetes Pods configuration. - `ENV_PATH` - path to the file, in form `env=value` which will be sourced and available for Codex at run. That is useful for Kubernetes Pods configuration.
- `NAT_IP_AUTO` - when set to `true`, will set `CODEX_NAT` variable with container internal IP address. It also is useful for Kubernetes Pods configuration, when we perform automated tests. - `NAT_IP_AUTO` - when set to `true`, will set `CODEX_NAT` variable with container internal IP address. It also is useful for Kubernetes Pods configuration, when we perform automated tests.
- `NAT_PUBLIC_IP_AUTO` - used to set `CODEX_NAT` to public IP address using lookup services, like [ip.codex.storage](https://ip.codex.storage). Can be used for Docker/Kubernetes, and binary as well, to set public IP in auto mode. - `NAT_PUBLIC_IP_AUTO` - used to set `CODEX_NAT` to public IP address using lookup services, like [ip.codex.storage](https://ip.codex.storage). Can be used for Docker/Kubernetes to set public IP in auto mode.
- `PRIV_KEY` - can be used to pass ethereum private key, which will be saved and passed as a value of the `CODEX_ETH_PRIVATE_KEY` variable. It should be considered as unsafe option and used for testing purposes only. - `PRIV_KEY` - can be used to pass ethereum private key, which will be saved and passed as a value of the `CODEX_ETH_PRIVATE_KEY` variable. It should be considered as unsafe option and used for testing purposes only.
- When we set `prover` sub-command, entrypoint will run `cirdl` tool to download ceremony files, required by [Codex storage node](#codex-storage-node). - When we set `prover` sub-command, entrypoint will run `cirdl` tool to download ceremony files, required by [Codex storage node](#codex-storage-node).
@ -411,7 +429,7 @@ When we are running Codex using Docker with default [bridge network](https://doc
If your Internet router does not support [Full Cone NAT](https://learningnetwork.cisco.com/s/question/0D56e0000CWxJ9sCQF/lets-explain-in-details-full-cone-nat-restricted-cone-nat-and-symmetric-nat-terminologies-vs-cisco-nat-terminologies), you might have an issue and peer discovery and data transport will not work or might work unexpected. If your Internet router does not support [Full Cone NAT](https://learningnetwork.cisco.com/s/question/0D56e0000CWxJ9sCQF/lets-explain-in-details-full-cone-nat-restricted-cone-nat-and-symmetric-nat-terminologies-vs-cisco-nat-terminologies), you might have an issue and peer discovery and data transport will not work or might work unexpected.
In that case, we should consider the following solutions: In that case, we should consider the following solutions:
- Use [host network](https://docs.docker.com/engine/network/drivers/host/) which is supported only in Linux - Use [host network](https://docs.docker.com/engine/network/drivers/host/) for Docker, which is supported only in Linux
- Run [Using binary](#using-binary) - Run [Using binary](#using-binary)
- Use VM/VPS in the Cloud to run Docker with bridge or host network - Use VM/VPS in the Cloud to run Docker with bridge or host network
@ -422,6 +440,26 @@ And we basically can use same options we [used for binary](#using-binary) and ad
[Codex storage node](#codex-storage-node) [Codex storage node](#codex-storage-node)
1. Create ethereum key file 1. Create ethereum key file
<details>
<summary>example</summary>
> [!CAUTION]
> Please use key generation service for demo purpose only.
```shell
response=$(curl -s https://key.codex.storage)
awk -F ': ' '/private/ {print $2}' <<<"${response}" > eth.key
awk -F ': ' '/address/ {print $2}' <<<"${response}" > eth.address
chmod 600 eth.key
```
Show your ethereum address:
```shell
cat eth.address
```
```
0x412665aFAb17768cd9aACE6E00537Cc6D5524Da9
```
</details>
2. Run Codex: 2. Run Codex:
```shell ```shell
@ -432,9 +470,6 @@ docker run \
-p 8070:8070 \ -p 8070:8070 \
-p 8080:8080 \ -p 8080:8080 \
-p 8090:8090/udp \ -p 8090:8090/udp \
-e CODEX_DATA_DIR=/datadir \
-e CODEX_ETH_PROVIDER=https://rpc.testnet.codex.storage \
-e CODEX_MARKETPLACE_ADDRESS=0xCDef8d6884557be4F68dC265b6bB2E3e52a6C9d6 \
codexstorage/nim-codex:latest \ codexstorage/nim-codex:latest \
codex \ codex \
--data-dir=/datadir \ --data-dir=/datadir \
@ -450,17 +485,12 @@ docker run \
--eth-private-key=/opt/eth.key \ --eth-private-key=/opt/eth.key \
--marketplace-address=0xCDef8d6884557be4F68dC265b6bB2E3e52a6C9d6 \ --marketplace-address=0xCDef8d6884557be4F68dC265b6bB2E3e52a6C9d6 \
prover \ prover \
--circom-r1cs=/datadir/circuits/proof_main.r1cs \ --circuit-dir=/datadir/circuits
--circom-wasm=/datadir/circuits/proof_main.wasm \
--circom-zkey=/datadir/circuits/proof_main.zkey
``` ```
> [!NOTE] > [!NOTE]
> You would need to pass a bootstrap nodes, blockchain RPC endpoint and marketplace address based on the [network](/networks/networks) you would like to join. > You would need to pass a bootstrap nodes, blockchain RPC endpoint and marketplace address based on the [network](/networks/networks) you would like to join.
> [!NOTE]
> Currently, [Docker entrypoint](#docker-entrypoint) support just environment variables to download circuit files. We will update it soon, so we can get rid of the duplicate options.
### Using Docker Compose ### Using Docker Compose
For Docker Compose, it is more suitable to use [environment variables](#environment-variables) for Codex configuration and we can reuse commands from example above, for Docker. For Docker Compose, it is more suitable to use [environment variables](#environment-variables) for Codex configuration and we can reuse commands from example above, for Docker.
@ -468,6 +498,26 @@ For Docker Compose, it is more suitable to use [environment variables](#environm
[Codex storage node](#codex-storage-node) [Codex storage node](#codex-storage-node)
1. Create ethereum key file 1. Create ethereum key file
<details>
<summary>example</summary>
> [!CAUTION]
> Please use key generation service for demo purpose only.
```shell
response=$(curl -s https://key.codex.storage)
awk -F ': ' '/private/ {print $2}' <<<"${response}" > eth.key
awk -F ': ' '/address/ {print $2}' <<<"${response}" > eth.address
chmod 600 eth.key
```
Show your ethereum address:
```shell
cat eth.address
```
```
0x412665aFAb17768cd9aACE6E00537Cc6D5524Da9
```
</details>
2. Create `docker-compose.yaml` file: 2. Create `docker-compose.yaml` file:
```yaml ```yaml
@ -516,7 +566,7 @@ For Docker Compose, it is more suitable to use [environment variables](#environm
3. Run Codex: 3. Run Codex:
```shell ```shell
docker compose up -d docker compose up
``` ```
> [!NOTE] > [!NOTE]
@ -529,7 +579,7 @@ Helm chart code is available in [helm-charts](https://github.com/codex-storage/h
## Known issues ## Known issues
[^multivalue-env-var]: Environment variables like `CODEX_BOOTSTRAP_NODE` and `CODEX_LISTEN_ADDRS` does not support multiple values. Please check [[Feature request] Support multiple SPR records via environment variable #525](https://github.com/codex-storage/nim-codex/issues/525), for more information. [^multivalue-env-var]: Environment variables like `CODEX_BOOTSTRAP_NODE` and `CODEX_LISTEN_ADDRS` does not support multiple values. Please check [[Feature request] Support multiple SPR records via environment variable #525](https://github.com/codex-storage/nim-codex/issues/525), for more information.
[^sub-commands]: Sub-commands configuration like `persistence` and `persistence prover` can't be done via environment variables for now. [^sub-commands]: Sub-commands `persistence` and `persistence prover` can't be set via environment variables.
[^data-dir]: We should set data-dir explicitly when we use GitHub releases - [[BUG] Change codex default datadir from compile-time to run-time #923](https://github.com/codex-storage/nim-codex/issues/923) [^data-dir]: We should set data-dir explicitly when we use GitHub releases - [[BUG] Change codex default datadir from compile-time to run-time #923](https://github.com/codex-storage/nim-codex/issues/923)
[^port-forwarding]: [NAT traversal #753](https://github.com/codex-storage/nim-codex/issues/753) is not implemented yet and we would need to setup port-forwarding for discovery and transport protocols. [^port-forwarding]: [NAT traversal #753](https://github.com/codex-storage/nim-codex/issues/753) is not implemented yet and we would need to setup port-forwarding for discovery and transport protocols.
[^eth-account]: Please ignore `--eth-account` CLI option - [Drop support for --eth-account #727](https://github.com/codex-storage/nim-codex/issues/727). [^eth-account]: Please ignore `--eth-account` CLI option - [Drop support for --eth-account #727](https://github.com/codex-storage/nim-codex/issues/727).