status-im
/
nim-codex
mirror of https://github.com/status-im/nim-codex.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity

Remove moved docs (#930) 

* Remove moved document

* Update main Readme and point links to the documentation site
This commit is contained in:
Slava 2024-10-03 11:55:54 +03:00 committed by GitHub
parent 5ace105a66
commit b491906005
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 10 additions and 383 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

149
README.md
View File

@ -1,8 +1,6 @@
# Codex Decentralized Durability Engine
> The Codex project aims to create a decentralized durability engine that
> allows persisting data in p2p networks. In other words, it allows storing
> files and data with predictable durability guarantees for later retrieval.
> The Codex project aims to create a decentralized durability engine that allows persisting data in p2p networks. In other words, it allows storing files and data with predictable durability guarantees for later retrieval.
> WARNING: This project is under active development and is considered pre-alpha.
@ -26,8 +24,7 @@ To build the project, clone it and run:
make update && make
```
The executable will be placed under the `build` directory under the project
root.
The executable will be placed under the `build` directory under the project root.
Run the client with:
@ -38,145 +35,19 @@ build/codex
It is possible to configure a Codex node in several ways:
1. CLI options
2. Env. variable
3. Config
2. Environment variables
3. Configuration file
The order of priority is the same as above:
Cli arguments > Env variables > Config file values.
The order of priority is the same as above: CLI options --> Environment variables --> Configuration file.
### Environment variables
Please check [documentation](https://docs.codex.storage/learn/run#configuration) for more information.
In order to set a configuration option using environment variables,
first find the desired CLI option and then transform it in the following way:
1. prepend it with `CODEX_`
2. make it uppercase
3. replace `-` with `_`
For example, to configure `--log-level`, use `CODEX_LOG_LEVEL` as the
environment variable name.
### Configuration file
A [TOML](https://toml.io/en/) configuration file can also be used to set
configuration values. Configuration option names and corresponding values are
placed in the file, separated by `=`. Configuration option names can be
obtained from the `codex --help` command, and should not include
the `--` prefix. For example, a node's log level (`--log-level`) can be
configured using TOML as follows:
```toml
log-level = "trace"
```
The Codex node can then read the configuration from this file using
the `--config-file` CLI parameter, like
`codex --config-file=/path/to/your/config.toml`.
### CLI Options
```
$ build/codex --help
Usage:
codex [OPTIONS]... command
The following options are available:
--config-file Loads the configuration from a TOML file [=none].
--log-level Sets the log level [=info].
--metrics Enable the metrics server [=false].
--metrics-address Listening address of the metrics server [=127.0.0.1].
--metrics-port Listening HTTP port of the metrics server [=8008].
-d, --data-dir The directory where codex will store configuration and data.
-i, --listen-addrs Multi Addresses to listen on [=/ip4/0.0.0.0/tcp/0].
-a, --nat IP Addresses to announce behind a NAT [=127.0.0.1].
-e, --disc-ip Discovery listen address [=0.0.0.0].
-u, --disc-port Discovery (UDP) port [=8090].
--net-privkey Source of network (secp256k1) private key file path or name [=key].
-b, --bootstrap-node Specifies one or more bootstrap nodes to use when connecting to the network.
--max-peers The maximum number of peers to connect to [=160].
--agent-string Node agent string which is used as identifier in network [=Codex].
--api-bindaddr The REST API bind address [=127.0.0.1].
-p, --api-port The REST Api port [=8080].
--api-cors-origin The REST Api CORS allowed origin for downloading data. '*' will allow all
origins, '' will allow none. [=Disallow all cross origin requests to download
data].
--repo-kind Backend for main repo store (fs, sqlite, leveldb) [=fs].
-q, --storage-quota The size of the total storage quota dedicated to the node [=$DefaultQuotaBytes].
-t, --block-ttl Default block timeout in seconds - 0 disables the ttl [=$DefaultBlockTtl].
--block-mi Time interval in seconds - determines frequency of block maintenance cycle: how
often blocks are checked for expiration and cleanup
[=$DefaultBlockMaintenanceInterval].
--block-mn Number of blocks to check every maintenance cycle [=1000].
-c, --cache-size The size of the block cache, 0 disables the cache - might help on slow hardrives
[=0].
Available sub-commands:
codex persistence [OPTIONS]... command
The following options are available:
--eth-provider The URL of the JSON-RPC API of the Ethereum node [=ws://localhost:8545].
--eth-account The Ethereum account that is used for storage contracts.
--eth-private-key File containing Ethereum private key for storage contracts.
--marketplace-address Address of deployed Marketplace contract.
--validator Enables validator, requires an Ethereum node [=false].
--validator-max-slots Maximum number of slots that the validator monitors [=1000].
If set to 0, the validator will not limit the maximum number of slots it
monitors.
--validator-groups Slot validation groups [=ValidationGroups.none].
A number indicating total number of groups into which the whole slot id space
will be divided. The value must be in the range [2, 65535]. If not provided, the
validator will observe the whole slot id space and the value of the
--validator-group-index parameter will be ignored. Powers of twos are advised
for even distribution.
--validator-group-index Slot validation group index [=0].
The value provided must be in the range [0, validatorGroups). Ignored when
--validator-groups is not provided. Only slot ids satisfying condition [(slotId
mod validationGroups) == groupIndex] will be observed by the validator.
Available sub-commands:
codex persistence prover [OPTIONS]...
The following options are available:
--circom-r1cs The r1cs file for the storage circuit.
--circom-wasm The wasm file for the storage circuit.
--circom-zkey The zkey file for the storage circuit.
--circom-no-zkey Ignore the zkey file - use only for testing! [=false].
--proof-samples Number of samples to prove [=5].
--max-slot-depth The maximum depth of the slot tree [=32].
--max-dataset-depth The maximum depth of the dataset tree [=8].
--max-block-depth The maximum depth of the network block merkle tree [=5].
--max-cell-elements The maximum number of elements in a cell [=67].
```
#### Logging
Codex uses [Chronicles](https://github.com/status-im/nim-chronicles) logging
library, which allows great flexibility in working with logs.
Chronicles has the concept of topics, which categorize log entries into
semantic groups.
Using the `log-level` parameter, you can set the top-level log level like
`--log-level="trace"`, but more importantly, you can set log levels for
specific topics like `--log-level="info; trace: marketplace,node; error: blockexchange"`,
which sets the top-level log level to `info` and then for topics
`marketplace` and `node` sets the level to `trace` and so on.
### Guides
## Guides
To get acquainted with Codex, consider:
* running the simple [Codex Two-Client Test](docs/TwoClientTest.md) for
a start, and;
* if you are feeling more adventurous, try
[Running a Local Codex Network with Marketplace Support](docs/Marketplace.md)
using a local blockchain as well.
* running the simple [Codex Two-Client Test](https://docs.codex.storage/learn/local-two-client-test) for a start, and;
* if you are feeling more adventurous, try [Running a Local Codex Network with Marketplace Support](https://docs.codex.storage/learn/local-marketplace) using a local blockchain as well.
## API
The client exposes a REST API that can be used to interact with the clients.
Overview of the API can be found on [api.codex.storage](https://api.codex.storage).
The client exposes a REST API that can be used to interact with the clients. Overview of the API can be found on [api.codex.storage](https://api.codex.storage).

68
docs/DownloadFlow.md
View File

@ -1,68 +0,0 @@
# Download Flow
Sequence of interactions that result in dat blocks being transferred across the network.
## Local Store
When data is available in the local blockstore,
```mermaid
sequenceDiagram
actor Alice
participant API
Alice->>API: Download(CID)
API->>+Node/StoreStream: Retrieve(CID)
loop Get manifest block, then data blocks
Node/StoreStream->>NetworkStore: GetBlock(CID)
NetworkStore->>LocalStore: GetBlock(CID)
LocalStore->>NetworkStore: Block
NetworkStore->>Node/StoreStream: Block
end
Node/StoreStream->>Node/StoreStream: Handle erasure coding
Node/StoreStream->>-API: Data stream
API->>Alice: Stream download of block
```
## Network Store
When data is not found ih the local blockstore, the block-exchange engine is used to discover the location of the block within the network. Connection will be established to the node(s) that have the block, and exchange can take place.
```mermaid
sequenceDiagram
box
actor Alice
participant API
participant Node/StoreStream
participant NetworkStore
participant Discovery
participant Engine
end
box
participant OtherNode
end
Alice->>API: Download(CID)
API->>+Node/StoreStream: Retrieve(CID)
Node/StoreStream->>-API: Data stream
API->>Alice: Download stream begins
loop Get manifest block, then data blocks
Node/StoreStream->>NetworkStore: GetBlock(CID)
NetworkStore->>Engine: RequestBlock(CID)
opt CID not known
Engine->>Discovery: Discovery Block
Discovery->>Discovery: Locates peers who provide block
Discovery->>Engine: Peers
Engine->>Engine: Update peers admin
end
Engine->>Engine: Select optimal peer
Engine->>OtherNode: Send WantHave list
OtherNode->>Engine: Send BlockPresence
Engine->>Engine: Update peers admin
Engine->>Engine: Decide to buy block
Engine->>OtherNode: Send WantBlock list
OtherNode->>Engine: Send Block
Engine->>NetworkStore: Block
NetworkStore->>NetworkStore: Add to Local store
NetworkStore->>Node/StoreStream: Resolve Block
Node/StoreStream->>Node/StoreStream: Handle erasure coding
Node/StoreStream->>API: Push data to stream
end
API->>Alice: Download stream finishes
```

176
docs/TwoClientTest.md
View File

@ -1,176 +0,0 @@
# Codex Two-Client Test
The two-client test is a manual test you can perform to check your setup and familiarize yourself with the Codex API. These steps will guide you through running and connecting two nodes, in order to upload a file to one and then download that file from the other. This test also includes running a local blockchain node in order to have the Marketplace functionality available. However, running a local blockchain node is not strictly necessary, and you can skip steps marked as optional if you choose not start a local blockchain node.
## Prerequisite
Make sure you have built the client, and can run it as explained in the [README](../README.md).
## Steps
### 0. Setup blockchain node (optional)
You need to have installed NodeJS and npm in order to spinup a local blockchain node.
Go to directory `vendor/codex-contracts-eth` and run these two commands:
```
npm ci
npm start
```
This will launch a local Ganache blockchain.
### 1. Launch Node #1
Open a terminal and run:
- Mac/Unx: `"build/codex" --data-dir="$(pwd)/Data1" --listen-addrs="/ip4/127.0.0.1/tcp/8070" --api-port=8080 --disc-port=8090`
- Windows: `"build/codex.exe" --data-dir="Data1" --listen-addrs="/ip4/127.0.0.1/tcp/8070" --api-port=8080 --disc-port=8090`
Optionally, if you want to use the Marketplace blockchain functionality, you need to also include these flags: `--persistence --eth-account=<account>`, where `account` can be one following:
- `0x70997970C51812dc3A010C7d01b50e0d17dc79C8`
- `0x3C44CdDdB6a900fa2b585dd299e03d12FA4293BC`
- `0x90F79bf6EB2c4f870365E785982E1f101E93b906`
- `0x15d34AAf54267DB7D7c367839AAf71A00a2C6A65`
**For each node use a different account!**
| Argument | Description |
|----------------|-----------------------------------------------------------------------|
| `data-dir` | We specify a relative path where the node will store its data. |
| `listen-addrs` | Multiaddress where the node will accept connections from other nodes. |
| `api-port` | Port on localhost where the node will expose its API. |
| `disc-port` | Port the node will use for its discovery service. |
| `persistence` | Enables Marketplace functionality. Requires a blockchain connection. |
| `eth-account` | Defines which blockchain account the node should use. |
Codex uses sane defaults for most of its arguments. Here we specify some explicitly for the purpose of this walk-through.
### 2. Sign of life
Run the command :
```bash
curl -X GET http://127.0.0.1:8080/api/codex/v1/debug/info
```
This GET request will return the node's debug information. The response will be in JSON and should look like:
```json
{
"id": "16Uiu2HAmJ3TSfPnrJNedHy2DMsjTqwBiVAQQqPo579DuMgGxmG99",
"addrs": [
"/ip4/127.0.0.1/tcp/8070"
],
"repo": "/Users/user/projects/nim-codex/Data1",
"spr": "spr:CiUIAhIhA1AL2J7EWfg7x77iOrR9YYBisY6CDtU2nEhuwDaQyjpkEgIDARo8CicAJQgCEiEDUAvYnsRZ-DvHvuI6tH1hgGKxjoIO1TacSG7ANpDKOmQQ2MWasAYaCwoJBH8AAAGRAh-aKkYwRAIgB2ooPfAyzWEJDe8hD2OXKOBnyTOPakc4GzqKqjM2OGoCICraQLPWf0oSEuvmSroFebVQx-3SDtMqDoIyWhjq1XFF",
"announceAddresses": [
"/ip4/127.0.0.1/tcp/8070"
],
"table": {
"localNode": {
"nodeId": "f6e6d48fa7cd171688249a57de0c1aba15e88308c07538c91e1310c9f48c860a",
"peerId": "16Uiu2HAmJ3TSfPnrJNedHy2DMsjTqwBiVAQQqPo579DuMgGxmG99",
"record": "...",
"address": "0.0.0.0:8090",
"seen": false
},
"nodes": []
},
"codex": {
"version": "untagged build",
"revision": "b3e626a5"
}
}
```
| Field | Description |
| ------- | ---------------------------------------------------------------------------------------- |
| `id` | Id of the node. Also referred to as 'peerId'. |
| `addrs` | Multiaddresses currently open to accept connections from other nodes. |
| `repo` | Path of this node's data folder. |
| `spr` | Signed Peer Record, encoded information about this node and its location in the network. |
| `announceAddresses` | Multiaddresses used for annoucning this node
| `table` | Table of nodes present in the node's DHT
| `codex` | Codex version information
### 3. Launch Node #2
We will need the signed peer record (SPR) from the first node that you got in the previous step.
Replace `<SPR HERE>` in the following command with the SPR returned from the previous command. (Note that it should include the `spr:` at the beginning.)
Open a new terminal and run:
- Mac/Linux: `"build/codex" --data-dir="$(pwd)/Data2" --listen-addrs=/ip4/127.0.0.1/tcp/8071 --api-port=8081 --disc-port=8091 --bootstrap-node=<SPR HERE>`
- Windows: `"build/codex.exe" --data-dir="Data2" --listen-addrs=/ip4/127.0.0.1/tcp/8071 --api-port=8081 --disc-port=8091 --bootstrap-node=<SPR HERE>`
Alternatively on Mac, Linux, or MSYS2 and a recent Codex binary you can run it in one command like:
```sh
"build/codex" --data-dir="$(pwd)/Data2" --listen-addrs=/ip4/127.0.0.1/tcp/8071 --api-port=8081 --disc-port=8091 --bootstrap-node=$(curl -H "Accept: text/plain" http://127.0.0.1:8080/api/codex/v1/spr)
```
Notice we're using a new data-dir, and we've increased each port number by one. This is needed so that the new node won't try to open ports already in use by the first node.
We're now also including the `bootstrap-node` argument. This allows us to link the new node to another one, bootstrapping our own little peer-to-peer network. (SPR strings always start with "spr:".)
### 4. Connect The Two
Normally the two nodes will automatically connect. If they do not automatically connect or you want to manually connect nodes you can use the peerId to connect nodes.
You can get the first node's peer id by running the following command and finding the `"peerId"` in the results:
```bash
curl -X GET -H "Accept: text/plain" http://127.0.0.1:8081/api/codex/v1/debug/info
```
Next replace `<PEER ID HERE>` in the following command with the peerId returned from the previous command:
```bash
curl -X GET http://127.0.0.1:8080/api/codex/v1/connect/<PEER ID HERE>?addrs=/ip4/127.0.0.1/tcp/8071
```
Alternatively on Mac, Linux, or MSYS2 and a recent Codex binary you can run it in one command like:
```bash
curl -X GET http://127.0.0.1:8080/api/codex/v1/connect/$(curl -X GET -H "Accept: text/plain" http://127.0.0.1:8081/api/codex/v1/peerid)\?addrs=/ip4/127.0.0.1/tcp/8071
```
Notice that we are sending the peerId and the multiaddress of node 2 to the `/connect` endpoint of node 1. This provides node 1 all the information it needs to communicate with node 2. The response to this request should be `Successfully connected to peer`.
### 5. Upload The File
We're now ready to upload a file to the network. In this example we'll use node 1 for uploading and node 2 for downloading. But the reverse also works.
Next replace `<FILE PATH>` with the path to the file you want to upload in the following command:
```bash
curl -H "Content-Type: application/octet-stream" -H "Expect: 100-continue" -T "<FILE PATH>" 127.0.0.1:8080/api/codex/v1/data -X POST
```
(Hint: if curl is reluctant to show you the response, add `-o <FILENAME>` to write the result to a file.)
Depending on the file size this may take a moment. Codex is processing the file by cutting it into blocks and generating erasure-recovery data. When the process is finished, the request will return the content-identifier (CID) of the uploaded file. It should look something like `zdj7WVxH8HHHenKtid8Vkgv5Z5eSUbCxxr8xguTUBMCBD8F2S`.
### 6. Download The File
Replace `<CID>` with the identifier returned in the previous step. Replace `<OUTPUT FILE>` with the filename where you want to store the downloaded file.
```bash
curl 127.0.0.1:8081/api/codex/v1/data/<CID>/network --output <OUTPUT FILE>
```
Notice we are connecting to the second node in order to download the file. The CID we provide contains the information needed to locate the file within the network.
### 7. Verify The Results
If your file is downloaded and identical to the file you uploaded, then this manual test has passed. Rejoice! If on the other hand that didn't happen or you were unable to complete any of these steps, please leave us a message detailing your troubles.
## Notes
When using the Ganache blockchain, there are some deviations from the expected behavior, mainly linked to how blocks are mined, which affects certain functionalities in the Sales module.
Therefore, if you are manually testing processes such as payout collection after a request is finished or proof submissions, you need to mine some blocks manually for it to work correctly. You can do this by using the following curl command:
```bash
$ curl -H "Content-Type: application/json" -X POST --data '{"jsonrpc":"2.0","method":"evm_mine","params":[],"id":67}' 127.0.0.1:8545
```