Decentralized Durability Engine
Go to file
Giuliano Mega 4619260dc1
Fix StoreStream so it doesn't return parity bytes (#838)
* fix storestream so it doesn\'t return parity bits for protected/verifiable manifests

* use Cid.example instead of creating a mock manually
2024-06-21 11:09:59 +00:00
.github Allow compiler version to be pinned consistently (#832) 2024-06-17 18:59:25 +00:00
benchmarks Pr add prover benchmark tool (#790) 2024-05-23 09:28:17 -07:00
codex Fix StoreStream so it doesn't return parity bytes (#838) 2024-06-21 11:09:59 +00:00
docker Updates rust in docker. Pins circom-compat to pinned FFI branch (#818) 2024-05-31 08:23:11 +00:00
docs Updates tutorial with entry on how to run Codex with its Markeplace (#765) 2024-05-22 16:43:33 +00:00
metrics Adding metrics (#203) 2022-08-23 10:11:21 -06:00
tests Fix StoreStream so it doesn't return parity bytes (#838) 2024-06-21 11:09:59 +00:00
vendor remove stale and misleading dependency version metadata (#833) 2024-06-17 14:31:10 +00:00
.dockerignore Docker build (#354) 2023-03-08 12:45:55 +01:00
.editorconfig Project setup 2021-02-02 19:29:52 +01:00
.gitignore refactor: multinode integration test refactor (#662) 2024-02-19 04:55:39 +00:00
.gitmodules Metadata in LevelDB (#806) 2024-05-30 06:57:10 +00:00
BUILDING.md Add Release workflow (#827) 2024-06-05 12:56:41 +00:00
Makefile Allow compiler version to be pinned consistently (#832) 2024-06-17 18:59:25 +00:00
README.md Updates tutorial with entry on how to run Codex with its Markeplace (#765) 2024-05-22 16:43:33 +00:00
build.nims Remove codex_use_hardhat compilation flag (#741) 2024-03-19 11:08:54 +00:00
codecov.yml [ci] disable pull-request comments by codecov 2022-05-19 15:23:35 +02:00
codex.nim Prover CLI updates (#735) 2024-03-12 09:57:13 +00:00
codex.nimble remove stale and misleading dependency version metadata (#833) 2024-06-17 14:31:10 +00:00
config.nims Wire up prover (#736) 2024-03-12 12:10:14 +00:00
env.sh add env.sh shim to project root (#34) 2021-12-20 13:12:18 -06:00
openapi.yaml Fix/spec intsizes (#829) 2024-06-06 09:44:58 +00:00

README.md

Codex Decentralized Durability Engine

The Codex project aims to create a decentralized durability engine that allows persisting data in p2p networks. In other words, it allows storing files and data with predictable durability guarantees for later retrieval.

WARNING: This project is under active development and is considered pre-alpha.

License: Apache License: MIT Stability: experimental CI Docker Codecov Discord Docker Pulls

Build and Run

For detailed instructions on preparing to build nim-codex see Building Codex.

To build the project, clone it and run:

make update && make

The executable will be placed under the build directory under the project root.

Run the client with:

build/codex

Configuration

It is possible to configure a Codex node in several ways:

  1. CLI options
  2. Env. variable
  3. Config

The order of priority is the same as above: Cli arguments > Env variables > Config file values.

Environment variables

In order to set a configuration option using environment variables, first find the desired CLI option and then transform it in the following way:

  1. prepend it with CODEX_
  2. make it uppercase
  3. replace - with _

For example, to configure --log-level, use CODEX_LOG_LEVEL as the environment variable name.

Configuration file

A TOML configuration file can also be used to set configuration values. Configuration option names and corresponding values are placed in the file, separated by =. Configuration option names can be obtained from the codex --help command, and should not include the -- prefix. For example, a node's log level (--log-level) can be configured using TOML as follows:

log-level = "trace"

The Codex node can then read the configuration from this file using the --config-file CLI parameter, like codex --config-file=/path/to/your/config.toml.

CLI Options

build/codex --help
Usage:

codex [OPTIONS]... command

The following options are available:

     --config-file          Loads the configuration from a TOML file [=none].
     --log-level            Sets the log level [=info].
     --metrics              Enable the metrics server [=false].
     --metrics-address      Listening address of the metrics server [=127.0.0.1].
     --metrics-port         Listening HTTP port of the metrics server [=8008].
 -d, --data-dir             The directory where codex will store configuration and data.
 -i, --listen-addrs         Multi Addresses to listen on [=/ip4/0.0.0.0/tcp/0].
 -a, --nat                  IP Addresses to announce behind a NAT [=127.0.0.1].
 -e, --disc-ip              Discovery listen address [=0.0.0.0].
 -u, --disc-port            Discovery (UDP) port [=8090].
     --net-privkey          Source of network (secp256k1) private key file path or name [=key].
 -b, --bootstrap-node       Specifies one or more bootstrap nodes to use when connecting to the network.
     --max-peers            The maximum number of peers to connect to [=160].
     --agent-string         Node agent string which is used as identifier in network [=Codex].
     --api-bindaddr         The REST API bind address [=127.0.0.1].
 -p, --api-port             The REST Api port [=8080].
     --repo-kind            Backend for main repo store (fs, sqlite) [=fs].
 -q, --storage-quota        The size of the total storage quota dedicated to the node [=8589934592].
 -t, --block-ttl            Default block timeout in seconds - 0 disables the ttl [=$DefaultBlockTtl].
     --block-mi             Time interval in seconds - determines frequency of block maintenance cycle: how
                            often blocks are checked for expiration and cleanup
                            [=$DefaultBlockMaintenanceInterval].
     --block-mn             Number of blocks to check every maintenance cycle [=1000].
 -c, --cache-size           The size of the block cache, 0 disables the cache - might help on slow hardrives
                            [=0].

Available sub-commands:

codex persistence [OPTIONS]... command

The following options are available:

     --eth-provider         The URL of the JSON-RPC API of the Ethereum node [=ws://localhost:8545].
     --eth-account          The Ethereum account that is used for storage contracts.
     --eth-private-key      File containing Ethereum private key for storage contracts.
     --marketplace-address  Address of deployed Marketplace contract.
     --validator            Enables validator, requires an Ethereum node [=false].
     --validator-max-slots  Maximum number of slots that the validator monitors [=1000].

Available sub-commands:

codex persistence prover [OPTIONS]...

The following options are available:

     --circom-r1cs          The r1cs file for the storage circuit.
     --circom-wasm          The wasm file for the storage circuit.
     --circom-zkey          The zkey file for the storage circuit.
     --circom-no-zkey       Ignore the zkey file - use only for testing! [=false].
     --proof-samples        Number of samples to prove [=5].
     --max-slot-depth       The maximum depth of the slot tree [=32].
     --max-dataset-depth    The maximum depth of the dataset tree [=8].
     --max-block-depth      The maximum depth of the network block merkle tree [=5].
     --max-cell-elements    The maximum number of elements in a cell [=67].

Logging

Codex uses Chronicles logging library, which allows great flexibility in working with logs. Chronicles has the concept of topics, which categorize log entries into semantic groups.

Using the log-level parameter, you can set the top-level log level like --log-level="trace", but more importantly, you can set log levels for specific topics like --log-level="info; trace: marketplace,node; error: blockexchange", which sets the top-level log level to info and then for topics marketplace and node sets the level to trace and so on.

Guides

To get acquainted with Codex, consider:

API

The client exposes a REST API that can be used to interact with the clients. Overview of the API can be found on api.codex.storage.