Decentralized Durability Engine
Go to file
Ben Bierens 0c3d1dd563
Verifiable manifests (#642)
* Adds test for encoding/decoding protected manifest

* Setting up verifiable manifest

* mysterious mysteries

* Successful encoding test for verifiable manifests

* extracts toF out of users of manifest code

* Update codex/manifest/coders.nim

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* Review comments by Dmitriy

* Adds missing verifiable print to $ method.

* Replace poseidon2 F type with int as temporary stand-in for verification hashes

* Replaces verification hash placeholder with CID

---------

Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2023-12-12 08:11:54 +00:00
.github Run C-Tests on master commits (#636) 2023-11-27 13:32:13 +02:00
codex Verifiable manifests (#642) 2023-12-12 08:11:54 +00:00
docker Adds option to automatically announce codex by its public IP (#621) 2023-11-17 13:48:25 +00:00
docs Adds endpoint for listing files (manifests) in node. Useful for demo UI. (#599) 2023-11-09 08:47:09 +00:00
metrics Adding metrics (#203) 2022-08-23 10:11:21 -06:00
tests Verifiable manifests (#642) 2023-12-12 08:11:54 +00:00
vendor Verifiable manifests (#642) 2023-12-12 08:11:54 +00:00
.dockerignore Docker build (#354) 2023-03-08 12:45:55 +01:00
.editorconfig Project setup 2021-02-02 19:29:52 +01:00
.gitignore Bump deps take2 (#492) 2023-08-01 16:47:57 -07:00
.gitmodules Bump dht (#628) 2023-11-27 09:23:25 -08:00
BUILDING.md Update links to codex-storage organization (#420) 2023-05-23 23:01:13 +03:00
Makefile Taiko L2 (#483) 2023-09-13 16:17:56 +02:00
README.md Update badges (#533) 2023-08-27 07:22:09 +03:00
atlas.lock Bump deps take2 (#492) 2023-08-01 16:47:57 -07:00
build.nims Taiko L2 (#483) 2023-09-13 16:17:56 +02:00
codecov.yml [ci] disable pull-request comments by codecov 2022-05-19 15:23:35 +02:00
codex.nim Plumbing in conf types (#472) 2023-07-06 16:23:27 -07:00
codex.nimble Add poseidon2 dependency (#625) 2023-11-20 09:51:36 +01:00
config.nims Verifiable manifests (#642) 2023-12-12 08:11:54 +00:00
env.sh add env.sh shim to project root (#34) 2021-12-20 13:12:18 -06:00
nimble.lock Sync submodule dependencies and lock file (#134) 2022-07-19 09:31:32 -06:00
openapi.yaml Add get active slot /slots/{slotId} to REST api, use utils/json (#645) 2023-12-07 12:16:36 +11:00

README.md

Codex Decentralized Durability Engine

The Codex project aims to create a decentralized durability engine that allows persisting data in p2p networks. In other words, it allows storing files and data with predictable durability guarantees for later retrieval.

WARNING: This project is under active development and is considered pre-alpha.

License: Apache License: MIT Stability: experimental CI Docker Codecov Discord Docker Pulls

Build and Run

For detailed instructions on preparing to build nim-codex see Building Codex.

To build the project, clone it and run:

make update && make

The executable will be placed under the build directory under the project root.

Run the client with:

build/codex

Configuration

It is possible to configure a Codex node in several ways:

  1. CLI options
  2. Env. variable
  3. Config

The order of priority is the same as above: Cli arguments > Env variables > Config file values.

Environment variables

In order to set a configuration option using environment variables, first find the desired CLI option and then transform it in the following way:

  1. prepend it with CODEX_
  2. make it uppercase
  3. replace - with _

For example, to configure --log-level, use CODEX_LOG_LEVEL as the environment variable name.

Configuration file

A TOML configuration file can also be used to set configuration values. Configuration option names and corresponding values are placed in the file, separated by =. Configuration option names can be obtained from the codex --help command, and should not include the -- prefix. For example, a node's log level (--log-level) can be configured using TOML as follows:

log-level = "trace"

The Codex node can then read the configuration from this file using the --config-file CLI parameter, like codex --config-file=/path/to/your/config.toml.

CLI Options

build/codex --help
Usage:

codex [OPTIONS]... command

The following options are available:

     --config-file          Loads the configuration from a TOML file [=none].
     --log-level            Sets the log level [=info].
     --metrics              Enable the metrics server [=false].
     --metrics-address      Listening address of the metrics server [=127.0.0.1].
     --metrics-port         Listening HTTP port of the metrics server [=8008].
 -d, --data-dir             The directory where codex will store configuration and data..
 -i, --listen-addrs         Multi Addresses to listen on [=/ip4/0.0.0.0/tcp/0].
 -a, --nat                  IP Addresses to announce behind a NAT [=127.0.0.1].
 -e, --disc-ip              Discovery listen address [=0.0.0.0].
 -u, --disc-port            Discovery (UDP) port [=8090].
     --net-privkey          Source of network (secp256k1) private key file path or name [=key].
 -b, --bootstrap-node       Specifies one or more bootstrap nodes to use when connecting to the network..
     --max-peers            The maximum number of peers to connect to [=160].
     --agent-string         Node agent string which is used as identifier in network [=Codex].
     --api-bindaddr         The REST API bind address [=127.0.0.1].
 -p, --api-port             The REST Api port [=8080].
     --repo-kind            backend for main repo store (fs, sqlite) [=fs].
 -q, --storage-quota        The size of the total storage quota dedicated to the node [=8589934592].
 -t, --block-ttl            Default block timeout in seconds - 0 disables the ttl [=$DefaultBlockTtl].
     --block-mi             Time interval in seconds - determines frequency of block maintenance cycle: how
                            often blocks are checked for expiration and cleanup.
                            [=$DefaultBlockMaintenanceInterval].
     --block-mn             Number of blocks to check every maintenance cycle. [=1000].
 -c, --cache-size           The size in MiB of the block cache, 0 disables the cache - might help on slow
                            hardrives [=0].
     --persistence          Enables persistence mechanism, requires an Ethereum node [=false].
     --eth-provider         The URL of the JSON-RPC API of the Ethereum node [=ws://localhost:8545].
     --eth-account          The Ethereum account that is used for storage contracts [=EthAddress.none].
     --eth-deployment       The json file describing the contract deployment [=string.none].
     --validator            Enables validator, requires an Ethereum node [=false].
     --validator-max-slots  Maximum number of slots that the validator monitors [=1000].

Available sub-commands:

codex initNode

Logging

Codex uses Chronicles logging library, which allows great flexibility in working with logs. Chronicles has the concept of topics, which categorize log entries into semantic groups.

Using the log-level parameter, you can set the top-level log level like --log-level="trace", but more importantly, you can set log levels for specific topics like --log-level="info; trace: marketplace,node; error: blockexchange", which sets the top-level log level to info and then for topics marketplace and node sets the level to trace and so on.

Example: running two Codex clients

To get acquainted with Codex, consider running the manual two-client test described HERE.

API

The client exposes a REST API that can be used to interact with the clients. Overview of the API can be found on api.codex.storage.