Decentralized Storage
Go to file
Ben Bierens 9c8a59d150
Blockstore maintenance (#347)
* setting up

* Implements timer utility

* applies async and cancellation to timer loop

* Sets up mocktimer and mockblockstore to set up first maintenance module test

* wip: first test that calls blockChecker

* wip: adding user type to timer callback

* Chronicles doesn't like type-arguments? Disables logging in timer module for now

* Implementing block check test for blockMaintainer module

* Sets up additional tests for blockmaintainer

* Removes generic from timer module. Implements numberOfBlocks per interval in blockMaintainer.

* Implements blockMaintainer

* Sets up tests for blockChecker

* Some comments by Mark

* Cleanup repostore tests

* Setting up the use of std/times for block TTL tracking

* repostore adds expiration timestamp

* Defaults the repostore clock to the system clock

* Applies updates to repostore interface.

* Implements retrieving of block expiration information from repostore

* Sets up tests for maintenance module behavior

* Implements block maintenance module

* Wires maintenance module into codex. Sets up integration tests for block expiration

* Sets up test for missing behavior: removing timestamp metadata on block delete

* Implements removing of expiration metadata in repoStore

* Fixes integration tests for block expiration

* Adds block expiration tests to integration test run

* Handled some comments by Dmitriy

* Review comment by Dmitriy: Removes seq[cid] from runBlockCheck

* Review comment by Dmitriy: Moves key formatting methods to keyutils.

* Review comment by Dmitriy: Encodes durations using chronos

* Fixes conversion of TTL type in conf.

* Review comments by Dmitriy

* Adds unit tests for keyUtils.

* Adds test coverage for exception in maintenance module
2023-03-08 16:04:54 +01:00
.github Docker build (#354) 2023-03-08 12:45:55 +01:00
codex Blockstore maintenance (#347) 2023-03-08 16:04:54 +01:00
docker Docker build (#354) 2023-03-08 12:45:55 +01:00
docs Docs/update setup run and test steps (#342) 2023-02-09 10:56:01 +01:00
metrics Adding metrics (#203) 2022-08-23 10:11:21 -06:00
tests Blockstore maintenance (#347) 2023-03-08 16:04:54 +01:00
vendor [marketplace] Load sales state from chain (#306) 2023-03-08 14:34:26 +01:00
.dockerignore Docker build (#354) 2023-03-08 12:45:55 +01:00
.editorconfig Project setup 2021-02-02 19:29:52 +01:00
.gitignore Docker build (#354) 2023-03-08 12:45:55 +01:00
.gitmodules [marketplace] support latest contracts changes (#327) 2023-01-19 16:58:04 +11:00
BUILDING.md Docs/update setup run and test steps (#342) 2023-02-09 10:56:01 +01:00
Makefile Speed up development (#303) 2022-11-14 14:50:00 +01:00
README.md updating readme with latest changes (#331) 2023-02-15 13:40:21 -06:00
codecov.yml [ci] disable pull-request comments by codecov 2022-05-19 15:23:35 +02:00
codex.nim don't crash on shutdown (#323) 2022-12-05 09:00:02 -06:00
codex.nimble Fix nimble requirements (#305) 2022-11-16 13:10:23 +01:00
config.nims [build] fix nimbus-build-system (#343) 2023-02-08 09:48:27 +01:00
env.sh add env.sh shim to project root (#34) 2021-12-20 13:12:18 -06:00
nim.cfg Disable ObservableStores warning 2021-11-16 16:51:24 +01:00
nimble.lock Sync submodule dependencies and lock file (#134) 2022-07-19 09:31:32 -06:00

README.md

Codex Decentralized Durability Engine

The Codex project aims to create a decentralized durability engine that allows persisting data in p2p networks. In other words, it allows storing files and data with predictable durability guarantees for later retrieval.

WARNING: This project is under active development and is considered pre-alpha.

License: Apache License: MIT Stability: experimental CI Codecov Discord

Build and Run

For detailed instructions on preparing to build nim-codex see Building Codex.

To build the project, clone it and run:

make update && make exec

The executable will be placed under the build directory under the project root.

Run the client with:

build/codex

CLI Options

build/codex --help
Usage:

codex [OPTIONS]... command

The following options are available:

     --log-level            Sets the log level [=INFO].
     --metrics              Enable the metrics server [=false].
     --metrics-address      Listening address of the metrics server [=127.0.0.1].
     --metrics-port         Listening HTTP port of the metrics server [=8008].
 -d, --data-dir             The directory where codex will store configuration and data..
 -i, --listen-addrs         Multi Addresses to listen on [=/ip4/0.0.0.0/tcp/0].
 -a, --nat                  IP Addresses to announce behind a NAT [=127.0.0.1].
 -e, --disc-ip              Discovery listen address [=0.0.0.0].
 -u, --disc-port            Discovery (UDP) port [=8090].
     --net-privkey          Source of network (secp256k1) private key file path or name [=key].
 -b, --bootstrap-node       Specifies one or more bootstrap nodes to use when connecting to the network..
     --max-peers            The maximum number of peers to connect to [=160].
     --agent-string         Node agent string which is used as identifier in network [=Codex].
 -p, --api-port             The REST Api port [=8080].
 -q, --storage-quota        The size of the total storage quota dedicated to the node [=8589934592].
 -t, --block-ttl            Default block timeout in seconds - 0 disables the ttl [=86400].
 -c, --cache-size           The size in MiB of the block cache, 0 disables the cache - might help on slow
                            hardrives [=0].
     --persistence          Enables persistence mechanism, requires an Ethereum node [=false].
     --eth-provider         The URL of the JSON-RPC API of the Ethereum node [=ws://localhost:8545].
     --eth-account          The Ethereum account that is used for storage contracts [=EthAddress.none].
     --eth-deployment       The json file describing the contract deployment [=string.none].

Available sub-commands:

codex initNode

Example: running two Codex clients

To get acquainted with Codex, consider running the manual two-client test described HERE.

Interacting with the client

The client exposes a REST API that can be used to interact with the clients. These commands could be invoked with any HTTP client, however the following endpoints assume the use of the curl command.

/api/codex/v1/connect/{peerId}

Connect to a peer identified by its peer id. Takes an optional addrs parameter with a list of valid multiaddresses. If addrs is absent, the peer will be discovered over the DHT.

Example:

curl "127.0.0.1:8080/api/codex/v1/connect/<peer id>?addrs=<multiaddress>"

/api/codex/v1/download/{id}

Download data identified by a Cid.

Example:

 curl -vvv "127.0.0.1:8080/api/codex/v1/download/<Cid of the content>" --output <name of output file>

/api/codex/v1/upload

Upload a file, upon success returns the Cid of the uploaded file.

Example:

curl -vvv -H "content-type: application/octet-stream" -H Expect: -T "<path to file>" "127.0.0.1:8080/api/codex/v1/upload" -X POST

/api/codex/v1/debug/info

Get useful node info such as its peer id, address and SPR.

Example:

curl -vvv "127.0.0.1:8080/api/codex/v1/info"

/api/codex/v1/debug/chronicles/loglevel

Set chronicles log level and topic filtering at runtime. The request format is ?level=<MAIN LEVEL>;<FILTER LEVEL>:<TOPIC> - e.g. ?level=DEBUG;TRACE:codex

Example:

curl -X POST -H 'Content-Type: text/plain' -vvv "127.0.0.1:8080/api/codex/v1/debug/chronicles/loglevel?level=DEBUG;TRACE:codex" -d ""