Decentralized Durability Engine
Go to file
markspanbroek d56eb6aee1
Validator (#387)
* [contracts] Add SlotFreed event

* [integration] allow test node to be stopped twice

* [cli] add --validator option

* [contracts] remove dead code

* [contracts] instantiate OnChainMarket and OnChainClock only once

* [contracts] add Validation

* [sales] remove duplicate import

* [market] add missing import

* [market] subscribe to all SlotFilled events

* [market] add freeSlot()

* [sales] fix warnings

* [market] subscribe to SlotFreed events

* [contracts] fix warning

* [validator] keep track of filled slots

* [validation] remove slots that have ended

* [proving] absorb Proofs into Market

Both Proofs and Market are abstractions around
the Marketplace contract, having them separately
is more trouble than it's worth at the moment.

* [market] add markProofAsMissing()

* [clock] speed up waiting for clock in tests

* [validator] mark proofs as missing

* [timer] fix error on node shutdown

* [cli] handle --persistence and --validator separately

* [market] allow retrieval of proof timeout value

* [validator] do not subscribe to SlotFreed events

Freed slots are already handled in
removeSlotsThatHaveEnded(), and onSlotsFreed()
interfered with its iterator.

* [validator] Start validation at the start of a new period

To decrease the likelihood that we hit the validation timeout.

* [validator] do not mark proofs as missing after timeout

* [market] check whether proof can be marked as missing

* [validator] simplify validation

Simulate a transaction to mark proof as missing, instead
of trying to keep track of all the conditions that may
lead to a proof being marked as missing.

* [build] use nim-ethers PR #40

Uses "pending" blocktag instead of "latest" blocktag
for better simulation of transactions before sending
them.

https://github.com/status-im/nim-ethers/pull/40

* [integration] integration test for validator

* [validator] monitor a maximum number of slots

Adds cli parameter --validator-max-slots.

* [market] fix missing collateral argument

After rebasing, add the new argument to fillSlot calls.

* [build] update to nim-ethers 0.2.5

* [validator] use Set instead of Table to keep track of slots

* [validator] add logging

* [validator] add test for slot failure

* [market] use "pending" blocktag to use more up to date block time

* [contracts] remove unused import

* [validator] fix: wait until after period ends

The smart contract checks that 'end < block.timestamp',
so we need to wait until the block timestamp is greater
than the period end.
2023-04-19 15:06:00 +02:00
.github Creates marketplace_address.json during docker start. (#397) 2023-04-18 14:17:24 +02:00
codex Validator (#387) 2023-04-19 15:06:00 +02:00
docker Creates marketplace_address.json during docker start. (#397) 2023-04-18 14:17:24 +02:00
docs fix: approving tokens transfer when creating request (#385) 2023-03-30 11:34:38 +02:00
metrics Adding metrics (#203) 2022-08-23 10:11:21 -06:00
tests Validator (#387) 2023-04-19 15:06:00 +02:00
vendor Validator (#387) 2023-04-19 15:06:00 +02:00
.dockerignore Docker build (#354) 2023-03-08 12:45:55 +01:00
.editorconfig Project setup 2021-02-02 19:29:52 +01:00
.gitignore docs: openapi specification (#355) 2023-03-15 14:10:53 +01:00
.gitmodules [marketplace] support latest contracts changes (#327) 2023-01-19 16:58:04 +11:00
BUILDING.md Update references to `make exec` (#384) 2023-03-27 18:08:13 -06:00
Makefile running `make` should build exec (#356) 2023-03-13 17:56:03 -06:00
README.md docs: api documentation linked (#382) 2023-03-29 14:55:17 +02:00
codecov.yml [ci] disable pull-request comments by codecov 2022-05-19 15:23:35 +02:00
codex.nim Fix warnings (drops Nim 1.2) (#348) 2023-03-09 12:23:45 +01:00
codex.nimble Fix nimble requirements (#305) 2022-11-16 13:10:23 +01:00
config.nims Enable stylecheck (#353) 2023-03-10 08:02:54 +01:00
env.sh add env.sh shim to project root (#34) 2021-12-20 13:12:18 -06:00
nimble.lock Sync submodule dependencies and lock file (#134) 2022-07-19 09:31:32 -06:00
openapi.yaml feat: collateral per slot (#390) 2023-04-14 11:04:17 +02:00

README.md

Codex Decentralized Durability Engine

The Codex project aims to create a decentralized durability engine that allows persisting data in p2p networks. In other words, it allows storing files and data with predictable durability guarantees for later retrieval.

WARNING: This project is under active development and is considered pre-alpha.

License: Apache License: MIT Stability: experimental CI Codecov Discord

Build and Run

For detailed instructions on preparing to build nim-codex see Building Codex.

To build the project, clone it and run:

make update && make

The executable will be placed under the build directory under the project root.

Run the client with:

build/codex

CLI Options

build/codex --help
Usage:

codex [OPTIONS]... command

The following options are available:

     --log-level            Sets the log level [=INFO].
     --metrics              Enable the metrics server [=false].
     --metrics-address      Listening address of the metrics server [=127.0.0.1].
     --metrics-port         Listening HTTP port of the metrics server [=8008].
 -d, --data-dir             The directory where codex will store configuration and data..
 -i, --listen-addrs         Multi Addresses to listen on [=/ip4/0.0.0.0/tcp/0].
 -a, --nat                  IP Addresses to announce behind a NAT [=127.0.0.1].
 -e, --disc-ip              Discovery listen address [=0.0.0.0].
 -u, --disc-port            Discovery (UDP) port [=8090].
     --net-privkey          Source of network (secp256k1) private key file path or name [=key].
 -b, --bootstrap-node       Specifies one or more bootstrap nodes to use when connecting to the network..
     --max-peers            The maximum number of peers to connect to [=160].
     --agent-string         Node agent string which is used as identifier in network [=Codex].
 -p, --api-port             The REST Api port [=8080].
     --repo-kind            backend for main repo store (fs, sqlite) [=fs].
 -q, --storage-quota        The size of the total storage quota dedicated to the node [=8589934592].
 -t, --block-ttl            Default block timeout in seconds - 0 disables the ttl [=86400].
 -c, --cache-size           The size in MiB of the block cache, 0 disables the cache - might help on slow
                            hardrives [=0].
     --persistence          Enables persistence mechanism, requires an Ethereum node [=false].
     --eth-provider         The URL of the JSON-RPC API of the Ethereum node [=ws://localhost:8545].
     --eth-account          The Ethereum account that is used for storage contracts [=EthAddress.none].
     --eth-deployment       The json file describing the contract deployment [=string.none].

Available sub-commands:

codex initNode

Example: running two Codex clients

To get acquainted with Codex, consider running the manual two-client test described HERE.

API

The client exposes a REST API that can be used to interact with the clients. Overview of the API can be found on api.codex.storage.