3b520fc0c6
Support logging to file Log the entire config and fix build error Process slot queue on reservation callback onMarkUnused Add Reservation object, rename reserve > create refactor Reservations api to include Reservation CRUD All tests that use the Reservation module updated - add requestId and slotIndex to Reservation (hopefully these will prove to be useful when we persist Reservations until request are completed, to add back bytes to Availability) - add querying of all reservations, with accompanying tests - change from find to findAvailabilities - move onCleanUp from SalesContext to SalesAgent as it was getting overwritten for each slot processed - remove sales agent AFTER deleting reservation, as this was causing some SIGSEGVs - retrofit testsales and testslotqueue to match updated Reservations module API Add deletion of inactive reservations on sales load clean up add exception message detail util Apply to onStore errors as we are seeing undetailed errors in the dist tests logs add missing file change slotsize to reflect current implemenation Fix slotSize to reduce by one block Revert change to slotSize that reduces it by one block Add additional test check for querying reservations/availabilities filter past requests based on availability Because availability filtering on push was removed, when availability is added and past storage request events are queried, those requests need to be filtered by availability before being added to the queue. Revert "filter past requests based on availability" This reverts commit 0c2362658b523e0de425794b1fa30ebd53bd30ae. Add debugging for dist tests Add cancel on error during filling state When calling fillSlot, any transaction errors that occur (possibly during estimate gas) will cause that tx to be replaced with a cancellation transaction (a 0-valued tx to ourselves). more debug logging fix build wait for fillSlot to be mined (so that a revert error can be extracted) fix confirmation of fillSlot add more debugging moar debugging debugging: change echo to trace bump ethers to add chronicles fix contracts tests switch to earlier nim-ethers commit bump json-rpc and nim-ethers bump nim-ethers to prevent parsing newHeads log from crashing sales state machine moar debugging moar debugging moar debugging bump ethers to fix "key not found: data" error ethers debug logging bump ethers - better Transaction object deserialization bump ethers to replay tx with past tx format bump ethers to add serialization for PastTransaction ethers bump: better logging, separate logic for revert reason string fix build ethers: update revert reason retreival remove unneeded confirm ethers: try replay without decrementing blockNumber ethers: include type and chainId in replayed txs ethers: add gas into and remove type from replayed txs ensure gas is being serialized in Transaction ethers: fix build error bump ethers: rebased on top of cancel tx due to estimateGas error PR Fix proving with latest ethers bump (tested on master) Update multinode suite for better simulateFailedProofs enabling, add proofs test Improve multinode suite for better debug options, including logging to file There is a 503 "sales unavailable" error improve multinode test suite, add logging to file
Codex Decentralized Durability Engine
The Codex project aims to create a decentralized durability engine that allows persisting data in p2p networks. In other words, it allows storing files and data with predictable durability guarantees for later retrieval.
WARNING: This project is under active development and is considered pre-alpha.
Build and Run
For detailed instructions on preparing to build nim-codex see Building Codex.
To build the project, clone it and run:
make update && make
The executable will be placed under the build directory under the project root.
Run the client with:
build/codex
Configuration
It is possible to configure a Codex node in several ways:
- CLI options
- Env. variable
- Config
The order of priority is the same as above: Cli arguments > Env variables > Config file values.
Environment variables
In order to set a configuration option using environment variables, first find the desired CLI option and then transform it in the following way:
- prepend it with
CODEX_ - make it uppercase
- replace
-with_
For example, to configure --log-level, use CODEX_LOG_LEVEL as the environment variable name.
Configuration file
A TOML configuration file can also be used to set configuration values. Configuration option names and corresponding values are placed in the file, separated by =. Configuration option names can be obtained from the codex --help command, and should not include the -- prefix. For example, a node's log level (--log-level) can be configured using TOML as follows:
log-level = "trace"
The Codex node can then read the configuration from this file using the --config-file CLI parameter, like codex --config-file=/path/to/your/config.toml.
CLI Options
build/codex --help
Usage:
codex [OPTIONS]... command
The following options are available:
--config-file Loads the configuration from a TOML file [=none].
--log-level Sets the log level [=info].
--metrics Enable the metrics server [=false].
--metrics-address Listening address of the metrics server [=127.0.0.1].
--metrics-port Listening HTTP port of the metrics server [=8008].
-d, --data-dir The directory where codex will store configuration and data..
-i, --listen-addrs Multi Addresses to listen on [=/ip4/0.0.0.0/tcp/0].
-a, --nat IP Addresses to announce behind a NAT [=127.0.0.1].
-e, --disc-ip Discovery listen address [=0.0.0.0].
-u, --disc-port Discovery (UDP) port [=8090].
--net-privkey Source of network (secp256k1) private key file path or name [=key].
-b, --bootstrap-node Specifies one or more bootstrap nodes to use when connecting to the network..
--max-peers The maximum number of peers to connect to [=160].
--agent-string Node agent string which is used as identifier in network [=Codex].
--api-bindaddr The REST API bind address [=127.0.0.1].
-p, --api-port The REST Api port [=8080].
--repo-kind backend for main repo store (fs, sqlite) [=fs].
-q, --storage-quota The size of the total storage quota dedicated to the node [=8589934592].
-t, --block-ttl Default block timeout in seconds - 0 disables the ttl [=$DefaultBlockTtl].
--block-mi Time interval in seconds - determines frequency of block maintenance cycle: how
often blocks are checked for expiration and cleanup.
[=$DefaultBlockMaintenanceInterval].
--block-mn Number of blocks to check every maintenance cycle. [=1000].
-c, --cache-size The size in MiB of the block cache, 0 disables the cache - might help on slow
hardrives [=0].
--persistence Enables persistence mechanism, requires an Ethereum node [=false].
--eth-provider The URL of the JSON-RPC API of the Ethereum node [=ws://localhost:8545].
--eth-account The Ethereum account that is used for storage contracts [=EthAddress.none].
--eth-deployment The json file describing the contract deployment [=string.none].
--validator Enables validator, requires an Ethereum node [=false].
--validator-max-slots Maximum number of slots that the validator monitors [=1000].
Available sub-commands:
codex initNode
Logging
Codex uses Chronicles logging library, which allows great flexibility in working with logs. Chronicles has the concept of topics, which categorize log entries into semantic groups.
Using the log-level parameter, you can set the top-level log level like --log-level="trace", but more importantly,
you can set log levels for specific topics like --log-level="info; trace: marketplace,node; error: blockexchange",
which sets the top-level log level to info and then for topics marketplace and node sets the level to trace and so on.
Example: running two Codex clients
To get acquainted with Codex, consider running the manual two-client test described HERE.
API
The client exposes a REST API that can be used to interact with the clients. Overview of the API can be found on api.codex.storage.