Decentralized Durability Engine
Go to file
Marcin Czenko 92a0eda79a
Validator historical state restoration (#922)
* adds a new overload of queryPastEvents allowing to query past events based on timestamp in the past

* adds state restoration to validator

* refactors a bit to get the tests back to work

* replaces deprecated generic methods from Market with methods for specific event types

* Refactors binary search

* adds market tests for querying past SlotFilled events and binary search

* Takes into account that <<earliest>> block available is not necessarily the genesis block

* Adds more logging and makes testing earliest block boundary more reliable

* adds validation tests for historical state restoration

* adds mockprovider to simplify and improve testing of the edge conditions

* adds slot reservation to the new tests after rebasing

* adds validation groups and group index in logs of validator

* adds integration test with two validators

* adds comment on how to enable logging in integration test executable itself

* testIntegration: makes list is running nodes injected and available in the body of the test

* validation: adds integration test for historical state

* adds more logging to validator

* integration test: validator only looks 30 days back for historical state

* adds logging of the slotState when removing slots during validation

* review and refactor validator integration tests

* adds validation to the set of integration tests

* Fixes mistyped name of the mock provider module in testMarket

* Fixes a typo in the name of the validation suite in integration tests

* Makes validation unit test a bit easier to follow

* better use of logScopes to reduce duplication

* improves timing and clarifies the test conditions

* uses http as default RPC provider for nodes running in integration tests as a workaround for dropped subscriptions

* simplifies the validation integration tests by waiting for failed request instead of tracking slots

* adds config option allowing selectively to set different provider url

* Brings back the default settings for RPC provider in integration tests

* use http RPC provider for clients in validation integration tests

* fine-tune the tests

* Makes validator integration test more robust - adds extra tracking

* brings tracking of marketplace event back to validator integration test

* refactors integration tests

* deletes tmp file

* adds <<return>> after forcing integration test to fail preliminarily

* re-enables all integration tests and matrix

* stops debug output in CI

* allows to choose a different RPC provider for a given integration test suite

* fixes signature of <<getBlock>> method in mockProvider

* adds missing import which seem to be braking integration tests on windows

* makes sure that clients, SPs, and validators use the same provider url

* makes validator integration tests using http at 127.0.0.1:8545

* testvalidator: stop resubscribing as we are now using http polling as rpc provider

* applying review comments

* groups queryPastStorage overrides together (review comment)

* groups the historical validation tests into a sub suite

* removes the temporary extensions in marketplacesuite and multinodesuite allowing to specify provider url

* simplifies validation integration tests

* Removes debug logs when waiting for request to fail

* Renaming waitForRequestFailed => waitForRequestToFail

* renames blockNumberForBlocksAgo to pastBlockTag and makes it private

* removes redundant debugging logs

* refines logging in validation

* removes dev logging from mockmarket

* improves exception handling in provider helper procs and prepares for extraction to a separate module

* Uses chronos instead of std/times for Duration

* extracts provider and binary search helpers to a separate module

* removes redundant log entry params from validator

* unifies the notation to consistently use method call syntax

* reuses ProviderError from nim-ethers in the provider extension

* clarifies the comment in multinodesuite

* uses == operator to check the predefined tags and raises exception when `BlockTag.pending` is requested.

* when waiting for request to fail, we break on any request state that is not Started

* removes tests that were moved to testProvider from testMarket

* extracts tests that use MockProvider to a separate async suite

* improves performance of the historical state restoration

* removing redundant log messages in validator (groupIndex and groups)

* adds testProvider to testContracts group

* removes unused import in testMarket
2024-12-14 05:07:55 +00:00
.github Validator historical state restoration (#922) 2024-12-14 05:07:55 +00:00
benchmarks Pr add prover benchmark tool (#790) 2024-05-23 09:28:17 -07:00
codex Validator historical state restoration (#922) 2024-12-14 05:07:55 +00:00
docker Enables stacktrace in docker images (#1013) 2024-12-02 09:32:53 +00:00
metrics Adding metrics (#203) 2022-08-23 10:11:21 -06:00
nix nix: make derivation and update shell (#1003) 2024-12-09 17:07:01 +00:00
tests Validator historical state restoration (#922) 2024-12-14 05:07:55 +00:00
tools/cirdl chore: update dependencies, especially nim-ethers to chronos v4 compatible version (#968) 2024-10-30 10:40:17 +00:00
vendor feat: repair is rewarded (#1022) 2024-12-12 20:19:56 +00:00
.dockerignore Docker build (#354) 2023-03-08 12:45:55 +01:00
.editorconfig Project setup 2021-02-02 19:29:52 +01:00
.gitignore Add MIT/Apache licenses (#861) 2024-08-13 15:38:17 +00:00
.gitmodules Rework circuit downloader (#882) 2024-09-23 14:37:17 +00:00
LICENSE-APACHEv2 Remove extra license file (#876) 2024-08-19 09:48:03 +00:00
LICENSE-MIT Add MIT/Apache licenses (#861) 2024-08-13 15:38:17 +00:00
Makefile nix: make derivation and update shell (#1003) 2024-12-09 17:07:01 +00:00
README.md Remove moved docs (#930) 2024-10-03 08:55:54 +00:00
build.nims Validator historical state restoration (#922) 2024-12-14 05:07:55 +00:00
codecov.yml [ci] disable pull-request comments by codecov 2022-05-19 15:23:35 +02:00
codex.nim Rework circuit downloader (#882) 2024-09-23 14:37:17 +00:00
codex.nimble remove stale and misleading dependency version metadata (#833) 2024-06-17 14:31:10 +00:00
config.nims Chronos v4 Update (v3 Compat Mode) (#814) 2024-07-18 21:04:33 +00:00
env.sh add env.sh shim to project root (#34) 2021-12-20 13:12:18 -06:00
flake.lock nix: make derivation and update shell (#1003) 2024-12-09 17:07:01 +00:00
flake.nix nix: make derivation and update shell (#1003) 2024-12-09 17:07:01 +00:00
openapi.yaml chore(openapi): update the openapi spec for the manifest object (#972) 2024-11-15 13:21:18 +00:00

README.md

Codex Decentralized Durability Engine

The Codex project aims to create a decentralized durability engine that allows persisting data in p2p networks. In other words, it allows storing files and data with predictable durability guarantees for later retrieval.

WARNING: This project is under active development and is considered pre-alpha.

License: Apache License: MIT Stability: experimental CI Docker Codecov Discord Docker Pulls

Build and Run

For detailed instructions on preparing to build nim-codex see Build Codex.

To build the project, clone it and run:

make update && make

The executable will be placed under the build directory under the project root.

Run the client with:

build/codex

Configuration

It is possible to configure a Codex node in several ways:

  1. CLI options
  2. Environment variables
  3. Configuration file

The order of priority is the same as above: CLI options --> Environment variables --> Configuration file.

Please check documentation for more information.

Guides

To get acquainted with Codex, consider:

API

The client exposes a REST API that can be used to interact with the clients. Overview of the API can be found on api.codex.storage.