db8f81cd63
A classic encoding of a merkle tree is to store the layers consecutively in memory breadth-first. This encoding has several advantages: * Good performance for accessing successive nodes, such as when constructing the tree or serializing it * Significantly lower memory usage - avoids the per-node allocation overhead which otherwise more than doubles the memory usage for "regular" 32-byte hashes * Less memory management - a single memory allocation can reserve memory for the whole tree meaning that there are fewer allocations to keep track of * Simplified buffer lifetimes - with all memory allocated up-front, there's no need for cross-thread memory management or transfers While we're here, we can clean up a few other things in the implementation: * Move async implementation to `merkletree` so that it doesn't have to be repeated * Factor tree construction into preparation and computation - the latter is the part offloaded onto a different thread * Simplify task posting - `threadpools` already creates a "task" from the worker function call * Deprecate several high-overhead accessors that presumably are only needed in tests
Codex Decentralized Durability Engine
The Codex project aims to create a decentralized durability engine that allows persisting data in p2p networks. In other words, it allows storing files and data with predictable durability guarantees for later retrieval.
WARNING: This project is under active development and is considered pre-alpha.
Build and Run
For detailed instructions on preparing to build nim-codex see Build Codex.
To build the project, clone it and run:
make update && make
The executable will be placed under the build directory under the project root.
Run the client with:
build/codex
Configuration
It is possible to configure a Codex node in several ways:
- CLI options
- Environment variables
- Configuration file
The order of priority is the same as above: CLI options --> Environment variables --> Configuration file.
Please check documentation for more information.
Guides
To get acquainted with Codex, consider:
- running the simple Codex Two-Client Test for a start, and;
- if you are feeling more adventurous, try Running a Local Codex Network with Marketplace Support using a local blockchain as well.
API
The client exposes a REST API that can be used to interact with the clients. Overview of the API can be found on api.codex.storage.
Bindings
Codex provides a C API that can be wrapped by other languages. The bindings is located in the library folder.
Currently, only a Go binding is included.
Build the C library
make libcodex
This produces the shared library under build/.
Run the Go example
Build the Go example:
go build -o codex-go examples/golang/codex.go
Export the library path:
export LD_LIBRARY_PATH=build
Run the example:
./codex-go
Static vs Dynamic build
By default, Codex builds a dynamic library (libcodex.so), which you can load at runtime.
If you prefer a static library (libcodex.a), set the STATIC flag:
# Build dynamic (default)
make libcodex
# Build static
make STATIC=1 libcodex
Limitation
Callbacks must be fast and non-blocking; otherwise, the working thread will hang and prevent other requests from being processed.
Contributing and development
Feel free to dive in, contributions are welcomed! Open an issue or submit PRs.
Linting and formatting
nim-codex uses nph for formatting our code and it is required to adhere to its styling.
If you are setting up fresh setup, in order to get nph run make build-nph.
In order to format files run make nph/<file/folder you want to format>.
If you want you can install Git pre-commit hook using make install-nph-commit, which will format modified files prior committing them.
If you are using VSCode and the NimLang extension you can enable "Format On Save" (eq. the nim.formatOnSave property) that will format the files using nph.