99a78a41ee
For minimal correctness, we must ensure that buffers that cross the thread boundary are allocated and deallocated within the same thread and that there is no reference counting going on during the computation. To get there with minimal changes: * Preallocate a buffer for the outcome of the merkle tree computation * Pass pointers instead of `ref` types between threads * Avoid relying on isolation - this is an ORC-only feature * Add `SharedBuf` as a simple "view" type that allows working with a set of values while at the same time avoiding allocations and refcounts - the view checks for out-of-bounds acccess much like a seq, but the user is responsible for managing lifetime (which in this case is simple since all that needs to happen is for the task to complete) * In order not to upset the code too much, use a simple linear packer for the hashes that simply copies the values back and forth * Block cancellation and panic if the thread signalling mechanism fails - cancelling the task itself would require inserting cancellation points in the computation The worker task relies on a nuance, namely that calling a closure procedure does not count as a reference-counting event - while this works, it can be brittle in "general" code since it's easy to make copy of the closure itself by accident - the refactoring necessary for addressing this point is beyond the scope of this change however.
Codex Decentralized Durability Engine
The Codex project aims to create a decentralized durability engine that allows persisting data in p2p networks. In other words, it allows storing files and data with predictable durability guarantees for later retrieval.
WARNING: This project is under active development and is considered pre-alpha.
Build and Run
For detailed instructions on preparing to build nim-codex see Build Codex.
To build the project, clone it and run:
make update && make
The executable will be placed under the build directory under the project root.
Run the client with:
build/codex
Configuration
It is possible to configure a Codex node in several ways:
- CLI options
- Environment variables
- Configuration file
The order of priority is the same as above: CLI options --> Environment variables --> Configuration file.
Please check documentation for more information.
Guides
To get acquainted with Codex, consider:
- running the simple Codex Two-Client Test for a start, and;
- if you are feeling more adventurous, try Running a Local Codex Network with Marketplace Support using a local blockchain as well.
API
The client exposes a REST API that can be used to interact with the clients. Overview of the API can be found on api.codex.storage.
Bindings
Codex provides a C API that can be wrapped by other languages. The bindings is located in the library folder.
Currently, only a Go binding is included.
Build the C library
make libcodex
This produces the shared library under build/.
Run the Go example
Build the Go example:
go build -o codex-go examples/golang/codex.go
Export the library path:
export LD_LIBRARY_PATH=build
Run the example:
./codex-go
Static vs Dynamic build
By default, Codex builds a dynamic library (libcodex.so), which you can load at runtime.
If you prefer a static library (libcodex.a), set the STATIC flag:
# Build dynamic (default)
make libcodex
# Build static
make STATIC=1 libcodex
Limitation
Callbacks must be fast and non-blocking; otherwise, the working thread will hang and prevent other requests from being processed.
Contributing and development
Feel free to dive in, contributions are welcomed! Open an issue or submit PRs.
Linting and formatting
nim-codex uses nph for formatting our code and it is required to adhere to its styling.
If you are setting up fresh setup, in order to get nph run make build-nph.
In order to format files run make nph/<file/folder you want to format>.
If you want you can install Git pre-commit hook using make install-nph-commit, which will format modified files prior committing them.
If you are using VSCode and the NimLang extension you can enable "Format On Save" (eq. the nim.formatOnSave property) that will format the files using nph.