diff --git a/README.md b/README.md index 6b23404..747281c 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,14 @@ Proof Aggregation ================================ +**WARNING**: This repository contains work-in-progress prototypes, and has not received careful code review. It is NOT ready for production use. -This repository contains all work related to proof aggregation (currently only local proof aggregation). + +This repository contains all work related to the Codex proof system which supports proof aggregation (currently only local proof aggregation). + +## Quick Usage +see [`codex-storage-proofs-circuits`](../codex-storage-proofs-circuits) to look at the circuits. +see [`proof-input`](./proof-input) to test the circuits. +see [`workflow`](../workflow) for an overview of the whole workflow and how to use the circuits and run them. Repository organization ----------------- @@ -14,8 +21,11 @@ Repository organization - [`workflow`](./workflow) contains the scripts and example code to generate input, run the circuits, generate a proof, and verify the proof. +- [`goldibear_experiments`](./goldibear_experiments) contains experiments with using [Plonky2_Goldibear](https://github.com/telosnetwork/plonky2_goldibear/tree/main). + +- [`recursion_experiments`](./recursion_experiments) contains experiment with multiple recursion approaches prior to settling with the uniform 2-to-1 tree aggregation. + Documentation ----------------- -See the write-ups on [plonky2 storage proofs](https://hackmd.io/@NQdG6IOmQE6astjwhJ6ACw/rJSsScfAR) and [proof recursion](https://hackmd.io/@NQdG6IOmQE6astjwhJ6ACw/rk85D2HX1e) +See the write-ups on [plonky2 storage proofs](https://hackmd.io/@NQdG6IOmQE6astjwhJ6ACw/rJSsScfAR). -**WARNING**: This repository contains work-in-progress prototypes, and has not received careful code review. It is NOT ready for production use. diff --git a/codex-plonky2-circuits/README.md b/codex-plonky2-circuits/README.md index 14e94f0..98933ff 100644 --- a/codex-plonky2-circuits/README.md +++ b/codex-plonky2-circuits/README.md @@ -17,7 +17,11 @@ This crate is an implementation of the [codex storage proofs circuits](https://g - [`utils`](./src/circuits/utils.rs) contains helper functions. -- [`recursion`](./src/recursion) contains various approaches of implementing recursion circuits for aggregating proofs. +- [`circuit_helper`](./src/circuit_helper) contains a general trait for all Plonky2 circuit to automate the building and proving. + +- [`error`](./src/error.rs) contains the list of error related to the circuits. + +- [`uniform recursion`](./src/recursion/uniform) contains the uniform (2-to-1 tree) recursion circuits for aggregating proofs. ## Usage diff --git a/proof-input/.gitignore b/proof-input/.gitignore index a3547b9..88ac17f 100644 --- a/proof-input/.gitignore +++ b/proof-input/.gitignore @@ -11,3 +11,7 @@ pgo-data.profdata # MacOS nuisances .DS_Store + +# circuit files +/prover_data +/verifier_data \ No newline at end of file diff --git a/proof-input/README.md b/proof-input/README.md index 2dae891..52b418e 100644 --- a/proof-input/README.md +++ b/proof-input/README.md @@ -8,11 +8,15 @@ the [`plonky2 codex proof circuits`](../codex-plonky2-circuits). Currently only - [`merkle_tree`](./src/merkle_tree) is the implementation of "safe" merkle tree used in codex, consistent with the one [here](https://github.com/codex-storage/nim-codex/blob/master/codex/merkletree/merkletree.nim). +- [`data_structs`](./src/data_structs.rs) contains the data structures for the codex storage, this is used to generate proof input. + - [`gen_input`](./src/gen_input.rs) contains the necessary function to generate the proof input. -- [`json`](./src/serialization) contains the serialization functions to read and write the proof input from/to json files. +- [`recursion`](./src/recursion) contains the tests for the uniform (2-to-1 tree) recursion. -- [`params`](./src/params.rs) is the test parameters used to generate the input. The params include circuit params as well. +- [`serialization`](./src/serialization) contains the serialization functions to read and write circuit data, input, and proofs. + +- [`params`](./src/params.rs) is the test parameters used to generate the input. - [`sponge`](./src/sponge.rs) contains the non-circuit version of hash function (with and without padding) used to hash cells and during sampling. diff --git a/workflow/README.md b/workflow/README.md index 5fbb307..328f7e2 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -2,27 +2,28 @@ WARNING: This is a work-in-progress prototype, and has not received careful code review. This implementation is NOT ready for production use. This crate guides you through generating the circuit input, -exporting it to a JSON file, importing from a JSON file, -running the circuits to generate a proof, and finally verify the proof. +building the circuit, +running the circuits to generate a proof, aggregating multiple proofs, and finally verify the proof. This crate can be used to: - Generate circuit input from **fake data** with given params. -- Build the plonky2 codex storage proof circuits. +- Build the Plonky2 codex storage proof circuits. - Generate a proof with given proof input in JSON file. +- Aggregate multiple proofs with 2-to-1 tree like aggregation to generate a final proof. - Verify the proof. ## Code organization -- [`gen_input`](./src/bin/gen_input.rs) contains the main function to generated input with the given params as environment variables. +- [`gen_input`](./src/bin/gen_input.rs) contains the main function to generate input with the given params as environment variables. -- [`build_circ`](./src/bin/build_circ.rs) contains the main function to generated input with the given params as environment variables. +- [`build_circ`](./src/bin/build_circ.rs) contains the main function to build the storage proof (Sampling) circuits. -- [`prove`](./src/bin/prove.rs) contains the main function to generated input with the given params as environment variables. +- [`prove`](./src/bin/prove.rs) contains the main function to generate a single proof. -- [`prove_and_verify`](./src/bin/verify) contains the main function to generated input with the given params as environment variables. +- [`aggregate`](./src/bin/aggregate.rs) contains the main function to aggregate `k` (default = 4) proofs. -- [`aggregate`](./src/bin/aggregate.rs) contains the main function to generate the sampling proofs and aggregate `k` of them. +- [`verify`](./src/bin/verify) contains the main function to verify the storage proof (Sampling) proof. ## Usage @@ -54,7 +55,7 @@ export MAXDEPTH=32 # Maximum depth of the slot tree export MAXSLOTS=256 # Maximum number of slots export CELLSIZE=2048 # Cell size in bytes export BLOCKSIZE=65536 # Block size in bytes -export NSAMPLES=5 # Number of samples to prove +export NSAMPLES=100 # Number of samples to prove export ENTROPY=1234567 # External randomness export SEED=12345 # Seed for creating fake data @@ -63,17 +64,9 @@ export NSLOTS=11 # Number of slots in the dataset export SLOTINDEX=3 # Which slot to prove (0..NSLOTS-1) export NCELLS=512 # Number of cells in this slot ``` -Alternatively, for testing you can just use the default parameters as follows: -```rust -use proof_input::params::Params; - -fn main() { - let params = Params::default(); -} -``` #### Step 2: Run the Script -Once the params are set, you can run the script to generate the [`JSON file`](./input.json). +Once the params are set, you can run the script to generate the circuit input (with fake data). ```bash sudo bash ./scripts/gen_input.sh @@ -87,9 +80,8 @@ sudo bash ./scripts/build_circuit.sh To see the source code of how to build the circuit, see [`build_circ`](./src/bin/build_circ.rs). ### Generate the Proof -After generating the circuit input (in a JSON file), -you can run the circuits to generate the proofs. -First make sure you have the [`JSON file`](./input.json), then follow the steps: +After generating the circuit input (in a JSON file), you can run the circuits to generate the proofs. +First make sure you have the circuit data and input from previous scripts then follow the steps: #### Step 1: Setting Up Circuit Parameters Parameters for the circuit can be defined in [`circ_params.sh`](scripts/circ_params.sh). @@ -99,7 +91,7 @@ export MAX_DEPTH=32 # maximum depth of the slot tree export MAX_LOG2_N_SLOTS=8 # Depth of the dataset tree = ceiling_log2(max_slots) export BLOCK_TREE_DEPTH=5 # depth of the mini tree (block tree) export N_FIELD_ELEMS_PER_CELL=272 # number of field elements per cell -export N_SAMPLES=5 # number of samples to prove +export N_SAMPLES=100 # number of samples to prove ``` #### Step 2: Run the Script @@ -110,6 +102,14 @@ You can also see the time taken to generate the proof. sudo bash ./scripts/prove.sh ``` +### Verify the proof +To verify the generated proof, run the following script. +Make sure that you generate the circuit data and proof prior to this. + +```bash +sudo bash ./scripts/verify.sh +``` + ### Build, Prove, and Verify To automate the whole process, you can run the following script the script builds the circuit, loads the JSON circuit input, generates the proof, and verifies it. @@ -119,10 +119,12 @@ Make sure that you generate the circuit input prior to this so that you have the ```bash sudo bash ./scripts/prove_and_verify.sh ``` -To inspect the source code, see [`prove_and_verify`](./src/bin/verify). +This script simply runs all previous scripts. -### Generate K Proofs and aggregate them -To do this, you can run the following script: +### Aggregate K Proofs +To do this, you can run the following script. +Note that we don't actually generate `K` proofs but rather just clone the single generated proof which is enough for testing. +Make sure a proof is already generated using the script for proving above. ```bash sudo bash ./scripts/aggregate.sh