workflow description and scripts (WIP, mostly copied from codex-storage-proofs-circuits)

ceremony/.gitignore

@@ -0,0 +1,2 @@
+.DS_Store
+*.ptau

ceremony/README.md

@@ -0,0 +1,7 @@
+
+Place the "powers of tau" ceremony file(s) here.
+
+In the RLN case, normally a size `2^13 = 8192` ceremony should be enough.
+
+You can find a set of pre-made public ceremony files in the README of the
+[`snarkjs` repo](https://github.com/iden3/snarkjs).

workflow/.gitignore

@@ -0,0 +1,2 @@
+.DS_Store
+build

workflow/PROOFS.md

@@ -0,0 +1,164 @@
+
+The cogs of a ZK proof
+======================
+
+The goal of this document is to describe all the different moving parts 
+(circuit, witness, public input, etc) of a ZK proof through a simple example,
+hopefully giving some intuition about how these cogs fit together.
+
+
+The example: Fibonacci sequence
+-------------------------------
+
+ZK proofs (or more precisely: "succint arguments") are all about convicing
+another party (the "verifier"), about the truth of some statement, without
+telling them _everything_.
+
+In our toy example, the statement will be the 1000th element of a Fibonacci
+sequence.
+
+A Fibonacci sequence is generated deterministically from its first two elements,
+let's call them `U` and `V`. These are numbers. The sequence is then defined as
+
+    a[0] = U
+    a[1] = V
+    a[n] = a[n-1] + a[n-2]
+
+In the standard Fibonacci sequence, we have `U=0` and `V=1` (or sometimes `U=V=1`,
+which results in the same sequence but shifted by one). But we can consider 
+this generalized setting with `U` and `V` being arbitrary numbers.
+
+The statement I want to convince you, is that for a given `U` (which I tell you,
+say `U=53`), I know a value for `V` (which I don't tell you), such that `a[1000]`
+is some given value (of course, I tell you this value too, otherwise there is
+nothing to be convinced about).
+
+Pretty simple so far.
+
+
+The statement
+-------------
+
+A proof always start with a statement. Usually, the statement is descibed by
+a computer program (in our case, the program is computing the Fibonacci sequence).
+A program normally has some inputs and some ouputs. Some of the inputs are
+public (I tell you what they are), some others can be secret (I don't tell you what 
+they are). And the statement is _usually_ that if you run this program with the given 
+inputs, then the program runs normally (it doesn't throw an error) and you get the 
+given output.
+
+Note: you can freely move things from outputs to inputs. In our case, the output
+would be `a[1000]`, but equivalently, I can make another program where there is
+a third input `W`, and the program just throws an error at the end if `a[1000] != W`.
+Because of this, we often just say IO for input/output, and "public IO" for the
+part of it I tell you, and "private IO" or "secret IO" for the part I don't tell 
+you.
+
+So far we have:
+
+- the statement (a computer program)
+- the public IO
+- and the private IO
+
+The computer program is often called a "circuit", because for technical reasons
+it often needs to be described as an "arithmetic circuit", which is something
+similar to digital electronics circuits (but in math). This gets a bit more
+complicated with zkVM-s, where the circuit itself is something like a CPU, and your
+statement is an actual program running on that (virtual) CPU.
+
+
+The witness
+-----------
+
+How essentially all proof system works is to translate the statement you want
+the prove into some math equations, and then prove that I know a solution for
+those equations. Because of this, the prover needs to know not only the input
+and the output, but all the intermediate results too - essentially a very detailed
+log of all individual operations happening in the computer program describing 
+the statement. This huge logfile is called "the witness", and in practice it's
+usually just a large array of numbers (more precisely, finite field elements).
+
+Unfortunately, the word "witness" is a bit overloaded: Sometimes it refers only
+to the private IO, and sometimes to the whole log. While these are kind of the same
+from a holistic point of view, it still makes sense to distinguish them, and
+I will call the whole execution log the witness.
+
+Creating a proof always start by running the program which descibes statement,
+and creating this very detailed log, the witness (sometimes also called a 
+"transcript"). This step is usually called the "witness generation".
+
+
+Trusted setup
+-------------
+
+Some, but not all, proof systems requires something called a "trusted setup".
+This is some structured piece of information produced by an external party, 
+which has some secret encoded in it. It's important that the secret is
+deleted (hence the word "trusted") after generating this setup, because anybody 
+in possesion of the secret can cheat and create fake proofs. 
+
+For example in the popular scheme called "KZG commitments", the secret is
+a number (a field element) usually called `tau`, and the result of the 
+trusted setup is a sequence
+
+    [ g , tau*g , tau^2*g , ... , tau^N*g ]
+
+where `g` is a fixed group element. Because of this, the procedure generating
+this sequence is called a "powers-of-tau ceremony". Assuming the discrete 
+logarithm problem is hard in the group, it's not feasible to compute `tau` 
+itself given that sequence, but it's easy to compute `f(tau)*g` for a polynomial
+`f` with degree at most `N`.
+
+In practice such a trusted setup can be accomplished using multiparty computation
+(MPC), so that if there is at least _one_ honest participant, then the resulting
+sequence is safe to use. These events are called "ceremonies" (in early days
+people actually had to do this in person, and at the end they ceremonially destroyed 
+the computer which contained the random secret `tau`).
+
+There are two kinds of trusted setups: universal and circuit-specific. An universal
+one can be used to prove many different statements, while a circuit-specific one
+can be used to prove a single fixed statement, or circuit (but with different inputs).
+
+For example Groth16 needs a circuit-specific trusted setup, Plonk+KZG only needs 
+a universal trusted setup, while Plonk+FRI does not need any such setup at all. 
+However usually systems with trusted setups are more efficient (for example Groth16 has
+the smallest proofs among all known proof system). In case of Groth16, the circuit-specific 
+setup is generated from an already existing universal setup and the circuit.
+
+
+The files
+---------
+
+Using the popular [`circom`](https://docs.circom.io/) + [`snarkjs`](https://github.com/iden3/snarkjs) 
+ecosystem, all the above parts are in different files, which are produced by
+different commands.
+
+These are:
+
+- `.circom` files contain the source code of your circuit (both the equations and 
+  the actual program, interleaved together)
+- `input.json` contains the circuit inputs (both public and private inputs)
+- `.r1cs` file contains the circuit, that is, the statement you want to prove
+  (R1CS is short for Rank-1 Constraint System, a specific form of a "circuit").
+  This is one output of the `circom` compiler, which reads the above source code and produces `.r1cs` files
+- `.wtns` files contain the witness. This is generated by the so-called "witness generator", which is 
+  another output of the `circom` compiler. Essentially the compiler separates the equations (goes into `.r1cs`)
+  and running the program (goes into the witness generator, which in this case can be either a WASM or a C++ program).
+  When you run the witness generator, it takes the `input.json` and produces the `.wtns` file
+- `.ptau` files are containing universal trusted setups
+- `.zkey` files are called a "prover key" (it contains everything the prover needs), and 
+  in case of Groth16 it also corresponds to a circuit-specific trusted setup.
+- the prover takes the `.zkey` and the `.wtns` files, and produces two outputs: a `proof.json` 
+  containing the proof itself, and a `public.json` containing only the public input (so this
+  is copied from `input.json`, but the latter also contains the private inputs)
+- from the `.zkey` file, a "verification key" can be extracted (this is again `.json` file),
+  which contains everything the verifier needs (in particular, it contains something like a 
+  hash of the circuit. At least once this has to be checked against a source code, trusted setup
+  and resulting `.zkey`, so that you actually know which statement you verify!
+  However you don't want to do this agin and again, because the statement - the circuit - can be very big).
+  In practice the verifier key is often hardcoded in the on-chain verifier contract.
+- then finally the verifier takes this verification key, the proof file, and the public input file
+  (all `.json` files here), and outputs either "valid" or "invalid"
+
+The command line workflow of doing all this is described in the file
+[README.md](README.md).

workflow/README.md

@@ -0,0 +1,134 @@
+
+Guide though the whole proof workflow
+-------------------------------------
+
+The workflow described below is implemented with shell scripts in this directory.
+So the below is more like an explanation.
+
+To run the full workflow:
+
+- set the parameters by editing `params.sh`
+- run `setup.sh` to do the circuit-specific setup
+- run `prove.sh` to generate input, compute witness and create (and verify) the proof
+
+NOTE: the examples below assume `bash`. In particular, it won't work with `zsh` 
+(which is the dafault on newer macOS)! Because, you know, reasons...
+
+To have an overview of what all the different steps and files are, see [PROOFS.md](PROOFS.md).
+
+### Preliminaries
+
+- install `circom`, `snarkjs`, `rapidsnark`: <https://docs.circom.io/getting-started/installation>
+- furthermore install `circom-witnesscalc`: <https://github.com/iden3/circom-witnesscalc/> (note: we need the legacy `build-circuit` version!)
+- install Nim: <https://nim-lang.org/>
+
+TODO: fix this: 
+Build the Nim cli proof input generator:
+
+    $ cd ../reference/nim/proof_input/
+    $ nimble build -d:release cli
+    $ cd ../../../workflow
+
+### Powers of tau setup
+
+Either download a ready-to-use "powers of tau" setup file (section 7), or generate one
+youself using `snarkjs` (sections 1..7), see the README here: <https://github.com/iden3/snarkjs>
+
+Size `2^13` (file size about 10MB) should be big enough:
+
+    $ cd ../ceremony
+    $ wget https://storage.googleapis.com/zkevm/ptau/powersOfTau28_hez_final_13.ptau
+    $ cd ../workflow
+
+Note: generating this yourself will probably take quite some time (though this size is relatively small, so maybe not that bad).
+
+### Set the parameters
+
+There are quite a few parameters (run `cli --help` too see them), it's probably
+best to collect them into a parameter file. Check out `params.sh` and `cli_args.sh` 
+to see one way to do that.
+
+You can edit `params.sh` to your taste before running the workflow scripts.
+
+### Compile the circuit
+
+Create a build directory so we don't pollute the repo:
+
+    $ mkdir -p build
+    $ cd build
+
+After that, the first real step is to create the main component:
+
+    $ source ../cli_args.sh && ../../reference/nim/proof_input/cli $CLI_ARGS -v --circom="rln_main.circom"
+
+Then compile the circuit:
+
+    $ export CIRCUIT_LIBDIRS="-l../../circuit/lib -l../../circuit/poseidon2 -l../../circuit/codex"
+    $ circom --r1cs --wasm --O2 ${CIRCUIT_LIBDIRS} rln_main.circom
+
+### Extract the witness computation graph
+
+    $ build-circuit rln_main.circom rln_main.graph ${CIRCUIT_LIBDIRS}
+
+### Do the circuit-specific setup
+
+See the [`snarkjs` README](https://github.com/iden3/snarkjs) for an overview of
+the whole process.
+
+    $ snarkjs groth16 setup rln_main.r1cs ../../ceremony/powersOfTau28_hez_final_21.ptau rln_main_0000.zkey
+    $ snarkjs zkey contribute rln_main_0000.zkey rln_main_0001.zkey --name="1st Contributor Name"
+
+NOTE: with large circuits, javascript can run out of heap. You can increase the
+heap limit with (but as this is a small circuit, this is not necessary):
+
+    $ NODE_OPTIONS="--max-old-space-size=8192" snarkjs groth16 setup <...>
+
+You can add more contributors here if you want.
+
+Finally rename the last contributions result and export the verification key:
+
+    $ rm rln_main_0000.zkey
+    $ mv rln_main_0001.zkey rln_main.zkey
+    
+    $ snarkjs zkey export verificationkey rln_main.zkey rln_main_verification_key.json
+
+NOTE: You have redo all the above if you change any of the five parameters the circuit 
+depends on (these are: maxdepth, maxslots, cellsize, blocksize, nsamples).
+
+### Generate an input to the circuit
+
+    $ source ../cli_args.sh && ../../reference/nim/proof_input/cli $CLI_ARGS -v --output=input.json
+
+### Generate the witness
+
+    $ cd rln_main_js
+    $ time node generate_witness.js rln_main.wasm ../input.json ../witness.wtns
+    $ cd ..
+
+### Create the proof
+
+Using `snarkjs` (very slow, but more portable):
+
+    $ snarkjs groth16 prove rln_main.zkey witness.wtns proof.json public.json
+
+Or using `rapidsnark` (fast, but not very portable):
+
+    $ rapidsnark rln_main.zkey witness.wtns proof.json public.json
+
+Or using `nim-groth16` (experimental):
+
+    $ nim-groth16 -p -z=rln_main.zkey -w=witness.wtns -o=proof.json -i=public.json
+    
+The output of this step will consist of:
+
+- `proof.json` containing the proof itself
+- `public.json` containing the public inputs
+
+### Verify the proof (on CPU)
+
+    $ snarkjs groth16 verify rln_main_verification_key.json public.json proof.json
+
+### Generate solidity verifier contract
+
+    $ snarkjs zkey export solidityverifier rln_main.zkey verifier.sol
+

workflow/cli_args.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+MY_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+source ${MY_DIR}/params.sh
+
+CLI_ARGS="--merkle_depth=${MERKLE_DEPTH} \
+ --limit_bits=${LIMIT_BITS} \
+ --seed=$SEED"
+
+if [[ "$1" == "--export" ]]
+then
+  echo "exporting CLI_ARGS"
+  export CLI_ARGS
+fi

workflow/params.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+MERKLE_DEPTH=20    # depth of the Merkle tree
+LIMIT_BITS=16      # log2 of the maximal possible rate limit per epoch
+
+SEED=1234567       # seed for creating fake data

workflow/paths.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+
+ORIG=`pwd`
+
+NIMCLI_DIR="${ORIG}/../test_data/proof_input/"
+PTAU_DIR="${ORIG}/../ceremony"
+
+CIRCUIT_ROOT="${ORIG}/../circuit/"
+CIRCUIT_POS_DIR="${CIRCUIT_ROOT}/poseidon2/"
+
+PTAU_FILE="powersOfTau28_hez_final_13.ptau"
+PTAU_PATH="${PTAU_DIR}/${PTAU_FILE}"
+
+CIRCUIT_MAIN="rln_main"

workflow/prove.sh

@@ -0,0 +1,91 @@
+#!/bin/bash
+
+source ./paths.sh
+source ./cli_args.sh
+
+# --- setup build directory ---
+
+mkdir -p build 
+cd build
+
+# --- export the witness computation graph ---
+
+if command -v build-circuit
+then
+  CIRCUIT_INCLUDES="-l${CIRCUIT_LIB_DIR} -l${CIRCUIT_POS_DIR} -l${CIRCUIT_PRF_DIR}"
+  build-circuit ${CIRCUIT_MAIN}.circom ${CIRCUIT_MAIN}_graph.bin ${CIRCUIT_INCLUDES}
+else
+  echo " "
+  echo "\`circom-witnesscalc\` not found; skipping graph extraction"
+fi
+
+# --- generate input for the circuit ---
+
+echo ""
+echo "generating the input for the proof circuit..."
+${NIMCLI_DIR}/cli $CLI_ARGS -v --output=input.json
+
+# --- generate the witness ---
+
+start=`date +%s`
+echo ""
+echo "generating the witness..."
+cd ${CIRCUIT_MAIN}_js
+time node generate_witness.js ${CIRCUIT_MAIN}.wasm ../input.json ../witness.wtns
+cd ${ORIG}/build
+end=`date +%s`
+echo "Generating the witness took `expr $end - $start` seconds."
+
+# --- create the proof ---
+
+PROVER="snarkjs"
+
+RS=`which rapidsnark`
+if [[ ! -z "$RS" ]]
+then
+  PROVER="rapidsnark"
+fi
+
+# PROVER="zikkurat"
+# PROVER="nim"
+
+echo ""
+echo "creating the proof... using prover: \`$PROVER\`"
+
+start=`date +%s`
+case $PROVER in
+  snarkjs)
+    time snarkjs groth16 prove ${CIRCUIT_MAIN}.zkey witness.wtns proof.json public.json
+    ;;
+  rapidsnark)
+    time rapidsnark ${CIRCUIT_MAIN}.zkey witness.wtns proof.json public.json
+    ;;
+  nim)
+    time nim-groth16 -tpv --zkey=${CIRCUIT_MAIN}.zkey --wtns=witness.wtns -o=proof.json -i=public.json
+    ;;
+  zikkurat)
+    time zikkurat-groth16 -tpv --zkey=${CIRCUIT_MAIN}.zkey --wtns=witness.wtns # -o=proof.json -i=public.json
+    ;;
+  *)
+    echo "unknown prover \`$PROVER\`"
+    exit 99
+    ;;
+esac
+end=`date +%s`
+echo "Creating the proof took `expr $end - $start` seconds."
+
+# --- verify the proof ---
+
+echo ""
+echo "verifying the proof:"
+snarkjs groth16 verify ${CIRCUIT_MAIN}_verification_key.json public.json proof.json
+
+# --- create solidity verifier contract ---
+
+echo ""
+echo "creating solidity verifier contract:"
+snarkjs zkey export solidityverifier ${CIRCUIT_MAIN}.zkey verifier.sol
+
+# --- finish  ---
+
+cd $ORIG

workflow/setup.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+source ./paths.sh
+source ./cli_args.sh
+
+# --- setup build directory ---
+
+mkdir -p build
+cd build
+
+# --- generate the main component ---
+
+### ${NIMCLI_DIR}/cli $CLI_ARGS -v --circom=${CIRCUIT_MAIN}.circom
+
+cp ${CIRCUIT_ROOT}/example_main.circom ./${CIRCUIT_MAIN}.circom
+
+# --- compile the circuit ---
+
+echo ""
+start=`date +%s`
+CIRCUIT_INCLUDES="-l${CIRCUIT_ROOT} -l${CIRCUIT_POS_DIR}"
+circom --r1cs --wasm --O2 ${CIRCUIT_INCLUDES} ${CIRCUIT_MAIN}.circom
+end=`date +%s`
+echo "Compiling the circuit took `expr $end - $start` seconds."
+echo ""
+
+# --- extract the witness computation graph ---
+
+build-circuit --O2 ${CIRCUIT_MAIN}.circom ${CIRCUIT_MAIN}.graph ${CIRCUIT_INCLUDES} 
+
+# --- circuit specific setup ---
+
+start=`date +%s`
+
+NODE_OPTIONS="--max-old-space-size=8192" snarkjs groth16 setup ${CIRCUIT_MAIN}.r1cs $PTAU_PATH ${CIRCUIT_MAIN}_0000.zkey
+echo "some_entropy_75289v3b7rcawcsyiur" | \
+NODE_OPTIONS="--max-old-space-size=8192" snarkjs zkey contribute ${CIRCUIT_MAIN}_0000.zkey ${CIRCUIT_MAIN}_0001.zkey --name="1st Contributor Name"
+
+rm ${CIRCUIT_MAIN}_0000.zkey
+mv ${CIRCUIT_MAIN}_0001.zkey ${CIRCUIT_MAIN}.zkey
+snarkjs zkey export verificationkey ${CIRCUIT_MAIN}.zkey ${CIRCUIT_MAIN}_verification_key.json
+
+end=`date +%s`
+echo "The circuit specific setup took `expr $end - $start` seconds."
+
+# --- finish the setup ---
+
+cd $ORIG