research/tools/benchmarks/Equi-X/README.md

Equi-X

A benchmarking framework for the Equi-X proof-of-work algorithm — tevador's Equihash(n=60, k=3) over the HashX pseudo-random hash function (the client-puzzle used by Tor onion-service PoW, proposal 327).

It benchmarks multiple implementations side by side — the reference C (tevador/equix + hashx) and the Rust (Tor arti equix / hashx crates) — across all parameters (runtime backend, operation, challenge, difficulty) and measures execution time and cost (solve/verify throughput, peak memory, JIT-compile overhead, and difficulty/effort cost). It is extensible to any other implementation via a small language-agnostic plugin protocol.

What it measures

• Solve & verify time + throughput — the asymmetric core of the PoW.
• Peak RSS memory — per implementation and runtime.
• JIT compile overhead — interpreter vs compiled HashX, isolated.
• Difficulty / effort sweep — Tor prop-327 effort: expected attempts & time to reach a target difficulty.
• DoS-protection effectiveness — the attacker-solve vs defender-verify asymmetry on this system, with a verdict (effective from what effort). This figure is per-core (derived as 1/latency from a single serial op).
• Sustained throughput under concurrency (--full) — the complementary measured answer: N worker processes run at once (N = 1, 2, 4, … up to the core count) to capture real memory-bandwidth contention, reporting the machine's true aggregate solves/s and verifies/s and the saturation knee. Additive — it never overwrites the per-core estimate above.
• Mining rate vs difficulty (--full, or standalone via configs/mining.toml) — the measured whole-machine token mint rate at each effort target (pooled over many independent nonce ranges, one streaming search per core), the basis for "control the mint rate by setting difficulty". Outputs mining.csv + a report section; best measured under idle (scripts/run_when_idle.sh <cmd> gates on CPU idle).
• Compiler-flag comparison — the same C impl built under different -O levels / -march / -flto / gcc-vs-clang, compared side by side.

Two companion documents build on the measurements: docs/findings.md (the full findings report: how Equi-X works, message-exchange schemas, DoS and mining usage with the measured numbers) and docs/difficulty-control.md (closed-loop difficulty controllers — mint-rate + load — with a simulator calibrated on the measured curve: python -m equix_bench.difficulty_control).

Every generated plot compares the implementations (C vs Rust) on the same axes, and a correctness cross-check proves the implementations agree (solutions from one verify under the other; effort values match byte-for-byte).

Architecture

flowchart TB
    cfg["configs/*.toml<br/>parameter matrix"]

    subgraph harness["Python harness — equix_bench"]
        direction TB
        config["config.py<br/>expand matrix → cells"]
        registry["registry.py<br/>adapter manifests"]
        runner["runner.py<br/>spawn 1 process / cell"]
        stats["stats.py<br/>aggregate median/p95"]
        crosscheck["crosscheck.py<br/>interop gate"]
        dos["dosprotect.py<br/>DoS asymmetry"]
        report["report.py<br/>plots + report.md"]
    end

    subgraph runners["Runners — JSON job/result over stdio"]
        direction LR
        c["runners/c<br/>libequix + hashx"]
        rust["runners/rust<br/>equix + hashx crates"]
        plugin["your adapter<br/>(any language)"]
    end

    outputs["report.md · results.csv · raw/*.json<br/>plots/*.png (C-vs-Rust, faceted per CPU, DoS)"]

    cfg --> config
    registry --> runner
    config --> runner
    runner -- job JSON --> c
    runner -- job JSON --> rust
    runner -- job JSON --> plugin
    c -- result JSON --> stats
    rust -- result JSON --> stats
    plugin -- result JSON --> stats
    stats --> report
    crosscheck --> report
    dos --> report
    report --> outputs

    manifests["adapters/*.manifest.toml"] -.-> registry

• Runners wrap one implementation and speak a JSON-over-stdio protocol (adapters/README.md). One process per parameter cell keeps memory attributable.
• Harness (harness/equix_bench) expands the TOML matrix, runs every cell, aggregates statistics, cross-checks implementations, evaluates DoS-protection, and renders results/report.md, results/results.csv, and comparison plots.
• Extensible: add an implementation by dropping in a runner that speaks the protocol plus a manifest — including compiler-flag variants and future GPU runners.

Run everything (one command)

./scripts/run_all.sh            # bootstrap deps + build + test + benchmark + compiler variants
./scripts/run_all.sh --full     # deeper sweep (full config + effort sweep; takes longer)

Or via make — make benchmark depends on setup and test, so one command gets a correct-by-construction run:

make benchmark        # setup -> tests -> quick main benchmark
make benchmark-full   # setup -> tests -> full sweep (concurrency + mining)
make help             # all targets: check/test/compiler-flags/mining/control/clean/distclean

Copied or moved the repo (e.g. rsync'd to another machine)? setup.sh (and make setup) detects the stale CMake caches that a copy carries and cleans them automatically; make clean / make distclean are also available.

run_all.sh does the whole pipeline: installs/builds dependencies (setup.sh), runs the unit tests, runs the main C-vs-Rust benchmark (all operations + the DoS-protection verdict, with the correctness gate), then builds the compiler-flag variants and compares them. Outputs:

• results/main/report.md — C vs Rust: time, throughput, RSS, compile, effort, DoS (with --full also the concurrency and mining sections + concurrency.csv, mining.csv)
• results/compiler_flags/report.md — compiler-flag comparison

Flags: --out DIR, --no-variants, --no-setup, --no-tests (see --help).

Quick start (step by step)

# 1. Install any missing deps (cmake/compiler/cargo/python) + fetch + build.
#    setup.sh auto-installs via the system package manager / rustup when possible.
#    Use `--check` to only report what's missing; EQUIX_NO_AUTO_INSTALL=1 to disable.
./scripts/setup.sh          # installs the harness + pytest into a project .venv

# 2. Run the smoke benchmark (seconds).  Use the venv interpreter setup.sh made:
.venv/bin/python -m equix_bench run --config configs/smoke.toml --out results/
#    (or `make benchmark` / `scripts/run_all.sh`, which auto-prefer .venv)

# 3. Full sweep (minutes)
python -m equix_bench run --config configs/full.toml --out results/

# One-shot end-to-end check
./scripts/verify.sh

Outputs land in results/: report.md, results.csv, raw/results.json, and plots/*.png (each comparing C vs Rust).

DoS-protection evaluation

Equi-X is a DoS defense: requesters solve (expensive), the service verifies (cheap). Any run that includes effort + verify gets a DoS-protection section in the report — measured attacker-cost vs defender-cost on this system, the asymmetry factor, verify throughput, and a verdict ("effective from effort ≥ E*").

python -m equix_bench run --config configs/dos_protection.toml --out results/
# report.md -> "DoS-protection effectiveness (this system)" + dos_protection.png

Comparing compiler flags

Build the C runner (and libequix/hashx) under several compiler/optimization flag sets, then benchmark them as separate impls — every plot compares the variants:

./scripts/build_variants.sh   # gcc -O0/-O2/-O3/-march=native/-flto, clang -O3, ...
python -m equix_bench run --config configs/compiler_flags.toml --out results/

Each variant becomes equix-c-<name> via an auto-generated manifest in adapters/generated/; unbuilt variants (missing compiler) are skipped.

The main equix-c runner is not one fixed guess: setup.sh runs scripts/autotune_c_flags.sh, which builds the fast-tier flag candidates (-O2, -O3, -O3 -march=native, -O3 -flto) with the default compiler, benchmarks the JIT solve path, and installs the fastest binary as the main runner (recording the winning flags in build/runners/c/equix_runner.flags and build/provenance.json). Solve is JIT-dominated so the margin is usually small; when plain -O3 is within 1% of the best it is preferred (portable, no native lock-in). Skip the tuning with EQUIX_NO_AUTOTUNE=1 (falls back to -O3 -DNDEBUG).

Comparing multiple CPUs (and GPUs)

Every run records the CPU it executed on (model, arch) and shows it on each plot. To compare across machines, run on each and merge — no re-benchmarking:

python -m equix_bench run --config configs/full.toml --out runA/ --device-label host-a
# ...on another machine...
python -m equix_bench run --config configs/full.toml --out runB/ --device-label host-b
python -m equix_bench combine --inputs runA runB --out combined/

combine produces faceted plots (one panel per CPU, C-vs-Rust within each) plus xdev_* cross-CPU charts (x=CPU, series=implementation) for headline metrics. The concurrency and mining sections/figures are carried across every device too (faceted per CPU), not just the solve/verify/effort plots.

Automatic, when results from many devices are collected in one tree

Copy or rsync each device's output under a single directory, then combine the whole tree in one command — no need to list every run by hand:

scripts/combine_all.sh results-by-device --out combined/   # or: make combine ROOT=results-by-device

Discovery is layout-agnostic: any directory holding a raw/results.json is treated as a run, and each run's device identity comes from inside its records (CPU model + OS), so folder names are free-form and two machines are never conflated. Re-runs of the same device are de-duplicated automatically (newest wins). The layout can be anything, e.g.:

results-by-device/
  laptop-x1/main/{raw/results.json, concurrency.csv, mining.csv, ...}
  server-epyc/main/{raw/results.json, ...}
  rpi5/results/main/{raw/results.json, ...}

GPU? Equi-X/HashX is CPU-oriented by design (deliberately GPU/ASIC-hostile), so no GPU implementation is benchmarked or bundled. The framework is GPU-ready though: a runner reporting device: "gpu" plugs in via the adapter protocol and appears on every figure as another device automatically. See PARAMETERS.md §8.

Requirements

• C: cmake ≥ 3.10, gcc/clang. Rust: cargo/rustc. Python ≥ 3.11 (matplotlib).
• x86-64 or aarch64 for HashX JIT; other targets fall back to the interpreter.
• setup.sh installs the harness (matplotlib/numpy) and pytest into a project .venv. This sidesteps the Linux "externally-managed-environment" / "pytest not importable" failure (PEP-668): a virtualenv has its own writable site-packages, so pip just works — no pyenv or --break-system-packages needed. On Debian/Ubuntu it provisions python3-venv if missing, and falls back to system pip if a venv can't be created.

Platform support (Linux, macOS, ARM64, Raspberry Pi 5)

Runners and harness are portable C / Rust / Python. CI builds and runs the smoke benchmark on Linux (x86-64) and macOS (Apple Silicon); both are supported, along with ARM64 Linux (Raspberry Pi 5).

macOS (Intel & Apple Silicon)

• Interpreter runtime: fully supported on both Intel and Apple Silicon.
• JIT (compiled runtime): works on Linux and Intel macOS. On Apple Silicon the bundled HashX C JIT uses mmap+mprotect without MAP_JIT, which the kernel rejects — so the C runner detects this and uses the interpreter (try-compile → interpreter, must-compile → clean error), never crashing. The Rust impl JITs via dynasmrt (which handles Apple Silicon), so runtime_effective honestly reports what each impl actually ran.
• Platform specifics are handled: peak RSS via getrusage (macOS reports bytes, not Linux's KB — converted), CPU name via sysctl (no /proc), OS via uname.

ARM64 / Raspberry Pi 5

The framework is portable C / Rust / Python with no x86-specific code, so it runs on ARM64 including the Raspberry Pi 5:

• 64-bit OS (recommended for Pi 5): full support, including the JIT — HashX ships an aarch64 compiler backend (compiler_a64.c; the Rust side uses dynasmrt), so interpret, try-compile, and must-compile all work.
• 32-bit OS (armv7/armhf): runs interpreter-only — there is no 32-bit-ARM JIT, so try-compile transparently falls back to the interpreter and must-compile fails by design. Use a 64-bit OS on the Pi 5 to benchmark the JIT.
• CPU identification handles ARM /proc/cpuinfo (no model name field): the device label uses the board Model (e.g. raspberry-pi-5-model-b-rev-1-0-<kernel>).
• Comparing x86 vs ARM is a first-class use case — run on each and combine for faceted x86-vs-Pi figures (the arch is recorded per device).
• Caveat: the Pi 5 can thermal-throttle under sustained solving; use active cooling and watch the reported stddev/p95 for stability.

Build on the Pi exactly as elsewhere: ./scripts/setup.sh (it creates the .venv with the harness installed; use .venv/bin/python -m equix_bench ... to run).

Parameters

See PARAMETERS.md for a full description of every parameter (operation, runtime, challenge/nonce, effort/difficulty, repetitions/warmup) and its implications for time, memory, and cost.

Adding an implementation

Write a runner that speaks the protocol in adapters/README.md, drop a <name>.manifest.toml next to the examples, and add its name to a config's run.impls. No harness code changes required.

Licensing

The framework code (runners glue, harness) is MIT (LICENSE). The vendored tevador/equix + hashx (git submodule) and the Rust equix/hashx crates are LGPL-3.0-only; they are used as dependencies/submodules and are not relicensed here.