research/tools/simulators/tsi/tsi-sim-pernode/README.md

# tsi-sim-pernode — Cryptarchia TSI **per-node** network simulator (Phase 2)

> The reduced-model simulators (`../tsi-sim/`, `../tsi-sim-mc/`) collapse the network to one
> global canonical chain and one scalar `D_est` per epoch. **This package removes that
> collapse:** every one of the `N` nodes runs TSI individually with its **own** `D_est`, from
> its **own** partial view of the block tree under explicit message propagation over a peering
> graph. Its job is to *test* the reduced model's assumption that all honest nodes agree.

## What it models

- **Per-node lottery:** node `i` wins a slot with `φ_f(w_i / D_est_i)` — `D_est` is a length-`N`
  **vector**, each node self-updating from its own view (the reduced model's key reuse: the
  sparse sampler already takes a per-node probability vector).
- **Topology** (`topology`): three propagation models over the network.
  - `full_mesh`: every node one hop away with uniform latency `L` — reproduces the reduced
    model exactly (validation baseline).
  - `regular`: a random **d-regular** peering graph (configurable `degree`) with per-link
    latency (`link_latency_dist ∈ {fixed, uniform, exp, geo}`, all with mean
    `link_latency_mean`). A block reaches a node after the shortest **weighted** path from its
    producer (gossip flooding). Models **direct block gossip**.
  - `blend`: the **same** d-regular graph, but a block is first routed through the **Blend
    mixnet** before it is public — the producer picks `blend_hops` distinct relay nodes
    uniformly at random, the block hops `producer → r₁ → … → r_hops` over the graph, each relay
    waiting a `Uniform(0, blend_delay_max)` **mixing delay** before forwarding, and the last
    relay's forward is the final network-wide gossip that makes the block visible. Relays are
    blind forwarders (they learn the block only from that final gossip). The dominant latency is
    the per-hop mixing, not the graph transport — this is the multi-slot regime where forks and
    the stake underestimate appear and uncle references matter. Because the mixing delays are
    `Uniform`-bounded, the windowed fork choice stays **exact** (horizon
    `(blend_hops+1)·max_path_latency + blend_hops·blend_delay_max`).
- **Real-world latency (units).** Latency is in **slots** and a slot is **1 s**, so measured
  internet latencies (tens–hundreds of ms) are *fractions* of a slot; arrivals are therefore
  kept **sub-slot (float)**, not rounded to whole slots. `link_latency_dist=geo` draws each
  link from a geographic band mixture (`~15 ms` metro → `~200 ms` antipodal, EU↔EU ≪ EU↔AU),
  rescaled so `link_latency_mean` stays the mean-latency knob. So `regular` runs the realistic
  sub-slot direct-gossip regime (`~0.05–0.2` slot), where forks are rare, and `blend` runs the
  multi-slot Blend-mixnet regime, where per-hop mixing delays dominate.
- **Per-node views:** one global block tree plus an `(N × n_blocks)` **arrival matrix** `A`;
  each node builds on / measures density over the blocks that have arrived at it. Uncle refs
  are **baked at production** from the producer's view (faithful — immutable once adopted).
- **Metrics:** per-node `D_est` spread (`range`, `IQR`), canonical-chain **agreement**
  (window prefix vs current tip), mean accuracy, and — with `init_dest=heterogeneous` —
  transient re-convergence.

## Headline result

**Per-node `D_est` disagreement collapses to zero.** Because TSI reads density from a window
buried far past `k`-finality, and all nodes seed the recursion from a common hardcoded
genesis `D`, every node computes the **same** measured density `m` → **identical** `D_est`
(`range ≈ 0`, `agreement_window = 1`) — *even under a sparse graph with high latency and heavy
tip-level forking* (`agreement_tip` can drop well below 1). This **validates the reduced
model**. Topology/latency instead shift the shared *mean* accuracy (via fork rate → `q`),
which uncle references recover just as in the reduced model. (Injected heterogeneous
disagreement, which the real protocol never creates, is *preserved* by the common
multiplicative update — a cautionary note, not protocol behaviour.)

## Quick start

```bash
make install         # venv + editable install
make test            # unit + fast per-node checks
make verify          # per-node validation (parity, spread→0, agreement, topology effect)
# Run any configs/<name>.yaml by its stem (auto-discovered); each writes a dated runs/ folder:
make smoke           # tiny end-to-end grid + figures  (configs/smoke.yaml)
make default         # scaled-k divergence/topology sweep + figures  (configs/default.yaml)
make fullscale       # full-scale (true k) confirmation  (configs/fullscale.yaml)
make figures RESULTS=runs/<dir>/results.parquet          # re-render figures from a run
# Extra sweep flags: make fullscale SWEEP_ARGS="--batch-size 1 --mem-frac 0.6"
```

## Scale & performance

- **Representation:** one global block tree + `(N × n_blocks)` `float64` arrival matrix `A`
  (sub-slot arrivals); topology `path_latency[N,N]` (per-node Dijkstra, once per trajectory).
- **`n_blocks` is NOT `~10·k` in general — it tracks block production.** `n_blocks` is the number
  of lottery wins in an epoch, `≈ E·Σᵢφ(wᵢ/D_est)`. At equilibrium that is `~10·k` (≈ 22k at
  k=2160), but when `D_est` is far below the true stake — the **collapsed-estimate regime**, e.g. a
  small `genesis_d_factor` — `Σ(stake)/D_est = 1/genesis_d_factor` is large and block production
  explodes proportionally. At `genesis_d_factor=0.01`, genesis epoch-0 produces **~2.0M blocks**
  (100× equilibrium) → `A ≈ 15 GB` for a *single* worker; `D_est` self-corrects to equilibrium
  within ~2 epochs, so only the earliest epoch(s) are heavy. **Raising `genesis_d_factor` toward
  0.1–0.5 collapses this cost** (0.1 → ~0.22M blocks → ~1.6 GB; 0.5 → ~44k → ~0.4 GB) and does not
  change the equilibrium result, which is measured after burn-in.
- **Sliding-window prune (`prune_arrival`, default on):** the arrival matrix never needs per-node
  columns for blocks past the horizon — under deterministic latency a block with `slot ≤ t − H` has
  reached *every* node, so its column is finalized and dropped. We keep columns only for blocks
  inside `max(horizon, uncle_window)` slots in a base-offset buffer, turning the `O(N·n_blocks)`
  matrix into `O(N · keep-span-blocks)`. This is what makes the collapsed regime affordable: at
  N=1000/k=2160/`gdf=0.01` the buffer is ~tens of MB instead of the ~15 GB full matrix (fork choice,
  the parent clamp, uncle selection, and per-node tips all reconstruct exactly from it). It is
  **bit-identical** to the full matrix at `jitter_mean == 0` (proven by `test_prune_matches_full_matrix`
  across topologies/uncles/`gdf`); with jitter it falls back to the full matrix (whose safety clamp
  keeps the tree valid). Set `prune_arrival: false` to force the full matrix (the parity oracle).
  The measurement pass also argmaxes in node-row bands so it adds only a small temporary. Divergence
  sweeps run at scaled **k=256** (`configs/default.yaml`); full-scale k=2160 is validated to **N ≤ 2000**.
- **Worker sizing (auto, RAM-safe):** both `A` (`~N·n_blocks`, incl. the block explosion above)
  and `path_latency` (`~N²`) grow, so the sweep runner sizes the loky pool to fit a RAM budget
  (`--mem-frac`, default 0.7 of physical RAM) instead of blindly using every core. The per-worker
  estimate realises the seeded stake to compute the **genesis-epoch** block count
  (`expected_peak_blocks`), so it reflects a low-`genesis_d_factor` explosion rather than assuming
  `~10·k`. A **calibration probe** measures a real worker's peak RSS (one genesis epoch of the
  heaviest config in a spawned process) whenever the estimate is heavy or `N > 2000`
  (`--calibrate {auto,always,never}`, default `auto`; the probe bounds itself to physical RAM so it
  fails loud rather than freezing).
- **Fail-loud memory guard (`memguard.py`):** every worker checks size *before* allocating both
  big arrays — the `(N × n_blocks)` `A` (in `build_tree_pernode`) and the `(N × N)` `path_latency`
  (in `build_path_latency`, built first) — and raises `ArrivalMatrixTooLarge` if it would exceed
  the budget `TSI_ARRIVAL_BYTES_BUDGET`. The sweep sets that to each worker's RAM share; **unset or
  `0` is not "unlimited"** — it resolves to `DEFAULT_BUDGET_FRAC` (0.9) of physical RAM, so a bare
  `run_trajectory`, `tsi-verify`, the probe, or a `--mem-frac 0` run all keep an absolute
  per-process ceiling. So a mis-estimated block explosion (or a huge `N`) fails with a clear
  message instead of freezing the machine.
- **Cost:** dominated by the per-node fork choice (batched per slot) and the arrival-matrix
  fill; the sparse lottery is negligible. Across-config joblib **loky** parallelism reused.
- **Measurement optimisation (`measure.py`):** the per-node canonical/density/agreement pass
  was ~95% of an epoch. It is now **deduped by tip** (nodes sharing a tip share every derived
  quantity — high agreement collapses `N` to a handful of computations) and the per-tip chain
  walk runs as a cached **numba** kernel (pure-Python fallback if numba is absent). Exact —
  bit-identical to the naive loop (`test_measure`). Measured **~9× end-to-end** (heavy config
  11.3 s → 1.2 s) and ~14× on measurement-bound configs. numba comes via the `accel` extra
  (`pip install -e ".[dev,accel]"`, done by `make install`).
- **Windowed fork choice (`windowed_fork_choice`, default on):** bounds the block-tree build's
  per-slot candidate scan to a horizon of the max path latency plus the fully-propagated best
  tip, turning `O(n_blocks^2)` fork choice into `O(n_blocks*H)`. **Exact** when link latency is
  deterministic (`jitter_mean == 0`) — bit-identical to a full scan (parity test). With
  `jitter_mean > 0` it becomes a tiny approximation and **warns**; a safety clamp still keeps
  the tree valid, and `windowed_fork_choice=False` forces a guaranteed-exact full scan.
- **Reproducibility:** every draw spawns off `SeedSequence(hash(config))` — child 0 stake,
  1 graph, 2 init, 3+e epoch `e`. `graph_seed`/`degree`/`link_latency_*` are part of the
  config identity.
  - **numpy-version caveat:** the `accel` extra (numba) requires `numpy<2.5`, so installing it
    pins numpy to 2.4.x. numpy's `Generator.choice(replace=False)` is **not** stream-stable
    across the 2.4↔2.5 boundary, and the sparse lottery uses it heavily only in the degenerate
    *collapsed-estimate* regime (`D_est → 0` ⇒ win-prob → 1 ⇒ `count ≈ n_slots`). So a run on
    numpy 2.5 and a run on numpy 2.4 give **identical results for all normal configs** but can
    diverge chaotically in that one extreme regime (e.g. `degree=4, link_latency=8`, where the
    estimate has already collapsed to ~0.13 — off the safe chart). The differences are tiny
    (max |Δ mean_ratio| ≈ 3e-3) and change no conclusion; pin numpy if bit-reproducibility
    across environments is required.

## Layout

```
src/tsi_sim/   constants config rng stake lottery topology blocktree(+build_tree_pernode)
               uncles(+select_uncles_at_production) tsi(+update_D_vec) epoch engine metrics
               theory verify  plotting/{style, figures_pernode, make_figures}
configs/       smoke.yaml  default.yaml  fullscale.yaml
tests/         test_{pernode,config,rng,lottery,blocktree,uncles,tsi_counting,stake,
                     theory,latency,theory_convergence}.py
```