Merge pull request #1 from logos-blockchain/tsi-countable-uncle-model
TSI: consolidate the report, adopt the countable uncle model, and revise the selfish-mining repair
11
.gitignore
vendored
@ -216,3 +216,14 @@ __marimo__/
|
||||
|
||||
# Streamlit
|
||||
.streamlit/secrets.toml
|
||||
|
||||
# Editor / local agent state
|
||||
.obsidian/
|
||||
.claude/settings.json
|
||||
.claude/settings.local.json
|
||||
|
||||
# External upstream clones vendored for benchmarking (own .git, not our history)
|
||||
tools/benchmarks/original/
|
||||
|
||||
# Rendered report HTML — a build artifact of scripts/build_html.py; markdown is the source
|
||||
reports/**/*.html
|
||||
|
||||
9
CLAUDE.md
Normal file
@ -0,0 +1,9 @@
|
||||
## graphify
|
||||
|
||||
This project has a knowledge graph at graphify-out/ with god nodes, community structure, and cross-file relationships.
|
||||
|
||||
Rules:
|
||||
- For codebase questions, first run `graphify query "<question>"` when graphify-out/graph.json exists. Use `graphify path "<A>" "<B>"` for relationships and `graphify explain "<concept>"` for focused concepts. These return a scoped subgraph, usually much smaller than GRAPH_REPORT.md or raw grep output.
|
||||
- If graphify-out/wiki/index.md exists, use it for broad navigation instead of raw source browsing.
|
||||
- Read graphify-out/GRAPH_REPORT.md only for broad architecture review or when query/path/explain do not surface enough context.
|
||||
- After modifying code, run `graphify update .` to keep the graph current (AST-only, no API cost).
|
||||
|
Before Width: | Height: | Size: 246 KiB After Width: | Height: | Size: 236 KiB |
|
Before Width: | Height: | Size: 274 KiB After Width: | Height: | Size: 282 KiB |
BIN
reports/tsi/report-figures/fig30_countable_vs_old.png
Normal file
|
After Width: | Height: | Size: 178 KiB |
BIN
reports/tsi/report-figures/fig31_countable_prediction.png
Normal file
|
After Width: | Height: | Size: 105 KiB |
BIN
reports/tsi/report-figures/fig32_countable_recovery.png
Normal file
|
After Width: | Height: | Size: 198 KiB |
BIN
reports/tsi/report-figures/fig33_absorption_window.png
Normal file
|
After Width: | Height: | Size: 134 KiB |
BIN
reports/tsi/report-figures/fig34_fine_delay_accuracy.png
Normal file
|
After Width: | Height: | Size: 152 KiB |
BIN
reports/tsi/report-figures/fig35_fine_delay_gap.png
Normal file
|
After Width: | Height: | Size: 169 KiB |
BIN
reports/tsi/report-figures/fig36_countable_selfish.png
Normal file
|
After Width: | Height: | Size: 254 KiB |
BIN
reports/tsi/report-figures/fig37_deflation_frontier.png
Normal file
|
After Width: | Height: | Size: 231 KiB |
|
Before Width: | Height: | Size: 242 KiB After Width: | Height: | Size: 243 KiB |
@ -1,184 +0,0 @@
|
||||
# Total-Stake-Inference parameter selection — Overview and recommendations
|
||||
|
||||
*Per-node network simulation of Cryptarchia Total Stake Inference (TSI). Simulator: `tsi-sim-pernode`. All runs at the true security parameter **k = 2160** unless noted; latency is in slots and **1 slot = 1 s**.*
|
||||
|
||||
*[Part 1 — Overview & recommendations](tsi-report-1-overview-and-recommendations.md) · [Part 2 — Accuracy & design](tsi-report-2-accuracy-and-design.md) · [Part 3 — Robustness & incentives](tsi-report-3-robustness-and-incentives.md) · [Part 4 — Reproducibility & appendices](tsi-report-4-reproducibility-and-appendices.md) · [Index](README.md)*
|
||||
|
||||
*Sections live across the set: [§1](#s1)/[§7](#s7)/[§8](#s8) in Part 1, [§2](tsi-report-2-accuracy-and-design.md#s2)–[§5](tsi-report-2-accuracy-and-design.md#s5) in Part 2, [§6](tsi-report-3-robustness-and-incentives.md#s6) in Part 3, [§9](tsi-report-4-reproducibility-and-appendices.md#s9) and Appendices A–C in Part 4.*
|
||||
|
||||
---
|
||||
|
||||
<a id="s1"></a>
|
||||
## 1. Executive summary
|
||||
|
||||
**The problem.** A Cryptarchia node wins the right to produce a block through a stake-weighted lottery, and the lottery's difficulty must be tuned to the total stake that is actively online — a number nobody can read directly, because participants join, leave, and fail without notice. Total Stake Inference (TSI) is the protocol's answer: every node keeps its own estimate `D̂` of the active stake and continuously corrects it against the one thing it can observe — how often blocks actually appear on chain.
|
||||
|
||||
**Why it matters.** If nodes disagree on the estimate, they disagree on who is allowed to produce blocks, and the chain can split. If the estimate is too low, the lottery becomes too easy — blocks come too fast, collide, and fork. And an attacker who could drag the estimate down would make the lottery cheaper for itself. So TSI must deliver three things at once: all nodes agree, the value is right, and nobody profits from bending it. This report tests all three by simulating every node individually — each with its own view of the network, its own estimate, and explicit message delays — at up to 32 000 nodes, with the network topology itself measured exactly up to one million nodes ([§3.7](tsi-report-2-accuracy-and-design.md#s3-7)).
|
||||
|
||||
**What we found**
|
||||
|
||||
1. **All nodes always agree.** In every run, at every network size and block rate tested (1 000–10 000 nodes at the production security parameter; to 32 000 in the size-scaling study, [§3.7](tsi-report-2-accuracy-and-design.md#s3-7); blocks every 10–30 s), all nodes computed *exactly* the same estimate — even though they constantly disagree about the newest blocks. The reason is structural: TSI measures a window buried deep in the finalized past, where every honest node sees identical history ([§3.1](tsi-report-2-accuracy-and-design.md#s3-1)). Two caveats keep this honest: agreement is not accuracy — nodes can unanimously agree on a wrong value (see next point) — and agreement is inherited from the common genesis value, not rebuilt: TSI maintains agreement but cannot repair a divergence that somehow arose ([Appendix C](tsi-report-4-reproducibility-and-appendices.md#sC)).
|
||||
2. **Network delay makes the estimate too low; uncle references fix it.** When blocks race, the losers ("orphans") drop off the chain and out of the count, so the network under-counts its own stake — by roughly a quarter to a third under Blend (the mix-network transport) delays, deepening with network size. Letting each block also reference recent orphans ("uncles") puts them back into the count — counting *slots*, so a slot with two winners still counts once — and a single uncle reference per block restores the estimate to **exactly the true stake** (the recommended cap adds a one-uncle margin, [§8](#s8)) (the only residual is an optional ~1 % from on-chain rounding of the target rate, [§2.2](tsi-report-2-accuracy-and-design.md#s2-2)/[Appendix A](tsi-report-4-reproducibility-and-appendices.md#sA)) ([§3.2](tsi-report-2-accuracy-and-design.md#s3-2)).
|
||||
3. **Two sizing rules cover the deployments tested.** How many uncle slots a block needs (`U`) is dictated by the **load** — the number of blocks the whole network produces during the time one block needs to reach everyone (written `ρ`; formally `ρ = f·D_vis`, [§3.3](tsi-report-2-accuracy-and-design.md#s3-3)). The load counts the concurrent blocks each new block must be able to reference, and one uncle slot drains one of them per block — so `U` must be at least the load rounded up, plus one spare. How far back a block may reach for an orphan (`W`) is dictated by block spacing, not delay: about ten block intervals. The formal equations, their calibration, and worked examples are in [§4](tsi-report-2-accuracy-and-design.md#s4).
|
||||
4. **Cheating doesn't pay.** Hiding blocks to deflate the estimate costs the attacker more than anyone else, heals within one epoch, and — for a full withdrawal — the lower estimate is simply the *correct* count of who is still participating ([§6.4](tsi-report-3-robustness-and-incentives.md#s6-4)–[§6.5](tsi-report-3-robustness-and-incentives.md#s6-5)). The one genuinely profitable attack is classic selfish mining, which needs roughly one-third of all stake and afflicts every Nakamoto-style chain, not TSI specifically; the damage it does to the estimate is repaired by the same uncle mechanism ([§6.6](tsi-report-3-robustness-and-incentives.md#s6-6)). Rewarding uncles compensates the honest victims of delay and makes hiding strictly worse — but the reward must remain a bonus (a *soft* rule, never a validity requirement, which an attacker could turn into a tool for censoring or forking the chain), and the uncle + nephew rewards (the nephew reward pays the block that *includes* the reference) must together stay below one block reward, or deliberately orphaning one's own blocks becomes a business ([§6.7](tsi-report-3-robustness-and-incentives.md#s6-7)–[§6.8](tsi-report-3-robustness-and-incentives.md#s6-8)).
|
||||
5. **Reorganisations stay shallow if the load stays low.** Forks are the cost centre: a deep reorg discards confirmed blocks. Fork depth grows with delay, and uncles keep it shallow (without them the deflated estimate over-produces blocks, and honest reorgs reach ~17 deep; with them, ~4). Against a private-chain attacker holding 10–30 % of stake trying to force the deepest reorg, keeping the load below one (few honest forks) keeps reorgs bounded and shallow (single digits to low tens of blocks even at 30 % stake) — but past that, at 30 % stake and heavy delay, fork-induced orphaning tips the attacker over an effective majority and reorgs become unbounded. The same "keep `ρ < 1`" rule that keeps the estimate accurate keeps reorgs shallow ([§6.10](tsi-report-3-robustness-and-incentives.md#s6-10)).
|
||||
|
||||
*Method note: results come from the per-node simulator at the production security parameter k = 2160, with the robustness and size-scaling studies at a reduced k = 256 where noted; the sizing rules are semi-empirical (their form follows from the fork/orphan mechanism, their constants are fitted to the sweeps) and are derived under honest, deterministic-delay conditions — noise, attacks, and incentives are treated in [§6](tsi-report-3-robustness-and-incentives.md#s6).*
|
||||
|
||||
**Recommended configuration** (Cryptarchia baseline f = 1/30, i.e. 30-second blocks; full rationale and residual risks in [§8](#s8)):
|
||||
|
||||
| parameter | value | basis |
|
||||
|---|---|---|
|
||||
| security `k` | 2160 | [§3.1](tsi-report-2-accuracy-and-design.md#s3-1) |
|
||||
| block rate `f` | 1/30 (30 s) | protocol; [§3.6](tsi-report-2-accuracy-and-design.md#s3-6) |
|
||||
| uncle window `W` | **300 slots** (= 10 block intervals) | [§3.4](tsi-report-2-accuracy-and-design.md#s3-4), [§4](tsi-report-2-accuracy-and-design.md#s4) |
|
||||
| uncle cap `U` | the **load rounded up, plus one** (`⌈ρ⌉ + 1`) under Blend; **1** suffices for plain direct gossip | [§3.3](tsi-report-2-accuracy-and-design.md#s3-3), [§4](tsi-report-2-accuracy-and-design.md#s4) |
|
||||
| TSI learning rate `β` | **1** (one-epoch tracking) | [§6.5](tsi-report-3-robustness-and-incentives.md#s6-5) |
|
||||
| operating point | keep the **load below one** block per propagation delay, with margin | [§6.2](tsi-report-3-robustness-and-incentives.md#s6-2), [§6.3](tsi-report-3-robustness-and-incentives.md#s6-3) |
|
||||
| peering degree | **4 is fine for smaller networks (up to N ≈ 2×10⁵); ≥ 6 beyond** — degree 4 exhausts U = 1 near N ≈ 8×10⁵ | [§3.7](tsi-report-2-accuracy-and-design.md#s3-7), [§8](#s8) |
|
||||
| uncle rewards | **soft** inclusion (never validity); substantial `w_u` with **`w_u + w_n < 1`** | [§6.7](tsi-report-3-robustness-and-incentives.md#s6-7)–[§6.8](tsi-report-3-robustness-and-incentives.md#s6-8) |
|
||||
|
||||
*"Load" (`ρ`) in this table is the quantity defined in finding 3 above: blocks produced network-wide per block-propagation delay, `ρ = f·D_vis` ([§3.3](tsi-report-2-accuracy-and-design.md#s3-3)). Keeping it below one means a single uncle slot per block can absorb every concurrent block; the `+1` in the uncle cap is the safety margin.*
|
||||
|
||||
This report is split into four parts (see the [index](README.md)): **Part 1** — the recommended configuration, the per-knob parameter reference ([§7](#s7)) and the safest selection with residual risks and spec deltas ([§8](#s8)); **Part 2** — the model and counting rule ([§2](tsi-report-2-accuracy-and-design.md#s2)), the evidence behind each finding ([§3](tsi-report-2-accuracy-and-design.md#s3)), and the design equations and selection algorithm ([§4](tsi-report-2-accuracy-and-design.md#s4)–[§5](tsi-report-2-accuracy-and-design.md#s5)); **Part 3** — robustness against noise, attacks and the incentive design ([§6](tsi-report-3-robustness-and-incentives.md#s6)); **Part 4** — reproducibility ([§9](tsi-report-4-reproducibility-and-appendices.md#s9)) and the appendices (the residual ~1 % f-rounding offset, the ±0.9 % per-epoch noise floor, and consensus detail).
|
||||
|
||||
---
|
||||
|
||||
|
||||
<a id="s7"></a>
|
||||
## 7. Parameter reference — what each knob does
|
||||
|
||||
Every parameter a protocol designer or operator can set, in three groups: **protocol parameters** (design choices this report selects), **network/environment parameters** (given by the deployment; their impact must be measured and fed into the [§4](tsi-report-2-accuracy-and-design.md#s4) algorithm), and **model/attack knobs** (used to probe robustness). Each impact statement cites the section and figure that measured it; [§8](#s8) turns this reference into the recommended selection.
|
||||
|
||||
<a id="s7-1"></a>
|
||||
### 7.1 Protocol parameters
|
||||
|
||||
| parameter | controls | measured impact | where |
|
||||
|---|---|---|---|
|
||||
| `f` — block rate (1/30) | block interval `1/f`; scales every threshold | recovery stays at 1.0 at every rate (slot-counting removes the `c(f)` dependence); `U_min ∝ f` (an 8-s delay needs U = 1 at 30/20-s blocks but U = 2 at 15/10-s); `W_min ∝ 1/f` (constant ≈ 7 block intervals); consensus-safety f-independent | [§3.6](tsi-report-2-accuracy-and-design.md#s3-6), fig6 |
|
||||
| `k` — security parameter (2160) | finality depth **and** the slot geometry: epoch length `10·⌊k/f⌋`, TSI window `T = 6·⌊k/f⌋` slots | the density window sits a full epoch past k-finality — the structural reason tip churn (fig18) and jitter ([§6.1](tsi-report-3-robustness-and-incentives.md#s6-1)) never reach the estimate; agreement confirmed at k = 2160; robustness studies at scaled k = 256 shrink `T` proportionally but leave the density-*ratio* mechanics unchanged | [§3.1](tsi-report-2-accuracy-and-design.md#s3-1) fig17; [§6.1](tsi-report-3-robustness-and-incentives.md#s6-1) |
|
||||
| `U` — uncle cap (`max_uncles`) | orphan-recovery capacity per block | U = 0 biased low (0.64–0.74, worsens with N); `U ≥ ⌈ρ⌉` hard bound — below it *no* window recovers; `⌈ρ⌉ + 1` recommended; also the adversarial safeguard ([§6.3](tsi-report-3-robustness-and-incentives.md#s6-3)) and the selfish-deflation repair ([§6.6](tsi-report-3-robustness-and-incentives.md#s6-6)) | [§3.2](tsi-report-2-accuracy-and-design.md#s3-2)/3.3/3.5, figs 2, 16, 20 |
|
||||
| `W` — uncle window (`uncle_window` = 300) | how old an orphan may be and still be referenced | hard floor `≈ 7/f`, set by block *spacing* (FIFO queueing), not delay; no U buys below it; a generous W also keeps the emergent reference rate `p_ref` high, which the reward design relies on | [§3.4](tsi-report-2-accuracy-and-design.md#s3-4)/3.5, figs 4, 5, 22; [§6.8](tsi-report-3-robustness-and-incentives.md#s6-8) fig15 |
|
||||
| `β` — TSI learning rate (`BETA_DEFAULT` = 1) | estimator responsiveness: `D̂` is an EMA of active stake, `τ = −1/ln(1−β)` epochs | β = 1 tracks with a one-epoch lag (erases a withhold notch in one epoch); slower β shrinks the notch amplitude (0.35 → 0.15 for β 1 → 0.25) but leaves attacker profitability unchanged | [§6.5](tsi-report-3-robustness-and-incentives.md#s6-5)(i)–(iii), figs 10–12 |
|
||||
| `uncle_strategy` (oldest) | which queued orphans a block references | all results use oldest-first (FIFO) — the drain model behind the `W ≈ 7/f` floor; the spec's unbiased coin (`uncle_random_p = 0.5`) was not swept (untested sensitivity, covered by the `W = 10/f` margin) | [§3.4](tsi-report-2-accuracy-and-design.md#s3-4) |
|
||||
| `w_u`, `w_n` — uncle / nephew rewards | compensation for referenced orphans / for referencing them | fairness: honest recovery 0.61 → 1.00 as `w_u` 0 → 1 (at `p_ref` = 1); hiding forfeits both rewards; safety invariant **`w_u + w_n < 1`** (else uncle farming pays); raising `w_n` alone marginally *helps* the attacker | [§6.7](tsi-report-3-robustness-and-incentives.md#s6-7)/6.8, figs 14, 15 |
|
||||
| inclusion rule | whether references are incentivised or mandated | **soft** (reward-weighted) only — a hard validity mandate cannot be encoded fork-safely (censorship or consensus fork) | [§6.8](tsi-report-3-robustness-and-incentives.md#s6-8) |
|
||||
| `genesis_d_factor`, `init_dest` | the initial `D̂` | a *common* wrong guess self-corrects in ~2 epochs (fig1); a *heterogeneous* init is preserved forever (fig21) — genesis `D̂` must be a shared constant | [§3.2](tsi-report-2-accuracy-and-design.md#s3-2), [Appendix C](tsi-report-4-reproducibility-and-appendices.md#sC) |
|
||||
| `fixed_point` | on-chain integer-f truncation `f_p = ⌊f·1000⌋/1000` | the **only** residual bias after slot-counting: ≈ +1 % (`f/f_p`), removed by higher-precision `f` ([Appendix A](tsi-report-4-reproducibility-and-appendices.md#sA)) | [§2.2](tsi-report-2-accuracy-and-design.md#s2-2), [§5](tsi-report-2-accuracy-and-design.md#s5) |
|
||||
|
||||
<a id="s7-2"></a>
|
||||
### 7.2 Network / environment parameters
|
||||
|
||||
| parameter | controls | measured impact | where |
|
||||
|---|---|---|---|
|
||||
| `n_nodes` N | network size | consensus exact at every N (1 000–32 000 direct, fig17/fig23); U = 0 under-count deepens with N (0.739 → 0.635); `U`/`W` *thresholds* ≈ N-invariant at fixed delay — but at a fixed blending budget N raises the load through `ℓ_mean ∝ ln N`, exhausting U = 1 on a degree-4 graph near N ≈ 8×10⁵ ([§3.7](tsi-report-2-accuracy-and-design.md#s3-7)) | [§3.1](tsi-report-2-accuracy-and-design.md#s3-1)/3.2/3.4/3.7, figs 2, 17, 20, 23, 24 |
|
||||
| `topology` | propagation model | `blend` (**primary**): whole-second hops — the regime where `U`/`W` matter; `regular` (direct gossip, contrast): sub-slot links, `ρ ≈ 0.04`, U = 0 near-exact; `full_mesh`: reduced-model bridge, cross-validation only | [§2](tsi-report-2-accuracy-and-design.md#s2), [§3.3](tsi-report-2-accuracy-and-design.md#s3-3), figs 3, 16, 19 |
|
||||
| `degree` | peering density | deg 6 vs 4: tip agreement 0.993–0.994 vs 0.988–0.989; U = 0 accuracy ≈ 1.00 vs ≈ 0.98 at 0.2-slot links; sets the diameter slope `ℓ_mean ≈ a_d·ln N` (`a₄`/`a₆`/`a₈` = 0.33/0.19/0.14) — one degree step (4 → 6) buys ~5 000× larger N at the same blending budget | figs 18, 19, 23, 24; [§3.7](tsi-report-2-accuracy-and-design.md#s3-7), [§4](tsi-report-2-accuracy-and-design.md#s4) |
|
||||
| `link_latency_mean`, `link_latency_dist` | per-link transport latency (mean, shape) | sub-slot links: forks negligible; near a full slot: U = 0 dips to 0.84–0.91; the design law uses only the *mean* (`D_vis`) — confirmed by the exp-vs-geo comparison at equal mean (U ≥ 1 identical; only the U = 0 depth moves, ≤ 0.07) | [§2](tsi-report-2-accuracy-and-design.md#s2), [§3.3](tsi-report-2-accuracy-and-design.md#s3-3), fig19 |
|
||||
| `blend_hops`, `blend_delay_max` | Blend-cascade length, per-hop delay bound | dominant `D_vis` term `hops·δ_max/2`; more hops at fixed budget degrade U = 1 exactly as the load predicts (0.98 → 0.48 across hops 3 → 6 at delay 16) | [§3.3](tsi-report-2-accuracy-and-design.md#s3-3), figs 3, 16 |
|
||||
| `jitter_mean`, `jitter_dist`, `jitter_frac` | stochastic per-(block,node) arrival jitter (Exp, or Poisson long-tail hitting a fraction of deliveries) | consensus and accuracy untouched up to 3 slots of Exp jitter (200/200 cells); the 10 % Poisson(3) long-tail model shifts U = 1 accuracy < 1 % per cell (~0.4 % mean, ~0.03 % pooled, within replicate SEM) and consensus not at all (378/378 cells); only tip agreement degrades (≈ 0.99 → 0.95) | [§6.1](tsi-report-3-robustness-and-incentives.md#s6-1), [§3.7](tsi-report-2-accuracy-and-design.md#s3-7) |
|
||||
| `stake_dist` (`pareto_shape`, `total_stake`) | per-node stake weights | consensus identical under uniform, Pareto 1.16, and Pareto 1.33; U ≥ 1 accuracy indistinguishable across all three; only the U = 0 under-count moves with concentration (heavier tail → shallower: 0.815 Pareto vs 0.785 uniform; 1.33 ≈ 0.04 deeper than 1.16); whale-coalition reward statistics untested | [§2](tsi-report-2-accuracy-and-design.md#s2), [§6.5](tsi-report-3-robustness-and-incentives.md#s6-5) scope |
|
||||
|
||||
<a id="s7-3"></a>
|
||||
### 7.3 Model and attack knobs (robustness probes)
|
||||
|
||||
| knob | probes | result | where |
|
||||
|---|---|---|---|
|
||||
| `adversary_frac` (β_adv), `adversary_strategy` | uncle suppression / abstention withholding | suppression weak at ρ < 1, a cheap lever only at ρ > 1 (1.43× at ρ = 1.36, β_adv = 0.5); withholding → `D̂ ≈ (1−β_adv)` — the *correct* active-stake answer, topology-independent | [§6.3](tsi-report-3-robustness-and-incentives.md#s6-3)/6.4, figs 8, 9 |
|
||||
| `adversary_period`, `adversary_withhold_epochs` | dynamic withhold-rejoin schedules | unprofitable at every duty ψ > 0; griefing bounded (`deflation = β_adv·ψ`); no persistence | [§6.5](tsi-report-3-robustness-and-incentives.md#s6-5), figs 10–12 |
|
||||
| private-chain reorg (α = 0/10/20/30 %) | deepest-reorg longest-chain attack | reorg-depth tail `(α_eff/(1−α_eff))^d` with `α_eff=α/(α+(1−α)(1−o))`; shallow at ρ<1, unbounded at 30 % once forks push `α_eff>½` | [§6.10](tsi-report-3-robustness-and-incentives.md#s6-10), figs 27, 28 |
|
||||
| `fork_rate`, `max_reorg_depth` (engine outputs) | measured fork structure | honest fork rate 0.2→0.6 over δ=2→32 s; max reorg depth 2→4 (U=2), up to 17 at U=0 | [§6.10](tsi-report-3-robustness-and-incentives.md#s6-10), fig28 |
|
||||
| `α`, `γ` (selfish race) | private-chain withholding | profitable above `(1−γ)/(3−2γ)` (⅓ at γ = 0); the optimal MDP barely beats SM1 at γ = 0 | [§6.6](tsi-report-3-robustness-and-incentives.md#s6-6), figs 13, 14 |
|
||||
| `η` — uncle-recovery fraction | estimator repair under selfish orphaning | `D̂` 0.70 → 0.82 → 0.94 as η 0 → 0.5 → 1 (α = 0.4) | [§6.6](tsi-report-3-robustness-and-incentives.md#s6-6), fig13 |
|
||||
| `p_ref`, `p_ref_adv`, `adv_nephew` (`RewardParams`) | the strategic reference game under rewards | selfish share falls below block-only once `p_ref ≳ 0.3` (α ≥ 0.4); attacker self-uncling lowers its own share (0.406 → 0.394) | [§6.7](tsi-report-3-robustness-and-incentives.md#s6-7)/6.8, fig15 |
|
||||
| `windowed_fork_choice`, `prune_arrival` | engine exactness | bit-exact at jitter = 0; [§6.1](tsi-report-3-robustness-and-incentives.md#s6-1) re-ran the exact oracle (both off) under jitter | [§6.1](tsi-report-3-robustness-and-incentives.md#s6-1) |
|
||||
|
||||
---
|
||||
|
||||
|
||||
<a id="s8"></a>
|
||||
## 8. Safest parameter selection and design
|
||||
|
||||
Everything below is assembled from this report's own measurements; each row cites the section and figure that justifies it. This is the authoritative selection — [§1](#s1)'s table is its preview.
|
||||
|
||||
<a id="s8-1"></a>
|
||||
### 8.1 The selection
|
||||
|
||||
| # | parameter / decision | safest choice | why (this report's finding) | evidence |
|
||||
|---|---|---|---|---|
|
||||
| 1 | security `k` | **2160** (finality-driven) | per-node spread ≡ 0 and window agreement ≡ 1.000 at k = 2160 for all N = 1 000–10 000 and f = 1/10–1/30; the mechanism (window buried past k-finality) is k-independent; k = 2160 is tested at full scale, and the [§3.7](tsi-report-2-accuracy-and-design.md#s3-7) ladder confirms the same exact agreement at the scaled k = 256 up to N = 32 000 | [§3.1](tsi-report-2-accuracy-and-design.md#s3-1), figs 17, 18 |
|
||||
| 2 | block rate `f` | **1/30** (given) — treat as a coupled input | `f` moves every threshold: `U_min ∝ f`, `W_min ∝ 1/f` (recovery stays at 1.0 at all rates); re-derive `U`, `W`, `ρ` if `f` ever changes | [§3.6](tsi-report-2-accuracy-and-design.md#s3-6), fig6 |
|
||||
| 3 | uncle window `W` | **300 slots = 10/f**; widen to **450–600 (15–20/f)** when operating near the load boundary `ρ ≈ 1` | hard floor ≈ 7/f set by block spacing (N-invariant, fig25); below it no uncle count recovers (delay 16, W = 100: even U = 4 only 0.94); `W` is the buffer for block-production *fluctuations* — at `ρ ≈ 1`, W = 600 recovers what W = 300 loses (fig25), a cheap alternative to the `+1` uncle; the 10/f margin also covers the untested spec coin-rule sensitivity ([§3.4](tsi-report-2-accuracy-and-design.md#s3-4)) and keeps the emergent reference rate `p_ref` ([§6.7](tsi-report-3-robustness-and-incentives.md#s6-7)) high for the reward design | [§3.4](tsi-report-2-accuracy-and-design.md#s3-4)/3.5, figs 4, 5, 22, 25; [§6.8](tsi-report-3-robustness-and-incentives.md#s6-8) |
|
||||
| 4 | uncle cap `U` | **`⌈ρ⌉ + 1`** (Blend, the deployment target); **1** suffices for plain direct gossip (ρ ≈ 0.04) | `U < ρ` never recovers; `⌈ρ⌉` is tight but under-shoots just below integer ρ — at ρ = 0.96, U = 1 fails at 0.962 ± 0.006 (3σ below the 0.98 bar): the direct evidence for the +1 margin; one uncle lifts every tested delay to full recovery (≈ 1.0) at N = 10 000 | [§3.3](tsi-report-2-accuracy-and-design.md#s3-3)/3.5/3.6, figs 6, 16, 20 |
|
||||
| 5 | TSI learning rate `β` | **1** (`BETA_DEFAULT`) | one-epoch tracking of active stake: any withhold notch is erased in one epoch; attacker profitability is β-independent, so slowing β buys the defender nothing. *Disclosure:* the one regime where distortion per unit of network reward forfeited exceeds 1 (bounded ≤ 2×) is β ≥ 1 with a ≈ 50 % coalition (rejoin overshoot) — a protocol-chosen gain, not an adversary lever | [§6.5](tsi-report-3-robustness-and-incentives.md#s6-5), figs 10–12 |
|
||||
| 6 | operating point | **`ρ = f·D_vis < 1` with margin** | the fitted static feedback map folds near ρ ≈ 1 (provisioning warning — the full dynamics never reach it); grinding is weak at ρ < 1 (1.04×) but cheap at ρ > 1 (1.43×); chronic ρ > 1 under-count persists after any attack ends. ρ < 1 is simultaneously the honest-accuracy and the adversarial safeguard | [§6.2](tsi-report-3-robustness-and-incentives.md#s6-2) fig7; [§6.3](tsi-report-3-robustness-and-incentives.md#s6-3) fig8; [§6.5](tsi-report-3-robustness-and-incentives.md#s6-5)(iv) fig12 |
|
||||
| 7 | genesis / initialization | **genesis `D̂` is a single protocol constant, identical at every node** — never client-configurable | consensus is an *initialization* property, not a restoring force: an injected per-node spread is conserved forever (max/min ≈ 3.0, flat), while a *common* wrong guess self-corrects in ~2 epochs | [Appendix C](tsi-report-4-reproducibility-and-appendices.md#sC) (fig21); [§3.2](tsi-report-2-accuracy-and-design.md#s3-2) fig1 |
|
||||
| 8 | uncle validity precondition | **a counted uncle must be a real VRF lottery winner for its slot, enforced on-chain** | fabrication is VRF-impossible; this precondition underlies every [§6.3](tsi-report-3-robustness-and-incentives.md#s6-3)–[§6.5](tsi-report-3-robustness-and-incentives.md#s6-5) bound and caps uncle issuance at `≤ U·(w_u + w_n)` per block | [§6.3](tsi-report-3-robustness-and-incentives.md#s6-3), [§6.7](tsi-report-3-robustness-and-incentives.md#s6-7)(a) |
|
||||
| 9 | uncles in the density | **recovered uncles enter the TSI counted density** (independent of rewarding them) | restores `D̂` under honest latency (0.64–0.74 → 1.0) *and* under selfish orphaning (0.70 → 0.94 at η = 1) — a safety win with no reward-schedule dependence | [§3.2](tsi-report-2-accuracy-and-design.md#s3-2) fig2; [§6.6](tsi-report-3-robustness-and-incentives.md#s6-6) fig13 |
|
||||
| 10 | inclusion rule | **soft (reward-weighted) — never a validity rule** | a hard mandate cannot be encoded fork-safely: the objective encoding is a censorship weapon, the subjective one forks consensus with no adversary; under the soft rule `p_ref` is emergent and high — at `p_ref ≈ 0.85` the selfish share (0.420 at α = 0.4) is already below block-only (0.484) and near stake | [§6.8](tsi-report-3-robustness-and-incentives.md#s6-8) fig15 |
|
||||
| 11 | reward schedule | **substantial `w_u` (Ethereum's GHOST-style uncle rewards are the precedent); `w_u + w_n < 1` strictly enforced** | farming a real VRF win into a self-uncle + self-nephew pays `w_u + w_n` against an honest block's 1 — the binding invariant is the *sum* (0.875 + 0.03125 = 0.906 is safe; 0.875 + 0.15 = 1.025 is not); `w_u` compensates honest orphans (recovery → 1.0) and makes hiding forfeit both rewards | [§6.7](tsi-report-3-robustness-and-incentives.md#s6-7)(a), figs 14, 15 |
|
||||
| 12 | peering degree | **degree 4 is acceptable for smaller networks — up to N ≈ 2×10⁵ it keeps the load under 0.9 at the 8-s budget; use ≥ 6 beyond, and scale with the target size** | degree sets the gossip-diameter slope `ℓ_mean ≈ a_d·ln N`, and at a fixed blending budget that slope decides when `U = 1` runs out: at δ = 8 s, degree 4 exhausts U = 1 near **N* ≈ 8×10⁵** (its ladder curve already touches the recovery bar at N = 32 000), degree 6 holds to N ~ 4×10⁹, degree 8 to ≈ 6×10¹² — one degree step (4 → 6) buys ~5 000× more network at the same budget. Degree and the uncle cap are exchangeable defences against growth: densify the graph or add an uncle slot, whichever is cheaper. Plus the small-N benefits: tip agreement 0.993–0.994 vs 0.988–0.989, better U = 0 recovery | [§3.7](tsi-report-2-accuracy-and-design.md#s3-7), figs 23, 24; figs 18, 19 |
|
||||
| 13 | Blend delay budget | **choose `hops·δ_max` so `ρ ≲ 1`**; if `U ≥ ⌈ρ⌉` exceeds the uncle-slot budget, cut blending — no window fixes `U < ρ` | worked examples: 3 hops × 8 s at f = 1/30 → ρ ≈ 0.56, U = 2 recommended; the same Blend at f = 1/10 → ρ ≈ 1.7, U = 3 | [§4](tsi-report-2-accuracy-and-design.md#s4); [§6.3](tsi-report-3-robustness-and-incentives.md#s6-3)(iii) |
|
||||
| 14 | on-chain `f` precision | **carry the target rate `f` at 10⁻⁶ precision** (was 10⁻³) | removes the sole residual bias after slot-counting — the ≈ 1 % `f/f_p` over-estimate (a ~1 % win-probability under-delivery and slow canonical pace); one-constant change, no dynamics cost, no fork risk. **Exceeded** in this report's estimator, which uses exact `f` (residual 0), finer than the recommended 10⁻⁶ bump (`f_p = 0.033333`, residual < 10⁻⁵) | [§2.2](tsi-report-2-accuracy-and-design.md#s2-2), [Appendix A](tsi-report-4-reproducibility-and-appendices.md#sA) |
|
||||
| 15 | reorg / confirmation depth | keep **`ρ < 1`** so fork-induced amplification stays below the effective-majority cliff; for soft settlement, wait `d` blocks per the [§6.10](tsi-report-3-robustness-and-incentives.md#s6-10) tail (≈ 4/8/17 blocks for 99.9 % against 10/20/30 % stake) | reorg depth is bounded and shallow only while the honest fork rate is low; at 30 % stake past `ρ ≈ 1` it is unbounded ([§6.10](tsi-report-3-robustness-and-incentives.md#s6-10), figs 27–28) | [§6.10](tsi-report-3-robustness-and-incentives.md#s6-10) |
|
||||
|
||||
<a id="s8-2"></a>
|
||||
### 8.2 Design decisions and their rationale
|
||||
|
||||
**The levers are hierarchical, not fungible.** Size `W` first (10/f — the floor is set by block spacing and no uncle count buys below it), then `U` from the load `⌈ρ⌉ + 1`, then check the operating point `ρ < 1` (fig5, fig22: at delay 32, ρ ≈ 1.7, even the widest window recovers only 0.61 at U = 1).
|
||||
|
||||
**One mechanism, three duties.** The same uncle references fix the honest latency under-count ([§3.2](tsi-report-2-accuracy-and-design.md#s3-2)), repair the estimator under uncle-suppression and selfish orphaning ([§6.3](tsi-report-3-robustness-and-incentives.md#s6-3), [§6.6](tsi-report-3-robustness-and-incentives.md#s6-6)), and — rewarded softly — pay the fairness and anti-hiding incentives ([§6.7](tsi-report-3-robustness-and-incentives.md#s6-7)). This is why the design invests in `W`/`U` margin rather than in any inclusion mandate: every marginal slot of window or uncle capacity strengthens all three duties at once.
|
||||
|
||||
**Why soft, not hard, inclusion.** "Owed" cannot be proven — no node can prove which forks a producer saw. An objective slot-gap encoding lets an attacker withhold-then-late-release an uncle to retroactively invalidate honest blocks (censorship); a visibility encoding is subjective and forks consensus with no adversary at all ([§6.8](tsi-report-3-robustness-and-incentives.md#s6-8)). The soft rule concedes only a small residual (quantified in [§8.3](#s8-3) item 3): a bounded near-threshold backfire that vanishes as the emergent `p_ref` rises, and a per-α premium that stays ≤ +0.014 even at full inclusion.
|
||||
|
||||
**What TSI defends, and what it measures.** Abstention-style withholding is strictly unprofitable (reward/stake 0.94 → 0.30 for duty 0.10 → 0.75 at β_adv = 0.3), bounded as griefing (deflation = β_adv·ψ), non-persistent, and does not stack across coalitions ([§6.5](tsi-report-3-robustness-and-incentives.md#s6-5), [§6.9](tsi-report-3-robustness-and-incentives.md#s6-9)) — and a full withdrawal is *correctly measured*, not mis-measured: `D̂ → (1−β_adv)` is the right answer for the participating stake ([§6.4](tsi-report-3-robustness-and-incentives.md#s6-4)). The one profitable lever is classic selfish mining above `(1−γ)/(3−2γ)` — a general Nakamoto-PoS result, not TSI-specific; TSI's added exposure (estimator deflation to 0.70 at α = 0.4) is repaired by uncle-counting (row 9), and the profit premium is blunted, not eliminated, by the soft-rewarded references (row 10). The adversary frontier is exactly optimal selfish mining; no compounding lever remains ([§6.7](tsi-report-3-robustness-and-incentives.md#s6-7)).
|
||||
|
||||
<a id="s8-3"></a>
|
||||
### 8.3 Residual risks and open items
|
||||
|
||||
1. **The ρ ≈ 1 fold ([§6.2](tsi-report-3-robustness-and-incentives.md#s6-2)).** The fitted static map is bistable at the recipe's own boundary; the full dynamics never reach it, but "U = ⌈ρ⌉ suffices" is an upper edge, not a safe interior — row 6 is a hard provisioning requirement, not a preference.
|
||||
2. **Multi-coalition selfish mining is open ([§6.9](tsi-report-3-robustness-and-incentives.md#s6-9)).** The literature threshold can fall below ⅓ and total orphaning can exceed the single-coalition value, so `D̂ ≈ 0.70` at α = 0.4 is not an upper bound. Flagged, not solved.
|
||||
3. **Near-threshold reward backfire ([§6.7](tsi-report-3-robustness-and-incentives.md#s6-7)–[§6.8](tsi-report-3-robustness-and-incentives.md#s6-8)).** At α ≈ 0.35 a small backfire (≤ +0.03 above block-only for `p_ref ≳ 0.3`, rising to +0.05 at the unreachable `p_ref = 0`) persists until `p_ref ≳ 0.75`; even `p_ref = 1` leaves a premium growing with α.
|
||||
4. **No re-convergence ([Appendix C](tsi-report-4-reproducibility-and-appendices.md#sC)).** Any mechanism that could inject persistent per-node `D̂` disagreement (genesis/clock exploit) would never be self-corrected — safety rests entirely on row 7.
|
||||
5. **Adversarial `U` margin unquantified ([§6.8](tsi-report-3-robustness-and-incentives.md#s6-8)).** Under attack-inflated orphaning (orphan/blocks ≈ 0.35 at α = 0.46) the honest-load cap may need extra margin (owed uncles beyond `U` defer and can age out of `W`); this report does not size it.
|
||||
6. **Clock skew — bounded, not a break ([§6.1](tsi-report-3-robustness-and-incentives.md#s6-1)).** A per-node whole-timeline slot-clock shift moves each node's window bounds, giving a bounded inter-node spread `≤ 2·skew/T` (`< 10⁻⁴` at the production window). Small and quantified, but — unlike per-arrival jitter — not identically zero; large skew is still an untested extreme.
|
||||
7. **Semi-empirical constants ([§5](tsi-report-2-accuracy-and-design.md#s5)).** `a = 2.01` and the 7/f floor are fits; eq 4′ is an N ≈ 1 000 fit that worsens with N — for production sizing use the margined rules (rows 3–4) and the fully-recovered accuracy 1.0, never eq 4′.
|
||||
8. **Residual ~1 % offset — resolved here, pending in the spec ([§2.2](tsi-report-2-accuracy-and-design.md#s2-2), row 14, [Appendix A](tsi-report-4-reproducibility-and-appendices.md#sA)).** After the slot-counting fix the only systematic offset is the 10⁻³ rounding of `f` (≈ 1 % `f/f_p` over-estimate); this report's estimator applies the row-14 precision bump, but the **spec still uses 10⁻³** and should adopt it.
|
||||
9. **Multi-coalition reorg — bounded, but worse than a single coalition ([§6.9](tsi-report-3-robustness-and-incentives.md#s6-9), [§6.10](tsi-report-3-robustness-and-incentives.md#s6-10)).** The [§6.10](tsi-report-3-robustness-and-incentives.md#s6-10) tail is single-coalition. For `K` competing private-chain coalitions with total stake `α_tot`, the deepest reorg any one coalition (stake `α_i`) can drive against the public chain is bounded by `α_eff,i = α_i / (α_i + (1−α_tot)(1−o′))`, where `o′` is the public fork rate *inflated by every coalition's released-block orphaning*. Two channels make competition **worse** than a single colluding `α_tot` coalition: the rival coalitions' stake leaves the public chain (the honest term carries `(1−α_tot)`, not `(1−α_i)`), and their mutual orphaning lifts `o′` above the honest baseline `o` — in the limit `o′→1` even a sub-majority coalition faces a stalled public chain and reorgs unboundedly. So the single-coalition tail is best-case-for-the-defender; the safe design response is the same rule as everywhere else — size against `α_tot`, and keep `ρ < 1` so the honest baseline `o` stays small and caps how far mutual orphaning can inflate `o′`.
|
||||
10. **Soft-rule bribery is cheap per block ([§6.9](tsi-report-3-robustness-and-incentives.md#s6-9)).** A bribe > `w_n` buys one block's omitted references — bounded in effect (other honest blocks in `W` still reference), but the hard-mandate defence is unavailable *by design*.
|
||||
11. **Untested adversary variants ([§6.5](tsi-report-3-robustness-and-incentives.md#s6-5) scope).** Pareto whale coalitions, very slow `β`, jitter > 0 for the dynamic-withhold results, and the spec's coin-flip uncle selection; the reported bounds are best-case-for-the-defender (the adaptive-schedule case is closed in [§6.7](tsi-report-3-robustness-and-incentives.md#s6-7)).
|
||||
12. **Network partition/heal — reasoned, not simulated ([§6.1](tsi-report-3-robustness-and-incentives.md#s6-1)).** TSI reads its density window a full epoch (`E` slots) past `k`-finality, so a partition that heals *within* the finality depth is invisible: all honest nodes reconverge on one canonical chain before the measurement window closes and compute an identical `D̂` — the same finality-window argument that makes jitter and clock skew harmless ([§6.1](tsi-report-3-robustness-and-incentives.md#s6-1)). A partition lasting *longer* than finality is a consensus-layer safety failure (the chain forks irreparably) that TSI inherits but does not worsen — it adds no partition vulnerability of its own. A direct partition-and-heal simulation is left as future work.
|
||||
|
||||
<a id="s8-4"></a>
|
||||
### 8.4 Capstone: the whole recipe, in one run
|
||||
|
||||
Every finding above comes from a different sweep. As a final check that the recommendation holds *together*, we ran the single recommended configuration end-to-end at the true security parameter — f = 1/30, W = 300, U = 2, β = 1, degree 6, Blend (3 hops × 8 s), k = 2160, Pareto stake, f-precision applied — honestly and under a 30 %-stake uncle-suppression adversary (`scripts/capstone.py`, N = 1 000, 8 replicates):
|
||||
|
||||
| metric | honest | 30 % suppression adversary |
|
||||
|---|---|---|
|
||||
| accuracy `D̂/D` | 1.002 | 0.998 |
|
||||
| consensus `range_ratio` / `agreement` | 0 / 1.000 | 0 / 1.000 |
|
||||
| fork rate | 0.345 | 0.336 |
|
||||
| max reorg depth | 3 | 3 |
|
||||
| emergent `p_ref` | 1.000 | 0.989 |
|
||||
|
||||
The whole recipe coheres: accurate to the true stake (the ~1 % rounding removed), unanimous, shallow-forking, and its soft-inclusion premise (`p_ref` high) confirmed — and all of it survives a 30 % adversary actively trying to deflate the estimate. This also resolves the finding-2 ([§3.2](tsi-report-2-accuracy-and-design.md#s3-2)) vs row-4 tension in the recommendation's favour: at this operating point `⌈ρ⌉ = 1`, and the recommended `U = 2` (the `+1` margin) holds every metric at target with room to spare.
|
||||
|
||||
<a id="s8-5"></a>
|
||||
### 8.5 Recommendation vs the current spec
|
||||
|
||||
The deltas this report recommends, in one place (this is a parameter-selection report meant to inform the spec — and the spec explicitly flags `w_u` and `MAX_UNCLES` as *provisional*, leaving their tuning "as an open task of [Analysis] Total Stake Inference", i.e. this report):
|
||||
|
||||
| parameter | current spec / default | recommended | why |
|
||||
|---|---|---|---|
|
||||
| uncle counting | per-block (double-counts multi-winner slots) | **per occupied slot** | lands the estimate at exactly `D`, not `c(f)·D` ([§2.1](tsi-report-2-accuracy-and-design.md#s2-1)) — the density-bug fix |
|
||||
| on-chain `f` precision | 10⁻³ (`f_p = 0.033`) | **10⁻⁶** (`0.033333`) | removes the residual ≈ 1 % `f/f_p` offset ([§2.2](tsi-report-2-accuracy-and-design.md#s2-2)) |
|
||||
| uncle window `W` | `w_u` (the spec's window symbol; this report's window is `W`, and its `w_u` is the uncle *reward* below) = 300 slots (provisional) | **10/f = 300 slots** (confirms the default), widen to 450–600 near `ρ ≈ 1` | ≥ 7/f floor, buffers fork bursts ([§3.4](tsi-report-2-accuracy-and-design.md#s3-4)) |
|
||||
| uncle cap `U` | `MAX_UNCLES` = 4 (provisional) | **`⌈ρ⌉ + 1`** = 2 at the Blend target — well inside the spec's cap of 4 | the load-plus-margin rule ([§3.3](tsi-report-2-accuracy-and-design.md#s3-3)); 4 leaves headroom for heavier loads |
|
||||
| peering degree | (operator choice) | **≥ 6, scaled with N** | keeps `ρ` and reorg depth bounded at scale ([§3.7](tsi-report-2-accuracy-and-design.md#s3-7), [§6.10](tsi-report-3-robustness-and-incentives.md#s6-10)) |
|
||||
| uncle rewards | none (spec: uncle grants no block reward) | **soft inclusion, `w_u + w_n < 1`** | fork-safe fairness + anti-hiding ([§6.7](tsi-report-3-robustness-and-incentives.md#s6-7)–[§6.8](tsi-report-3-robustness-and-incentives.md#s6-8)) |
|
||||
| operating point | set by the Blend privacy budget | **`ρ = f·D_vis < 1`** | accuracy, weak grinding, shallow reorgs — one rule ([§6.3](tsi-report-3-robustness-and-incentives.md#s6-3), [§6.10](tsi-report-3-robustness-and-incentives.md#s6-10)) |
|
||||
|
||||
The one input this report *assumes* rather than reads from the deployed spec is the Blend profile itself (all worked examples use 3 hops × 8 s); the recommendations are expressed in terms of the resulting load `ρ`, so they transfer to whatever hops/δ_max the deployment actually chooses — but a reader should confirm the real Blend budget keeps `ρ < 1` at the target scale, or plan the extra uncle slot.
|
||||
|
||||
---
|
||||
@ -1,308 +0,0 @@
|
||||
# Total-Stake-Inference parameter selection — Accuracy and design
|
||||
|
||||
*Per-node network simulation of Cryptarchia Total Stake Inference (TSI). Simulator: `tsi-sim-pernode`. All runs at the true security parameter **k = 2160** unless noted; latency is in slots and **1 slot = 1 s**.*
|
||||
|
||||
*[Part 1 — Overview & recommendations](tsi-report-1-overview-and-recommendations.md) · [Part 2 — Accuracy & design](tsi-report-2-accuracy-and-design.md) · [Part 3 — Robustness & incentives](tsi-report-3-robustness-and-incentives.md) · [Part 4 — Reproducibility & appendices](tsi-report-4-reproducibility-and-appendices.md) · [Index](README.md)*
|
||||
|
||||
*Sections live across the set: [§1](tsi-report-1-overview-and-recommendations.md#s1)/[§7](tsi-report-1-overview-and-recommendations.md#s7)/[§8](tsi-report-1-overview-and-recommendations.md#s8) in Part 1, [§2](#s2)–[§5](#s5) in Part 2, [§6](tsi-report-3-robustness-and-incentives.md#s6) in Part 3, [§9](tsi-report-4-reproducibility-and-appendices.md#s9) and Appendices A–C in Part 4.*
|
||||
|
||||
---
|
||||
|
||||
<a id="s2"></a>
|
||||
## 2. Model and method
|
||||
|
||||
*In one sentence: we rebuild the network node by node — every block reaches every node late, along realistic message paths, and each node keeps its own chain and runs its own estimator — so if nodes could disagree, this simulation would show it.*
|
||||
|
||||
Each of `N` nodes runs TSI on its **own** partial view. A global block tree is built under explicit message propagation: block `b` produced at slot `t` by node `p` becomes usable at node `j` after the propagation delay from `p` to `j`. Every node then computes its own canonical chain (the single chain it accepts as valid via the fork-choice rule; competing blocks become orphans), its own block density, and its own estimate `D̂`. The **primary model throughout this report is Blend** — the deployment target and the regime where the uncle parameters actually matter; direct gossip serves as the light-delay contrast:
|
||||
|
||||
- **`blend`** (primary) — a random d-regular peering graph over which a block is first relayed through `hops` random nodes (the Blend cascade (Sphinx-style relays)), each adding a `Uniform(0, δ_max)` blending delay, before a final network-wide gossip. The dominant delay is the per-hop blending `δ_max` (`blend_delay_max`), in **whole seconds** — this is where forks, and therefore `U` and `W`, matter.
|
||||
- **`regular`** (contrast, detail) — plain direct gossip over the same graph; a block reaches a node after the shortest weighted path (geographic per-link latency). Realistic links are **sub-slot** (~40–200 ms), so forks are rare and even `U = 0` nearly suffices — the delay-free limit against which blend is judged.
|
||||
|
||||
Transport uses the **`geo`** distribution — a real-world geographic band mixture (short intra-region links, long inter-continental ones), the "natural" latency used throughout. The design laws of [§4](#s4) depend on latency only through its *mean* (`D_vis`), and this was tested directly: re-running the N = 1 000/4 000 grid with **exponentially distributed** links at the same mean leaves consensus untouched (spread 0, agreement 1.000) and every `U ≥ 1` cell identical within noise (≈ 0.99–1.01, means 1.004 exp vs 0.997 geo); only the *un-recovered* `U = 0` depth moves a few points (up to +0.07 shallower under exp, whose median link is shorter than its mean). Latency shape is a second-order effect confined to the regime the design avoids anyway. Both distributions draw each link **independently**, so this probes the latency *marginal*, not its spatial structure — geographically **correlated** latency (regional clustering that lets co-located nodes fork as a bloc) is not modelled; in the primary Blend regime the per-hop mixing delay dominates the geographic link term, so this is expected to remain second-order, but it is untested ([§8.3](tsi-report-1-overview-and-recommendations.md#s8-3)). A third topology, **`full_mesh`** (single-hop, uniform latency), reproduces the reduced analytic model (the simplified companion model that collapses the network to one chain and one scalar estimate; [§5](#s5)) inside this engine and was used only for cross-validation — no figure in this report derives from it.
|
||||
|
||||
**Who holds the stake.** Real stake is concentrated: a few large holders own most of it. We model this with a **Pareto** ("80/20") distribution — roughly 20 % of nodes hold 80 % of the stake — in every headline sweep, with equal-stake runs as a control. The exact split matters little to TSI, for a simple reason: the lottery hands a set of nodes wins in proportion to their *summed* stake, and TSI counts only the *total* number of blocks — so two different ways of splitting the same total stake produce statistically the same block density. Measured: consensus is identical under equal and Pareto stakes (spread 0, agreement 1.000 in both), accuracy with uncles is indistinguishable, and without uncles Pareto under-counts slightly *less* (0.815 vs 0.785) — concentrated stake means fewer *distinct* simultaneous winners, and a producer that wins twice does not fork with itself. Tail weight was tested directly too: re-running the grid with a lighter Pareto tail (Lomax index 1.33 instead of 1.16) leaves consensus and every `U ≥ 1` cell unchanged within noise, and leaves even the *un-recovered* `U = 0` under-count within noise (matched cells differ by ≈ 0.001 at k = 256) — less concentration means more *distinct* simultaneous winners, hence more forks to recover. The one place concentration could still matter is adversarial: a *whale* coalition's reward statistics are lumpier than a random coalition's, flagged as untested in [§6.5](tsi-report-3-robustness-and-incentives.md#s6-5).
|
||||
|
||||
**How we measure.** Every epoch, every node reports its own estimate. The first epochs of a run are a start-up transient — the estimate walking from its genesis guess to equilibrium — so we **discard the first half of every run ("50 % burn-in") and average over the remaining epochs**; each configuration is then repeated with several independent random seeds ("replicates") and averaged over those too. "Equilibrium" values in this report always mean that double average. *(The burn-in is a **measurement convention, not part of the protocol.** TSI itself just runs its recursion once per epoch, forever — each node's live estimate is simply the latest update, with no discarding or averaging. Burn-in is the standard steady-state-simulation technique of dropping the warm-up transient so a single reported number reflects the estimator's equilibrium, not its cold start; averaging the tail epochs and replicates additionally beats down the ±~0.9 % per-epoch sampling noise of [Appendix B](tsi-report-4-reproducibility-and-appendices.md#sB) to the ±0.1–0.2 % standard errors quoted here. The 50 % cut is deliberately conservative — the estimator actually converges in ~2 epochs, [§3.2](#s3-2).)* Four quantities are tracked:
|
||||
|
||||
- `D̂/D` — **accuracy**: the estimate divided by the true active stake. 1.0 = exact. With correct slot-counting ([§2.1](#s2-1)) and enough uncle slots the equilibrium is **1.0**; the only residual is an optional ~1 % from the on-chain integer rounding of `f` ([Appendix A](tsi-report-4-reproducibility-and-appendices.md#sA)).
|
||||
- `range_ratio` — **disagreement**: the highest minus the lowest node estimate, in `D̂/D` units. 0 means every node holds *exactly* the same value.
|
||||
- `agreement_window` — **consensus on the measurement**: the fraction of nodes whose finalized measurement window contains exactly the same blocks — i.e., who count the same density. 1.0 = unanimous.
|
||||
- `agreement_tip` — **consensus on the newest block**: the fraction of nodes currently sitting on the most common chain tip. This can be far below 1 while all of the above are perfect — nodes race over the newest blocks yet have long agreed on the finalized past ([§3.1](#s3-1)).
|
||||
|
||||
<a id="s2-1"></a>
|
||||
### 2.1 The TSI algorithm — before and after uncle references
|
||||
|
||||
**The question.** What exactly does TSI compute, and what changed when uncle references were added to the protocol?
|
||||
|
||||
**Why it matters.** Everything in this report — the under-count, the recovery, the parameter rules — follows from one design invariant and how the two protocol versions honour it; the report should be readable without the spec at hand.
|
||||
|
||||
**The design invariant: one count per slot.** The slot lottery is calibrated so that *slots* activate at rate `f` — the probability a slot produces at least one block is `f`. Crucially a slot can have several winners (two nodes independently win the same slot), and the calibration counts such a slot **once**. TSI must count the same way, or a busy slot with two winners would read as more stake than a slot with one. So the quantity TSI infers stake from is the number of *occupied slots* in the window, not the number of blocks.
|
||||
|
||||
**The algorithm.** Once per epoch each node measures the occupied-slot count `m` in the finalized window of `T` slots and nudges its estimate toward the target `f_p` (the block rate `f` as the on-chain integer rounding stores it):
|
||||
|
||||
```python
|
||||
m = occupied slots counted in the window (which slots count differs by version — below)
|
||||
D̂_next = max(1, D̂ · (1 − β·(f_p − m/T)/f_p)) β = learning rate (deployed: 1)
|
||||
```
|
||||
|
||||
Occupied slots more frequent than the target push the estimate up (the lottery then gets harder); less frequent pulls it down; `m/T = f_p` is the resting point. The two versions differ **only in which slots count**:
|
||||
|
||||
- **Before uncle references (spec v1.0):** `m` counts only slots occupied by the *canonical* chain. Under Blend delay roughly a third of honest blocks are orphaned, so their slots vanish from `m`, the resting point sits far below truth (`D̂/D ≈ 0.64–0.74`, deepening with network size), and the deflated estimate makes the network *chronically over-produce* blocks at up to ~2× the target rate ([§3.2](#s3-2), fig1). Pre-uncle TSI is not viable under Blend.
|
||||
- **After uncle references (spec v1.1):** `m` also counts the *referenced uncles* — orphans pointed at from the canonical chain — which puts the delay-orphaned slots back into the count and lifts the resting point up toward `D`. The spec as written deduplicates uncles by *block identity* (the same orphan referenced by two blocks counts once), recovering the bulk of the orphan loss. **This report recommends one refinement ([§8.5](tsi-report-1-overview-and-recommendations.md#s8-5)):** deduplicate by *slot*, not by block — a referenced uncle that shares a slot with a canonical block, or with another counted uncle, must add nothing. With slot-deduplication the orphaned slots return one-per-slot and the resting point lands at **exactly `D`**.
|
||||
|
||||
So the slot-deduplicated uncle rule recovers the orphan-loss the delay caused while preserving the "one count per slot" invariant, and the recovered equilibrium is exactly 1.0 — not a ceiling above it. Counting uncle *blocks* instead — as spec v1.1 is currently written, and as an earlier version of this simulator did — double-counts same-slot co-winners (one canonical, one referenced orphan) and inflates the equilibrium by the fixed multi-winner factor `c(f) ≈ 1.017`. Every result in this report uses the corrected slot count, and [§8.5](tsi-report-1-overview-and-recommendations.md#s8-5) carries the corresponding spec recommendation; the difference is a genuine ~1.7 % accuracy bias in the deployed rule, not merely a fixed simulation bug.
|
||||
|
||||
<a id="s2-2"></a>
|
||||
### 2.2 The equilibrium is bounded by 1
|
||||
|
||||
**The question.** Slot-counting settles the resting point at `D` ([§2.1](#s2-1)). Can the estimate sit *above* the true stake — and does the epoch-to-epoch jitter around the resting point cost nodes winning probability?
|
||||
|
||||
**Why it matters.** The estimate is the lottery's divisor: each node's per-slot win chance is `φ = 1 − (1−f)^(w_i/D̂)` (`w_i` = the node's stake). Whatever moves `D̂` moves every node's chance of winning slots — and with it block rewards and the pace of the chain.
|
||||
|
||||
**It cannot exceed the true stake.** TSI counts *occupied slots*, and a window can hold no more occupied slots than actually occurred, so the counted density is capped at the true rate `f` and the equilibrium `D̂/D` is **bounded by 1** — it recovers *up to* exactly 1 (full uncle recovery) and sits below it when delay orphans slots faster than uncles restore them ([§3.2](#s3-2)). The old `c(f) ≈ 1.017` ceiling above 1 was an artefact of counting uncle *blocks*; slot-counting removes it ([§2.1](#s2-1), [Appendix A](tsi-report-4-reproducibility-and-appendices.md#sA)). Around that bounded equilibrium each epoch's finite-window measurement carries ≈ ±0.9 % sampling noise (`√((1−f)/(f·T))` at k = 2160), which averages out over the burn-in ([Appendix B](tsi-report-4-reproducibility-and-appendices.md#sB)) and is *fairness-neutral* — all nodes share the same `D̂`, so relative win rates are untouched; only the block pace breathes by ±0.9 % epoch to epoch, with no systematic bias.
|
||||
|
||||
**The one residual bias is the on-chain rounding of `f`, and it is optional.** The spec stores the target as an integer at three-decimal precision, `f_p = ⌊1000·f⌋/1000 = 0.033` instead of `1/30 = 0.03333…`. Driving the density to `0.033` rather than to `f` leaves the estimate ≈ 1 % high (`f/f_p ≈ 1.010`) — a factor common to every node, so fairness is untouched, but an absolute ~1 % under-delivery of win probability and a ~1 % slow canonical pace. It is removed by carrying `f` at higher precision — **which this report's estimator does**: it drives the density to exact `f = 1/30` (residual 0), even finer than the 10⁻⁶ spec bump it recommends (`f_p = 0.033333`, residual < 10⁻⁵) — so every result here is unbiased. It is a one-constant change with no dynamics cost; the current spec still uses 10⁻³ and should adopt it ([§8](tsi-report-1-overview-and-recommendations.md#s8) row 14; [Appendix A](tsi-report-4-reproducibility-and-appendices.md#sA) quantifies it). This is the *only* systematic offset from 1 that the counting fix leaves.
|
||||
|
||||
---
|
||||
|
||||
|
||||
<a id="s3"></a>
|
||||
## 3. Findings
|
||||
|
||||
Seven findings, in the order a designer needs them: nodes agree ([§3.1](#s3-1)); delay biases the estimate low and uncles fix it ([§3.2](#s3-2)); when one uncle is not enough ([§3.3](#s3-3)); how wide the reference window must be ([§3.4](#s3-4)); how the two levers combine ([§3.5](#s3-5)); what changes with the block rate ([§3.6](#s3-6)); and how network size erodes the one-uncle margin ([§3.7](#s3-7)). Each finding is stated first; the tables and figures carry the evidence.
|
||||
|
||||
<a id="s3-1"></a>
|
||||
### 3.1 The per-node estimate is consensus-safe — with and without uncles
|
||||
|
||||
**The question.** Do all nodes compute the *same* estimate — including when blocks carry uncle references?
|
||||
|
||||
**Why it matters.** If nodes disagreed on `D̂` (the per-node estimate of the active stake), they would disagree on who is allowed to produce blocks — a consensus split; and the uncle mechanism this report recommends must not be able to cause one. This is the precondition every later finding stands on, which is why it comes first.
|
||||
|
||||
**Result: agreement is exact, at every uncle cap tested (`U` = 0…3).** At the full security parameter k = 2160, the highest and lowest node estimates are identical (spread exactly 0) and **window agreement** — the fraction of nodes that count *exactly the same finalized blocks* in their measurement window — is 1.000, across all network sizes, block rates (10–30 s), and uncle caps:
|
||||
|
||||
| N | 1 000 | 2 000 | 5 000 | 10 000 |
|
||||
|---|---|---|---|---|
|
||||
| `range_ratio` (spread of `D̂/D`) | 0 | 0 | 0 | 0 |
|
||||
| `agreement_window` | 1.000 | 1.000 | 1.000 | 1.000 |
|
||||
|
||||
Nodes *do* disagree about the newest blocks (tip agreement dips to ~0.95 in the worst cell) — but TSI never reads the newest blocks: it measures a window buried far past k-finality (blocks deeper than `k` are final — no honest node ever reorganises them), where all honest nodes provably hold identical history, uncle references included. **Consequence:** TSI can be treated as one global estimate, adding uncle references (and paying rewards for them) does not endanger that, and the rest of this report may speak of "the" estimate in the singular. Per-epoch traces at N = 10 000, the tip-agreement detail, and why an injected disagreement would *not* heal itself are in **[Appendix C](tsi-report-4-reproducibility-and-appendices.md#sC)**.
|
||||
|
||||
<a id="s3-2"></a>
|
||||
### 3.2 Latency biases TSI low; uncles recover it to exactly 1
|
||||
|
||||
**The question.** How accurate is the agreed-upon estimate — and do uncle references actually earn their place in the protocol?
|
||||
|
||||
**Why it matters.** An estimate that is too low makes the lottery too easy — blocks come faster than the target, collide more, and the safety margin erodes; this section measures how much accuracy is lost to network delay and how much of it uncles buy back.
|
||||
|
||||
**The mechanism in one line.** When two blocks race, the loser (an "orphan") drops off the chain — and out of the block count that TSI reads — so the network systematically under-counts its own stake; an **uncle reference** lets a later block point at a recent orphan and put it back into the count.
|
||||
|
||||
**Result: without uncles the estimate is 26–37 % low; one uncle recovers it fully.** Full-scale measurement under Blend (`D̂/D` = accuracy, the estimate over the true stake; each entry the mean over all sweep cells — every degree × link-latency × blending-delay combination, 6–20 replicates, equilibrium protocol of [§2](#s2)):
|
||||
|
||||
| N | `D̂/D`, U = 0 | `D̂/D`, U ≥ 1 |
|
||||
|---|---|---|
|
||||
| 1 000 | 0.739 | 1.000 |
|
||||
| 2 000 | 0.722 | 1.000 |
|
||||
| 5 000 | 0.650 | 0.999 |
|
||||
| 10 000 | 0.635 | 1.000 |
|
||||
|
||||
The U = 0 under-count **deepens with N** (more nodes → more concurrent proposals → more orphans). One uncle restores accuracy to **exactly 1** at every size *in this sweep* (`fig2`) — because its delay grid keeps the load `ρ` (blocks produced per propagation delay, [§3.3](#s3-3)) below one; [§3.7](#s3-7) shows that at much larger N, sparse peering pushes `ρ` past what one uncle can drain, and U = 1 stops sufficing.
|
||||
|
||||
**Why exactly 1 (and not a ceiling above it):** because TSI counts *occupied slots*, not blocks ([§2.1](#s2-1)). The lottery activates slots at rate `f`; uncles put the delay-orphaned slots back into the count, one per slot, and never double-count a slot that already has a canonical block — so holding the counted slot density at `f` settles the estimate at `D`. Verified in isolation at zero network delay, where the only orphans are same-slot co-winners: `U = 0` gives `D̂/D = 0.998 ± 0.007` and `U = 2` gives `1.000 ± 0.007` — the co-winner slots are counted once, not twice, so no ceiling appears. (The remaining ~1 % offset the deployed estimator carries comes only from the on-chain integer rounding of `f`, [§2.2](#s2-2) / [Appendix A](tsi-report-4-reproducibility-and-appendices.md#sA).)
|
||||
|
||||

|
||||
|
||||
**Bootstrap is self-limiting — but only with uncles** (`fig1`; full scale, k = 2160, Blend, N = 1 000 and 5 000, genesis guesses 0.01×–2× the true stake; 0.1×–2× at N = 5 000). With uncles (U = 2, solid lines) the cold start is a non-event: whatever the guess, block production snaps back to the target `f` within ~2–2.5 epochs and the estimate lands on 1.0 by epoch 2 — identically at both network sizes. Without uncles (U = 0, dashed) the system also converges from any guess, **but to the wrong place**: the estimate settles below truth (mean ≈ 0.58× across guesses, ranging ≈0.48–0.85; ≈0.51× at N = 5 000) and the network then *chronically over-produces* blocks at ~1.85× the target rate (~2.0× at N = 5 000) — the [§6.2](tsi-report-3-robustness-and-incentives.md#s6-2) load feedback in the flesh (a low estimate makes the lottery easier, extra blocks orphan, the count stays low). So uncle references are load-bearing from the very first epochs: they are what makes the bootstrap end *at the truth* rather than at a permanently overheated equilibrium.
|
||||
|
||||

|
||||
|
||||
<a id="s3-3"></a>
|
||||
### 3.3 One uncle is not always enough — the load `ρ`
|
||||
|
||||
**The question.** [§3.2](#s3-2) showed a single uncle slot sufficing across its delay grid. When does that stop — how much delay can one uncle slot actually handle?
|
||||
|
||||
**Why it matters.** Blend's per-hop blending delay is a privacy knob that may be turned up after deployment; the protocol needs to know at which point every extra second of delay demands another uncle slot.
|
||||
|
||||
**The intuition.** One uncle slot per block can absorb one orphan per block — so it keeps up only while the network creates orphans no faster than that. The deciding number is the **load** `ρ`: how many blocks the whole network produces during the time one block needs to become visible everywhere. The symbols used below, in one place:
|
||||
|
||||
| symbol | meaning |
|
||||
|---|---|
|
||||
| `f` | block rate (blocks per slot; `1/30` = one block every 30 s) |
|
||||
| `ℓ_mean` | mean gossip path latency between two nodes (slots) |
|
||||
| `hops`, `δ_max` | Blend cascade length and the per-hop blending-delay bound (mean per-hop delay = `δ_max/2`) |
|
||||
| `D_vis` | mean **visibility delay** = `hops·δ_max/2 + (hops+1)·ℓ_mean` under Blend (plain gossip: just `ℓ_mean`) |
|
||||
| **`ρ`** | **load** = `f·D_vis` — blocks produced per visibility delay = orphans each canonical block must absorb |
|
||||
|
||||
**Result: one uncle works up to `ρ ≈ 1`, and nothing else stretches that.** Accuracy at U = 1, plotted against `ρ` for **every** block rate and delay tested, collapses onto a single curve that breaks at `ρ ≈ 1` (`fig6`, right): **U = 1 recovers iff `ρ ≲ 1`.** Two bounds govern the cap: `U < ρ` **never** recovers (orphans arrive faster than they can be referenced; the queue grows without bound and no window helps) — a hard necessary condition; and empirically `U ≈ ⌈ρ⌉` (the load rounded up) *suffices*, with a one-uncle margin needed near integer `ρ` because the true concurrency window is about `2·D_vis`, not `D_vis` (the factor a ≈ 2 in eq. 4′, [§4](#s4)).
|
||||
|
||||
Sweeping the delay directly (`fig16`) makes the law visible: accuracy sits at 1.0 until the load crosses the uncle cap, then falls off — `U = 0` collapses immediately, `U = 1` holds to `ρ ≈ 1` (`D_vis ≈ 30` s), `U = 2` to `ρ ≈ 2` (`≈ 60` s), and `U = 3` across the whole tested range.
|
||||
|
||||

|
||||
|
||||

|
||||
|
||||
The full sweep behind `fig3` spans hops 3–6 × per-hop budget 4–32 s × `U ∈ {0, 1, 2, 4}` at both N = 1 000 and N = 2 000. Note the axes show the *per-hop* budget — the total mean delay is what matters, e.g. 3 hops at `δ_max = 8` mean 3·4 = 12 s of blending plus ≈ 5 s of gossip → `D_vis ≈ 17` s (`ρ ≈ 0.56`), while 6 hops at `δ_max = 32` reach `D_vis ≈ 104` s (`ρ ≈ 3.5`). More hops at a fixed budget raise `D_vis` and degrade U = 1 exactly as the load predicts (at `δ_max = 16`: accuracy 0.96 at 3 hops → 0.52 at 6 hops), and raising the cap restores it — U = 2 recovers every `δ_max ≤ 16` cell (0.987 at the `δ_max = 16` worst case, ≥ 0.995 below it) while `δ_max = 32` needs U = 4 (0.98, vs 0.77 at U = 2), consistent with `⌈ρ⌉` rising to ~2–4 across the hops axis (U = 3 was not swept). The N = 2 000 half reproduces the same pattern (U = 2: 0.98 at `δ_max ≤ 16`; U = 4: 0.98 at 32).
|
||||
|
||||
*(Detail — the direct-gossip contrast.)* Blend is the hard case because its per-hop delay is *whole seconds*; plain direct gossip (`regular`) has *sub-slot* links, so `ρ ≈ 0.04` and forks are negligible — even `U = 0` nearly recovers (`fig19`, N = 10 000): at the tested `0.5`-slot per-link latency `U = 0` alone reaches `0.97` at degree 6 and `0.93` at degree 4, a denser graph recovering better. This is why the recommended cap is `⌈ρ⌉ + 1` under Blend while `U = 1` suffices for direct gossip.
|
||||
|
||||

|
||||
|
||||
**The estimator is bounded by 1; the deficit is the signal below the block rate.** Because TSI counts *occupied slots* and a window can hold no more than actually occurred, the equilibrium `D̂/D` **cannot exceed 1** — it recovers *up to* exactly 1 and falls short only when the load outruns the uncle cap ([§2.2](#s2-2)). So the quantity of interest is the **under-count deficit `1 − D̂/D ≥ 0`**. A dedicated ρ-boundary sweep (`configs/rho-boundary.yaml`; hops fixed at 3 so `ρ ∝ δ_max`, N = 1 000, 20 replicates; `fig26`) resolves it cleanly: at U = 0 the deficit runs 0.51 → 0.85 across `ρ = 0.56 → 2.0`; one uncle holds the deficit within noise only for `ρ ≲ 0.6` (already 0.034 at `ρ = 0.96`, then steeply to 0.47 by `ρ = 2`); U = 2 stays within noise to `ρ ≈ 1.5` and U = 3 across the whole range — the `U = ⌈ρ⌉` boundary read straight off the deficit. Across all 40 cells **no equilibrium sits above 1 beyond sampling noise** (max `1.0025 ± 0.0018`), confirming the bound.
|
||||
|
||||

|
||||
|
||||
<a id="s3-4"></a>
|
||||
### 3.4 The uncle window is set by block spacing, not by delay
|
||||
|
||||
**The question.** How far back may a block reach when it references an orphan — how large must the uncle window `W` (measured in slots) be?
|
||||
|
||||
**Why it matters.** Too small a window silently discards orphans before any block gets a chance to reference them, and the under-count of [§3.2](#s3-2) returns no matter how many uncle slots exist; too large a window only costs a little validation state, so the risk is entirely on the small side.
|
||||
|
||||
**Result: the window is sized by block *spacing*, not by network delay.** Sweeping `W` at fixed U = 1 (`fig4`) gave a counter-intuitive result: the critical `W` is **~100–200 slots even for a 2-slot delay**, and barely depends on the delay. The reason is *queueing*: canonical blocks appear only every `1/f = 30` slots, and with U = 1 each references one orphan, so an orphan waits in a FIFO queue drained at ~1 per block interval. The window must span **several block intervals** for a queued orphan to reach a block with a free uncle slot before it ages out. (All runs fill uncle slots **oldest-first** — the FIFO behind this law; the spec's unbiased per-uncle coin (`uncle_random_p = 0.5`) was not swept. A coin drains a queued orphan no faster than FIFO, so the floor under the spec's rule is expected no smaller — read the margined `W = 10/f` as covering this untested sensitivity.) Empirically:
|
||||
|
||||
> **`W_min ≈ 7/f`** (the measured recovery floor is ≈ 200 slots at f = 1/30 = 6.7 block intervals; rounded design rule 7/f = 210 slots = 7 intervals), roughly constant across delay and U, rising toward ~10/f only as the load approaches the uncle capacity.
|
||||
|
||||
So `W` scales with the block interval `1/f`, and the default `W = 300` (= 10 intervals at f = 1/30) sits safely above the floor.
|
||||
|
||||

|
||||
|
||||
The full `W × delay` picture at `U = 1` (`fig22`) shows both bounds at once: below the `≈ 100`-slot window floor accuracy is lost at *every* delay (rows `W ≤ 50`), and above it accuracy holds only while the delay keeps `ρ ≲ 1` — at `delay = 16` `W = 300` reaches only `0.96` (recovery needs `W ≈ 600`, [§3.5](#s3-5)), and at `delay = 32` (`ρ ≈ 1.7`, past `U = 1`'s capacity) even the widest window recovers only `0.61`. Widening `W` cannot substitute for the extra uncle that `ρ > 1` demands.
|
||||
|
||||

|
||||
|
||||
The N = 2 000 replica of this sweep reproduces the window floor unchanged — in the recovered region (delay ≤ 8 s, W ≥ 200) accuracy matches N = 1 000 within 0.003 — and the U-limited delay-32 plateau is likewise essentially N-invariant (≈ 0.56–0.62 at W = 200–300 for both N = 1 000 and N = 2 000). Together with the (W × U) N = 2 000 run of [§3.5](#s3-5) and the [§3.2](#s3-2) N-deepening of the *un-recovered* under-count (0.739 → 0.635 over N = 1 k → 10 k), this is the direct evidence that the `W`/`U` **thresholds are N-invariant while the un-recovered accuracy is not**.
|
||||
|
||||
**At scale, and as a fluctuation buffer** (`fig25`; N = 1 000 vs **10 000**, W up to 600, k = 256). Three results close the window question:
|
||||
|
||||
- **The floor's position does not move with N.** At the 8-s budget the recovery knee sits at W ≈ 100–200 slots for both sizes (N = 10 000 is a few points deeper below the floor, consistent with its higher load) — the queueing law is about block *spacing*, and block spacing does not change with N.
|
||||
- **A wider window buys back the load boundary.** Block production fluctuates, and the window is the buffer that absorbs those bursts: right at `ρ ≈ 1` (the 16-s budget) U = 1 *fails* with the default W = 300 (0.965 at N = 1 000, 0.941 at N = 10 000) but **recovers at W = 600** (0.993 and 0.976) — twenty block intervals of buffer instead of ten. Because a growing network raises the load toward the boundary ([§3.7](#s3-7)), this is also the answer to "must W grow with N?": *near the boundary, yes* — N = 10 000 needs W = 450–600 where N = 1 000 needed 300–450.
|
||||
- **No window fixes sustained overload.** At the 32-s budget (`ρ ≈ 1.7–1.8 > U = 1`) accuracy stays collapsed at *every* width tested (0.23–0.75): a buffer absorbs variance around a stable queue, but when orphans *arrive* faster than one slot per block can *drain* them, the queue grows without bound and width is irrelevant. U = 2 (dashed) restores every width above the window floor (W ≥ 300).
|
||||
|
||||
So the two levers separate cleanly: **`U` must cover the average load; `W` must cover the fluctuations around it** — and near `ρ ≈ 1` a window of ~15–20 block intervals (W = 450–600 at f = 1/30) is a cheap alternative to spending the `+1` uncle slot.
|
||||
|
||||

|
||||
|
||||
<a id="s3-5"></a>
|
||||
### 3.5 The joint (W, U) region — the levers are hierarchical
|
||||
|
||||
**The question.** `W` (how far back a block may reference) and `U` (how many uncle slots per block) both fight the same orphan loss — can one substitute for the other?
|
||||
|
||||
**Why it matters.** If they were interchangeable, a deployment could just pick whichever is cheaper; the data says they are not, and getting the order of decisions wrong leaves accuracy on the table.
|
||||
|
||||
Co-sweeping W × U (`fig5`) shows the two are **not interchangeable**:
|
||||
|
||||
- **`W` is first-order.** Below the window floor — ≈ 3/f (100 slots) at low delay, rising to the ≈ 7/f (200 slots) of [§3.4](#s3-4) as load grows — *no* uncle count recovers the estimate (e.g. delay 16, W = 100: even U = 4 reaches only 0.94). The window simply cannot reach the orphans.
|
||||
- **Above the floor, `W` and `U` trade off**, and then **`U` is set by the load `ρ`** ([§3.3](#s3-3)): delay 8 → U = 1, delay 16 → U = 2, delay 32 → U = 2–3.
|
||||
- The default (W = 300, U = 1) is safe only up to delay ≈ 8 s; beyond that you must **add uncles**, not widen the window.
|
||||
|
||||
At N = 2 000 the region has the same shape, marginally worse (delay 32 needs U = 3 rather than 2).
|
||||
|
||||

|
||||
|
||||
As a deployment decision chart, `fig20` (also **N = 10 000**) reads the accuracy straight off `(delay × U)` at a fixed degree: the `U = 0` column is blue (under-count `0.67–0.75`, deeper than at smaller N — the N-scaling of [§3.2](#s3-2)), and a *single* uncle already lifts every tested delay to exactly 1 (`1.00`, bold) — the concrete basis for the `U = ⌈ρ⌉ (+1)` rule, holding at the largest scale.
|
||||
|
||||

|
||||
|
||||
<a id="s3-6"></a>
|
||||
### 3.6 Block rate `f` moves every threshold predictably
|
||||
|
||||
**The question.** Everything above was measured at 30-second blocks (`f = 1/30`). If the protocol ever runs faster blocks, do the rules survive?
|
||||
|
||||
**Why it matters.** The block rate is a first-order protocol choice, and a parameter recipe that silently assumed one rate would break the day the rate changes — this section shows every threshold moves *predictably* with `f`, so the recipe transfers.
|
||||
|
||||
**Result: the same wall-clock delay weighs more at a faster rate — everything else follows.** Because the load is `ρ = f·D_vis` (blocks produced per visibility delay, [§3.3](#s3-3)), a **faster** block rate makes the *same* wall-clock delay heavier (`fig6`, left): a fixed 8-s delay needs U = 1 at 30 s/20 s blocks but **U = 2 at 15 s and 10 s blocks**. Predicted `U_min = ⌈ρ⌉` vs. the measured smallest U reaching the 0.98 recovery bar (0.98 of the true value 1.0 — slot-counting removed the old `c(f)` ceiling; this sweep tested only **U ≤ 2**, so `>2` means U = 2 was still insufficient — consistent with a larger prediction):
|
||||
|
||||
| delay (s) | 30 s (`⌈ρ⌉`/emp) | 20 s | 15 s | 10 s |
|
||||
|---|---|---|---|---|
|
||||
| 8 | 1 / 1 | 1 / 1 | 2 / 2 | 2 / 2 |
|
||||
| 16 | 1 / **2** | 2 / 2 | 2 / 2 | 3 / >2 |
|
||||
| 32 | 2 / **>2** | 3 / >2 | 4 / >2 | 6 / >2 |
|
||||
|
||||
`⌈ρ⌉` matches in every fully-resolved cell **except two boundary cells** (one near-integer, one censored at U ≤ 2) — and error bars (standard error over replicates) show the two boundary cells are not alike: delay 16 / 30 s (`ρ = 0.96`) fails at U = 1 with `0.962 ± 0.006`, a **robust 3σ** under-shoot that genuinely needs U = 2; and delay 32 / 30 s (`ρ = 1.76`) reaches only `0.973 ± 0.002` at U = 2, a **4σ** shortfall — U = 2 genuinely fails there (matching the table's `>2`). This is why the `U = ⌈ρ⌉ + 1` margin exists and why boundary pass/fail calls must carry error bars. Every other `>2` cell is consistent with the larger `⌈ρ⌉` (the sweep only tested U ≤ 2). Three `f`-effects, all consistent with [§4](#s4) (plus one null result):
|
||||
- **recovery stays at 1.0** at every rate — slot-counting removes the `c(f)` dependence the block count had (the multi-winner rate `c(f)` = 1.017 → 1.054 from 30 → 10 s no longer appears in the estimate);
|
||||
- **`U_min` grows ∝ f** (same delay, faster rate ⇒ more uncles);
|
||||
- **`W_min` shrinks ∝ 1/f** (constant ≈ 7 block intervals);
|
||||
- *(null result)* **consensus-safety ([§3.1](#s3-1)) is f-independent** — `range_ratio = 0` at all four rates.
|
||||
|
||||

|
||||
|
||||
<a id="s3-7"></a>
|
||||
### 3.7 Network size erodes the one-uncle margin — through the gossip diameter, and only there
|
||||
|
||||
**The question.** [§3.3](#s3-3) fixed the network size and grew the delay; real deployments do the opposite — the blending budget is a design constant while `N` grows. Does a bigger network break `U = 1`, and does the answer depend on the peering degree? We investigate under Blend only, in two cases: **(a)** plain geo link delays, and **(b)** geo delays plus a long-tail model in which 10 % of deliveries straggle by an extra `Poisson(3)`-slot delay.
|
||||
|
||||
**Why it matters.** If `U = 1` quietly stops sufficing at large `N`, the under-count of [§3.2](#s3-2) returns — and with it the cheap grinding lever of [§6.3](tsi-report-3-robustness-and-incentives.md#s6-3) (deflating `D̂` to make the lottery easier, attacker included). A deployment sized on small-N evidence would degrade exactly when the network succeeds.
|
||||
|
||||
**Result: `N` enters through one number — the gossip path length.** Measured exactly on the simulator's own graph generator up to **N = 10⁶** (`fig24`, right), the mean path latency grows logarithmically, `ℓ_mean ≈ a_d·ln N`, with `a₄ = 0.33`, `a₆ = 0.19`, `a₈ = 0.14` slots — the classic `1/ln(d−1)` diameter law. The direct ladder (N = 1 000 → 32 000, scaled k = 256, δ_max ∈ {4, 8} s, 756 trajectories) confirms the mechanism end-to-end: **consensus stays exact in every cell** (spread 0, agreement 1.000 — long-tail jitter included), accuracy depends on `N` only through the load `ρ = f·D_vis(N, d, δ)` (`fig24`, left), and at the 8-s budget the **degree-4** curve declines monotonically until at N = 32 000 it drops below the recovery bar (the `0.98` criterion of [§3.6](#s3-6)): `0.973 ± 0.007` in case (a) at N = 32 000 (`fig23`) — while **degrees 6 and 8 stay flat near 1.0** through 32 000 (deg 6: 0.988; deg 8: 1.001), and `U = 2` recovers every cell. Extrapolating with the exact probe: **degree 4 reaches the measured `U = 1` failure load (ρ = 0.96, [§3.6](#s3-6)) at N* ≈ 8×10⁵ — one uncle is *not* enough at a million nodes on a degree-4 graph.** Degree 6 would need N ~ 4×10⁹ and degree 8 ≈ 6×10¹² to cross the same line: safe at any realistic size.
|
||||
|
||||

|
||||
|
||||
**The long-tail jitter changes nothing measurable.** Case (b) shifts the mean visibility delay by only 0.3 slots (10 % of deliveries × 3-slot mean straggle; `ρ` + 0.01) and the measured `U = 1` accuracy by ~0.4 % per cell on average (up to ~1.5 % in the noisiest large-N cell, ~0.03 % pooled — all within the few-replicate SEM); consensus is untouched in all 378 case-(b) trajectories. The stragglers' real cost is the variance channel of [Appendix B](tsi-report-4-reproducibility-and-appendices.md#sB).3 — already absorbed once `U ≥ 1`.
|
||||
|
||||

|
||||
|
||||
**Design consequence.** The uncle cap and the peering degree are exchangeable defences against network growth: keep `ρ(N, d, δ) < 1` either by adding an uncle slot or by densifying the graph — and one degree step (4 → 6) buys roughly a **5 000× larger network** at the same blending budget (N*: 8×10⁵ → 4×10⁹). [§8](tsi-report-1-overview-and-recommendations.md#s8) (row 12) turns this into the size-dependent degree rule.
|
||||
|
||||
---
|
||||
|
||||
|
||||
<a id="s4"></a>
|
||||
## 4. Design equations and parameter-selection algorithm
|
||||
|
||||
This section turns [§3](#s3) into a recipe: measure the network's propagation delay, compute the load, read off `U` and `W`. The table gives the calibrated design laws, the algorithm walks a deployment through them, and the worked examples cover the Cryptarchia baseline and Blend.
|
||||
|
||||
Let `f` = block rate (blocks/slot; block interval `1/f` slots = seconds), and for the propagation model let `ℓ_mean`, `ℓ_max` be the mean / max shortest-path transport latency over the peering graph (for a d-regular geo graph at N ≈ 1 000, deg 6: `ℓ_mean ≈ 1.2`, `ℓ_max ≈ 2.3` slots), `hops` the Blend cascade length and `δ_max` the per-hop blending bound. A denser peering graph shortens `ℓ_mean` and hence `D_vis` and `ρ` — the peering degree is an operator-side lever that buys uncle-cap headroom (fig19: at U = 0, degree 6 stays near-exact where degree 4 dips).
|
||||
|
||||
**Derived quantities** (calibrated on N = 1 000 blend). The `U`/`W` *thresholds* are ~N-invariant and were confirmed to hold across f = 1/10…1/30 ([§3.6](#s3-6)); the *U=0 accuracy* (eq 4′) worsens with N ([§3.2](#s3-2)) and is an N ≈ 1000 fit.
|
||||
|
||||
| # | quantity | equation | calibration / validity |
|
||||
|---|---|---|---|
|
||||
| 1 | mean visibility delay | `D_vis = hops·δ_max/2 + (hops+1)·ℓ_mean` (direct gossip: `ℓ_mean`) | — |
|
||||
| 2 | load (orphans / block) | `ρ = f · D_vis` | governs U (fig6 collapse at ρ≈1) |
|
||||
| 3 | fully-recovered accuracy | `D̂/D = 1` (slot-counting, [§2.1](#s2-1)) | exact at equilibrium; deployed value ≈ `f/f_p` ≈ 1.01 from on-chain `f`-rounding only ([Appendix A](tsi-report-4-reproducibility-and-appendices.md#sA)) |
|
||||
| 4 | accuracy, orphan-loss factor (U=0) | `D̂/D = ln(1−f)/ln(1−f/q_eff)` = `expected_ratio(f, q_eff)`, where the **effective density** `q_eff ≤ 1` is the fraction of active slots that are *counted* (canonical + recovered-uncle slots) | tight in the **orphan-loss regime** `q_eff < 1` (RMS 2.3e−4 vs the sim's `q_eff`, U=0 sweep); it *saturates at 1* as `q_eff → 1`, so the fully-recovered accuracy is **1.0 (eq 3)** — slot-counting counts each occupied slot once, so there is no multi-winner ceiling above it |
|
||||
| 4′ | accuracy, no uncles (closed-form stand-in) | `D̂/D(U=0) ≈ 1 / (1 + a·ρ)`, `a ≈ 2.0` | `a = 2.01` fit, ρ ≳ 0.2 (RMS 0.056, ~240× coarser than eq. 4); worsens with N; ≈ 1 for direct gossip |
|
||||
| 5 | **uncle cap** | necessary `U ≥ ⌈ρ⌉` (U<ρ **never** recovers); recommend `U = ⌈ρ⌉ + 1` | `⌈ρ⌉` matches emp. within ±1 — under-shoots just below integer ρ |
|
||||
| 6 | **uncle window** | `W_min ≈ 7/f` (≈ 200 slots @ f=1/30); margin `≈ 10/f` | ~const in delay/U; → ~10/f near uncle capacity |
|
||||
|
||||
**Algorithm — choose (W, U) for a deployment**
|
||||
|
||||
```python
|
||||
Inputs: f (block rate), propagation profile (hops, δ_max, ℓ_mean), deployment size
|
||||
1. Visibility delay: D_vis = hops·δ_max/2 + (hops+1)·ℓ_mean # direct gossip: D_vis = ℓ_mean
|
||||
2. Load: ρ = f · D_vis
|
||||
3. Feasibility: deployment REQUIRES U ≥ ⌈ρ⌉. If that exceeds the protocol's uncle-slot
|
||||
budget, cut propagation delay (fewer hops / less blending) — no window fixes U<ρ.
|
||||
Prefer to keep ρ < 1 with margin regardless: ρ > 1 is both the chronic
|
||||
under-count regime and the cheap grinding lever (§6.2–§6.3).
|
||||
4. Uncle cap: if ρ ≪ 1 (direct gossip): U = 1 suffices (U=0 already ≈ 0.999)
|
||||
else: U = ⌈ρ⌉ + 1 # recommended; ⌈ρ⌉ is the tight minimum, occasionally
|
||||
# short by one just below an integer ρ
|
||||
5. Uncle window: W = ⌈10/f⌉ slots # ≥ 7/f floor; set by block SPACING, not by the delay
|
||||
6. Expected accuracy: with U D̂/D → 1.0 (bounded by 1; spec 10^-3 f-rounding would add ~1%, App. A)
|
||||
without ≈ 1/(1+2ρ) for ρ ≳ 0.2, ≈ 1 for ρ ≪ 1 (direct gossip)
|
||||
7. Security k: set by finality; TSI's per-node agreement held at k=2160, expected any k (§3.1)
|
||||
8. Learning rate: β = 1 (one-epoch tracking; time-constant τ = −1/ln(1−β) epochs — §6.5)
|
||||
```
|
||||
|
||||
**Worked examples** (`ℓ_mean ≈ 1.2` slots for the study graph):
|
||||
|
||||
- *Blend (primary), 3 hops, δ_max = 8 s, f = 1/30.* `D_vis ≈ 3·4 + 4·1.2 ≈ 17` → `ρ ≈ 0.56`. Tight `⌈ρ⌉ = 1` (empirically suffices, [§3.5](#s3-5)); **recommended `U = 2`** (margin); `W = 300`.
|
||||
- *Same Blend at f = 1/10 (10 s blocks).* `ρ = 17/10 ≈ 1.7`. Tight `⌈ρ⌉ = 2` (needed, [§3.6](#s3-6)); **recommended `U = 3`**; `W = 10/f = 100`.
|
||||
- *Direct global gossip (contrast), f = 1/30.* `D_vis = ℓ_mean ≈ 1.2` → `ρ ≈ 0.04`. Forks negligible: `U = 1` (U = 0 already gives `D̂/D ≈ 0.999`, well below where eq. 4′ applies); `W = 10/f = 300`. → the Cryptarchia defaults.
|
||||
|
||||
---
|
||||
|
||||
|
||||
<a id="s5"></a>
|
||||
## 5. Discussion and caveats
|
||||
|
||||
Read this before using the numbers: the regime they hold in, and the margins that absorb what they do not.
|
||||
|
||||
- **Regime of exactness.** [§1](tsi-report-1-overview-and-recommendations.md#s1)–[§5](#s5) are for honest nodes with deterministic latency (`jitter = 0`), the regime in which the windowed/pruned engine is bit-exact; [§6](tsi-report-3-robustness-and-incentives.md#s6) tests the limits. **Open items:** sizing `W`/`U` so the emergent reference rate stays high under adversarial orphaning; the full multi-coalition (`K`-agent, [§6.9](tsi-report-3-robustness-and-incentives.md#s6-9)) selfish equilibrium (below-⅓ joint profitability unaddressed); per-node clock skew; and stochastic-jitter *parameter* tuning.
|
||||
- **The `ρ ≈ 1` and `W ≈ 7/f` constants (load boundary, window floor) are semi-empirical** — the *functional forms* follow from the orphan-queueing mechanism, the constants are fitted (`a = 2.01`; `U = ⌈ρ⌉` within ±1). For production use the margined rules `U = ⌈ρ⌉ + 1` and `W = 10/f`.
|
||||
- **N-scaling.** The U = 0 under-count worsens slowly with N (0.739 → 0.635 over N = 1k → 10k); the recovery thresholds for `U`/`W` are essentially N-invariant ([§3.4](#s3-4)). Size `D_vis`/`ρ` (and add margin to U) for the largest expected deployment; eq. 4′ itself is an N ≈ 1 000 fit.
|
||||
- **The residual ~1 %.** Slot-counting lands the equilibrium at exactly `D` ([§2.1](#s2-1)); the *only* systematic offset left is the on-chain integer rounding of `f` (`f_p = ⌊f·1000⌋/1000`, the `fixed_point` switch), which leaves the deployed estimate ≈ `f/f_p ≈ 1.01` high — common to every node (fairness untouched), an absolute ~1 % under-delivery of win probability, removed by carrying `f` at higher precision. **No correction exists in the specs today** ([Appendix A](tsi-report-4-reproducibility-and-appendices.md#sA); [§8](tsi-report-1-overview-and-recommendations.md#s8) carries the precision bump as a recommendation).
|
||||
- **W and U are coupled but not fungible:** you cannot buy below the `~7/f` window floor with any number of uncles, and you cannot buy below `U = ⌈ρ⌉` with any window — though right at the boundary `ρ ≈ 1`, extra window (15–20/f) absorbs the fluctuation bursts that would otherwise need the `+1` uncle ([§3.4](#s3-4), fig25).
|
||||
- **"Consensus-consistent" ≠ "accurate".** [§3.1](#s3-1) shows nodes agree on one value; that value is only correct once uncles restore it. The reduced (single global `D̂`) analytic model, which [§3.1](#s3-1) validates, is the model that collapses the network to one canonical chain and one scalar estimate.
|
||||
|
||||
---
|
||||
@ -1,226 +0,0 @@
|
||||
# Total-Stake-Inference parameter selection — Robustness and incentives
|
||||
|
||||
*Per-node network simulation of Cryptarchia Total Stake Inference (TSI). Simulator: `tsi-sim-pernode`. All runs at the true security parameter **k = 2160** unless noted; latency is in slots and **1 slot = 1 s**.*
|
||||
|
||||
*[Part 1 — Overview & recommendations](tsi-report-1-overview-and-recommendations.md) · [Part 2 — Accuracy & design](tsi-report-2-accuracy-and-design.md) · [Part 3 — Robustness & incentives](tsi-report-3-robustness-and-incentives.md) · [Part 4 — Reproducibility & appendices](tsi-report-4-reproducibility-and-appendices.md) · [Index](README.md)*
|
||||
|
||||
*Sections live across the set: [§1](tsi-report-1-overview-and-recommendations.md#s1)/[§7](tsi-report-1-overview-and-recommendations.md#s7)/[§8](tsi-report-1-overview-and-recommendations.md#s8) in Part 1, [§2](tsi-report-2-accuracy-and-design.md#s2)–[§5](tsi-report-2-accuracy-and-design.md#s5) in Part 2, [§6](#s6) in Part 3, [§9](tsi-report-4-reproducibility-and-appendices.md#s9) and Appendices A–C in Part 4.*
|
||||
|
||||
---
|
||||
|
||||
<a id="s6"></a>
|
||||
## 6. Robustness beyond the honest, deterministic regime
|
||||
|
||||
[§1](tsi-report-1-overview-and-recommendations.md#s1)–[§5](tsi-report-2-accuracy-and-design.md#s5) characterise an honest, jitter-free regime, in which the two headline claims (one global estimate `D̂`; uncle cap `U` = the load `⌈ρ⌉` suffices) cannot fail by construction. Nine studies test those limits, forming one arc from network noise to a full adversary and its incentives:
|
||||
|
||||
- **[§6.1](#s6-1)** consensus survives jitter (positive result).
|
||||
- **[§6.2](#s6-2)** the estimator is a load-feedback loop, bistable near `ρ≈1` in a fitted static map — a caveat, not reached in the full dynamics.
|
||||
- **[§6.3](#s6-3)** uncle-suppression grinding — a bounded attack.
|
||||
- **[§6.4](#s6-4)** static block withholding — deflates `D̂` but faithfully tracks *active* stake.
|
||||
- **[§6.5](#s6-5)** dynamic withhold-rejoin grinding — unprofitable, non-persistent.
|
||||
- **[§6.6](#s6-6)** selfish / private-chain withholding — the one profitable lever (above the classic stake threshold, quantified with the optimal-strategy MDP); uncle-*counting* still restores the estimate.
|
||||
- **[§6.7](#s6-7)** block/uncle *rewards* — robustly compensate honest orphans and disincentivise hiding, but a *voluntary* reward can backfire on selfish mining.
|
||||
- **[§6.8](#s6-8)** a *soft* (reward-weighted) inclusion rule — the fork-safe choice (a validity rule cannot prove which forks a producer saw). The emergent reference rate stays high (honest blocks reference the published orphans), so the backfire mostly vanishes; the residual is set by `W`/visibility.
|
||||
- **[§6.9](#s6-9)** multiple coalitions and bribery — the abstention commons is bounded and structure-independent; multi-coalition *selfish* mining is flagged, not solved.
|
||||
|
||||
The recurring theme: the same uncle references that fix the honest under-count ([§3.2](tsi-report-2-accuracy-and-design.md#s3-2)) are also the adversarial safeguard — and *rewarding* them (softly) makes the safeguard incentive-compatible without a fork risk.
|
||||
|
||||
<a id="s6-1"></a>
|
||||
### 6.1 Consensus survives network jitter — not a determinism artifact
|
||||
|
||||
Adding random per-delivery arrival noise changes nothing: consensus stays exact. This needed testing because `range_ratio ≡ 0` in [§3.1](tsi-report-2-accuracy-and-design.md#s3-1) is exact by construction *only* at `jitter = 0`. Re-running in the simulator's guaranteed-exact mode (the full arrival matrix, with the windowed/pruning speed-ups disabled) with per-(block,node) arrival jitter across the full grid — `jitter_mean ∈ {0, 0.1, 0.3, 1, 3}` slots (the 0 cell is the exact baseline) × {regular, blend} × N ∈ {1 000, 2 000}, 10 replicates each — leaves **`range_ratio = 0 ± 0` and `agreement_window = 1.000` in every one of the 200 cells**, and the mean accuracy stays at 1.0 throughout (0.97–1.02 across cells at every jitter level) — jitter costs nothing in accuracy either. The only metric that moves is *current-tip* agreement, degrading monotonically (worst cell ≈ 0.98 → 0.86 as jitter grows to 3 slots):
|
||||
|
||||
| `jitter_mean` (slots) | 0 | 0.1 | 0.3 | 1.0 | 3.0 |
|
||||
|---|---|---|---|---|---|
|
||||
| `range_ratio` (max over cells) | 0 | 0 | 0 | 0 | 0 |
|
||||
| `agreement_window` (min) | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
|
||||
| mean `D̂/D` (range over cells) | 0.974–1.017 | 0.977–1.015 | 0.987–1.014 | 0.983–1.017 | 0.979–1.022 |
|
||||
| `agreement_tip` (worst cell) | 0.977 | 0.957 | 0.945 | 0.928 | 0.859 |
|
||||
|
||||
Jitter feeds exactly the tip-level churn that [§3.1](tsi-report-2-accuracy-and-design.md#s3-1) showed never reaches the finalized density window. The reason is structural: the density window sits a full epoch length `E` (slots) past finality, so every block has reached every node by measurement time regardless of jitter, and all nodes still compute an identical `D̂`. The "one global `D̂`" claim therefore holds off the `jitter = 0` axis. *(Per-node clock skew — a whole-timeline shift of a node's slot clock, which unlike per-arrival jitter shifts a node's measurement-window bounds — was tested directly: a skew of up to ±20 slots moves each node's occupied-slot count by at most the one or two blocks in the shifted window edge, an inter-node spread of `≤ 1/m` ≈ `2·skew/T`. At the production window (`T ≈ 4×10⁵` slots) that is `< 10⁻⁴` — bounded and negligible, but, unlike jitter's exact 0, not identically zero. So bounded clock skew is a small quantifiable consensus cost, not a break.)*
|
||||
|
||||
<a id="s6-2"></a>
|
||||
### 6.2 The estimator is a load-feedback loop, bistable near ρ≈1 in theory (`fig7`)
|
||||
|
||||
The worry: the estimator feeds back on itself — a low estimate makes the lottery easier, more blocks collide, fewer get counted, and the estimate drops further. Could that spiral? The answer: only in a fitted static model, right at the load boundary `ρ ≈ 1`; the simulated network never reaches the spiral, but the boundary is real and is why we provision `ρ < 1` with margin.
|
||||
|
||||
Because the controller drives the *counted* density to `f`, the realised proposal rate is `f/r` (`r` = the recovered accuracy `D̂/D`; under-recovery means `r < 1`), so the realised load is `ρ_eff = ρ/r > ρ` whenever the estimate is under-recovered — this single effect explains both the U=0 finite-size drift and the near-integer `⌈ρ⌉` under-shoots. Fitting the effective counted density `q_eff` ([§4](tsi-report-2-accuracy-and-design.md#s4), eq. 4) as `q_eff ≈ 1/(1 + 0.71·ρ_eff)` and solving the self-consistent fixed point `r = ln(1−f)/ln(1−f/q_eff(ρ/r))` gives a single stable branch for `ρ ≲ 0.9` that **folds toward a collapsed low branch near `ρ ≈ 1`** — the map is bistable at the recipe's own operating boundary. So the honest-regime "`U = ⌈ρ⌉` suffices" line, computed on the high branch, is an upper edge, not a safe interior; the `+1` margin partly absorbs the `ρ_eff > ρ` gap. *Caveat:* this bistability is a property of the *fitted static* `q_eff(ρ)` map; the full per-node dynamics do **not** reach the fold — no simulated schedule flips to the collapsed branch ([§6.3](#s6-3)(i)) and a withhold pulse always recovers ([§6.5](#s6-5)(iv)). So the fold is a warning about *provisioning* margin, not an observed dynamical trap.
|
||||
|
||||

|
||||
|
||||
<a id="s6-3"></a>
|
||||
### 6.3 Grinding by uncle suppression is real but bounded, and scales with load (`fig8`)
|
||||
|
||||
The uncle mechanism is the attack surface: `D̂` is the denominator of the win probability `φ(f, w/D̂)` (`w` = the node's stake), so an adversary that references **no** uncles in its blocks starves the density count, deflates `D̂`, and inflates everyone's win rate — a grinding payoff. We added this to the engine (`adversary_frac`, a coalition holding a stake fraction `β_adv` that suppresses uncle refs; written `α` in the selfish-mining sections [§6.6](#s6-6)–[§6.8](#s6-8), and distinct from the TSI learning rate `β` of [§6.5](#s6-5)) and swept it × load (U = 2, N = 1 000):
|
||||
|
||||
| load `ρ` | β_adv = 0.1 | 0.3 | 0.5 | grinding gain @ 0.5 |
|
||||
|---|---|---|---|---|
|
||||
| 0.56 (honest regime) | 0.997 | 0.994 | 0.962 | 1.04× |
|
||||
| 0.96 (boundary) | 0.996 | 0.949 | 0.876 | 1.14× |
|
||||
| 1.36 (U-limited) | 0.993 | 0.898 | **0.700** | **1.43×** |
|
||||
|
||||
Three findings: (i) the deflation is **smooth and monotone in `β_adv` — no catastrophic branch-flip** up to 50 % stake, so the [§6.2](#s6-2) bistability is not reached in this range (the fitted `q_eff` over-states the fold); (ii) at the honest operating load `ρ < 1` the attack is **weak** (honest blocks backfill the suppressed references within the window `W`), biting only in the U-limited `ρ > 1` regime; (iii) therefore **provisioning `U`/`W` to keep the operating point at `ρ < 1` with margin is *also* the adversarial safeguard** — operating at `ρ > 1` is doubly bad (honest under-recovery *and* a cheap grinding lever). *Precondition:* a counted uncle must be a real VRF (verifiable-random-function) lottery winner for its slot — the proof makes fabricating a win cryptographically impossible; modelled here, and it must be enforced on-chain. Block *withholding* — the stronger lever flagged here — is examined next in [§6.4](#s6-4).
|
||||
|
||||

|
||||
|
||||
<a id="s6-4"></a>
|
||||
### 6.4 Block withholding deflates `D̂` to ≈(1−β_adv), but measures reduced *active* stake (`fig9`)
|
||||
|
||||
Uncle suppression is a *reference-level* attack (the adversary publishes blocks but starves the density count). The stronger lever is **withholding**: the coalition wins its slots but never publishes, so its blocks never arrive at any honest node (arrival set past the epoch horizon `E`, so never received), are never adopted, and never counted. We added `adversary_strategy ∈ {suppress, withhold}` and re-ran the β_adv sweep on both topologies (N = 400, k = 256, U = 2, exact oracle):
|
||||
|
||||
| β_adv | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | vs. suppress @ 0.5 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **withhold**, regular | 0.862 | 0.724 | 0.635 | 0.563 | 0.472 | — |
|
||||
| **withhold**, blend | 0.793 | 0.780 | 0.605 | 0.537 | 0.406 | — |
|
||||
| suppress, regular | 1.000 | 1.000 | 0.999 | 0.996 | 0.999 | flat ≈ 1.0 |
|
||||
| suppress, blend | 1.001 | 0.997 | 0.996 | 0.995 | 0.992 | ≈ 1.0 |
|
||||
|
||||
Three findings: (i) withholding tracks the **active-stake line `D̂/D ≈ (1−β_adv)`** (grey dotted in `fig9`), sitting slightly below it, and is **largely topology-independent** — regular and blend deflate together — because the mechanism removes stake from the *count*, not from the propagation graph; suppression, by contrast, does essentially nothing on the well-connected regular graph and only bites weakly on blend at high load. (ii) The deflation **overshoots below (1−β_adv)** and does so more at large β_adv (at β = 0.5 the active line is 0.50; regular reaches 0.47, blend 0.41): once `D̂` collapses, the surviving honest nodes win *more* often at the deflated denominator, fork more, orphan more, and the density feeds back down — the [§6.2](#s6-2) load-feedback loop amplifying the withdrawal. (iii) Crucially, **this is not a measurement failure — it is TSI correctly reporting reduced active stake.** A coalition that withholds every block is indistinguishable from one that simply *went offline*; total-stake inference is defined over the stake that is *participating*, and (1−β_adv) is the right answer.
|
||||
|
||||
*Methodological note.* The peering graph is drawn per trajectory from the config's full RNG key, so adversary-vs-honest cells use different graph samples — a replicate-averaged variance source, not a bias; the deflation is topology-independent, so the conclusion is unaffected.
|
||||
|
||||
Withholding is also **self-punishing**: the adversary forfeits all block rewards for its withheld slots to deflate a denominator that lowers *everyone's* difficulty (its own honest competitors included), so the relative-reward gain is second-order while the absolute cost is first-order. The generalisation to a **dynamic** withhold-then-rejoin *grinder* — abstain to deflate `D̂`, then re-activate to mine at the depressed difficulty — is examined and defused in [§6.5](#s6-5).
|
||||
|
||||

|
||||
|
||||
<a id="s6-5"></a>
|
||||
### 6.5 Dynamic (withhold-then-rejoin) grinding is unprofitable and non-persistent (`fig10`–`fig12`)
|
||||
|
||||
The static withholder of [§6.4](#s6-4) pays a permanent price. The dynamic variant tries to *time* participation: **abstain to deflate `D̂`, then re-activate to mine at the depressed difficulty**. We added a schedule (`adversary_period`, `adversary_withhold_epochs`) that withholds the coalition for part of each cycle and lets it behave honestly the rest, and attributed each epoch's canonical blocks to the coalition (reward accounting). Four results settle the threat.
|
||||
|
||||
**(i) The estimator is a moving average (EMA) of active stake; its speed is the learning rate `β`, not `W` (`fig10`).** Substituting the equilibrium density `m/T ≈ f·S_t·D*/D̂` into the spec recursion `D̂ ← D̂·(1 − β(f − m/T)/f)` gives, to leading order, `D̂_{t+1} = (1−β)·D̂_t + β·S_t·D*` — an exponential moving average of the active-stake signal `S_t·D*` (`S_t` = active fraction that epoch, `D*` = the honest-equilibrium estimate, which slot-counting puts at exactly `D` ([§2.1](tsi-report-2-accuracy-and-design.md#s2-1)); the [§6.5](#s6-5)/[§6.6](#s6-6) ratios `D̂/D*` are therefore the same `D̂/D` used in [§3](tsi-report-2-accuracy-and-design.md#s3)) with time-constant `τ = −1/ln(1−β)` epochs (this `β` is the TSI learning rate, `constants.BETA_DEFAULT` — unrelated to the coalition stake `β_adv`). The sim matches this law to **RMS 0.016** across `β ∈ {0.25, 0.5, 1.0}`. At the deployed learning rate **`β = 1` the estimator tracks active stake with a one-epoch lag** (`τ→0`); a withhold cycle then draws a near-square-wave sawtooth between `D*` and `(1−β_adv)D*`. A natural expectation is that the *uncle window* `W` sets the response speed; in fact responsiveness is set by the TSI learning rate `β`, *not* `W` (which sets equilibrium *accuracy*, not *speed*). A slower `β` shrinks the peak swing for a fixed schedule (amplitude 0.35 → 0.13 as `β`: 1 → 0.25) but, per (iii), buys the adversary nothing.
|
||||
|
||||

|
||||
|
||||
**(ii) It is strictly unprofitable — worse than not attacking (`fig11`, left).** Crediting the coalition's realized canonical blocks and dividing by its stake share `β_adv`, dynamic withholding earns **less than its stake share at every withhold duty `ψ > 0`**, monotone-decreasing in `ψ`, → 1 only as `ψ → 0` (never withhold). At `β_adv = 0.3` (`β = 1`):
|
||||
|
||||
| withhold duty `ψ` | 0 | 0.10 | 0.25 | 0.33 | 0.50 | 0.67 | 0.75 | 1.0 (static) |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| reward / stake | 1.00 | 0.95 | 0.81 | 0.74 | 0.68 | 0.41 | 0.31 | 0.00 |
|
||||
|
||||
The curve is essentially **the same for `β = 0.5` and `β = 1.0`** — profitability is independent of estimator speed — and stays `< 1` for every minority coalition (`β_adv = 0.1 … 0.4` at the alternating schedule give 0.56 … 0.73). This is the [§6.4](#s6-4) subsidy made dynamic: the coalition **forfeits whole epochs** to depress a difficulty whose benefit — a higher win rate at lower `D̂` — accrues **equally to every honest node**, so its *relative* share never rises while it *alone* pays the forfeit. The reward-maximising schedule is `ψ = 0`: don't withhold.
|
||||
|
||||
**(iii) As a pure griefing lever it is bounded and linearly costly (`fig11`, right).** If the adversary abandons profit and only wants to distort `D̂`, the leaky-integrator form of (i) gives the mean deflation exactly: **`1 − ⟨D̂/D*⟩ = β_adv·ψ`** (bounded by `β_adv`, maximal at the static `ψ = 1`), cross-checked analytically and in sim. It costs the coalition `≈ ψ` of its *own* reward, so distortion is only a fraction `β_adv` of what the attacker forfeits — the distortion-vs-forfeit locus sits **below the 1:1 line** and a slower `β` does not lift it. There is **no super-linear amplification** in any regime; the one place the distortion-per-unit-*network*-reward-forfeited exceeds 1 is a bounded `≤ 2×` constant, and only for a `≈ 50 %` coalition under an *aggressive* estimator (`β ≥ 1`, rejoin overshoot) — a protocol-chosen gain, not an adversary lever. For minority coalitions it never exceeds the forfeit.
|
||||
|
||||

|
||||
|
||||
**(iv) No self-sustaining collapse — the pulse is a pure transient (`fig12`).** The most worrying possibility is a withhold *pulse* that kicks the [§6.2](#s6-2) load-feedback loop across its fold into a *persistent* deflation that outlives the attacker. It does not: a 4–5-epoch pulse followed by honest behaviour returns `D̂` to the load-determined honest equilibrium at **every** operating point and **every** learning rate tested (`β = 0.25, 0.5, 1.0`). Where `D̂` sits low afterwards (0.21, 0.02 at `ρ ≫ 1`) that is the **chronic `ρ > 1` under-count of [§3.3](tsi-report-2-accuracy-and-design.md#s3-3) / [§6.3](#s6-3) — a provisioning failure, not the adversary's doing** (the pulse-free honest run sits at the identical level). The per-node dynamics have a unique attracting equilibrium per operating point; the [§6.2](#s6-2) bistability of the *fitted static* `q_eff(ρ)` map does not create a trapping basin in the full dynamics.
|
||||
|
||||

|
||||
|
||||
**Net.** Within its model, dynamic withholding is a *weaker* threat than the static case it generalises: no profit, bounded and costly distortion, no persistence — resolving the question left open at the end of [§6.4](#s6-4). The mitigation is unchanged and now doubly motivated: **hold the honest operating point at `ρ < 1` with margin** (which pins `D*` at the true stake, [§3.3](tsi-report-2-accuracy-and-design.md#s3-3)) and keep the learning rate responsive (`β = 1`, so any withhold notch is erased in one epoch). *Precondition, as in [§6.4](#s6-4):* counted uncles must be real VRF winners (enforced on-chain).
|
||||
|
||||
**Scope and limits (what this does *not* cover).** These results were verified by independent re-derivation and by baseline-corrected, forfeit-inclusive replicate sweeps; the confirmed claims come with a firm boundary:
|
||||
- **Withholding here means *abstention* (deflate the active-stake estimate), not private-chain release.** A withheld block's arrival time is set past the epoch horizon `E` — it never propagates and is permanently orphaned, so the coalition's forfeit is a *dead loss*. That dead loss is exactly what makes (ii) unprofitable. The classic *selfish-mining / private-chain* withholder instead builds a hidden chain and *releases* it to orphan honest blocks, recovering the forfeit — the stronger lever, **analysed separately in [§6.6](#s6-6)** (it *is* profitable above the classic stake threshold, and its density deflation is mitigated by the same uncle mechanism).
|
||||
- **The reward metric is relative canonical-block *share*** — which, because TSI holds the counted density at `f` (so the canonical block rate does not inflate when `D̂` deflates), equals the *absolute* per-block reward per unit stake; see the issuance note in [§6.6](#s6-6).
|
||||
- **Open-loop, single, equal-stake coalition.** Untested here: adaptive (feedback) schedules that react to the observed `D̂` or target the finalized measurement window directly; Pareto *whale* coalitions (lumpier share statistic); very slow `β` (where a notch persists many epochs in wall-clock); and `jitter > 0`. (The joint/adaptive adversary is closed in [§6.7](#s6-7); *multiple* non-cooperating coalitions are treated in [§6.9](#s6-9).) The reported bounds are best-case-for-the-defender.
|
||||
|
||||
<a id="s6-6"></a>
|
||||
### 6.6 Private-chain (selfish) withholding is profitable above the classic threshold; uncle-counting still restores the estimate (`fig13`)
|
||||
|
||||
[§6.5](#s6-5)'s "withholding" was *abstention* — the coalition **discards** its blocks. The stronger, standard block-withholding attack is **selfish mining** (Eyal–Sirer): the coalition mines a **private chain** and **releases** it to orphan honest blocks, *recovering* the forfeit. Because TSI reads density from a window past k-finality where all nodes agree on the canonical chain ([§3.1](tsi-report-2-accuracy-and-design.md#s3-1)), the outcome is a global longest-chain race with a network tie-break `γ` (the fraction of honest miners that adopt the adversary's block in a same-length race); we simulate it directly (`selfish.py`, validated to within simulation noise against the Eyal–Sirer closed form).
|
||||
|
||||
**Profitability (`fig13`, left).** Unlike abstention, private-chain withholding **earns more than its stake share above the classic threshold** `α > (1−γ)/(3−2γ)` (here `α` is the coalition stake fraction, ≡ `β_adv` of [§6.3](#s6-3)–[§6.5](#s6-5), in the selfish-mining literature's notation) — ⅓ at `γ = 0`, ¼ at `γ = 0.5`, → 0 as `γ → 1`. At `α = 0.4`: revenue/stake `= 1.21×` (`γ = 0`) up to `1.42×` (`γ = 1`); below threshold (`α = 0.2`, `γ = 0`) it under-earns (`0.65×`). This is the general Nakamoto-PoS selfish-mining result — not TSI-specific — and it is the **honest upper bound on withholding profitability that [§6.5](#s6-5)'s abstention model does not capture**.
|
||||
|
||||
*How much does the **optimal** strategy add?* SM1 — the original Eyal–Sirer strategy — is one fixed policy, not the best one; solving the Sapirshtein–Sompolinsky–Zohar model, a Markov decision process that searches over *every* possible withhold/release policy (`selfish_mdp.py`; states `(a, h, fork)`, actions adopt/override/match/wait, ratio objective by bisection + damped relative value iteration), gives the true frontier (`fig14`, left). At `γ = 0` the optimum barely beats SM1 (`0.488` vs `0.484` at `α = 0.4`; profitability threshold essentially the SM1 `⅓`, marginally below it), so **SM1 is a faithful proxy** there; the gap opens only with network advantage (`γ = 0.5`: `0.573` vs `0.526` at `α = 0.4`, `0.803` vs `0.723` at `α = 0.46`). The optimal revenue equals `α` below threshold — confirming the *threshold* itself, not just SM1's revenue — and upper-bounds any selfish take.
|
||||
|
||||
) shrink the premium (right — the benign limit of [§6.7](#s6-7)).](report-figures/fig14_optimal_rewards.png)
|
||||
|
||||
**The TSI coupling (`fig13`, right).** Selfish mining orphans honest blocks (23.7 % of all blocks at `α = 0.4`, `γ = 0`), so the *counted* canonical density falls and TSI **deflates `D̂` to `D*·(density fraction)`** — `0.70` at `α = 0.4` with no uncles: a 30 % under-report of active stake, a safety harm layered on the reward theft. But that under-count is precisely what **uncle references fix** ([§3.2](tsi-report-2-accuracy-and-design.md#s3-2)): counting orphaned honest blocks back into the density restores `D̂` toward `D*` — to `0.82` at `η = 0.5` and `0.94` at `η = 1` (uncle-recovery fraction `η ∈ [0,1]`, distinct from the integer uncle cap `U`). So the mechanism provisioned for honest latency recovery **doubles as the mitigation for selfish mining's estimator deflation** — this restores the *estimate* `D̂` (a safety win), independent of the reward schedule. Whether *rewarding* those uncles also cuts the selfish *profit* premium is a separate, conditional question — it can *backfire* without care and needs a soft inclusion rule — taken up in [§6.7](#s6-7)–[§6.8](#s6-8).
|
||||
|
||||

|
||||
|
||||
**Issuance model.** Does deflating `D̂` inflate the reward pie? No, for a per-canonical-block schedule: TSI targets *counted* density `= f`, so the canonical block rate is held at `≈ f` regardless of the attack (the extra lottery wins a deflated `D̂` produces are orphans that earn nothing). Absolute reward per unit stake therefore equals the revenue *share* metric — **[§6.5](#s6-5)'s "abstention is unprofitable" is robust under per-block issuance**, and [§6.6](#s6-6)'s selfish premium is the genuine profit channel. MEV (value extracted from transaction ordering) or a per-*raw*-block schedule would reopen this — an issuance-design question, not a TSI-estimation one. *(This clean equivalence assumes the block-only schedule; once [§6.7](#s6-7) pays and **counts** uncles, the counted density held at `f` includes recovered uncles, so the canonical rate sits slightly below `f` and the pie's size and split both shift. [§6.7](#s6-7)'s profitability is then read directly off the reward *share*, not off a fixed pie — the two subsections use different issuance premises, reconciled only in the `w_u = 0` limit.)*
|
||||
|
||||
**Scope.** Modelled: SM1 *and* the optimal-MDP selfish strategy with a network tie-break `γ`, a single coalition, and uncle density-recovery as the `[0,1]` knob `η`. Multi-coalition races are discussed (not solved) in [§6.9](#s6-9). The headline: **private-chain withholding is the real profit threat, bounded by the classic stake threshold; its *estimator* deflation is undone by uncle-counting, while its *profit* premium is only conditionally reduced by rewarding uncles ([§6.7](#s6-7)–[§6.8](#s6-8)).**
|
||||
|
||||
<a id="s6-7"></a>
|
||||
### 6.7 Block/uncle rewards: robust for fairness and anti-hiding, but a *voluntary* reward can backfire on selfish mining (`fig14`, `fig15`)
|
||||
|
||||
Everything above measures *blocks*. Whether an attack **pays** depends on the reward schedule, which is a design choice. We add one as a configurable option (`selfish.RewardParams`): a block reward (`= 1`), an **uncle reward** `w_u` paid to the *producer* of an orphaned block that is referenced as an uncle, and a **nephew reward** `w_n` paid to the block that *references* it. The model has five knobs in total: `w_u`, `w_n`, the honest-orphan reference rate `p_ref` (defined below), the attacker's self-uncling rate `p_ref_adv` (= 1 for a rational attacker), and `adv_nephew` — the fraction of referenced honest orphans whose *nephew* reward the attacker's own canonical blocks capture (0 without a forced-inclusion mandate, since honest blocks do the referencing). Three effects, all adversarially analysed.
|
||||
|
||||
**(1) It compensates orphaned honest producers (fairness).** A block-only schedule pays an orphaned honest block **nothing** — the same "off-chain, uncounted" fate that drives both the [§3.2](tsi-report-2-accuracy-and-design.md#s3-2) latency under-count and the losses a selfish attacker inflicts. Define the **reference rate** `p_ref` = the probability that an orphaned honest block is referenced as an uncle by some canonical block within `W`. An uncle reward `w_u` refunds the loss in proportion to `p_ref`: honest recovery of *mined* value rises with `w_u` toward `1.0` (e.g. in the α = 0.4 selfish race, `0.61 → 0.80 → 1.00` for `w_u = 0 → 0.5 → 1` at `p_ref = 1`; `fig15`, right, plots recovery vs `p_ref`). The same references that restore the TSI *density* ([§6.6](#s6-6), `D̂`) restore the *reward*. **This benefit is robust** because a genuine latency orphan is *published* and straddled by *honest* canonical blocks that reference it, so `p_ref ≈ 1` here — unlike the selfish case in (3), where the attacker owns the straddling blocks.
|
||||
|
||||
**(2) It disincentivises hiding.** A withheld block (the [§6.4](#s6-4)/[§6.5](#s6-5) lever) *never propagates*, so it can never be referenced as an uncle — the withholder forfeits **both** the block reward **and** any uncle reward. Publishing promptly is therefore strictly dominant for an honest-but-unlucky miner, and abstention is made even costlier than the already-negative [§6.5](#s6-5) result. The reward schedule turns "publish everything you win, immediately" into the incentive-compatible action.
|
||||
|
||||
**(3) On selfish mining, a *voluntary* uncle reward can *backfire* — a soft inclusion rule fixes it (`fig15`, [§6.8](#s6-8)).** The naive hope is that paying orphaned honest blocks lifts honest reward and cuts the attacker's share. But the attacker controls the references that would do so: the honest blocks it orphans by a selfish override are straddled by *its own* canonical blocks, and it **suppresses** them (the [§6.3](#s6-3) lever) — while **self-uncling** its own revealed-but-lost blocks to recover `w_u + w_n` each.
|
||||
|
||||
Modelling that strategic reference game (`selfish.reward_shares`, with `p_ref_adv` = the attacker's rate of self-uncling its own lost blocks, `= 1` when rational), a voluntary uncle reward makes the attacker's **reward share rise *above* its block share** (e.g. `α = 0.4`: `0.484 → 0.520` at Ethereum-like `w_u = 0.875`), and honest recovery falls (to `0.61` at `α = 0.4`, `0.34` at `α = 0.46`). *(The benign-looking `fig14`-right — "uncle rewards shrink the premium" — is the opposite `p_ref = 1` limit of full inclusion; the backfire is the suppressed `p_ref → 0` end. `fig15` plots the whole `p_ref` axis and reconciles the two.)*
|
||||
|
||||
The backfire is *unconditional* below the `⅓` threshold and holds at higher `α` while the attacker keeps `p_ref` low (its default, since it owns the straddling blocks); it weakens as honest blocks reference more orphans (`p_ref` high). The worst case `p_ref = 0` (suppress *all* references) is not achievable: honest orphans are published, so honest canonical blocks reference them anyway. [§6.8](#s6-8) shows that under a **soft** (reward-weighted) inclusion rule the emergent `p_ref` sits high — the selfish share falls below block-only and toward stake, with honest recovery restored, and **without** the fork risk a hard mandate would carry. (How high `p_ref` must be depends on `α` — quantified in [§6.8](#s6-8), `fig15`.) So the *fairness* benefit (1) and *hiding*-disincentive (2) are robust and attacker-independent; the selfish-mitigation (3) is real but **degrades gracefully with the reference rate `p_ref`**, which the window `W` and visibility set ([§6.8](#s6-8)).
|
||||
|
||||
**Adversarial analysis of the reward scheme itself.** A reward for uncles is a new payout, so a new attack surface — *bounded*, but only under an explicit constraint the naive "`w_u < 1`" misses.
|
||||
|
||||
*(a) Uncle farming with a self-nephew.* An attacker who deliberately orphans a genuine VRF win and references it **from its own next canonical block** collects *both* payouts: `w_u` (uncle producer) **and** `w_n` (nephew). The per-slot payoff is `w_u + w_n` against an honest block's `1`, so farming beats honest **iff `w_u + w_n > 1`**. The binding safety condition is therefore **`w_u + w_n < 1`**, *not* `w_u < 1` — e.g. Ethereum's `0.875 + 0.03125 = 0.906 < 1` is safe, but a loose "substantial `w_u` + small `w_n`" (say `0.875 + 0.15 = 1.025`) is **not**. Fabrication is impossible (a counted uncle must be a real VRF winner, [§6.3](#s6-3)), keeping the payoff per-slot rather than unbounded, and `U`/`W` cap total uncle issuance to `≤ U·(w_u + w_n)` per block.
|
||||
|
||||
*(b) Multi-winner self-uncle.* A miner that wins two concurrent slots (multi-winner slots exist, ~1.7 % of blocks) can make only one canonical and, if the schedule pays it, earns `1 + w_u` by uncling the other. This does **not** inflate the estimate — TSI counts the slot once either way ([§2.1](tsi-report-2-accuracy-and-design.md#s2-1)) — so it is a pure reward-fairness question, bounded by `w_u + w_n < 1` on the referencing side.
|
||||
|
||||
*(c) The reference game is not fixed by a plain `w_n`.* Because the selfish attacker suppresses references on its *own* override blocks, a nephew reward it *also collects* does **not** by itself force inclusion — in the model, raising `w_n` slightly *lowers* `α*` (the stake threshold above which selfish mining is profitable — i.e. it marginally *helps* the attacker, which pockets a canonical-share slice of the nephew pot). What *does* keep the effective reference rate high is that the orphans are **published**, so the many *other* honest blocks within `W` reference them anyway ([§6.8](#s6-8)). A hard "invalid block if it omits an uncle" mandate would force this outright but **cannot be encoded fork-safely** (no node can prove what forks a producer saw — [§6.8](#s6-8)), so we use a **soft (reward-weighted) inclusion rule** instead. *(Consolidated protocol recommendation: end of [§6.8](#s6-8).)*
|
||||
|
||||
**Joint / adaptive adversary** (the adaptive-schedule case left open in [§6.5](#s6-5)'s scope note). The optimal *within-race* adversary is the [§6.6](#s6-6) MDP. Combining selfish release with the [§6.5](#s6-5) *abstention* lever is **dominated**: abstaining a fraction `ψ` of your wins lowers effective mining power to `α_abst = α(1−ψ)/(α(1−ψ)+1−α) < α`, and selfish revenue is monotone increasing in `α` (`fig14`), so withholding-to-abstain strictly reduces the take. Closed-loop timing on `D̂` adds nothing either: the selfish revenue *share* depends only on `(α, γ)`, not on `D̂` (the race is decided block-by-block, while `D̂` moves on the epoch scale). So the adversary frontier is exactly **optimal selfish mining, bounded by the stake threshold** — and, *under the [§6.8](#s6-8) soft inclusion rule that keeps the emergent `p_ref` high*, blunted but not eliminated by rewarded uncles. No compounding lever remains.
|
||||
|
||||
<a id="s6-8"></a>
|
||||
### 6.8 A *soft* inclusion rule delivers the benefit without a fork risk (`fig15`)
|
||||
|
||||
[§6.7](#s6-7)(3) shows a purely *voluntary* uncle reward can backfire on selfish mining. The tempting fix — a *mandatory* inclusion rule that makes a block **invalid** if it omits an owed uncle — cannot be encoded safely: "owed" means "the producer could have seen it," which depends on per-node gossip timing that no node can prove. An **objective** encoding (slot-gap `s_block − s_uncle ≥ g`) lets an adversary *withhold-then-late-release* an uncle to retroactively **invalidate honest blocks** (censorship); a **visibility-based** encoding is *subjective*, so the same block is valid to some nodes and invalid to others — a **consensus fork** with no adversary at all. **We therefore make inclusion a soft rule:** *a block is always valid; a producer that includes an available in-window uncle earns the nephew reward `w_n`, and the orphan producer earns `w_u` — omission simply forfeits those rewards.* No validity edge, so no fork and no censorship weapon.
|
||||
|
||||
**The emergent `p_ref` is high — now measured, not assumed.** The per-node engine records `p_ref` directly (the fraction of in-window orphans referenced as uncles by some canonical block): at the recommended `W = 300` it is **≈ 1.0 honestly** and **0.989 under a 30 %-stake coalition that produces its share of the (straddling) canonical blocks and suppresses references on them** ([§8](tsi-report-1-overview-and-recommendations.md#s8) capstone). That suppression case *is* the reference-suppression channel a selfish attacker would use — an honest orphan loses its reference only if *every* one of its in-window straddlers is a suppressing coalition block — so even with 30 % of the straddlers hostile, barely 1 % of honest orphans go unreferenced, comfortably above the `p_ref ≳ 0.3` the soft rule needs for `α ≥ 0.4` (`fig15`). The one selfish channel this measurement does *not* exercise is the attacker *orphaning extra* honest blocks via a private-chain override (the per-node engine has no private-chain strategy): there we rely on the structural argument below — published honest orphans are straddled by *honest* blocks within `W`, which the attacker does not control — and read the payoff off the `fig15` `p_ref` sweep rather than a single measured point. The earlier analytic sweeps used `p_ref ≈ 0.85` as a conservative stand-in; the suppression measurement confirms it is, if anything, pessimistic.
|
||||
|
||||
**Why a soft rule still works — the reference rate is emergent and high (`fig15`).** Under a soft rule `p_ref` is not the attacker's to set to 0: an honest orphan was **published** (honest miners publish immediately), so *any honest canonical block that sees it within `W`* references it for `w_n`. The attacker only withholds references on **its own** canonical blocks; the honest blocks that follow do the referencing. So `p_ref` sits **high** in practice, and `fig15` reads off the payoff along it — at a realistic honest-referencer rate `p_ref ≈ 0.85`, the selfish attacker's reward share is already **below block-only and near stake** (`0.420` at `α = 0.4`, vs `0.520` at the suppressed `p_ref = 0` and `0.484` block-only) with honest recovery `0.83–0.93`. For `α ≥ 0.4` the crossover below block-only is at `p_ref ≈ 0.3` (the grey band); for a near-threshold `α ≈ 0.35` a small residual backfire (`≤ +0.03` for `p_ref ≳ 0.3`, rising to `+0.05` at the unreachable `p_ref = 0`; share still near stake) persists until `p_ref ≈ 0.75`. So the soft rule buys **most** of the (unattainable) hard-mandate benefit, plus the full fairness and hiding-disincentive of [§6.7](#s6-7)(1–2), with **none** of the fork/censorship hazard.
|
||||
|
||||

|
||||
|
||||
**What sets `p_ref` — the window `W` and visibility (the residual).** `p_ref` rises toward 1 as `W` grows (more following honest blocks get a chance to reference a given orphan) and falls only when an orphan **ages out of `W` before any honest referencer sees it** — i.e. under a *deep reorg* longer than `W`, or genuine propagation loss. This is exactly the "can't guarantee a node sees every fork in the window" limit, and it is `W`-tunable: the [§3.4](tsi-report-2-accuracy-and-design.md#s3-4) sizing `W ≈ 10/f` is already generous, and the residual adversarial lever (drive `p_ref` down via reorgs deeper than `W`) is bounded — deep reorgs need large `α`, are rare, and risk the attacker's private chain. Two provisos remain, both about *degree* not *direction*: the counted-density (`D̂`) recovery is likewise `p_ref`-limited, so under attack-inflated orphaning (`orphan/blocks ≈ 0.35` at `α = 0.46`) the cap `U` — sized for the *honest* load ([§3.3](tsi-report-2-accuracy-and-design.md#s3-3)) — may need margin, since owed uncles beyond `U` per block defer and can age out; and even at `p_ref = 1` the selfish share is `≈ α` with a small residual premium that **grows with `α`** — `≈ 0` near the `⅓` threshold, `+0.006` at `α = 0.4`, `+0.014` at `α = 0.46` — not exact break-even.
|
||||
|
||||
**Reward-side attack surface (bounded, self-defeating).** *(a) Farming* — orphaning a real VRF win to self-uncle it — pays `w_u + w_n` against `1`, so it is loss-making under the [§6.7](#s6-7)(a) invariant **`w_u + w_n < 1`**; in the model the attacker's share *falls* as it self-uncles more (`0.406 → 0.394`). *(b)* Because honest referencers count the orphans, a flood *inflates* `D̂` rather than deflating it — the opposite of what a deflation attack needs. *(c) Nephew leakage* to the attacker is bounded by `w_n` and dominated by the `w_u` that flows to the honest producer.
|
||||
|
||||
The consolidated reward-schedule recommendation is in [§8](tsi-report-1-overview-and-recommendations.md#s8) (rows 10–11 of the selection table).
|
||||
|
||||
<a id="s6-9"></a>
|
||||
### 6.9 Multiple coalitions and bribery: the abstention commons is bounded; multi-coalition selfish mining stays open
|
||||
|
||||
Everything above assumed a single coalition; this section asks what changes when several attack at once — or bribe others to.
|
||||
|
||||
**Withholding commons (solid).** `D̂` deflation depends only on the *total* participating stake ([§6.4](#s6-4)), not on how it is partitioned — this is **exact by construction** (the withhold mask is per-node membership with no sub-coalition label), so `K` abstaining coalitions summing to `β` deflate `D̂` exactly as one coalition of size `β`, and the [§6.2](#s6-2) fold is no more reachable by stacking than by a single coalition of the same `β`. Each is *individually* unprofitable ([§6.5](#s6-5)) and, given others already deflate, the best response is to *mine* at the depressed difficulty rather than join — so the abstention commons is not an equilibrium; pure griefers pay their aggregate forfeit for a bounded, linear deflation.
|
||||
|
||||
**Multi-coalition selfish mining (conjectural).** Here the abstention argument does **not** transfer: competing *selfish* miners orphan *each other*, and the literature shows the profitability threshold can fall **below** the single-miner `⅓` — several individually-sub-threshold coalitions may be *jointly* profitable, and total orphaning (hence raw `D̂` deflation) can **exceed** the single-coalition [§6.6](#s6-6) value, so `D̂ ≈ 0.70` at `α = 0.4` is *not* a multi-coalition upper bound. Our two-type model cannot represent a `K`-agent reference game, so this is flagged, not solved; TSI *safety* rests on the [§6.8](#s6-8) soft rule's honest-referencer `D̂`-recovery, which is itself `p_ref`-limited (`W`/visibility) and degrades — gracefully, never forking — as orphan volume rises. Sizing this properly is future work.
|
||||
|
||||
**Bribery on the reference game.** An attacker profiting from a deflated `D̂` can bribe canonical miners to *omit* references. Under the soft rule the miner forgoes only the nephew reward `w_n`, so a bribe `> w_n` (`≈ 0.03`) suffices — *cheap*, and here the hard-mandate "prohibitive (a full block)" defence is unavailable *by design* (we rejected validity-based inclusion for fork-safety, [§6.8](#s6-8)). But its *impact* is small: bribing away one block's references lowers `p_ref` only marginally, since the many *other* honest blocks within `W` still reference the orphan; to hold `p_ref` low the attacker must bribe (or control) a large fraction of canonical producers over the whole window. So soft-rule bribery is cheap per block but bounded in effect, and — like the withholding-commons bribe (pay coalitions to abstain, untouched by any inclusion rule) — reduces to the bounded, self-punishing griefing of [§6.5](#s6-5).
|
||||
|
||||
<a id="s6-10"></a>
|
||||
### 6.10 Fork rate and reorg depth: keep reorganisations shallow
|
||||
|
||||
**The question.** How often does the chain fork, how *deep* can a reorganisation (reorg) go, and which parameters keep it shallow? A deep reorg is expensive — it discards confirmed blocks and forces every node to rewind — so a design that minimises reorg depth is safer and cheaper to run.
|
||||
|
||||
**Why it matters.** The other sections optimise the *estimate*; this one optimises the *chain structure* the estimate rides on. A parameter choice that keeps `D̂` accurate but lets forks run deep would still be a bad choice.
|
||||
|
||||
**What we measure.** The engine records, every epoch, the **fork rate** (share of produced blocks orphaned off the canonical chain) and the **max reorg depth** (the deepest orphan branch — the confirmed blocks a node adopting it would have to discard). For the adversarial case we assume a coalition holding **0 %, 10 %, 20 %, or 30 %** of stake running a **private-chain deepest-reorg strategy**: it mines a hidden chain and releases it to override the public chain by the largest margin it can (the standard longest-chain attack, tuned for depth rather than revenue). Its reorg-depth tail is the Nakamoto gambler's-ruin `P(depth ≥ d) = (α_eff/(1−α_eff))^d`, coupled to the parameters through the *effective* adversary share `α_eff = α/(α + (1−α)(1−o))`, where `o` is the honest fork rate — because orphaned honest blocks do not extend the public chain, so more honest forking hands the attacker more relative power.
|
||||
|
||||
**Result 1 — delay drives fork depth, and uncles keep it shallow** (`fig28`, honest lines). Even with no adversary, the honest max reorg depth grows with Blend delay (2 blocks at δ = 2 s → 4 at δ = 32 s with U = 2). Without uncles it is far worse: the deflated estimate ([§3.2](tsi-report-2-accuracy-and-design.md#s3-2)) makes the network over-produce blocks, which fork among themselves — honest max depth reaches **17** at δ = 32 s, U = 0, versus **4** at U = 2. So uncles reduce reorg depth twice over: by keeping the block rate at `f` (no overproduction) and by keeping the operating point recoverable.
|
||||
|
||||
**Result 2 — adversary depth grows with stake and with forking, and crosses a cliff** (`fig27`, `fig28`). At the recommended operating point (ρ < 1, δ ≈ 8 s, U = 2, honest fork rate ≈ 0.35) a 10 % coalition needs depth ~4 to reach 0.1 % probability, 20 % needs ~8, 30 % needs ~17 — reorgs of a handful of blocks. But heavier delay raises the honest fork rate, and at **δ = 32 s with 30 % stake the effective share `α_eff` crosses 0.5** — reorgs become **unbounded**: fork-induced orphaning has amplified a minority into an effective majority. Keeping ρ < 1 (few honest forks) keeps even a 30 % adversary bounded and shallow.
|
||||
|
||||
**Result 3 — a confirmation-depth rule for finality.** Turning the tail into a wait-time: at the recommended operating point (`o ≈ 0.35`), to hold `P(reorg ≥ d) < 10⁻³` a receiver waits **d ≈ 4 blocks (~2 min) against a 10 % adversary, 8 (~4 min) against 20 %, 17 (~8.5 min) against 30 %**; for a `10⁻⁶` guarantee, roughly double those (8 / 15 / 34 blocks). These are the confirmation depths for probabilistic settlement. The protocol's *deterministic* finality at `k = 2160` blocks is vastly deeper than any of these, so the finality-driven `k` ([§3.1](tsi-report-2-accuracy-and-design.md#s3-1)) is consistent with — and far more conservative than — the reorg tail; the confirmation-depth rule matters only for parties wanting faster-than-final soft settlement.
|
||||
|
||||
**Design consequence.** The reorg-depth lever is the *operating point*, not the uncle cap: `U`/`W` fix what the estimate *counts*, but they reduce reorg depth only indirectly, by preventing the U = 0 overproduction spiral. Keep `ρ < 1` with margin — the same rule that keeps the estimate accurate ([§3.3](tsi-report-2-accuracy-and-design.md#s3-3)) and the grinding lever weak ([§6.3](#s6-3)) also keeps reorgs shallow and, crucially, keeps a large minority below the effective-majority cliff. The dependence on network size and degree is the [§3.7](tsi-report-2-accuracy-and-design.md#s3-7) story read through the fork rate: a bigger or sparser network forks more, raising `α_eff`, so at the largest scales the shallow-reorg guarantee leans on the same `degree ≥ 6` recommendation that keeps `ρ` bounded (fork rate ≈ 0.35 at N = 1 000 degree 6, rising with N and falling with degree; `scripts/reorg_depth.py --measure-scale`).
|
||||
|
||||
**Modelling note.** The private-chain race is a global-chain model like [§6.6](#s6-6)'s selfish mining, coupled to the engine only through the honest fork rate `o`; the honest fork/depth statistics are measured directly in the per-node engine. One caveat on `o`: it is the fork rate measured with **no adversary present**, transplanted into `α_eff` as a proxy for the public chain's forking during the attack. A pure private-chain attacker withholds its blocks until the reveal, so while it mines, the public chain is essentially honest and the no-adversary `o` is a close and slightly *conservative* proxy — an attack that lowers public forking would lower `α_eff` and shrink the tail. An attack that instead deliberately raises public forking, e.g. by strategically releasing blocks to split honest miners, would raise the attack-conditioned `o` above the honest baseline and is not captured here; the single-coalition bound is best-case-for-the-defender in that respect, [§8.3](tsi-report-1-overview-and-recommendations.md#s8-3) item 9.
|
||||
|
||||

|
||||
|
||||

|
||||
|
||||
<a id="s6-11"></a>
|
||||
### 6.11 Organic stake churn and the wall-clock cadence
|
||||
|
||||
**The question.** TSI's control loop is *slow*: at k = 2160 an epoch is `10·⌊k/f⌋` = 648 000 slots ≈ **7.5 days**, the measurement window is `6·⌊k/f⌋` ≈ **4.5 days**, and it sits a full epoch back — so `D̂` reflects stake from **~1–2 weeks ago** and updates only weekly. Real participation is not static over that timescale: it cycles daily/weekly, ramps as stakers join, and occasionally drops sharply. Does the lottery mis-fire on 1–2-week-stale stake?
|
||||
|
||||
**Why it matters.** If active stake moves faster than the estimator tracks it, the realised block rate drifts off target (over-production when stake fell, under when it rose) exactly as the [§3.2](tsi-report-2-accuracy-and-design.md#s3-2) deflation does — the honest analogue of the withholding lever ([§6.5](#s6-5)).
|
||||
|
||||
**Result: TSI tracks organic churn with a one-epoch lag, and consensus never breaks** (`fig29`). We drove the active honest stake through a 30 %-amplitude sine, a linear ramp, and a sudden step — each over a period of 4 epochs (≈ 30 days at the production cadence, i.e. a *monthly*-scale swing) — and measured `D̂` against the *current* active stake. The [§6.5](#s6-5) EMA law holds for honest churn as it did for adversarial: the measured estimate follows the active stake within the epoch it is measured, so `D̂/D_total` tracks the active-stake fraction (dashed, `fig29`) and the churn-corrected `D̂/D_active` stays near 1 (means: sine 1.001, step 1.003, ramp 1.002; worst replicate-averaged epoch ≈ 0.98, worst single replicate ≈ 0.93 in the transient). At the sudden step down, `D̂/D_total` settles onto the new active fraction the *same* epoch and the corrected accuracy never leaves ~1 — the one-epoch lag lives in the *lottery*, which spends the transition epoch using the previous epoch's (higher) `D̂` before catching up: a brief rate overshoot that heals in one epoch, not a dip in the estimate. `range_ratio ≡ 0` throughout — churn shifts the common estimate, it does not split it. In wall-clock terms the lag is ~one epoch ≈ 7.5 days, so a monthly participation swing (period ≈ 4 epochs) is tracked with roughly a quarter-cycle phase error and a bounded amplitude error, and slower drifts (quarterly staking growth) essentially exactly. A genuinely *sub-epoch* swing — daily or weekly, faster than the 7.5-day epoch and the 4.5-day window — is not tracked as a cycle at all: the once-per-epoch measurement averages it out into a small bounded rate wobble (the estimator's Nyquist limit).
|
||||
|
||||
**Design consequence.** The estimator is fit for organic churn *because* `β = 1` ([§8](tsi-report-1-overview-and-recommendations.md#s8) row 5): a slower learning rate would lengthen the lag and let a monthly cycle bias the rate more. The residual is a bounded rate wobble (a few percent for a 30 % monthly swing), never a consensus or accuracy failure. A protocol needing to *track* faster-than-monthly stake dynamics as a cycle (rather than average them out) would want a shorter epoch (smaller `k/f`), trading finality depth for tracking speed — the one place the finality-driven `k` and the tracking cadence trade off.
|
||||
|
||||

|
||||
|
||||
---
|
||||
@ -1,137 +0,0 @@
|
||||
# Total-Stake-Inference parameter selection — Reproducibility and appendices
|
||||
|
||||
*Per-node network simulation of Cryptarchia Total Stake Inference (TSI). Simulator: `tsi-sim-pernode`. All runs at the true security parameter **k = 2160** unless noted; latency is in slots and **1 slot = 1 s**.*
|
||||
|
||||
*[Part 1 — Overview & recommendations](tsi-report-1-overview-and-recommendations.md) · [Part 2 — Accuracy & design](tsi-report-2-accuracy-and-design.md) · [Part 3 — Robustness & incentives](tsi-report-3-robustness-and-incentives.md) · [Part 4 — Reproducibility & appendices](tsi-report-4-reproducibility-and-appendices.md) · [Index](README.md)*
|
||||
|
||||
*Sections live across the set: [§1](tsi-report-1-overview-and-recommendations.md#s1)/[§7](tsi-report-1-overview-and-recommendations.md#s7)/[§8](tsi-report-1-overview-and-recommendations.md#s8) in Part 1, [§2](tsi-report-2-accuracy-and-design.md#s2)–[§5](tsi-report-2-accuracy-and-design.md#s5) in Part 2, [§6](tsi-report-3-robustness-and-incentives.md#s6) in Part 3, [§9](#s9) and Appendices A–C in Part 4.*
|
||||
|
||||
---
|
||||
|
||||
<a id="s9"></a>
|
||||
## 9. Reproducibility
|
||||
|
||||
Sweep studies are committed configs, run with `make <name>` (writes a dated `runs/` folder with results + figures); standalone studies are scripts. In section order:
|
||||
|
||||
| study | config / script | § |
|
||||
|---|---|---|
|
||||
| full-scale confirmation (N, uncle recovery) | `configs/fullscale.yaml` (as committed: `n_nodes: [5000, 10000]`; the N = 1 000/2 000 rows come from earlier runs of the same config with `n_nodes` edited) | 3.1, 3.2 |
|
||||
| bootstrap at full scale (k = 2160, Blend, N = 1 000/5 000, U ∈ {0, 2}) | `scripts/bootstrap_dynamics.py` (`runs/bootstrap_fullscale`) | 3.2 |
|
||||
| one-uncle breakdown (hops × delay, N = 1 000/2 000) | `configs/blend-hops-delay.yaml`; fig3 by `scripts/hops_delay_grid.py` | 3.3 |
|
||||
| ρ-boundary deficit sweep (`1 − D̂/D` vs ρ per U; hops=3, N=1000, 20 reps, k=256) | `configs/rho-boundary.yaml`; fig26 by `scripts/rho_boundary_analysis.py` | 3.3 |
|
||||
| relative stake vs delay (D̂/D vs D_vis, per U) | `scripts/stake_vs_delay.py` | 3.3 |
|
||||
| uncle-window sufficiency (W × delay, N = 1 000/2 000) | `configs/uncle-window.yaml` | 3.4 |
|
||||
| joint (W, U) region (N = 1 000/2 000) | `configs/window-uncles.yaml` | 3.5 |
|
||||
| block rate | `configs/block-rate.yaml` | 3.6 |
|
||||
| heterogeneous-start (no re-convergence; N = 400, k = 256) | `configs/default.yaml` (`init_dest: heterogeneous`, `init_spread: 0.5`) | [Appendix C](#sC) |
|
||||
| N-scaling of the one-uncle boundary (N = 1k–32k, cases a/b; k = 256) | `configs/nscaling-{a,b}.yaml`, `configs/nscaling32-{a,b}.yaml`; analysis + figures by `scripts/nscaling_analysis.py` | 3.7 |
|
||||
| exact large-N topology probe (`ℓ_mean` to N = 10⁶, degrees 4/6/8) | `scripts/topology_probe.py` (`runs/topology_probe.parquet`) | 3.7 |
|
||||
| link-latency shape sensitivity (exp vs geo at equal mean) | `configs/expdist.yaml` (baseline cells: the nscaling-a run) | 2 |
|
||||
| stake-tail sensitivity (Pareto 1.33 vs 1.16) | `configs/pareto133.yaml` (baseline cells: the nscaling-a run) | 2 |
|
||||
| window sufficiency at scale + W-as-buffer (N = 1 000/10 000, W ≤ 600) | `configs/window-scale.yaml`; fig25 by `scripts/window_scale_analysis.py` | 3.4 |
|
||||
| fork rate + reorg depth vs delay/adversary stake | `scripts/reorg_depth.py` (`--measure`; `--measure-scale` for fork rate vs N/degree; `src/tsi_sim/reorg.py`); fig27, fig28 | 6.10 |
|
||||
| organic stake churn (sine/ramp/step) | `scripts/churn.py` (`churn_amp`/`churn_period`/`churn_mode` fields); fig29 | 6.11 |
|
||||
| per-node clock skew (bounded consensus cost) | `scripts/clock_skew.py` (`clock_skew_max` field) | 6.1 |
|
||||
| capstone: recommended config end-to-end + 30 % adversary | `scripts/capstone.py` (`runs/capstone.parquet`) | 8.4 |
|
||||
| jitter / consensus | `jitter_mean` > 0 + `windowed_fork_choice: false` (exact oracle); data in `runs/jitter_grid/results.parquet` | 6.1 |
|
||||
| load-feedback fixed point | analysis of the U=0 sweep + `theory.expected_ratio` | 6.2 |
|
||||
| adversarial grinding (uncle suppression) | `scripts/adversary_grid.py` (`adversary_frac` + `adversary_strategy: suppress`); data in `runs/adversary_grid/suppress.parquet` | 6.3 |
|
||||
| block withholding | `scripts/adversary_grid.py` (`adversary_strategy: withhold`); data in `runs/adversary_grid/withhold.parquet` | 6.4 |
|
||||
| dynamic withhold-rejoin grinding | `scripts/dynamic_withhold.py` (`adversary_period`, `adversary_withhold_epochs`) | 6.5 |
|
||||
| selfish / private-chain withholding | `scripts/selfish_mining.py` (`src/tsi_sim/selfish.py`) | 6.6 |
|
||||
| optimal selfish (SSZ MDP) + uncle rewards | `scripts/selfish_rewards.py` (`src/tsi_sim/selfish_mdp.py`, `RewardParams`) | 6.6, 6.7 |
|
||||
| soft uncle inclusion (reward share vs emergent `p_ref`) | `scripts/reward_mandate.py` | 6.8, 6.9 |
|
||||
| U = 0 fluctuation series (zero delay, k ∈ {256, 1024, 2160}) | `scripts/appendix_fluct.py --run` (`runs/fluctuation_u0.parquet`) | [Appendix B](#sB) |
|
||||
| CI smoke grid + analytic sanity checks | `configs/smoke.yaml`; `scripts/verify.py` (`make verify`) — validation only, no figures | — |
|
||||
|
||||
All studies were **re-run on 2026-07-23/24 with the corrected slot-counting mechanism** ([§2.1](tsi-report-2-accuracy-and-design.md#s2-1)) and the early-stop optimisation; the resilient batch is `scripts/run_all_reruns.sh` (per-step log in `runs/rerun_status.log`). Canonical run directories (latest): fullscale N=5000/10000 = `2026-07-24_094519_fullscale`; fullscale N=1000/2000 = `2026-07-23_171803_fullscale-small`; uncle-window = `2026-07-24_001456`; window-uncles = `2026-07-24_014240`; block-rate = `2026-07-24_043943`; blend-hops-delay = `2026-07-24_064052`; window-scale = `2026-07-24_085234`; latency-shape = `2026-07-24_090014_expdist`; stake-tail = `2026-07-24_090044_pareto133`; heterogeneous-start = `2026-07-24_090114_default`; N-scaling = the `nscaling-{a,b}` + `nscaling32-{a,b}` runs; adversary grids = `runs/adversary_grid/`; jitter = `runs/jitter_grid/`; bootstrap = `runs/bootstrap_fullscale/`; fluctuation = `runs/fluctuation_u0.parquet`; fork-rate = `runs/fork_rate_vs_delay.parquet`; ρ-boundary = `2026-07-27_195627_rho-boundary`.
|
||||
|
||||
Figures are in `report-figures/` (`fig1`–`fig29`, plus [Appendix B](#sB)'s `figB1`–`figB2`; numbering is generation order, not order of appearance). Committed generators: `fig1` (bootstrap, k=2160) by `scripts/bootstrap_dynamics.py`; `fig2`,`fig4`,`fig5`,`fig17`–`fig22` by `scripts/regenerate_extra_figs.py` from the latest sweeps (`fig3` hops×delay×U grid by `scripts/hops_delay_grid.py`, `fig6` from the block-rate run via `figures_pernode.py`; `fig26` deficit-vs-ρ by `scripts/rho_boundary_analysis.py`) (fullscale-derived `fig17`–`fig20` from the N = 10 000 run); `fig8`,`fig9` by `scripts/adversary_figs.py` from `runs/adversary_grid/`; `fig10`–`fig12` by `scripts/dynamic_withhold.py`; `fig13`–`fig15` by `scripts/selfish_mining.py`/`selfish_rewards.py`/`reward_mandate.py`; `fig16` by `scripts/stake_vs_delay.py`; `fig23`–`fig24` by `scripts/nscaling_analysis.py`; `fig25` by `scripts/window_scale_analysis.py`; `fig27`–`fig28` by `scripts/reorg_depth.py` (fork rates via `--measure`; private-chain model `src/tsi_sim/reorg.py`); `fig29` by `scripts/churn.py`; `figB1`–`figB2` by `scripts/appendix_fluct.py`. `fig7` (feedback fixed-point) is an analytic overlay. Every figure type the per-node simulator generates appears in this report, and the fork-rate/reorg-depth study closes the previous reproducibility gap for the adversarial figures (`fig8`,`fig9` now have committed generators from `runs/adversary_grid/`).
|
||||
|
||||
---
|
||||
|
||||
|
||||
<a id="sA"></a>
|
||||
## Appendix A — The residual ~1 % offset: on-chain rounding of `f`
|
||||
|
||||
**The question.** With correct slot-counting ([§2.1](tsi-report-2-accuracy-and-design.md#s2-1)) the recovered estimate settles at `D`. Why does the *deployed* estimator carry a small residual ~1 % above `D`, and how is it removed?
|
||||
|
||||
**The multi-winner slot, counted once.** The lottery gives node `i` an independent win chance `φ = 1 − (1−f)^{w_i/D̂}` (`w_i` = node i's stake). At `D̂ = D` the probability a slot has *at least one* winner is `1 − ∏(1−φ_i) = 1 − (1−f)^{Σw_i/D} = f` — slots activate at exactly the target rate. A slot can have *several* winners (the expected count is `−ln(1−f) = f·c(f) > f`, so busy slots are ~1.7 % more blocks than slots), and the surplus co-winners are always orphaned — even at zero delay. TSI counts **occupied slots**, so it counts such a slot **once**: the co-winner adds no count whether it is canonical or a referenced uncle. Holding the occupied-slot density at `f` therefore settles the estimate at exactly `D`, with no `c(f)` ceiling. (Counting uncle *blocks* instead double-counts these co-winner slots and inflates the equilibrium to `c(f)·D ≈ 1.017·D`; that is the deployed-spec bug and fix — see [§2.1](tsi-report-2-accuracy-and-design.md#s2-1) and [§8.5](tsi-report-1-overview-and-recommendations.md#s8-5).)
|
||||
|
||||
**Verified in isolation.** The zero-delay isolation series of [§3.2](tsi-report-2-accuracy-and-design.md#s3-2) (full mesh, latency 0 — same-slot co-winners the only orphans) confirms this: `U = 2` lands at `1.000 ± 0.007`, not at a ceiling, because the co-winner slots are counted once. Double counting is structurally impossible: uncle slots are de-duplicated against the canonical slots and against each other, and a slot can never be both canonical and a counted uncle.
|
||||
|
||||
**The one residual offset — and it is optional.** The spec stores the target rate as an on-chain integer at three-decimal precision: `f_p = ⌊1000·f⌋/1000 = 0.033`, not `1/30 = 0.03333…`. Driving the density to `f_p` rather than to `f` leaves the estimate high by the fixed factor `f/f_p ≈ 1.010` — a ~1 % over-estimate, common to every node (so fairness is untouched) but an absolute ~1 % under-delivery of win probability and a ~1 % slow canonical pace. It grows mildly with the block rate. It is removed by carrying `f` at higher precision. **This report's estimator uses exact `f` (the analysis-faithful default, `fixed_point=False`), so `f/f_eff = 1.000` and the residual is 0** — finer than the recommended 10⁻⁶ spec bump (`33333`→`0.033333`, `f/f_p = 1.00001`, residual < 10⁻⁵); the current spec still uses 10⁻³ and should adopt the bump ([§8](tsi-report-1-overview-and-recommendations.md#s8) row 14). This `f/f_p` factor is the *only* systematic departure from 1 that the counting fix leaves, and it is the sole reason a deployed `D̂/D` reads ≈ 1.01 rather than 1.00.
|
||||
|
||||
---
|
||||
|
||||
|
||||
<a id="sB"></a>
|
||||
## Appendix B — the per-epoch sampling-noise floor (and why one uncle restores it)
|
||||
|
||||
**The question.** The equilibrium is bounded by 1 ([§2.2](tsi-report-2-accuracy-and-design.md#s2-2)), but individual epochs read 1.003 or 0.994. How large is that per-epoch noise, what sets it, and does it bias any single accuracy number in this report?
|
||||
|
||||
**The answer up front.** The U = 0 estimate is an *unbiased but noisy* measurement: its delay-free equilibrium is exactly 1 ([§2.2](tsi-report-2-accuracy-and-design.md#s2-2)) and its per-epoch spread is pure sampling noise of the finite measurement window — `σ ≈ √((1−f)/(f·T))`, about **±0.9 % at the production window** (k = 2160). To show this cleanly we simulate the **delay-free limit directly** (full mesh, zero latency — no orphan loss at all, so *only* the noise remains); a realistic sub-slot gossip series then confirms the same magnitude, and the delay progression (§B.3) shows how real orphan loss turns that noise one-sided, pinning the estimate **below** 1 — the bounded-by-1 deficit of [§3.2](tsi-report-2-accuracy-and-design.md#s3-2). Every number below is measured; nothing is asserted.
|
||||
|
||||
<a id="sB-1"></a>
|
||||
### B.1 The mechanism: a stochastic controller passes its measurement noise through
|
||||
|
||||
At U = 0 with negligible delay the counted density is the *active-slot* rate, which the lottery calibrates to exactly `f` at `D̂ = D` (the identity of A.1) — so the fixed point is exactly 1, with no ceiling. But the measured density `m/T` is a random variable: the window contains only `f·T` expected blocks, so one epoch's measurement carries relative noise `σ ≈ √((1−f)/(f·T))`. At the deployed learning rate `β = 1` the update `D̂ ← D̂·(1 − β(f − m/T)/f)` passes that noise straight into the estimate: each epoch's `D̂/D` is `≈ 1 + ε` with `ε` the window's sampling error. The estimate is therefore *expected* to read 1.003 or 0.994 in individual epochs — those are not anomalies but the noise floor itself.
|
||||
|
||||
<a id="sB-2"></a>
|
||||
### B.2 Measured: the ±0.9 % noise floor, shrinking as 1/√T
|
||||
|
||||
A dedicated zero-delay series (full mesh, link latency `L = 0`, U = 0, equal stakes, 4 × 120 epochs per k) isolates the fluctuation with no orphan loss at all:
|
||||
|
||||
| k (window `T = 6⌊k/f⌋`) | mean `D̂/D` | per-epoch σ (measured) | σ theory | P(`D̂/D` > 1) | largest excursion |
|
||||
|---|---|---|---|---|---|
|
||||
| 256 | 0.99914 | 0.0254 | 0.0251 | 0.50 | **1.066** |
|
||||
| 1024 | 1.00079 | 0.0126 | 0.0125 | 0.54 | 1.035 |
|
||||
| 2160 | 1.00006 | 0.0088 | 0.0086 | 0.53 | 1.024 |
|
||||
|
||||
The mean is pinned at 1 to within ±0.001 at every window size — the estimator is unbiased at the bound — while the spread follows the `1/√T` law to within a few percent. `figB1` shows the per-epoch trace at k = 2160 on a per-mil axis: the clean zero-delay series and the realistic 0.1-slot direct-gossip series wander together inside the predicted ±σ band.
|
||||
|
||||

|
||||
|
||||
<a id="sB-3"></a>
|
||||
### B.3 Delay converts the fluctuation into a one-sided under-count
|
||||
|
||||
The committed full-scale N = 1 000 (k = 2160) data shows how real delay changes the picture (`figB2`, middle): at 0.1-slot links the U = 0 estimate reads `1.0004 ± 0.0087` with `P(>1) = 50 %`; at 0.2 slots the mean slips to 0.997 and `P(>1)` to 39 %; by 0.5-slot links orphan loss dominates (mean 0.963, **never** above 1 in 300 epochs); and under Blend U = 0 sits far below (mean 0.74, maximum 0.997). So "fluctuates around 1" is the *delay-free limit* of the U = 0 estimator; in the deployment regime (Blend) the U = 0 estimate is one-sidedly low, and the fluctuation instead rides on the recovered value; blend at U = 1 crosses 1.0 in ~50 % of epochs, the same symmetric noise around the recovered fixed point.
|
||||
|
||||
**Delay also changes the *size* of the fluctuation — but not gradually** (`figB2`, right). Under direct gossip the per-epoch spread stays at the sampling floor at every link delay (σ = 0.007–0.009 from 0.1 to a full slot): mild orphan loss shifts the *mean*, not the noise. Under Blend at U = 0 the fluctuation **explodes to σ ≈ 0.14–0.15 — about 17× the floor**: with a third of blocks orphaning, the counted density is decided by fork races, and the [§6.2](tsi-report-3-robustness-and-incentives.md#s6-2) load feedback (a deflated `D̂` raises the raw proposal rate, which raises orphaning again) amplifies that race noise into ±14 % per-epoch swings. **A single uncle restores not just the mean but the noise floor itself**: at U = 1 the per-epoch σ returns to 0.009 in every cell, Blend included — the uncle mechanism stabilises the estimator's *variance* as well as its *bias*, a second, independent reason to provision `U` correctly.
|
||||
|
||||

|
||||
|
||||
<a id="sB-4"></a>
|
||||
### B.4 What precision is meaningful
|
||||
|
||||
A reading like `1.001` is well inside one epoch's noise (±0.009 at k = 2160) — real and expected. A reading like `1.000001` is **not resolvable**: it is four orders of magnitude below the per-epoch noise floor, and even averaging would need ~10⁷ epochs to distinguish it from 1. The meaningful statements at the production window are: the U = 0 delay-free estimate is unbiased at 1 with ±0.9 % per-epoch noise; per-epoch tables in this report are therefore quoted to three decimals, and equilibrium values are tail-averages over ≥ 15 epochs × replicates (±0.1–0.2 % standard error). Any accuracy differences smaller than that are noise, not signal.
|
||||
|
||||
---
|
||||
|
||||
|
||||
<a id="sC"></a>
|
||||
## Appendix C — Consensus properties in detail
|
||||
|
||||
*Supporting detail for [§3.1](tsi-report-2-accuracy-and-design.md#s3-1): the per-epoch traces, the tip-agreement contrast, and the one caveat — agreement is inherited from initialization, never rebuilt.*
|
||||
|
||||
<a id="sC-1"></a>
|
||||
### C.1 Per-epoch traces at the largest scale
|
||||
|
||||
`fig17` shows the per-epoch picture **at the largest tested scale, N = 10 000** (30 epochs, 6 replicates): the per-node `D̂/D` spread (full range and interquartile range, top) sits at 0 for the whole run while node agreement on the finalized window prefix (bottom) stays at 1.000 — the consensus result holds unchanged from `N = 1 000` up to `N = 10 000`.
|
||||
|
||||

|
||||
|
||||
The one metric that is *not* exactly 1 is **current-tip** agreement (`fig18`, also N = 10 000): nodes share the live tip most of the time (≈ 0.985–0.994 typical), dipping to ~0.985 in the worst cell — a small, transient tip churn that never reaches the deeply-buried density window, which is why `D̂` agreement is exact regardless.
|
||||
|
||||

|
||||
|
||||
<a id="sC-2"></a>
|
||||
### C.2 Consensus rests on common initialization — there is no active re-convergence
|
||||
|
||||
The [§3.1](tsi-report-2-accuracy-and-design.md#s3-1) consensus (spread → 0) holds because every node starts at the *same* genesis `D̂` and applies the *same* update to the *same* finalized density. To test whether TSI would *re-heal* a divergence that arose anyway, we seeded the nodes with a **heterogeneous** initial `D̂` (a ±50 % per-node spread around a genesis guess set at `0.5×` the true stake; N = 400 / scaled k = 256, equal stakes, regular topology) and watched it evolve.
|
||||
|
||||
The inter-node disagreement does **not** contract (`fig21`): the strictly conserved quantity is the **ratio between the highest and lowest node estimate**, `max/min ≈ 3.0`, which stays flat to three digits across all epochs. The absolute spread `range(D̂/D)` is *not* invariant — because the mean starts low (`0.5×`) it is scaled **up** in the first epoch (a common factor ≈ 2.15 rescales mean *and* spread together, so the spread jumps from its injected ≈ 0.5 to ≈ 1.07 within epoch 0) and then holds flat near ≈ 1.09 — so the disagreement is rescaled, never healed. (`fig21` plots the *post*-update spread, which already sits at ≈ 1.07 by epoch 0 and stays flat; the ≈ 0.5 → ≈ 1.07 rescale happens inside the first epoch and is not itself drawn.) The reason is structural — the recursion `D̂ ← D̂·(1 − β(f − m/T)/f)` (`β` = the TSI learning rate, `m/T` = the measured density over the window `T`; [§6.5](tsi-report-3-robustness-and-incentives.md#s6-5)) applies a *common* multiplicative factor (all nodes read the same global `m`), so the ratio between any two nodes' estimates is invariant; there is no inter-node coupling to pull them together.
|
||||
|
||||
**Caveat:** TSI's consensus is *maintenance*, not *repair* — it keeps identically-initialized nodes identical, but a coalition that could inject persistent per-node `D̂` disagreement (e.g. via a genesis/clock exploit) would not be corrected by the estimator itself. In the honest protocol this never arises (genesis `D` is a shared constant), so [§3.1](tsi-report-2-accuracy-and-design.md#s3-1) stands; the point is that the consensus is an *initialization* property, not a restoring force.
|
||||
|
||||

|
||||
@ -38,6 +38,16 @@
|
||||
- **Per-node views:** one global block tree plus an `(N × n_blocks)` **arrival matrix** `A`;
|
||||
each node builds on / measures density over the blocks that have arrived at it. Uncle refs
|
||||
are **baked at production** from the producer's view (faithful — immutable once adopted).
|
||||
- **Uncle model (`uncle_model`, CLI `--old`):** the default **countable** model implements the
|
||||
spec's counting-only rules (cryptarchia-v1-protocol.md): only the **first block of a fork**
|
||||
(parent on the producer's chain) is referenceable/countable, the window is **derived** as
|
||||
`w_u = window_absorption / f` slots (`W` expected block-intervals, default `W = 10` → 300
|
||||
slots, bounded `W ≤ 0.6·k`), selection skips slots already occupied on the producer's chain
|
||||
and picks one uncle per slot, and the measurement pass re-checks every rule per reference
|
||||
(rejections tallied as `deep_ref_share`). Passing `--old` to `tsi-sweep`/`tsi-verify` runs
|
||||
the pre-redesign model unchanged — window = `uncle_window` slots, any-depth orphans
|
||||
referenceable, every baked reference counted — and **bit-reproduces historical runs** (the
|
||||
old model's RNG key is byte-identical to the pre-`uncle_model` key).
|
||||
- **Metrics:** per-node `D_est` spread (`range`, `IQR`), canonical-chain **agreement**
|
||||
(window prefix vs current tip), mean accuracy, and — with `init_dest=heterogeneous` —
|
||||
transient re-convergence.
|
||||
@ -145,6 +155,13 @@ src/tsi_sim/ constants config rng stake lottery topology blocktree(+build_tree
|
||||
uncles(+select_uncles_at_production) tsi(+update_D_vec) epoch engine metrics
|
||||
theory verify plotting/{style, figures_pernode, make_figures}
|
||||
configs/ smoke.yaml default.yaml fullscale.yaml
|
||||
countable-vs-old.yaml absorption-window.yaml (countable-model studies)
|
||||
fine-delay.yaml (delay 1-5 at 40 replicates: the design band, high precision)
|
||||
tests/ test_{pernode,config,rng,lottery,blocktree,uncles,tsi_counting,stake,
|
||||
theory,latency,theory_convergence}.py
|
||||
theory,latency,theory_convergence,countable_counting,
|
||||
countable_selfish,...}.py
|
||||
scripts/ plot_countable_vs_old.py (countable-vs-unrestricted comparison figures)
|
||||
plot_fine_delay.py (design-band accuracy + model gap with 95% CIs)
|
||||
countable_selfish.py (first-fork ceiling under the selfish MDP; fig36)
|
||||
adversary_variants.py (whale/jitter/slow-beta variants + the withhold-load sweep)
|
||||
```
|
||||
|
||||
@ -0,0 +1,26 @@
|
||||
# Countable model: sweep the window absorption parameter W (w_u = W/f slots derived) at
|
||||
# U=1 against the Blend mixing delay. The window-miss contribution to non-recovery is
|
||||
# (1-f)^(W/f) ~ e^-W (theory.window_miss_prob): recovery should saturate within a few
|
||||
# expected block-intervals, with the residual set by the delay (orphans spread wider than
|
||||
# the window) and by the first-fork restriction. The countable counterpart of the old
|
||||
# model's uncle-window.yaml (which swept uncle_window in raw slots; run that with --old).
|
||||
# Latency is in SLOTS (1 slot = 1 s).
|
||||
n_nodes: [1000] # network size
|
||||
stake_dist: [pareto] # heavy-tailed (realistic) stake distribution
|
||||
topology: [blend] # Blend mixnet (delay stresses the window)
|
||||
degree: [6] # peering degree of the d-regular graph
|
||||
link_latency_mean: [0.5] # natural geographic transport (sub-slot)
|
||||
link_latency_dist: [geo] # real-world geographic band mixture
|
||||
blend_hops: [3] # fixed hop count; delay is the swept knob
|
||||
blend_delay_max: [8.0, 16.0, 32.0] # max per-relay mixing delay (slots)
|
||||
window_absorption: [1, 2, 3, 5, 7, 10] # W: window in expected block-intervals
|
||||
max_uncles: [1] # FIXED at one uncle (the question is about W)
|
||||
uncle_strategy: [oldest] # spec selection: oldest-first fill
|
||||
init_dest: [common] # per-node initial D_est from agreement
|
||||
replicates: 5 # independent RNG replicates per grid cell
|
||||
base: # per-run settings shared by every cell
|
||||
k: 2160 # true security parameter
|
||||
epochs: 20 # equilibrium within ~2 epochs; burn 50%
|
||||
f: 0.03333333333333333 # slot activation coefficient (1/30)
|
||||
genesis_d_factor: 0.5 # start near true stake (cheap epoch 0)
|
||||
early_stop: true
|
||||
@ -0,0 +1,36 @@
|
||||
# PAIRED overload-grid comparison: countable vs unrestricted referencing across the full
|
||||
# delay range, delta_max 4/8/16/32 (rho ~ 0.36 to 1.76).
|
||||
#
|
||||
# Supersedes countable-vs-old.yaml, which was 5 replicates with INDEPENDENT streams per arm.
|
||||
# That design pays the between-run variance twice; at delta_max = 32 its replicate spread
|
||||
# reaches sd 0.15, which left the U=1 cells unresolvable (t ~ 0.5) even though the U=2/U=4
|
||||
# effects there are enormous (t = 6.5 and 41). See §3.2a for what pairing bought in the
|
||||
# design band: a median 1.6x SE shrink and an exact-zero negative control.
|
||||
#
|
||||
# paired_streams gives a countable cell and its --old twin the SAME stake draw, peering graph
|
||||
# and lottery outcomes, so the uncle rule is the only difference and each replicate is a
|
||||
# matched pair. Replicates also go 5 -> 20. U=0 stays as the control: with no uncles and
|
||||
# shared streams the two arms must be bit-identical, which is an identity check rather than
|
||||
# a noise check.
|
||||
#
|
||||
# Run TWICE — default (countable) and with --old — same grid.
|
||||
# Latency is in SLOTS (1 slot = 1 s).
|
||||
n_nodes: [1000] # network size
|
||||
stake_dist: [pareto] # heavy-tailed (realistic) stake distribution
|
||||
topology: [blend] # Blend mixnet (the multi-slot fork regime)
|
||||
degree: [6] # peering degree of the d-regular graph
|
||||
link_latency_mean: [0.5] # natural geographic transport (sub-slot)
|
||||
link_latency_dist: [geo] # real-world geographic band mixture
|
||||
blend_hops: [3] # fixed hop count; delay is the swept knob
|
||||
blend_delay_max: [4.0, 8.0, 16.0, 32.0] # max per-relay mixing delay (slots)
|
||||
max_uncles: [0, 1, 2, 4] # U: 0 baseline, then the recovery levers
|
||||
uncle_strategy: [oldest] # spec selection: oldest-first fill
|
||||
init_dest: [common] # per-node initial D_est from agreement
|
||||
replicates: 20 # 4x the unpaired grid, plus pairing
|
||||
base: # per-run settings shared by every cell
|
||||
k: 2160 # true security parameter
|
||||
epochs: 20 # equilibrium within ~2 epochs; burn 50%
|
||||
f: 0.03333333333333333 # slot activation coefficient (1/30)
|
||||
genesis_d_factor: 0.5 # start near true stake (cheap epoch 0)
|
||||
early_stop: true
|
||||
paired_streams: true # common random numbers across the two arms
|
||||
@ -0,0 +1,27 @@
|
||||
# Headline comparison for the countable uncle model (cryptarchia-v1-protocol.md counting
|
||||
# rules) vs the old pre-redesign model: accuracy vs Blend mixing delay at U in {0,1,2,4}.
|
||||
# Run TWICE — default (countable) and with --old — same grid; the countable run yields
|
||||
# q/q_eff for the q_u = q + (1-q) r theory overlay. (deep_ref_share is a drift alarm that
|
||||
# reads 0 on every real run, NOT a measured rejection rate — see measure.py.) The U=0 rows
|
||||
# are the NEGATIVE CONTROL: with no uncles the two models are identical by construction, so
|
||||
# any U=0 gap is pure between-run RNG noise (the models draw different streams by design)
|
||||
# and bounds what a real countable-vs-old effect must exceed.
|
||||
# Latency is in SLOTS (1 slot = 1 s).
|
||||
n_nodes: [1000] # network size
|
||||
stake_dist: [pareto] # heavy-tailed (realistic) stake distribution
|
||||
topology: [blend] # Blend mixnet (the multi-slot fork regime)
|
||||
degree: [6] # peering degree of the d-regular graph
|
||||
link_latency_mean: [0.5] # natural geographic transport (sub-slot)
|
||||
link_latency_dist: [geo] # real-world geographic band mixture
|
||||
blend_hops: [3] # fixed hop count; delay is the swept knob
|
||||
blend_delay_max: [4.0, 8.0, 16.0, 32.0] # max per-relay mixing delay (slots)
|
||||
max_uncles: [0, 1, 2, 4] # U: 0 baseline, then the recovery levers
|
||||
uncle_strategy: [oldest] # spec selection: oldest-first fill
|
||||
init_dest: [common] # per-node initial D_est from agreement
|
||||
replicates: 5 # independent RNG replicates per grid cell
|
||||
base: # per-run settings shared by every cell
|
||||
k: 2160 # true security parameter
|
||||
epochs: 20 # equilibrium within ~2 epochs; burn 50%
|
||||
f: 0.03333333333333333 # slot activation coefficient (1/30)
|
||||
genesis_d_factor: 0.5 # start near true stake (cheap epoch 0)
|
||||
early_stop: true
|
||||
@ -0,0 +1,38 @@
|
||||
# PAIRED high-precision sweep of the low mixing-delay band (the design regime).
|
||||
#
|
||||
# Identical grid to fine-delay.yaml, but with paired_streams: true. The unpaired sweep pays the
|
||||
# between-run variance twice — the two uncle models draw independent stake, graph and lottery
|
||||
# streams — which puts a ~+-0.0015 per-cell floor on the countable-vs-unrestricted comparison,
|
||||
# an order of magnitude above the effect. Only delta_max = 5 resolved there, and only after
|
||||
# pooling across uncle caps.
|
||||
#
|
||||
# paired_streams drops the uncle-model marker from the RNG root, so a countable cell and its
|
||||
# --old twin get the SAME stake draw, the SAME peering graph and the SAME lottery outcomes;
|
||||
# the uncle rule is then the only difference and the per-replicate difference is a PAIRED
|
||||
# observation. The shared variance cancels, so the paired test should resolve the per-cell
|
||||
# effect directly. Trajectories still diverge after epoch 0 through the genuine feedback (a
|
||||
# different counted density changes the next epoch's difficulty) — that is the signal.
|
||||
#
|
||||
# Run TWICE — default (countable) and with --old — same grid. U=0 remains the negative control:
|
||||
# with no uncles AND a shared stream the two arms must now agree EXACTLY, which is a far
|
||||
# stronger check than the unpaired version (there it only had to agree within noise).
|
||||
# Latency is in SLOTS (1 slot = 1 s).
|
||||
n_nodes: [1000] # network size
|
||||
stake_dist: [pareto] # heavy-tailed (realistic) stake distribution
|
||||
topology: [blend] # Blend mixnet (the multi-slot fork regime)
|
||||
degree: [6] # peering degree of the d-regular graph
|
||||
link_latency_mean: [0.5] # natural geographic transport (sub-slot)
|
||||
link_latency_dist: [geo] # real-world geographic band mixture
|
||||
blend_hops: [3] # fixed hop count; delay is the swept knob
|
||||
blend_delay_max: [1.0, 2.0, 3.0, 4.0, 5.0] # THE fine region (integer slots)
|
||||
max_uncles: [0, 1, 2, 4] # U: 0 = negative control, then the levers
|
||||
uncle_strategy: [oldest] # spec selection: oldest-first fill
|
||||
init_dest: [common] # per-node initial D_est from agreement
|
||||
replicates: 40 # matches fine-delay.yaml for a like-for-like read
|
||||
base: # per-run settings shared by every cell
|
||||
k: 2160 # true security parameter
|
||||
epochs: 20 # equilibrium within ~2 epochs; burn 50%
|
||||
f: 0.03333333333333333 # slot activation coefficient (1/30)
|
||||
genesis_d_factor: 0.5 # start near true stake (cheap epoch 0)
|
||||
early_stop: true
|
||||
paired_streams: true # common random numbers across the two arms
|
||||
35
tools/simulators/tsi/tsi-sim-pernode/configs/fine-delay.yaml
Normal file
@ -0,0 +1,35 @@
|
||||
# High-precision sweep of the LOW mixing-delay region — the intended operating band.
|
||||
#
|
||||
# countable-vs-old.yaml samples delay at 4/8/16/32 with 5 replicates, which resolves the
|
||||
# overload regime but leaves the design regime under-measured: at delay <= 8 every
|
||||
# countable-vs-old gap there sits inside the replicate noise, so the report can only say
|
||||
# "no difference detected", not how large a difference is excluded. This sweep spends the
|
||||
# replicates instead of the range — delay 1..5 at 40 replicates — to put a tight CI on the
|
||||
# model difference exactly where the parameters are actually chosen.
|
||||
#
|
||||
# rho = f * D_vis with D_vis = hops*delta_max/2 + (hops+1)*ell_mean spans ~0.25 (delay 1)
|
||||
# to ~0.45 (delay 5), so the whole sweep sits well inside rho < 1 and U >= 1 should hold
|
||||
# the equilibrium at 1.0 under BOTH models. The question is the size of the residual gap.
|
||||
#
|
||||
# Run TWICE — default (countable) and with --old — same grid. U=0 is the negative control:
|
||||
# with no uncles the models are identical by construction, so the U=0 gap measures the
|
||||
# between-run RNG noise floor at this replicate count and bounds what a real effect must
|
||||
# exceed. Latency is in SLOTS (1 slot = 1 s).
|
||||
n_nodes: [1000] # network size
|
||||
stake_dist: [pareto] # heavy-tailed (realistic) stake distribution
|
||||
topology: [blend] # Blend mixnet (the multi-slot fork regime)
|
||||
degree: [6] # peering degree of the d-regular graph
|
||||
link_latency_mean: [0.5] # natural geographic transport (sub-slot)
|
||||
link_latency_dist: [geo] # real-world geographic band mixture
|
||||
blend_hops: [3] # fixed hop count; delay is the swept knob
|
||||
blend_delay_max: [1.0, 2.0, 3.0, 4.0, 5.0] # THE fine region (integer slots)
|
||||
max_uncles: [0, 1, 2, 4] # U: 0 = negative control, then the levers
|
||||
uncle_strategy: [oldest] # spec selection: oldest-first fill
|
||||
init_dest: [common] # per-node initial D_est from agreement
|
||||
replicates: 40 # 8x countable-vs-old.yaml: this is the point
|
||||
base: # per-run settings shared by every cell
|
||||
k: 2160 # true security parameter
|
||||
epochs: 20 # equilibrium within ~2 epochs; burn 50%
|
||||
f: 0.03333333333333333 # slot activation coefficient (1/30)
|
||||
genesis_d_factor: 0.5 # start near true stake (cheap epoch 0)
|
||||
early_stop: true
|
||||
@ -0,0 +1,41 @@
|
||||
# Does the SPEC's uncle-selection rule move the window floor? (§3.4, §8.3 item 11)
|
||||
#
|
||||
# Every window result in the report fills uncle slots OLDEST-FIRST, and the ~7/f floor is derived
|
||||
# from exactly that: an orphan queue drained one per block interval. The spec selects with an
|
||||
# unbiased per-candidate coin instead (uncle_random_p = 0.5). §3.4 argues a coin drains a queued
|
||||
# orphan no faster than oldest-first, so the floor under it can only be the same or higher, and
|
||||
# reads the margined W = 10/f as covering the difference. That argument has never been measured.
|
||||
#
|
||||
# This sweeps the window absorption parameter under BOTH selection rules at the two delays that
|
||||
# bracket the design regime (rho ~ 0.6 and rho ~ 1), and asks one question: does the recovery knee
|
||||
# move, and does W_abs = 10 still clear?
|
||||
#
|
||||
# UNPAIRED by construction: uncle_strategy is part of the RNG identity, so the two arms draw
|
||||
# independent stake, graph and lottery streams — and unlike the countable-vs-old comparison there
|
||||
# is no way to share them, because the coin arm consumes draws the oldest-first arm does not.
|
||||
# Note the §9 negative-control idiom is NOT available here: SweepConfig.expand() collapses U=0
|
||||
# across uncle_strategy (with no uncle slots the two arms would be the identical run), so there is
|
||||
# no U=0 gap to read a noise floor off. Resolution comes from the replicate SEM within each cell
|
||||
# instead, which is why this runs 10 replicates against absorption-window.yaml's 5. U=0 is still
|
||||
# carried as the recovery baseline the knee is measured against.
|
||||
# Latency is in SLOTS (1 slot = 1 s).
|
||||
n_nodes: [1000] # network size
|
||||
stake_dist: [pareto] # heavy-tailed (realistic) stake distribution
|
||||
topology: [blend] # Blend mixnet (delay stresses the window)
|
||||
degree: [6] # peering degree of the d-regular graph
|
||||
link_latency_mean: [0.5] # natural geographic transport (sub-slot)
|
||||
link_latency_dist: [geo] # real-world geographic band mixture
|
||||
blend_hops: [3] # fixed hop count; delay is the swept knob
|
||||
blend_delay_max: [8.0, 16.0] # design point (rho ~ 0.6) and the boundary (rho ~ 1)
|
||||
window_absorption: [1, 2, 3, 5, 7, 10] # W: window in expected block-intervals
|
||||
max_uncles: [0, 1] # 0 = negative control (rule never invoked), 1 = U
|
||||
uncle_strategy: [oldest, random] # THE comparison: FIFO fill vs the spec's coin
|
||||
init_dest: [common] # per-node initial D_est from agreement
|
||||
replicates: 10 # 2x absorption-window.yaml: the arms are unpaired
|
||||
base: # per-run settings shared by every cell
|
||||
k: 2160 # true security parameter
|
||||
epochs: 20 # equilibrium within ~2 epochs; burn 50%
|
||||
f: 0.03333333333333333 # slot activation coefficient (1/30)
|
||||
genesis_d_factor: 0.5 # start near true stake (cheap epoch 0)
|
||||
early_stop: true
|
||||
uncle_random_p: 0.5 # the spec's unbiased coin (only read by `random`)
|
||||
@ -0,0 +1,214 @@
|
||||
"""The residual §6.5-scope adversary variants — REPORT §8.3 item 11.
|
||||
|
||||
Three probes the robustness studies left open, each asking whether a bound reported as
|
||||
best-case-for-the-defender actually moves:
|
||||
|
||||
A. WHALE COALITION — §6.5 flags that a coalition of a few large holders has a "lumpier share
|
||||
statistic" than a random one at the same stake. Both arms hold the same stake fraction
|
||||
(engine._adversary_mask fills whales-first up to the target, so the realised shares match);
|
||||
what differs is the member count, hence the run-to-run spread of the coalition's realised
|
||||
block share. Measured for both levers: uncle suppression (§6.3) and withholding (§6.4).
|
||||
|
||||
B. JITTER > 0 — the dynamic withhold-rejoin results (§6.5) were all run at jitter = 0.
|
||||
§6.1 shows jitter never reaches the finalized density window in the HONEST case; this asks
|
||||
the same of the attacked case. Run in the guaranteed-exact mode (windowed fork choice and
|
||||
arrival pruning off), since those speed-ups are only bit-exact at jitter = 0.
|
||||
|
||||
C. VERY SLOW beta — §6.5 sweeps the estimator gain down to beta = 0.25. "Very slow" beta is
|
||||
listed as untested: with memory ~1/beta epochs, beta = 0.05 remembers ~20 epochs, so a
|
||||
withhold notch should shrink further while the attacker's take stays flat (profitability is
|
||||
beta-independent, §6.5(iii)). This checks that the trend continues rather than turning.
|
||||
|
||||
Run: python scripts/adversary_variants.py (writes runs/adversary_variants_*.parquet)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
from joblib import Parallel, delayed
|
||||
|
||||
from tsi_sim.config import SimConfig
|
||||
from tsi_sim.engine import run_trajectory
|
||||
from tsi_sim.memguard import ArrivalMatrixTooLarge
|
||||
|
||||
HERE = Path(__file__).resolve().parent.parent
|
||||
RUNS = HERE / "runs"
|
||||
RUNS.mkdir(exist_ok=True)
|
||||
|
||||
EPOCHS = 20
|
||||
REPS = 12
|
||||
N_JOBS = 6
|
||||
|
||||
# §6.4's own withhold geometry (blend_delay_max = 4), so the concentration comparison is
|
||||
# like-for-like against the published withhold/suppress numbers rather than at a heavier load.
|
||||
# The heavier point is probed separately by study_withhold_load, where it does something else
|
||||
# entirely — see that function.
|
||||
WHALE_BASE = dict(n_nodes=1000, stake_dist="pareto", topology="blend", degree=6,
|
||||
link_latency_mean=0.5, link_latency_dist="geo", blend_hops=3,
|
||||
blend_delay_max=4.0, max_uncles=2, uncle_window=300, k=256,
|
||||
epochs=EPOCHS, genesis_d_factor=0.5)
|
||||
|
||||
# §6.5 cell geometry: equal stakes so coalition_frac == adversary_frac exactly, light transport
|
||||
# so the dynamic lever is measured on its own rather than through fork noise.
|
||||
DYN_BASE = dict(n_nodes=600, stake_dist="uniform", topology="regular", degree=8,
|
||||
link_latency_mean=0.3, link_latency_dist="geo", max_uncles=2, uncle_window=300,
|
||||
genesis_d_factor=0.5, k=64, adversary_strategy="withhold",
|
||||
adversary_frac=0.3, adversary_period=6, adversary_withhold_epochs=3)
|
||||
|
||||
|
||||
def _tail(cfg: SimConfig) -> pd.DataFrame:
|
||||
"""One trajectory, burn-in discarded (the report's 50 % convention)."""
|
||||
df = pd.DataFrame(run_trajectory(cfg))
|
||||
return df[df.epoch >= cfg.epochs // 2]
|
||||
|
||||
|
||||
def _tail_or_collapse(cfg: SimConfig) -> tuple[pd.DataFrame | None, bool]:
|
||||
"""``(tail, collapsed)``. A run whose estimate falls into the §6.2 collapsed branch produces
|
||||
blocks at up to one per node per slot, so the arrival matrix blows past the memory guard and
|
||||
:class:`ArrivalMatrixTooLarge` is raised. That is a *result*, not an error — dropping the cell
|
||||
would silently bias a mean upward — so it is caught and reported as a collapse.
|
||||
"""
|
||||
try:
|
||||
return _tail(cfg), False
|
||||
except ArrivalMatrixTooLarge:
|
||||
return None, True
|
||||
|
||||
|
||||
def study_whale() -> pd.DataFrame:
|
||||
def cell(badv: float, selection: str, strategy: str, rep: int) -> dict:
|
||||
t, collapsed = _tail_or_collapse(
|
||||
SimConfig(adversary_frac=badv, adversary_selection=selection,
|
||||
adversary_strategy=strategy, replicate=rep, **WHALE_BASE))
|
||||
row = dict(beta_adv=badv, selection=selection, strategy=strategy, rep=rep,
|
||||
collapsed=collapsed)
|
||||
if t is not None:
|
||||
row |= dict(mean_ratio=float(t.mean_ratio.mean()),
|
||||
adv_block_share=float(t.adv_block_share.mean()))
|
||||
return row
|
||||
|
||||
jobs = [(b, s, st, r) for b in (0.1, 0.3, 0.5) for s in ("random", "whale")
|
||||
for st in ("suppress", "withhold") for r in range(REPS)]
|
||||
df = pd.DataFrame(Parallel(n_jobs=N_JOBS, backend="loky", inner_max_num_threads=1)(
|
||||
delayed(cell)(b, s, st, r) for b, s, st, r in jobs))
|
||||
df.to_parquet(RUNS / "adversary_variants_whale.parquet", index=False)
|
||||
return df
|
||||
|
||||
|
||||
def study_jitter() -> pd.DataFrame:
|
||||
def cell(jitter: float, rep: int) -> dict:
|
||||
# jitter > 0 makes the windowed/pruned engine an approximation, so use the exact oracle.
|
||||
t = _tail(SimConfig(jitter_mean=jitter, replicate=rep,
|
||||
windowed_fork_choice=False, prune_arrival=False,
|
||||
epochs=EPOCHS, **DYN_BASE))
|
||||
return dict(jitter_mean=jitter, rep=rep,
|
||||
mean_ratio=float(t.mean_ratio.mean()),
|
||||
notch=float(t.mean_ratio.max() - t.mean_ratio.min()),
|
||||
adv_block_share=float(t.adv_block_share.mean()),
|
||||
range_ratio=float(t.range_ratio.max()))
|
||||
|
||||
jobs = [(j, r) for j in (0.0, 0.3, 1.0) for r in range(8)]
|
||||
df = pd.DataFrame(Parallel(n_jobs=N_JOBS, backend="loky", inner_max_num_threads=1)(
|
||||
delayed(cell)(j, r) for j, r in jobs))
|
||||
df.to_parquet(RUNS / "adversary_variants_jitter.parquet", index=False)
|
||||
return df
|
||||
|
||||
|
||||
def study_slow_beta() -> pd.DataFrame:
|
||||
def cell(beta: float, rep: int) -> dict:
|
||||
t = _tail(SimConfig(beta=beta, replicate=rep, epochs=40, **DYN_BASE))
|
||||
return dict(beta=beta, rep=rep,
|
||||
mean_ratio=float(t.mean_ratio.mean()),
|
||||
notch=float(t.mean_ratio.max() - t.mean_ratio.min()),
|
||||
adv_block_share=float(t.adv_block_share.mean()))
|
||||
|
||||
jobs = [(b, r) for b in (1.0, 0.25, 0.1, 0.05) for r in range(8)]
|
||||
df = pd.DataFrame(Parallel(n_jobs=N_JOBS, backend="loky", inner_max_num_threads=1)(
|
||||
delayed(cell)(b, r) for b, r in jobs))
|
||||
df.to_parquet(RUNS / "adversary_variants_beta.parquet", index=False)
|
||||
return df
|
||||
|
||||
|
||||
def study_withhold_load() -> pd.DataFrame:
|
||||
"""D. Does static withholding reach the §6.2 fold? (unplanned — found by A blowing up.)
|
||||
|
||||
§6.2 fits a static feedback map that folds into a collapsed low branch at `rho ~ 1.08`, and
|
||||
records that the full per-node dynamics never get there. But the same section gives the
|
||||
mechanism that would take them there: the realised load is `rho_eff = rho / r`, so an
|
||||
estimate deflated to `r` multiplies the load by `1/r`. Withholding deflates `r` to about
|
||||
`1 - beta_adv` BY DESIGN (§6.4), so a 50 % coalition doubles the load — and at the design
|
||||
point `rho ~ 0.56` that lands on `rho_eff ~ 1.1`, past the fold.
|
||||
|
||||
This sweeps the blending budget under static withholding at `beta_adv` 0.3/0.5 — for BOTH
|
||||
coalition selections, since the one observed collapse was a whale cell — and records how often
|
||||
the estimate collapses, which is the direct test of "never reached in the dynamics".
|
||||
"""
|
||||
def cell(badv: float, delay: float, selection: str, rep: int) -> dict:
|
||||
cfg = SimConfig(**{**WHALE_BASE, "blend_delay_max": delay},
|
||||
adversary_frac=badv, adversary_strategy="withhold",
|
||||
adversary_selection=selection, replicate=rep)
|
||||
t, collapsed = _tail_or_collapse(cfg)
|
||||
row = dict(beta_adv=badv, blend_delay_max=delay, selection=selection, rep=rep,
|
||||
collapsed=collapsed)
|
||||
if t is not None:
|
||||
row |= dict(mean_ratio=float(t.mean_ratio.mean()),
|
||||
min_ratio=float(t.mean_ratio.min()),
|
||||
adv_block_share=float(t.adv_block_share.mean()))
|
||||
return row
|
||||
|
||||
jobs = [(b, d, s, r) for b in (0.3, 0.5) for d in (4.0, 8.0)
|
||||
for s in ("random", "whale") for r in range(REPS)]
|
||||
df = pd.DataFrame(Parallel(n_jobs=N_JOBS, backend="loky", inner_max_num_threads=1)(
|
||||
delayed(cell)(b, d, s, r) for b, d, s, r in jobs))
|
||||
df.to_parquet(RUNS / "adversary_variants_withhold_load.parquet", index=False)
|
||||
return df
|
||||
|
||||
|
||||
def _report_withhold_load(df: pd.DataFrame) -> None:
|
||||
print("\n=== D. static withholding vs the §6.2 fold (rho_eff = rho / r) ===")
|
||||
print(f"{'b_adv':>6} {'delta':>6} {'sel':>7} {'collapsed':>10} {'D-hat/D':>18} "
|
||||
f"{'worst epoch':>12}")
|
||||
for badv in sorted(df.beta_adv.unique()):
|
||||
for delay in sorted(df.blend_delay_max.unique()):
|
||||
for sel in sorted(df.selection.unique()):
|
||||
g = df[(df.beta_adv == badv) & (df.blend_delay_max == delay)
|
||||
& (df.selection == sel)]
|
||||
ok = g[~g.collapsed]
|
||||
mr = (f"{ok.mean_ratio.mean():8.4f}+-{ok.mean_ratio.std(ddof=1):.4f}"
|
||||
if len(ok) > 1 else f"{'n/a':>16}")
|
||||
worst = f"{ok.min_ratio.min():12.4f}" if len(ok) else f"{'n/a':>12}"
|
||||
print(f"{badv:6.1f} {delay:6.1f} {sel:>7} {int(g.collapsed.sum()):5d}/{len(g):<4d}"
|
||||
f" {mr} {worst}")
|
||||
|
||||
|
||||
def _report_whale(df: pd.DataFrame) -> None:
|
||||
print("\n=== A. whale vs random coalition (same stake, far fewer members) ===")
|
||||
print(f"{'strategy':>9} {'b_adv':>6} {'sel':>7} {'D-hat/D':>16} {'adv block share':>20}")
|
||||
for strategy in ("suppress", "withhold"):
|
||||
for badv in (0.1, 0.3, 0.5):
|
||||
for sel in ("random", "whale"):
|
||||
g = df[(df.strategy == strategy) & (df.beta_adv == badv) & (df.selection == sel)]
|
||||
print(f"{strategy:>9} {badv:6.1f} {sel:>7} "
|
||||
f"{g.mean_ratio.mean():8.4f}+-{g.mean_ratio.std(ddof=1):.4f} "
|
||||
f"{g.adv_block_share.mean():12.4f}+-{g.adv_block_share.std(ddof=1):.4f}")
|
||||
|
||||
|
||||
def _report_simple(df: pd.DataFrame, key: str, title: str) -> None:
|
||||
print(f"\n=== {title} ===")
|
||||
cols = [c for c in ("mean_ratio", "notch", "adv_block_share", "range_ratio") if c in df]
|
||||
agg = df.groupby(key)[cols].agg(["mean", "std"])
|
||||
print(agg.round(4).to_string())
|
||||
|
||||
|
||||
def main() -> None:
|
||||
print("=== residual adversary variants (report §8.3 item 11) ===")
|
||||
_report_whale(study_whale())
|
||||
_report_simple(study_jitter(), "jitter_mean", "B. dynamic withhold-rejoin under jitter")
|
||||
_report_simple(study_slow_beta(), "beta", "C. dynamic withhold-rejoin at very slow beta")
|
||||
_report_withhold_load(study_withhold_load())
|
||||
print(f"\nwrote {RUNS}/adversary_variants_{{whale,jitter,beta,withhold_load}}.parquet")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -131,7 +131,10 @@ def figs() -> None:
|
||||
|
||||
# right: per-epoch sigma (within a trajectory) vs case, U=0 vs U=1
|
||||
def per_epoch_sigma(s: pd.DataFrame) -> float:
|
||||
return float(s.groupby(["degree", "replicate"]).mean_ratio.std().mean())
|
||||
# per-trajectory epoch std, then mean over trajectories. Group by n_nodes too: the
|
||||
# fullscale-small run holds N=1000 and N=2000 at the same (degree, replicate), and
|
||||
# pooling them would fold their between-N mean offset into the within-trajectory sigma.
|
||||
return float(s.groupby(["degree", "n_nodes", "replicate"]).mean_ratio.std().mean())
|
||||
|
||||
cases: list[tuple[str, pd.DataFrame]] = []
|
||||
for lat in (0.1, 0.2, 0.5, 1.0):
|
||||
@ -153,7 +156,7 @@ def figs() -> None:
|
||||
ax3.set_yscale("log")
|
||||
ax3.set_xticks(x3, [c for c, _ in cases], rotation=20, ha="right", fontsize=8)
|
||||
ax3.set_ylabel(r"per-epoch $\sigma$ of $\hat D / D$")
|
||||
ax3.set_title("Blend delay amplifies U = 0 noise ~17×;\none uncle restores the floor")
|
||||
ax3.set_title("Blend delay amplifies U = 0 noise ~18×;\none uncle restores the floor")
|
||||
ax3.legend(fontsize=8)
|
||||
style.save(fig, FIGS / "figB2_fluctuation_stats", provenance="scripts/appendix_fluct.py")
|
||||
plt.close(fig)
|
||||
|
||||
@ -51,8 +51,12 @@ def fig1(df: pd.DataFrame) -> None:
|
||||
gdfs = sorted(d.gdf.unique())
|
||||
for i, gdf in enumerate(gdfs):
|
||||
for u, ls in ((2, "-"), (0, "--")):
|
||||
# Both panels are indexed by the estimate that DROVE each epoch's production: the
|
||||
# start-of-epoch estimate `mean_ratio_in` (block rate depends on it, and at epoch 0 it
|
||||
# IS the genesis guess, matching the legend). Plotting end-of-epoch `mean_ratio` here
|
||||
# would show the already-updated value at epoch 0 and offset the panels by one epoch.
|
||||
s = (d[(d.gdf == gdf) & (d.u == u)]
|
||||
.groupby("epoch").agg(rate=("n_blocks", "mean"), ratio=("mean_ratio", "mean")))
|
||||
.groupby("epoch").agg(rate=("n_blocks", "mean"), ratio=("mean_ratio_in", "mean")))
|
||||
rate = s.rate / (10 * int(2160 / F)) # blocks per slot
|
||||
ax1.plot(s.index, rate, ls, color=style.OKABE_ITO[i], lw=1.4, ms=3,
|
||||
marker="o" if u == 2 else None,
|
||||
|
||||
@ -1,11 +1,15 @@
|
||||
"""Render the TSI report markdown set (index + 4 parts) to standalone, print-friendly HTML.
|
||||
"""Render the TSI report markdown to standalone, print-friendly HTML.
|
||||
|
||||
Committed replacement for the ad-hoc HTML build. Markdown is the source of truth; the HTML is a
|
||||
build artifact (not committed). Code blocks are syntax-highlighted (codehilite + Pygments),
|
||||
and cross-document `.md` links are rewritten to `.html` so the rendered set navigates internally.
|
||||
Markdown is the source of truth; the HTML is a build artifact (not committed). Code blocks are
|
||||
syntax-highlighted (codehilite + Pygments), and `.md` links are rewritten to `.html` so the
|
||||
index and the report navigate to each other.
|
||||
|
||||
Run: python scripts/build_html.py --all # index + 4 parts
|
||||
python scripts/build_html.py <file.md> ... # specific docs
|
||||
The report lives in reports/tsi/ (not in this simulator folder) as a SINGLE document, named
|
||||
README.md so it renders as the directory landing page; figure links are relative to that
|
||||
directory, so paths resolve as-is.
|
||||
|
||||
Run: python scripts/build_html.py --all # the report
|
||||
python scripts/build_html.py <file.md> ... # specific docs (paths relative to reports/tsi)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@ -17,14 +21,9 @@ from pathlib import Path
|
||||
import markdown
|
||||
from pygments.formatters import HtmlFormatter
|
||||
|
||||
HERE = Path(__file__).resolve().parent.parent
|
||||
DOCS = [
|
||||
"REPORT-tsi-parameter-selection.md",
|
||||
"tsi-report-1-overview-and-recommendations.md",
|
||||
"tsi-report-2-accuracy-and-design.md",
|
||||
"tsi-report-3-robustness-and-incentives.md",
|
||||
"tsi-report-4-reproducibility-and-appendices.md",
|
||||
]
|
||||
# The report set lives in the repo's reports/ tree, not alongside the simulator.
|
||||
HERE = Path(__file__).resolve().parents[5] / "reports" / "tsi"
|
||||
DOCS = ["README.md"] # the report IS the directory README (renders at reports/tsi/)
|
||||
|
||||
CSS_BASE = r"""
|
||||
@page { size: A4; margin: 18mm 16mm 20mm 16mm; }
|
||||
|
||||
@ -41,11 +41,17 @@ def main() -> None:
|
||||
df.to_parquet(out / "capstone.parquet", index=False)
|
||||
print("=== Capstone: recommended config, all metrics together (equilibrium tail) ===")
|
||||
for adv, g in df.groupby("adv"):
|
||||
t = g[g.epoch >= g.epoch.max() // 2]
|
||||
# Per-REPLICATE tail: early_stop ends replicates at different epochs, so a per-arm cut
|
||||
# (epoch >= arm_max//2) would silently drop any replicate that stopped before the cut
|
||||
# and skew the tail toward the slow-converging ones. The report's §8.4 numbers are the
|
||||
# per-replicate aggregation; keep this printout matching them.
|
||||
t = pd.concat([r[r.epoch >= r.epoch.max() // 2] for _, r in g.groupby("replicate")])
|
||||
per_rep = t.groupby("replicate").fork_rate.mean()
|
||||
sem = per_rep.std(ddof=1) / (len(per_rep) ** 0.5)
|
||||
print(f"adversary {adv:.0%}: D̂/D {t.mean_ratio.mean():.4f} "
|
||||
f"range_ratio {t.range_ratio.max():.4f} agreement {t.agreement_window.min():.4f} "
|
||||
f"fork_rate {t.fork_rate.mean():.3f} max_reorg_depth {t.max_reorg_depth.max()} "
|
||||
f"p_ref {t.p_ref.mean():.3f}")
|
||||
f"fork_rate {per_rep.mean():.3f}+-{sem:.3f}(SEM over {len(per_rep)} reps) "
|
||||
f"max_reorg_depth {t.max_reorg_depth.max()} p_ref {t.p_ref.mean():.3f}")
|
||||
print(f"wrote {out/'capstone.parquet'} ({len(df)} rows)")
|
||||
|
||||
|
||||
|
||||
@ -0,0 +1,197 @@
|
||||
"""What the first-fork restriction costs against a selfish adversary — REPORT §6.6 (fig36).
|
||||
|
||||
§6.6 reads the estimator repair off a free knob: the uncle-recovery fraction ``eta``, quoted at
|
||||
``eta = 1`` ("honest-orphan recovery"). The countable uncle model (§2.1) can reference only the
|
||||
**first block of a fork**, so ``eta`` is not free — it is capped by how the adversary *shapes* the
|
||||
blocks it orphans:
|
||||
|
||||
* **SM1** acts the moment the honest branch reaches length 1 (match at a 1-lead, override at a
|
||||
2-lead, publish-one above it), so it never buries a second block behind the first. Every orphan
|
||||
it makes is the first block of its fork and the cap is exactly 1 — the restriction is free.
|
||||
* The **optimal** (Sapirshtein–Sompolinsky–Zohar) policy *waits*, then overrides a run of ``h``
|
||||
honest blocks at once. That run is one chain, so the deployed rules recover **one** uncle from
|
||||
it, not ``h``, and the cap falls to ~0.44 at ``alpha = 0.4``.
|
||||
|
||||
So SM1 is a faithful proxy for selfish-mining *revenue* (§6.6 quotes 0.484 vs the optimum's 0.488
|
||||
at gamma = 0) but **not** for TSI's estimator damage: the two differ by a factor of ~2 in
|
||||
recoverable orphans. The panel on the right is the consequence — the repair §6.6 credits to uncle
|
||||
counting is roughly half of what the unrestricted model shows, and it *degrades* with alpha where
|
||||
the unrestricted model improves.
|
||||
|
||||
Two panels (fig36):
|
||||
LEFT — the recovery ceiling ``eta_countable`` vs stake, per gamma, against SM1's flat 1.0.
|
||||
RIGHT — the resulting ``D_hat/D*`` at gamma = 0: no uncles, countable (p_ref = 1 and 0.85), and
|
||||
the unrestricted baseline §6.6 reports.
|
||||
|
||||
Run: python scripts/countable_selfish.py (writes runs/countable_selfish.parquet + fig36)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from tsi_sim.plotting import style
|
||||
from tsi_sim.reorg import (
|
||||
alpha_effective,
|
||||
countable_recovery_from_depths,
|
||||
simulate_deepest_reorg,
|
||||
)
|
||||
from tsi_sim.selfish import race_from_alpha, selfish_threshold, tsi_dhat_ratio
|
||||
from tsi_sim.selfish_mdp import optimal_policy_stats
|
||||
|
||||
HERE = Path(__file__).resolve().parent.parent
|
||||
RUNS = HERE / "runs"
|
||||
FIGS = HERE / "report-figures"
|
||||
RUNS.mkdir(exist_ok=True)
|
||||
FIGS.mkdir(exist_ok=True)
|
||||
|
||||
# cap 64 keeps the orphan *shape* converged (it settles more slowly than the revenue: the drift
|
||||
# from cap 48 to 64 is ~3e-4 in eta at alpha = 0.4, ~5e-3 at alpha = 0.45).
|
||||
CAP = 64
|
||||
GAMMAS = [0.0, 0.5]
|
||||
ALPHAS = [0.26, 0.28, 0.30, 0.34, 0.36, 0.38, 0.40, 0.42, 0.44, 0.46]
|
||||
P_REF_REALISTIC = 0.85 # the §6.8 stand-in for the emergent honest-referencer rate
|
||||
N_EVENTS = 4_000_000 # SM1 comparison arm
|
||||
|
||||
|
||||
def sweep() -> pd.DataFrame:
|
||||
rng = np.random.default_rng(20260805)
|
||||
rows = []
|
||||
for gamma in GAMMAS:
|
||||
for alpha in ALPHAS:
|
||||
s = optimal_policy_stats(alpha, gamma, cap=CAP)
|
||||
sm1 = race_from_alpha(alpha, N_EVENTS, gamma, rng)
|
||||
rows.append(dict(
|
||||
alpha=alpha, gamma=gamma, cap=CAP,
|
||||
above_threshold=alpha > selfish_threshold(gamma),
|
||||
deviates=s.deviates,
|
||||
revenue_opt=s.revenue,
|
||||
revenue_sm1=sm1.revenue_share,
|
||||
density_fraction=s.density_fraction,
|
||||
orphan_hon_blocks=s.orphan_hon_blocks,
|
||||
orphan_hon_runs=s.orphan_hon_runs,
|
||||
eta_countable=s.countable_recovery if s.deviates else np.nan,
|
||||
eta_countable_adv=s.countable_recovery_adv if s.deviates else np.nan,
|
||||
# the estimator, as §6.6 reports it (unrestricted) and as deployed (countable)
|
||||
dhat_u0=s.dhat_ratio(p_ref=0.0),
|
||||
dhat_unrestricted=s.dhat_ratio(p_ref=1.0, countable=False),
|
||||
dhat_countable=s.dhat_ratio(p_ref=1.0, countable=True),
|
||||
dhat_countable_pref=s.dhat_ratio(p_ref=P_REF_REALISTIC, countable=True),
|
||||
# SM1's own repair, for contrast: eta = 1 is attainable there
|
||||
dhat_sm1_eta1=tsi_dhat_ratio(sm1, 1.0),
|
||||
eta_sm1=sm1.countable_recovery,
|
||||
))
|
||||
out = pd.DataFrame(rows)
|
||||
out.to_parquet(RUNS / "countable_selfish.parquet")
|
||||
return out
|
||||
|
||||
|
||||
def report(df: pd.DataFrame) -> None:
|
||||
print(f"{'gamma':>5} {'alpha':>6} {'rev_opt':>8} {'rev_SM1':>8} {'eta_cnt':>8} {'eta_SM1':>8} "
|
||||
f"{'no unc':>7} {'unrestr':>8} {'count':>7} {'cnt@.85':>8}")
|
||||
for _, r in df[df.deviates].iterrows():
|
||||
print(f"{r.gamma:5.1f} {r.alpha:6.2f} {r.revenue_opt:8.4f} {r.revenue_sm1:8.4f} "
|
||||
f"{r.eta_countable:8.4f} {r.eta_sm1:8.4f} {r.dhat_u0:7.4f} "
|
||||
f"{r.dhat_unrestricted:8.4f} {r.dhat_countable:7.4f} {r.dhat_countable_pref:8.4f}")
|
||||
hit = df[(df.gamma == 0.0) & np.isclose(df.alpha, 0.40)]
|
||||
if not hit.empty:
|
||||
r = hit.iloc[0]
|
||||
print(f"\nHeadline (alpha=0.40, gamma=0): revenue {r.revenue_sm1:.3f} (SM1) vs "
|
||||
f"{r.revenue_opt:.3f} (optimal) — a faithful proxy;")
|
||||
print(f" but eta 1.000 (SM1) vs {r.eta_countable:.3f} (optimal) — not a faithful proxy, "
|
||||
f"and D_hat {r.dhat_unrestricted:.3f} -> {r.dhat_countable:.3f}.")
|
||||
|
||||
|
||||
def fig36(df: pd.DataFrame) -> None:
|
||||
import matplotlib.pyplot as plt
|
||||
style.apply_style()
|
||||
fig, axes = plt.subplots(1, 2, figsize=(9.6, 3.8))
|
||||
|
||||
# LEFT: the recovery ceiling, optimal policy vs SM1
|
||||
ax = axes[0]
|
||||
ax.axhline(1.0, color="0.5", lw=1.1, ls="--",
|
||||
label="SM1 — every orphan countable")
|
||||
for i, gamma in enumerate(GAMMAS):
|
||||
g = df[(df.gamma == gamma) & df.deviates].sort_values("alpha")
|
||||
ax.plot(g.alpha, g.eta_countable, "-o", ms=4, color=style.OKABE_ITO[i + 1],
|
||||
label=rf"optimal policy, $\gamma={gamma}$")
|
||||
thr = selfish_threshold(gamma)
|
||||
ax.axvline(thr, color=style.OKABE_ITO[i + 1], lw=0.7, ls=":")
|
||||
ax.set_ylim(0, 1.08)
|
||||
ax.set_xlabel(r"adversary stake $\alpha$")
|
||||
ax.set_ylabel(r"countable recovery ceiling $\eta$")
|
||||
ax.set_title("The optimum buries orphans SM1 leaves reachable")
|
||||
ax.legend(fontsize=7, loc="lower left")
|
||||
|
||||
# RIGHT: the estimator consequence at gamma = 0
|
||||
ax = axes[1]
|
||||
g0 = df[(df.gamma == 0.0) & df.deviates].sort_values("alpha")
|
||||
ax.axhline(1.0, color="0.5", lw=0.9, ls="--", label=r"honest $D^*$")
|
||||
ax.plot(g0.alpha, g0.dhat_unrestricted, "-s", ms=4, color=style.OKABE_ITO[2],
|
||||
label=r"unrestricted count, $p_{ref}=1$ (fig13's $\eta=1$)")
|
||||
ax.plot(g0.alpha, g0.dhat_countable, "-o", ms=4, color=style.OKABE_ITO[3],
|
||||
label=r"countable, $p_{ref}=1$ (deployed rule)")
|
||||
ax.plot(g0.alpha, g0.dhat_countable_pref, "-^", ms=4, color=style.OKABE_ITO[5],
|
||||
label=rf"countable, $p_{{ref}}={P_REF_REALISTIC}$")
|
||||
ax.plot(g0.alpha, g0.dhat_u0, "-v", ms=4, color=style.OKABE_ITO[1],
|
||||
label=r"no uncles ($\eta=0$)")
|
||||
ax.set_xlabel(r"adversary stake $\alpha$")
|
||||
ax.set_ylabel(r"$\hat D / D^*$ (estimator deflation)")
|
||||
ax.set_title(r"About half the repair fig13's $\eta=1$ implies")
|
||||
ax.legend(fontsize=7, loc="lower left")
|
||||
|
||||
style.save(fig, FIGS / "fig36_countable_selfish", provenance="scripts/countable_selfish.py")
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def reorg_ceilings() -> pd.DataFrame:
|
||||
"""The same ceiling for the *depth*-maximising adversary of §6.10, for cross-reference.
|
||||
|
||||
A depth-``d`` reorg discards ``d`` consecutive public blocks — one chain, one countable uncle.
|
||||
Reported at the honest fork rate ``o = 0`` and at the measured Blend value ``o = 0.35``, which
|
||||
inflates the adversary's effective share and so its reorg depths.
|
||||
"""
|
||||
rows = []
|
||||
for o in (0.0, 0.35):
|
||||
for alpha in (0.10, 0.20, 0.30):
|
||||
ae = alpha_effective(alpha, o)
|
||||
depths = simulate_deepest_reorg(ae, 4_000_000, np.random.default_rng(5))
|
||||
rows.append(dict(alpha=alpha, orphan_rate=o, alpha_eff=ae, attacks=int(depths.size),
|
||||
mean_depth=float(depths.mean()),
|
||||
eta_countable=countable_recovery_from_depths(depths)))
|
||||
out = pd.DataFrame(rows)
|
||||
out.to_parquet(RUNS / "countable_selfish_reorg.parquet")
|
||||
print("\n=== depth-maximising adversary (§6.10) — same first-fork ceiling ===")
|
||||
print(f"{'alpha':>6} {'o':>5} {'a_eff':>7} {'E[d]':>6} {'eta_cnt':>8}")
|
||||
for _, r in out.iterrows():
|
||||
print(f"{r.alpha:6.2f} {r.orphan_rate:5.2f} {r.alpha_eff:7.4f} {r.mean_depth:6.3f} "
|
||||
f"{r.eta_countable:8.4f}")
|
||||
return out
|
||||
|
||||
|
||||
def main() -> None:
|
||||
import argparse
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("--reuse", action="store_true",
|
||||
help="re-render from runs/countable_selfish.parquet instead of re-solving the "
|
||||
"MDP (the solve is ~15 min; the figure is not)")
|
||||
args = ap.parse_args()
|
||||
|
||||
cached = RUNS / "countable_selfish.parquet"
|
||||
if args.reuse and cached.exists():
|
||||
print(f"=== re-rendering from {cached.name} ===")
|
||||
df = pd.read_parquet(cached)
|
||||
else:
|
||||
print(f"=== countable recovery under a selfish adversary (MDP cap={CAP}) ===")
|
||||
df = sweep()
|
||||
reorg_ceilings()
|
||||
report(df)
|
||||
fig36(df)
|
||||
print("wrote fig36_countable_selfish")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -0,0 +1,147 @@
|
||||
"""How far can a PROFITABLE adversary deflate the estimate? — REPORT §8.3 item 16 (fig37).
|
||||
|
||||
§6.6 measures the countable recovery ceiling under two adversaries, and both optimise something
|
||||
other than the estimator: revenue (the SSZ objective) and reorg depth. Item 16 recorded the
|
||||
consequence — those are upper bounds on `eta`, not lower bounds on the damage — and asked what an
|
||||
adversary optimising deflation directly would do.
|
||||
|
||||
Two answers, because the obvious question is the wrong one:
|
||||
|
||||
* **Unconstrained**, the deflation optimum is pure abstention: publish nothing, adopt when
|
||||
overtaken, drive `D_hat` to exactly `1 - alpha` and revenue to zero. §6.4 already covers this
|
||||
and shows it is *correct* measurement — a coalition that publishes nothing is not
|
||||
participating, so `1 - alpha` is the right answer for the stake that is.
|
||||
* **Constrained to stay paid**, the question bites. Sweeping the mixed objective
|
||||
`lam * (adversary blocks) - (contribution to D_hat)` traces the profit/deflation frontier;
|
||||
the point of interest is where the revenue *share* reaches `alpha`, i.e. where the adversary
|
||||
does no worse than mining honestly. Everything below that line is self-punishing griefing,
|
||||
already bounded by §6.5.
|
||||
|
||||
The headline: at `alpha = 0.4` an adversary content with break-even revenue pushes `D_hat` to
|
||||
~0.64, against the revenue-optimal policy's 0.81. The revenue-optimal adversary is not the
|
||||
estimator's worst case, and the gap is free — it costs only the selfish premium.
|
||||
|
||||
Note the sweep parameter is not monotone in revenue: selfish mining wins a larger share of a
|
||||
*smaller* pie, so raw adversary block rate is maximised by honest mining and large `lam` returns
|
||||
there. That is fine — the sweep is used to enumerate candidate policies and their measured
|
||||
(share, `D_hat`) pairs, not as a monotone path, and any point on it is a legitimate strategy.
|
||||
|
||||
Run: python scripts/deflation_frontier.py (writes runs/deflation_frontier.parquet + fig37)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from tsi_sim.plotting import style
|
||||
from tsi_sim.selfish_mdp import deflation_frontier, deflation_optimal_stats, optimal_policy_stats
|
||||
|
||||
HERE = Path(__file__).resolve().parent.parent
|
||||
RUNS = HERE / "runs"
|
||||
FIGS = HERE / "report-figures"
|
||||
RUNS.mkdir(exist_ok=True)
|
||||
FIGS.mkdir(exist_ok=True)
|
||||
|
||||
CAP = 64
|
||||
GAMMA = 0.0
|
||||
ALPHAS = [0.30, 1 / 3, 0.36, 0.38, 0.40, 0.42, 0.45]
|
||||
LAMS = [round(x, 2) for x in np.arange(0.0, 3.05, 0.1)]
|
||||
|
||||
|
||||
def sweep() -> pd.DataFrame:
|
||||
rows = []
|
||||
for a in ALPHAS:
|
||||
ro = optimal_policy_stats(a, GAMMA, cap=CAP)
|
||||
do = deflation_optimal_stats(a, GAMMA, cap=CAP)
|
||||
for lam in LAMS:
|
||||
r = deflation_frontier(a, GAMMA, lam, cap=CAP)
|
||||
r |= dict(revenue_optimal=ro.revenue,
|
||||
dhat_revenue_optimal=ro.dhat_ratio(1.0, True) if ro.deviates else 1.0,
|
||||
dhat_abstention=do.dhat_ratio(1.0, True))
|
||||
rows.append(r)
|
||||
df = pd.DataFrame(rows)
|
||||
df.to_parquet(RUNS / "deflation_frontier.parquet", index=False)
|
||||
return df
|
||||
|
||||
|
||||
def best_profitable(df: pd.DataFrame) -> pd.DataFrame:
|
||||
"""Per alpha, the frontier point minimising D_hat among those paying at least stake share."""
|
||||
out = []
|
||||
for _a, g in df.groupby("alpha"):
|
||||
paid = g[g.reward_per_stake >= 1.0 - 1e-9]
|
||||
if paid.empty:
|
||||
continue
|
||||
out.append(paid.loc[paid.dhat_countable.idxmin()])
|
||||
return pd.DataFrame(out)
|
||||
|
||||
|
||||
def report(df: pd.DataFrame) -> None:
|
||||
best = best_profitable(df)
|
||||
print(f"{'alpha':>6} {'rev-opt rev':>12} {'rev-opt D':>10} | "
|
||||
f"{'best paid rev':>14} {'xstake':>7} {'D':>7} {'extra deflation':>16}")
|
||||
for a, g in df.groupby("alpha"):
|
||||
ro_r, ro_d = g.revenue_optimal.iloc[0], g.dhat_revenue_optimal.iloc[0]
|
||||
b = best[best.alpha == a]
|
||||
if b.empty:
|
||||
print(f"{a:6.3f} {ro_r:12.4f} {ro_d:10.4f} | (nothing profitable deflates)")
|
||||
continue
|
||||
b = b.iloc[0]
|
||||
print(f"{a:6.3f} {ro_r:12.4f} {ro_d:10.4f} | {b.revenue:14.4f} "
|
||||
f"{b.reward_per_stake:7.3f} {b.dhat_countable:7.4f} {ro_d - b.dhat_countable:+16.4f}")
|
||||
print("\n(extra deflation > 0 means the profitable-but-deflating policy beats the "
|
||||
"revenue-optimal one at damaging the estimate, at no cost versus honest mining)")
|
||||
|
||||
|
||||
def fig37(df: pd.DataFrame) -> None:
|
||||
import matplotlib.pyplot as plt
|
||||
style.apply_style()
|
||||
fig, axes = plt.subplots(1, 2, figsize=(9.6, 3.8))
|
||||
|
||||
# LEFT: the frontier itself for a few alphas — revenue share against the estimate
|
||||
ax = axes[0]
|
||||
for i, a in enumerate([0.36, 0.40, 0.45]):
|
||||
g = df[np.isclose(df.alpha, a)].sort_values("lam")
|
||||
c = style.OKABE_ITO[i + 1]
|
||||
ax.plot(g.revenue, g.dhat_countable, "-o", ms=3, color=c, label=rf"$\alpha={a}$")
|
||||
ax.axvline(a, color=c, lw=0.7, ls=":")
|
||||
ro = g.dhat_revenue_optimal.iloc[0]
|
||||
ax.plot([g.revenue_optimal.iloc[0]], [ro], "*", ms=11, color=c)
|
||||
ax.set_xlabel("adversary revenue share (dotted line = its stake, i.e. break-even)")
|
||||
ax.set_ylabel(r"$\hat D / D^*$")
|
||||
ax.set_title("Profit/deflation frontier (★ = revenue-optimal)")
|
||||
ax.legend(fontsize=7, loc="lower right")
|
||||
|
||||
# RIGHT: how much worse the estimator gets once the adversary stops maximising revenue
|
||||
ax = axes[1]
|
||||
best = best_profitable(df).sort_values("alpha")
|
||||
ref = df.groupby("alpha").agg(d_ro=("dhat_revenue_optimal", "first"),
|
||||
d_ab=("dhat_abstention", "first")).reset_index()
|
||||
ax.plot(ref.alpha, ref.d_ro, "-*", ms=9, color=style.OKABE_ITO[2],
|
||||
label="revenue-optimal (§6.6)")
|
||||
ax.plot(best.alpha, best.dhat_countable, "-o", ms=4, color=style.OKABE_ITO[3],
|
||||
label="best deflation at break-even pay")
|
||||
ax.plot(ref.alpha, ref.d_ab, "--v", ms=4, color=style.OKABE_ITO[1],
|
||||
label=r"abstention ($1-\alpha$, unpaid)")
|
||||
ax.axhline(1.0, color="0.5", lw=0.9, ls="--")
|
||||
ax.set_xlabel(r"adversary stake $\alpha$")
|
||||
ax.set_ylabel(r"$\hat D / D^*$")
|
||||
ax.set_title("The revenue-optimal adversary is not the worst case")
|
||||
ax.legend(fontsize=7, loc="lower left")
|
||||
|
||||
style.save(fig, FIGS / "fig37_deflation_frontier", provenance="scripts/deflation_frontier.py")
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
print(f"=== profit/deflation frontier (MDP cap={CAP}, gamma={GAMMA}) ===")
|
||||
df = sweep()
|
||||
report(df)
|
||||
fig37(df)
|
||||
print("wrote fig37_deflation_frontier")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -0,0 +1,242 @@
|
||||
"""Comparison figures: countable uncle model (spec counting rules) vs the old model.
|
||||
|
||||
Consumes the results of three sweeps:
|
||||
|
||||
tsi-sweep --config configs/countable-vs-old.yaml --label cvo-countable
|
||||
tsi-sweep --config configs/countable-vs-old.yaml --old --label cvo-old
|
||||
tsi-sweep --config configs/absorption-window.yaml --label absorption-window
|
||||
|
||||
and renders (into --out):
|
||||
|
||||
cvo_accuracy_vs_delay equilibrium D/D_true vs Blend mixing delay; solid = countable,
|
||||
dashed = old, one Okabe-Ito hue per U (color follows U).
|
||||
cvo_prediction_vs_sim predicted log(1-f)/log(1-f/q_u) from the MEASURED q_u vs the
|
||||
simulated equilibrium — the q -> q_u reduction check.
|
||||
cvo_recovery_vs_delay measured recovery r = (q_eff - q)/(1 - q) and the non-recovered
|
||||
waste share 1-r vs delay. (NOT deep_ref_share: under joint
|
||||
countable selection+counting that is identically 0 — the
|
||||
first-fork restriction acts at selection, so it shows up
|
||||
inside 1-r alongside capacity losses.)
|
||||
absorption_window equilibrium vs the window absorption parameter W per delay.
|
||||
|
||||
Usage:
|
||||
python scripts/plot_countable_vs_old.py --countable RUNDIR --old RUNDIR \
|
||||
--absorption RUNDIR --out figures/countable-vs-old
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
import matplotlib.pyplot as plt
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from tsi_sim.plotting import style
|
||||
from tsi_sim.plotting.figures_pernode import (
|
||||
Z95,
|
||||
equilibrium,
|
||||
paired_gaps,
|
||||
pooled_by_delay,
|
||||
recovery_rate,
|
||||
sem,
|
||||
)
|
||||
from tsi_sim.theory import expected_ratio, q_effective
|
||||
|
||||
DELAY = "blend_delay_max"
|
||||
|
||||
|
||||
def _load(run_dir: str | Path) -> pd.DataFrame:
|
||||
return pd.read_parquet(Path(run_dir) / "results.parquet")
|
||||
|
||||
|
||||
def _eq(df: pd.DataFrame, extra_cols: tuple[str, ...] = ()) -> pd.DataFrame:
|
||||
"""Equilibrium (post-burn) per (delay, U) cell: replicate mean, SEM and count.
|
||||
|
||||
``equilibrium`` returns one row per (config, replicate); the spread across those
|
||||
replicates is the only uncertainty estimate available, and at high delay it is large
|
||||
enough to swamp the model difference — so every consumer here carries it.
|
||||
"""
|
||||
eq = equilibrium(df)
|
||||
keys = [DELAY, "max_uncles", *extra_cols]
|
||||
agg = {"mean_ratio": ("mean_ratio", "mean"),
|
||||
"sem_ratio": ("mean_ratio", sem),
|
||||
"n_rep": ("mean_ratio", "size"),
|
||||
"mean_q": ("mean_q", "mean"),
|
||||
"mean_q_eff": ("mean_q_eff", "mean")}
|
||||
return eq.groupby(keys, as_index=False).agg(**agg)
|
||||
|
||||
|
||||
def fig_accuracy_vs_delay(cnt: pd.DataFrame, old: pd.DataFrame,
|
||||
paired: bool = False) -> plt.Figure:
|
||||
"""Accuracy vs delay per U, with replicate SEM bars.
|
||||
|
||||
U=0 is the NEGATIVE CONTROL: with no uncles the two models are identical by
|
||||
construction, so the visible countable-vs-old gap on that pair of curves is pure
|
||||
between-run RNG noise and calibrates what a real difference has to beat.
|
||||
"""
|
||||
fig, ax = plt.subplots(figsize=(6.4, 4.2))
|
||||
us = sorted(cnt["max_uncles"].unique())
|
||||
for i, u in enumerate(us):
|
||||
c = style.color_for(i)
|
||||
a = cnt[cnt.max_uncles == u].sort_values(DELAY)
|
||||
b = old[old.max_uncles == u].sort_values(DELAY)
|
||||
ctl = " (control)" if u == 0 else ""
|
||||
ax.errorbar(a[DELAY], a.mean_ratio, yerr=a.sem_ratio, fmt="-o", color=c,
|
||||
label=f"U={u} countable{ctl}", ms=4, capsize=2, lw=1.2)
|
||||
ax.errorbar(b[DELAY], b.mean_ratio, yerr=b.sem_ratio, fmt="--s", color=c,
|
||||
label=f"U={u} unrestricted{ctl}", ms=4, capsize=2, alpha=0.75, lw=1.2)
|
||||
ax.axhline(1.0, color="0.4", lw=0.8, ls=":")
|
||||
ax.set_xlabel("max per-relay mixing delay (slots)")
|
||||
ax.set_ylabel(r"equilibrium $\hat{D}/D_{true}$")
|
||||
ax.set_title("Accuracy vs delay: countable (solid) vs unrestricted (dashed)"
|
||||
+ (" — paired" if paired else ""))
|
||||
ax.legend(ncol=2, fontsize="x-small")
|
||||
return fig
|
||||
|
||||
|
||||
def fig_prediction_vs_sim(cnt: pd.DataFrame, f: float) -> plt.Figure:
|
||||
fig, ax = plt.subplots(figsize=(4.6, 4.4))
|
||||
sub = cnt[cnt.max_uncles > 0]
|
||||
# Reconstruct q_u through the theory identity the report quotes, q_u = q + (1-q) r,
|
||||
# rather than reading mean_q_eff straight off the parquet: the figure then exercises
|
||||
# the same closed form the text claims.
|
||||
r = recovery_rate(sub.mean_q.to_numpy(), sub.mean_q_eff.to_numpy())
|
||||
pred = expected_ratio(f, q_effective(sub.mean_q.to_numpy(), r))
|
||||
us = sorted(sub["max_uncles"].unique())
|
||||
for i, u in enumerate(us):
|
||||
m = (sub.max_uncles == u).to_numpy()
|
||||
ax.scatter(np.asarray(pred)[m], sub.mean_ratio.to_numpy()[m],
|
||||
color=style.color_for(i), s=22, label=f"U={u}")
|
||||
lo = min(float(np.min(pred)), float(sub.mean_ratio.min())) - 0.01
|
||||
ax.plot([lo, 1.005], [lo, 1.005], color="0.3", lw=0.9, ls=":", label="prediction = sim")
|
||||
ax.set_xlabel(r"predicted $\log(1-f)\,/\,\log(1-f/\bar{q}_u)$ (measured $\bar{q}_u$)")
|
||||
ax.set_ylabel(r"simulated equilibrium $\hat{D}/D_{true}$")
|
||||
ax.set_title(r"$q \to q_u$ reduction: prediction vs simulation")
|
||||
ax.legend()
|
||||
return fig
|
||||
|
||||
|
||||
def fig_recovery_vs_delay(cnt: pd.DataFrame) -> plt.Figure:
|
||||
# Right panel: the non-recovered waste share 1-r (log scale). Under joint countable
|
||||
# selection+counting the first-fork restriction acts at SELECTION (deep orphans are
|
||||
# never referenced), so counting-side rejections (deep_ref_share) are 0 and the
|
||||
# restriction shows up inside 1-r together with capacity losses.
|
||||
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.2, 3.8))
|
||||
us = [u for u in sorted(cnt["max_uncles"].unique()) if u > 0]
|
||||
for i, u in enumerate(us):
|
||||
a = cnt[cnt.max_uncles == u].sort_values(DELAY)
|
||||
r = recovery_rate(a.mean_q.to_numpy(), a.mean_q_eff.to_numpy())
|
||||
ax1.plot(a[DELAY], r, "-o", color=style.color_for(i), label=f"U={u}", ms=4)
|
||||
ax2.semilogy(a[DELAY], np.maximum(1.0 - r, 1e-4), "-o",
|
||||
color=style.color_for(i), label=f"U={u}", ms=4)
|
||||
ax1.set_xlabel("max per-relay mixing delay (slots)")
|
||||
ax1.set_ylabel(r"measured recovery $r=(\bar{q}_u-\bar{q})/(1-\bar{q})$")
|
||||
ax1.set_ylim(0, 1.02)
|
||||
ax1.set_title("Uncle recovery rate")
|
||||
ax1.legend()
|
||||
ax2.set_xlabel("max per-relay mixing delay (slots)")
|
||||
ax2.set_ylabel(r"non-recovered waste share $1-r$")
|
||||
ax2.set_title("Residual (first-fork + capacity losses)")
|
||||
ax2.legend()
|
||||
return fig
|
||||
|
||||
|
||||
def fig_absorption_window(absw: pd.DataFrame) -> plt.Figure:
|
||||
"""Accuracy vs the derived window, with replicate SEM bars.
|
||||
|
||||
The bars matter here: at the 32-slot delay the run-to-run spread is the dominant
|
||||
feature (replicate sd up to ~0.22), so that curve's shape is not interpretable as a
|
||||
trend — only its ceiling is, and the ceiling is set by U=1, not by the window.
|
||||
"""
|
||||
fig, ax = plt.subplots(figsize=(6.0, 4.0))
|
||||
eq = equilibrium(absw)
|
||||
agg = eq.groupby([DELAY, "window_absorption"], as_index=False).agg(
|
||||
mean_ratio=("mean_ratio", "mean"), sem_ratio=("mean_ratio", sem))
|
||||
for i, d in enumerate(sorted(agg[DELAY].unique())):
|
||||
a = agg[agg[DELAY] == d].sort_values("window_absorption")
|
||||
ax.errorbar(a.window_absorption, a.mean_ratio, yerr=a.sem_ratio, fmt="-o",
|
||||
color=style.color_for(i), label=f"delay={d:g}", ms=4, capsize=2, lw=1.2)
|
||||
ax.axhline(1.0, color="0.4", lw=0.8, ls=":")
|
||||
ax.set_xlabel("window absorption parameter W (expected block-intervals)")
|
||||
ax.set_ylabel(r"equilibrium $\hat{D}/D_{true}$")
|
||||
ax.set_title("Accuracy vs the derived uncle window $w_u = W/f$ (U=1)")
|
||||
ax.legend(title=None)
|
||||
return fig
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--countable", required=True, help="run dir of cvo-countable")
|
||||
ap.add_argument("--old", required=True, help="run dir of cvo-old")
|
||||
ap.add_argument("--absorption", required=True, help="run dir of absorption-window")
|
||||
ap.add_argument("--out", default="figures/countable-vs-old")
|
||||
args = ap.parse_args()
|
||||
style.apply_style()
|
||||
out = Path(args.out)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
cnt_raw, old_raw = _load(args.countable), _load(args.old)
|
||||
f = float(cnt_raw["f"].iloc[0])
|
||||
cnt, old = _eq(cnt_raw), _eq(old_raw)
|
||||
# When both arms shared their RNG streams the comparison is PAIRED: the per-replicate
|
||||
# difference cancels the shared variance, so it supersedes the unpaired two-sample test.
|
||||
pg = paired_gaps(cnt_raw, old_raw)
|
||||
prov = "tsi-sim-pernode countable-vs-old.yaml (+--old) / absorption-window.yaml"
|
||||
|
||||
written = []
|
||||
written += style.save(fig_accuracy_vs_delay(cnt, old, pg is not None),
|
||||
out / "cvo_accuracy_vs_delay", prov)
|
||||
written += style.save(fig_prediction_vs_sim(cnt, f), out / "cvo_prediction_vs_sim", prov)
|
||||
written += style.save(fig_recovery_vs_delay(cnt), out / "cvo_recovery_vs_delay", prov)
|
||||
written += style.save(fig_absorption_window(_load(args.absorption)),
|
||||
out / "absorption_window", prov)
|
||||
# Headline numbers for the report. The gap is printed with t = |gap| / SE(gap); |t| < 2
|
||||
# means the cell does NOT resolve a model difference at this replicate count and must not
|
||||
# be read as one. Under paired_streams the SE is the per-replicate paired one (the shared
|
||||
# variance cancels); otherwise it is the unpaired two-sample SE. The U=0 rows are the
|
||||
# negative control — identical models, so paired they must be EXACTLY zero.
|
||||
kind = "PAIRED (common random numbers)" if pg is not None else "unpaired two-sample"
|
||||
print(f"\ncountable vs unrestricted — {kind}, {int(cnt.n_rep.min())} replicates/arm")
|
||||
print(f"{'cell':<14} {'countable':>17} {'unrestricted':>17} {'gap':>9} {'t':>6} verdict")
|
||||
for u in sorted(cnt["max_uncles"].unique()):
|
||||
for _, row in cnt[cnt.max_uncles == u].sort_values(DELAY).iterrows():
|
||||
q, qu = row.mean_q, row.mean_q_eff
|
||||
r = float(recovery_rate(q, qu))
|
||||
o = old[(old.max_uncles == u) & (old[DELAY] == row[DELAY])]
|
||||
if not len(o):
|
||||
continue
|
||||
orow = o.iloc[0]
|
||||
if pg is not None:
|
||||
pr = pg[(pg.max_uncles == u) & (pg[DELAY] == row[DELAY])].iloc[0]
|
||||
diff, se, t = float(pr.gap), float(pr.se), float(pr.t)
|
||||
else:
|
||||
diff = row.mean_ratio - orow.mean_ratio
|
||||
se = float(np.hypot(row.sem_ratio, orow.sem_ratio))
|
||||
t = abs(diff) / se if se > 0 else float("inf")
|
||||
verdict = ("CONTROL (exactly 0)" if u == 0 and diff == 0.0 else
|
||||
"CONTROL" if u == 0 else
|
||||
"resolved" if t >= 2 else "NOT RESOLVED (noise)")
|
||||
print(f"U={u} delay={row[DELAY]:>5g} "
|
||||
f"{row.mean_ratio:.4f}+-{row.sem_ratio:.4f} "
|
||||
f"{orow.mean_ratio:.4f}+-{orow.sem_ratio:.4f} "
|
||||
f"{diff:+.4f} {t:6.2f} {verdict}")
|
||||
print(f"{'':>14} q={q:.4f} q_u={qu:.4f} r={r:.4f} "
|
||||
f"pred={float(expected_ratio(f, q_effective(q, r))):.4f} "
|
||||
f"n_rep={int(row.n_rep)}")
|
||||
if pg is not None:
|
||||
print("\npooled over U>=1 per delay (the caps measure the same difference):")
|
||||
for _, r_ in pooled_by_delay(pg).iterrows():
|
||||
mark = " <-- resolved" if r_.t >= 2 else ""
|
||||
print(f" delay={r_[DELAY]:>5g}: {r_.gap:+.5f} +-{Z95 * r_.se:.5f} "
|
||||
f"t={r_.t:6.2f}{mark}")
|
||||
ctl = pg[pg.max_uncles == 0]
|
||||
ex, tot = int(ctl.n_zero.sum()), int(ctl.n_pair.sum())
|
||||
print(f" U=0 control must be exactly 0 under pairing: {ex}/{tot} pairs "
|
||||
f"-> {'PASSES' if ex == tot else 'FAILS'}")
|
||||
print(f"wrote {len(written)} files -> {out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
259
tools/simulators/tsi/tsi-sim-pernode/scripts/plot_fine_delay.py
Normal file
@ -0,0 +1,259 @@
|
||||
"""High-precision figures for the LOW mixing-delay band (the design regime).
|
||||
|
||||
countable-vs-old.yaml samples delay at 4/8/16/32 with 5 replicates. That resolves the
|
||||
overload regime but leaves the design regime under-measured: every countable-vs-unrestricted
|
||||
gap at delay <= 8 sits inside the replicate noise there, so the only honest statement is
|
||||
"no difference detected" — with no bound on how large an undetected difference could be.
|
||||
|
||||
fine-delay.yaml spends replicates instead of range (delay 1..5, 40 replicates) to turn that
|
||||
into a real bound. Consumes:
|
||||
|
||||
tsi-sweep --config configs/fine-delay.yaml --label fine-countable
|
||||
tsi-sweep --config configs/fine-delay.yaml --old --label fine-old
|
||||
|
||||
and renders (into --out):
|
||||
|
||||
fine_accuracy_vs_delay equilibrium D/D_true vs delay 1..5, countable (solid) vs
|
||||
unrestricted (dashed) per U, replicate-SEM bars.
|
||||
fine_gap_vs_delay THE precision figure: the countable - unrestricted gap with 95%
|
||||
CIs, against the U=0 negative-control band. A CI straddling zero
|
||||
means no difference at this power; the band shows the floor.
|
||||
|
||||
Usage:
|
||||
python scripts/plot_fine_delay.py --countable RUNDIR --old RUNDIR \
|
||||
--out figures/fine-delay
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
from statistics import NormalDist
|
||||
|
||||
import matplotlib.pyplot as plt
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from tsi_sim.plotting import style
|
||||
from tsi_sim.plotting.figures_pernode import (
|
||||
Z95,
|
||||
equilibrium,
|
||||
paired_gaps,
|
||||
pooled_by_delay,
|
||||
rho_for,
|
||||
sem,
|
||||
)
|
||||
|
||||
DELAY = "blend_delay_max"
|
||||
|
||||
|
||||
def _load(run_dir: str | Path) -> pd.DataFrame:
|
||||
return pd.read_parquet(Path(run_dir) / "results.parquet")
|
||||
|
||||
|
||||
def _cells(df: pd.DataFrame) -> pd.DataFrame:
|
||||
"""Per (delay, U): replicate mean, SEM and count of the equilibrium accuracy."""
|
||||
return equilibrium(df).groupby([DELAY, "max_uncles"], as_index=False).agg(
|
||||
mean_ratio=("mean_ratio", "mean"),
|
||||
sem_ratio=("mean_ratio", sem),
|
||||
n_rep=("mean_ratio", "size"),
|
||||
mean_q=("mean_q", "mean"),
|
||||
mean_q_eff=("mean_q_eff", "mean"))
|
||||
|
||||
|
||||
def vs_one(cells: pd.DataFrame) -> pd.DataFrame:
|
||||
"""Test each U >= 1 cell against the exact target 1.0.
|
||||
|
||||
An independent read on the same question the gap test asks: the report's claim is that
|
||||
uncle recovery restores the equilibrium to EXACTLY the true stake, so a systematic
|
||||
shortfall across uncle caps is the first-fork cost seen from the absolute side rather
|
||||
than differentially. 40 replicates give ~0.0005 resolution, enough to see 0.1%.
|
||||
"""
|
||||
u = cells[cells.max_uncles > 0].copy()
|
||||
u["dev"] = u.mean_ratio - 1.0
|
||||
u["t"] = u.dev / u.sem_ratio.replace(0, np.nan)
|
||||
return u.sort_values(["max_uncles", DELAY])
|
||||
|
||||
|
||||
def gaps(cnt: pd.DataFrame, old: pd.DataFrame) -> pd.DataFrame:
|
||||
"""countable - unrestricted per cell, with the unpaired SE and 95% CI half-width."""
|
||||
m = cnt.merge(old, on=[DELAY, "max_uncles"], suffixes=("_c", "_o"))
|
||||
m["gap"] = m.mean_ratio_c - m.mean_ratio_o
|
||||
m["se"] = np.hypot(m.sem_ratio_c, m.sem_ratio_o)
|
||||
m["ci95"] = Z95 * m.se
|
||||
m["t"] = np.where(m.se > 0, np.abs(m.gap) / m.se.replace(0, np.nan), np.inf)
|
||||
return m.sort_values(["max_uncles", DELAY])
|
||||
|
||||
|
||||
def fig_accuracy(cnt: pd.DataFrame, old: pd.DataFrame) -> plt.Figure:
|
||||
fig, ax = plt.subplots(figsize=(6.4, 4.2))
|
||||
for i, u in enumerate(sorted(cnt["max_uncles"].unique())):
|
||||
c = style.color_for(i)
|
||||
a = cnt[cnt.max_uncles == u].sort_values(DELAY)
|
||||
b = old[old.max_uncles == u].sort_values(DELAY)
|
||||
ctl = " (control)" if u == 0 else ""
|
||||
ax.errorbar(a[DELAY], a.mean_ratio, yerr=a.sem_ratio, fmt="-o", color=c,
|
||||
label=f"U={u} countable{ctl}", ms=4, capsize=2, lw=1.2)
|
||||
ax.errorbar(b[DELAY], b.mean_ratio, yerr=b.sem_ratio, fmt="--s", color=c,
|
||||
label=f"U={u} unrestricted{ctl}", ms=4, capsize=2, alpha=0.75, lw=1.2)
|
||||
ax.axhline(1.0, color="0.4", lw=0.8, ls=":")
|
||||
ax.set_xlabel("max per-relay mixing delay (slots)")
|
||||
ax.set_ylabel(r"equilibrium $\hat{D}/D_{true}$")
|
||||
ax.set_title("Design-regime accuracy: countable vs unrestricted referencing")
|
||||
ax.legend(ncol=2, fontsize="x-small")
|
||||
return fig
|
||||
|
||||
|
||||
def fig_gap(g: pd.DataFrame) -> plt.Figure:
|
||||
"""The countable − unrestricted gap with 95% CIs, zoomed to the U >= 1 scale.
|
||||
|
||||
The U=0 negative control is NOT plotted as a band here: with no uncles the models are
|
||||
identical by construction, but the unrecovered regime is so noisy that its CI (±0.025)
|
||||
is ~17x the entire range of the U >= 1 gaps and would fill the axes. Its magnitude is
|
||||
annotated instead — the point being that the control's noise floor lives far outside
|
||||
anything the uncle arms show, so those arms are measuring signal, not spread.
|
||||
"""
|
||||
fig, ax = plt.subplots(figsize=(6.6, 4.2))
|
||||
pooled = pooled_by_delay(g)
|
||||
for i, u in enumerate(sorted(g["max_uncles"].unique())):
|
||||
if u == 0:
|
||||
continue
|
||||
a = g[g.max_uncles == u].sort_values(DELAY)
|
||||
ax.errorbar(a[DELAY], a.gap, yerr=a.ci95, fmt="-o", color=style.color_for(i),
|
||||
label=f"U={u}", ms=4, capsize=3, lw=1.0, alpha=0.75)
|
||||
ax.errorbar(pooled[DELAY], pooled.gap, yerr=pooled.ci95, fmt="-D", color="0.15",
|
||||
label="pooled over U≥1", ms=5, capsize=4, lw=1.8, zorder=5)
|
||||
ax.axhline(0.0, color="0.3", lw=0.9, ls=":")
|
||||
ax.set_xlabel("max per-relay mixing delay (slots)")
|
||||
ax.set_ylabel(r"$\hat{D}/D$ gap: countable $-$ unrestricted")
|
||||
ax.set_title("The first-fork cost across the design band (95% CI)")
|
||||
ctl = g[g.max_uncles == 0]
|
||||
if len(ctl):
|
||||
band = float(ctl.ci95.max())
|
||||
span = float(np.abs(np.r_[g[g.max_uncles > 0].gap + g[g.max_uncles > 0].ci95,
|
||||
g[g.max_uncles > 0].gap - g[g.max_uncles > 0].ci95]).max())
|
||||
ax.set_ylim(-1.35 * span, 1.35 * span)
|
||||
# Under pairing the control is exactly 0 in every replicate pair (shared streams), so
|
||||
# quoting a CI for it is meaningless; unpaired, the width of that CI is the point.
|
||||
note = ("U=0 negative control: exactly 0.0 in all 200 replicate pairs "
|
||||
"(shared streams — an identity check, not a noise check)"
|
||||
if band == 0.0 else
|
||||
f"U=0 negative control (true gap = 0): 95% CI ±{band:.4f}, "
|
||||
f"{band / span:.0f}× outside this range")
|
||||
ax.text(0.015, 0.03, note, transform=ax.transAxes, fontsize=6.5, alpha=0.75)
|
||||
ax.legend(fontsize="x-small", ncol=2)
|
||||
return fig
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--countable", required=True, help="run dir of fine-countable")
|
||||
ap.add_argument("--old", required=True, help="run dir of fine-old")
|
||||
ap.add_argument("--out", default="figures/fine-delay")
|
||||
args = ap.parse_args()
|
||||
style.apply_style()
|
||||
out = Path(args.out)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
cnt_raw, old_raw = _load(args.countable), _load(args.old)
|
||||
cnt, old = _cells(cnt_raw), _cells(old_raw)
|
||||
unpaired = gaps(cnt, old)
|
||||
pg = paired_gaps(cnt_raw, old_raw)
|
||||
# The paired test supersedes the unpaired one when both arms share their streams: same
|
||||
# estimand, far smaller standard error. Keep the unpaired numbers for the variance-
|
||||
# reduction report below.
|
||||
g = unpaired if pg is None else unpaired.drop(columns=["gap", "se", "ci95", "t"]).merge(
|
||||
pg[[DELAY, "max_uncles", "gap", "se", "ci95", "t"]], on=[DELAY, "max_uncles"])
|
||||
prov = ("tsi-sim-pernode fine-delay%s.yaml (+--old)"
|
||||
% ("-paired" if pg is not None else ""))
|
||||
|
||||
# rho per delay, derived (never hand-substituted) — these are the report's axis labels.
|
||||
delays = sorted(cnt[DELAY].unique())
|
||||
print("load rho = f*D_vis per delay (measured ell_mean, see figures_pernode.rho_for):")
|
||||
print(" " + " ".join(f"delay={d:g}: rho={r:.3f}"
|
||||
for d, r in zip(delays, rho_for(cnt_raw, delays), strict=True)))
|
||||
print()
|
||||
|
||||
written = []
|
||||
written += style.save(fig_accuracy(cnt, old), out / "fine_accuracy_vs_delay", prov)
|
||||
written += style.save(fig_gap(g), out / "fine_gap_vs_delay", prov)
|
||||
|
||||
print(f"{'cell':<16} {'countable':>17} {'unrestricted':>17} "
|
||||
f"{'gap':>9} {'95% CI':>9} {'t':>6} verdict")
|
||||
for _, r in g.iterrows():
|
||||
verdict = ("CONTROL (true gap = 0)" if r.max_uncles == 0 else
|
||||
"resolved" if r.t >= 2 else "no difference resolved")
|
||||
print(f"U={int(r.max_uncles)} delay={r[DELAY]:>5g} "
|
||||
f"{r.mean_ratio_c:.4f}+-{r.sem_ratio_c:.4f} "
|
||||
f"{r.mean_ratio_o:.4f}+-{r.sem_ratio_o:.4f} "
|
||||
f"{r.gap:+.4f} +-{r.ci95:.4f} {r.t:6.2f} {verdict}")
|
||||
worst = g[g.max_uncles > 0]
|
||||
print(f"\nn_rep = {int(g.n_rep_c.min())}/{int(g.n_rep_o.min())} per arm")
|
||||
print(f"widest 95% CI half-width at U>=1: +-{worst.ci95.max():.4f} "
|
||||
f"({100 * worst.ci95.max():.2f} pp)")
|
||||
# Per-cell significance must be read against the number of cells tested — some are
|
||||
# expected to clear t=2 by chance alone — so quote the Bonferroni threshold for the grid
|
||||
# actually run, not a constant baked in for one grid size.
|
||||
bonf = NormalDist().inv_cdf(1 - 0.05 / (2 * len(worst)))
|
||||
print(f"per-cell: {int((worst.t >= 2).sum())}/{len(worst)} cells with t>=2 "
|
||||
f"(expected by chance {0.05 * len(worst):.2f}); max t = {worst.t.max():.2f} "
|
||||
f"vs Bonferroni threshold {bonf:.3f}")
|
||||
|
||||
print("\npooled over U>=1 (the three caps measure the same difference):")
|
||||
pooled = pooled_by_delay(g)
|
||||
for _, r in pooled.iterrows():
|
||||
mark = " <-- resolved" if r.t >= 2 else ""
|
||||
print(f" delay={r[DELAY]:>4g}: {r.gap:+.5f} +-{r.ci95:.5f} t={r.t:5.2f}{mark}")
|
||||
w = 1.0 / worst.se.to_numpy() ** 2
|
||||
allp = float((worst.gap.to_numpy() * w).sum() / w.sum())
|
||||
alle = float(np.sqrt(1.0 / w.sum()))
|
||||
print(f" whole band: {allp:+.5f} +-{Z95 * alle:.5f} t={abs(allp) / alle:.2f}")
|
||||
|
||||
if pg is not None:
|
||||
print("\n=== PAIRED design (common random numbers) ===")
|
||||
# Variance reduction actually achieved, per cell, vs the unpaired standard error.
|
||||
cmp = unpaired[[DELAY, "max_uncles", "se"]].merge(
|
||||
pg[[DELAY, "max_uncles", "se", "n_pair"]], on=[DELAY, "max_uncles"],
|
||||
suffixes=("_unpaired", "_paired"))
|
||||
u = cmp[cmp.max_uncles > 0]
|
||||
ratio = (u.se_unpaired / u.se_paired.replace(0, np.nan))
|
||||
print(f" SE shrink at U>=1: median {ratio.median():.1f}x, range "
|
||||
f"{ratio.min():.1f}-{ratio.max():.1f}x ({int(u.n_pair.min())} pairs/cell)")
|
||||
pctl = pg[pg.max_uncles == 0]
|
||||
exact, tot = int(pctl.n_zero.sum()), int(pctl.n_pair.sum())
|
||||
print(f" U=0 control under pairing must be EXACTLY zero: {exact}/{tot} pairs are 0.0"
|
||||
f" -> {'PASSES' if exact == tot else 'FAILS — streams are not shared'}")
|
||||
res = pg[(pg.max_uncles > 0) & (pg.t >= 2)]
|
||||
n_cells = len(pg[pg.max_uncles > 0])
|
||||
n_unpaired = int((unpaired[unpaired.max_uncles > 0].t >= 2).sum())
|
||||
print(f" per-cell resolved at |t|>=2: {len(res)}/{n_cells}"
|
||||
f" (same data, unpaired test: {n_unpaired}/{n_cells})")
|
||||
|
||||
ctl = g[g.max_uncles == 0]
|
||||
if len(ctl):
|
||||
# Under pairing the control gap is EXACTLY 0, so its se is 0 and t is 0/0. That is the
|
||||
# ideal outcome, not a failure — check the gap itself, and only fall back to the
|
||||
# t-based check when there is real spread to test (the unpaired case).
|
||||
worst = float(ctl.gap.abs().max())
|
||||
exact = bool((ctl.se == 0).all()) if "se" in ctl else False
|
||||
ok = (worst == 0.0) if exact else (float(ctl.t.max()) < 2)
|
||||
how = "identical by construction" if exact else "within noise"
|
||||
print(f"\nU=0 negative control (true gap = 0): |gap| up to {worst:.4g} ({how})"
|
||||
f" -> control {'PASSES' if ok else 'FAILS'}")
|
||||
|
||||
# Absolute test: does uncle recovery actually land on 1.0? Same question as the gap
|
||||
# test, asked without reference to the other model.
|
||||
for lbl, cells in (("countable", cnt), ("unrestricted", old)):
|
||||
v = vs_one(cells)
|
||||
lo = v[v.t <= -2]
|
||||
print(f"\nvs exact 1.0, {lbl}: {len(lo)}/{len(v)} cells significantly BELOW 1")
|
||||
if len(lo):
|
||||
for d, s in lo.groupby(DELAY):
|
||||
caps = "/".join(f"U={int(x)}" for x in sorted(s.max_uncles))
|
||||
print(f" delay={d:>4g}: {caps} dev {s.dev.min():+.5f}..{s.dev.max():+.5f} "
|
||||
f"t {s.t.min():.2f}..{s.t.max():.2f}")
|
||||
print(f"wrote {len(written)} files -> {out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -22,13 +22,11 @@ import pandas as pd
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
|
||||
|
||||
from tsi_sim.plotting import style # noqa: E402
|
||||
from tsi_sim.plotting.figures_pernode import graph_ell_mean # noqa: E402
|
||||
|
||||
HERE = Path(__file__).resolve().parent.parent
|
||||
RUNS = HERE / "runs"
|
||||
FIGS = HERE / "report-figures"
|
||||
F = 1.0 / 30.0
|
||||
HOPS = 3
|
||||
LMEAN = 1.2 # degree-6, N=1000 geo graph (matches §4)
|
||||
|
||||
|
||||
def load() -> pd.DataFrame:
|
||||
@ -37,9 +35,19 @@ def load() -> pd.DataFrame:
|
||||
keys = ["blend_delay_max", "max_uncles", "replicate"]
|
||||
df["emax"] = df.groupby(keys).epoch.transform("max")
|
||||
tail = df[df.epoch >= df.emax // 2]
|
||||
g = (tail.groupby(["blend_delay_max", "max_uncles"])
|
||||
# Per-trajectory tail mean FIRST, then mean + SEM ACROSS replicates. Pooling every
|
||||
# (replicate x tail-epoch) row instead would treat correlated within-trajectory epochs as
|
||||
# independent samples and understate the true replicate spread (by ~1.5x, up to ~3x).
|
||||
per_rep = (tail.groupby(keys, as_index=False).mean_ratio.mean())
|
||||
g = (per_rep.groupby(["blend_delay_max", "max_uncles"])
|
||||
.mean_ratio.agg(["mean", "sem"]).reset_index())
|
||||
g["rho"] = F * (HOPS * g.blend_delay_max / 2.0 + (HOPS + 1) * LMEAN)
|
||||
# Derive the rho axis from the run itself — f, hops, and the *measured* ell_mean — not from
|
||||
# hardcoded constants: rho = f*D_vis with D_vis = hops*delta_max/2 + (hops+1)*ell_mean. This
|
||||
# only re-labels the x-axis from the existing simulation data; it never re-simulates.
|
||||
f = float(df.f.iloc[0])
|
||||
hops = int(df.blend_hops.iloc[0])
|
||||
ell = graph_ell_mean(df)
|
||||
g["rho"] = f * (hops * g.blend_delay_max / 2.0 + (hops + 1) * ell)
|
||||
g["deficit"] = 1.0 - g["mean"]
|
||||
return g
|
||||
|
||||
@ -48,13 +56,22 @@ def fig26(g: pd.DataFrame) -> None:
|
||||
import matplotlib.pyplot as plt
|
||||
style.apply_style()
|
||||
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.6, 4.2))
|
||||
floor = 3e-4
|
||||
for i, U in enumerate((0, 1, 2, 3)):
|
||||
s = g[g.max_uncles == U].sort_values("rho")
|
||||
c = style.OKABE_ITO[i]
|
||||
# left: deficit (floored at a small positive value for the log axis)
|
||||
d = np.clip(s.deficit.values, 3e-4, None)
|
||||
ax1.plot(s.rho, d, "-o", ms=4, color=c, label=f"U = {U}")
|
||||
# right: accuracy, capped at 1.0
|
||||
# left: deficit on a log axis. A cell counts as a RESOLVED positive deficit only if it
|
||||
# is both positive and above its own 2*SEM noise level; unresolved cells (at/below noise,
|
||||
# or slightly negative because D̂/D sits a hair above 1 from sampling noise) are clamped to
|
||||
# the axis floor and drawn HOLLOW, so a point on the floor cannot be misread as a measured
|
||||
# deficit. A faint line joins the series for legibility.
|
||||
rho = s.rho.values
|
||||
d = np.clip(s.deficit.values, floor, None)
|
||||
resolved = (s.deficit.values > floor) & (s.deficit.values > 2.0 * s["sem"].values)
|
||||
ax1.plot(rho, d, "-", lw=0.8, color=c, alpha=0.5, zorder=0)
|
||||
ax1.plot(rho[resolved], d[resolved], "o", ms=4, color=c, label=f"U = {U}")
|
||||
ax1.plot(rho[~resolved], d[~resolved], "o", ms=4, mfc="none", mec=c)
|
||||
# right: accuracy, capped at 1.0, with the across-replicate SEM
|
||||
ax2.errorbar(s.rho, s["mean"], yerr=s["sem"], fmt="-o", ms=4, capsize=2,
|
||||
color=c, label=f"U = {U}")
|
||||
ax1.set_yscale("log")
|
||||
|
||||
@ -0,0 +1,204 @@
|
||||
"""Does the uncle cap need margin under a private-chain attack? — REPORT §8.3 item 5.
|
||||
|
||||
Item 5: "Under attack-inflated orphaning the honest-load cap may need extra margin (owed uncles
|
||||
beyond `U` defer and can age out of `W`); this report does not size it." It could not be sized
|
||||
before, because the per-node engine had no private-chain strategy (§6.8) — the selfish results
|
||||
came from a global race model in which uncle recovery is a free knob, not a queue with a cap.
|
||||
|
||||
With `adversary_strategy="selfish"` in the engine, the whole loop is present: the attack orphans
|
||||
honest blocks in runs, the survivors queue for the `U` uncle slots of each canonical block, and
|
||||
whatever does not drain within `W` ages out. This sweeps the cap against the attack to find the
|
||||
smallest `U` that still recovers, and compares it to the honest rule `U = ceil(rho) + 1`.
|
||||
|
||||
Three quantities separate the two failure modes the item conflates:
|
||||
|
||||
* ``p_ref_honest`` — of the honest blocks the attacker orphaned, how many got referenced at
|
||||
all. Falls for TWO different reasons, which is why the next column matters.
|
||||
* ``deep_ref_share`` — the share of examined references rejected by the first-fork rule. An
|
||||
override discards a *chain*, and only its first block is countable (§2.1), so this isolates
|
||||
"unreferenceable by construction" from "queue too small".
|
||||
* ``D_hat/D`` — what the estimator actually lands on, the thing the cap is sized to protect.
|
||||
|
||||
If raising `U` lifts recovery, the cap is the binding constraint and item 5 needs a bigger
|
||||
number. If it does not, the loss is structural and no cap buys it back.
|
||||
|
||||
Run: python scripts/selfish_uncle_margin.py (writes runs/selfish_uncle_margin.parquet)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from joblib import Parallel, delayed
|
||||
|
||||
from tsi_sim import lottery, topology
|
||||
from tsi_sim.blocktree import build_tree_pernode
|
||||
from tsi_sim.config import SimConfig
|
||||
from tsi_sim.engine import _adversary_mask, run_trajectory
|
||||
from tsi_sim.memguard import ArrivalMatrixTooLarge
|
||||
from tsi_sim.rng import rng_for, seedseq_for
|
||||
from tsi_sim.stake import make_stake
|
||||
|
||||
HERE = Path(__file__).resolve().parent.parent
|
||||
RUNS = HERE / "runs"
|
||||
RUNS.mkdir(exist_ok=True)
|
||||
|
||||
EPOCHS = 16
|
||||
REPS = 8
|
||||
N_JOBS = 6
|
||||
|
||||
BASE = dict(n_nodes=1000, stake_dist="pareto", topology="blend", degree=6,
|
||||
link_latency_mean=0.5, link_latency_dist="geo", blend_hops=3,
|
||||
k=256, epochs=EPOCHS, genesis_d_factor=0.5, early_stop=False,
|
||||
adversary_strategy="selfish")
|
||||
|
||||
ALPHAS = [0.0, 0.2, 0.3, 0.4]
|
||||
DELAYS = [8.0, 16.0] # rho ~ 0.56 (design point) and ~1.0 (the load boundary)
|
||||
CAPS = [1, 2, 3, 4] # spec allows up to MAX_UNCLES = 4
|
||||
WINDOWS = [10, 20] # W = 10/f (recommended) and the 20/f widening of §3.4
|
||||
|
||||
|
||||
def _cell(alpha: float, delay: float, u: int, w: int, rep: int) -> dict:
|
||||
cfg = SimConfig(**BASE, blend_delay_max=delay, max_uncles=u, window_absorption=w,
|
||||
adversary_frac=alpha, replicate=rep)
|
||||
row = dict(alpha=alpha, blend_delay_max=delay, max_uncles=u, window_absorption=w, rep=rep)
|
||||
try:
|
||||
t = pd.DataFrame(run_trajectory(cfg))
|
||||
t = t[t.epoch >= EPOCHS // 2]
|
||||
row |= dict(collapsed=False,
|
||||
mean_ratio=float(t.mean_ratio.mean()),
|
||||
fork_rate=float(t.fork_rate.mean()),
|
||||
p_ref=float(t.p_ref.mean()),
|
||||
p_ref_honest=float(t.p_ref_honest.mean()),
|
||||
deep_ref_share=float(t.deep_ref_share.mean()),
|
||||
max_reorg_depth=int(t.max_reorg_depth.max()),
|
||||
adv_share=float(t.adv_blocks.sum()
|
||||
/ max(t.adv_blocks.sum() + t.honest_blocks.sum(), 1)))
|
||||
except ArrivalMatrixTooLarge:
|
||||
row |= dict(collapsed=True)
|
||||
return row
|
||||
|
||||
|
||||
def sweep() -> pd.DataFrame:
|
||||
jobs = [(a, d, u, w, r) for a in ALPHAS for d in DELAYS for u in CAPS
|
||||
for w in WINDOWS for r in range(REPS)]
|
||||
df = pd.DataFrame(Parallel(n_jobs=N_JOBS, backend="loky", inner_max_num_threads=1)(
|
||||
delayed(_cell)(a, d, u, w, r) for a, d, u, w, r in jobs))
|
||||
df.to_parquet(RUNS / "selfish_uncle_margin.parquet", index=False)
|
||||
return df
|
||||
|
||||
|
||||
def _decompose_cell(alpha: float, u: int, rep: int, delay: float = 8.0, w: int = 10) -> dict:
|
||||
"""Split the honest orphans into "unreferenceable" and "eligible but unreferenced".
|
||||
|
||||
``p_ref_honest`` alone cannot answer item 5, because it falls for two unrelated reasons: a
|
||||
block can be *structurally* uncountable (buried behind the first block of an override, so no
|
||||
proposer may reference it — §2.1) or countable but starved of an uncle slot (the queue the
|
||||
cap `U` drains). Only the second is a cap-sizing problem. ``deep_ref_share`` does not
|
||||
separate them either: the proposer's candidate filter drops deep-fork blocks before they are
|
||||
ever proposed, so no deep reference is examined and the metric is 0 by construction here.
|
||||
This walks the tree and measures both directly.
|
||||
"""
|
||||
cfg = SimConfig(**{**BASE, "epochs": 4}, blend_delay_max=delay, max_uncles=u,
|
||||
window_absorption=w, adversary_frac=alpha, replicate=rep,
|
||||
prune_arrival=False, windowed_fork_choice=False)
|
||||
stake = make_stake(cfg, rng_for(cfg))
|
||||
mask = _adversary_mask(cfg, stake)
|
||||
flat = np.zeros(cfg.n_nodes, dtype=bool) if mask is None else mask
|
||||
kids = seedseq_for(cfg).spawn(cfg.epochs + 3)
|
||||
pl = topology.build_path_latency(cfg, np.random.default_rng(kids[1]))
|
||||
d_est = np.full(cfg.n_nodes, cfg.genesis_d_factor * float(stake.sum()))
|
||||
p = lottery.win_probs(stake, d_est, cfg.f)
|
||||
ws, wn = lottery.sample_wins(p, cfg.epoch_len, np.random.default_rng(kids[3]))
|
||||
slots, groups = lottery.group_by_slot(ws, wn)
|
||||
tree, A = build_tree_pernode(slots, groups, pl, cfg, np.random.default_rng(kids[4]),
|
||||
adversary_mask=mask)
|
||||
|
||||
E, T, nb = cfg.epoch_len, cfg.period_T, tree.n_blocks
|
||||
ids = np.arange(nb)
|
||||
arrived = (A <= E).any(axis=0)
|
||||
arrived[0] = True
|
||||
h = np.where(arrived, tree.height, np.iinfo(np.int64).min)
|
||||
canon = np.zeros(nb, dtype=bool)
|
||||
b = int(np.lexsort((-ids, -tree.slot, h))[-1])
|
||||
while b > 0:
|
||||
canon[b] = True
|
||||
b = int(tree.parent[b])
|
||||
canon[0] = True
|
||||
in_win = (tree.slot >= 0) & (tree.slot < T)
|
||||
hon_orph = in_win & ~canon & ~flat[tree.leader]
|
||||
countable = hon_orph & canon[tree.parent] # first block of its fork
|
||||
referenced = np.zeros(nb, dtype=bool)
|
||||
for cb in np.nonzero(canon)[0]:
|
||||
for un in tree.uncles[cb]:
|
||||
referenced[un] = True
|
||||
n, nc = int(hon_orph.sum()), int(countable.sum())
|
||||
return dict(alpha=alpha, max_uncles=u, rep=rep, honest_orphans=n,
|
||||
countable_share=(nc / n) if n else np.nan,
|
||||
referenced_of_countable=(int((countable & referenced).sum()) / nc)
|
||||
if nc else np.nan,
|
||||
referenced_of_all=(int((hon_orph & referenced).sum()) / n) if n else np.nan)
|
||||
|
||||
|
||||
def decompose(reps: int = 6) -> pd.DataFrame:
|
||||
jobs = [(a, u, r) for a in (0.0, 0.2, 0.3) for u in (1, 2, 4) for r in range(reps)]
|
||||
df = pd.DataFrame(Parallel(n_jobs=N_JOBS, backend="loky", inner_max_num_threads=1)(
|
||||
delayed(_decompose_cell)(a, u, r) for a, u, r in jobs))
|
||||
df.to_parquet(RUNS / "selfish_uncle_margin_decomp.parquet", index=False)
|
||||
return df
|
||||
|
||||
|
||||
def report_decomposition(df: pd.DataFrame) -> None:
|
||||
print("\n=== why p_ref_honest falls: structure vs queue (delta = 8, W = 10) ===")
|
||||
print(f"{'alpha':>6} {'U':>2} | {'countable share':>16} {'referenced OF those':>20}"
|
||||
f" {'referenced of all':>18}")
|
||||
for a in sorted(df.alpha.unique()):
|
||||
for u in sorted(df.max_uncles.unique()):
|
||||
g = df[(df.alpha == a) & (df.max_uncles == u)]
|
||||
print(f"{a:6.2f} {u:2d} | {g.countable_share.mean() * 100:14.1f}%"
|
||||
f" {g.referenced_of_countable.mean() * 100:18.1f}%"
|
||||
f" {g.referenced_of_all.mean() * 100:16.1f}%")
|
||||
|
||||
|
||||
BAR = 0.98 # the §3.6 recovery bar, as a fraction of the true stake
|
||||
|
||||
|
||||
def report(df: pd.DataFrame) -> None:
|
||||
ok = df[~df.collapsed]
|
||||
for w in WINDOWS:
|
||||
print(f"\n=== W = {w} block-intervals ===")
|
||||
print(f"{'delta':>6} {'alpha':>6} | " + " ".join(f"U={u}" for u in CAPS)
|
||||
+ " | smallest U >= bar p_ref_h deep_ref fork")
|
||||
for d in DELAYS:
|
||||
for a in ALPHAS:
|
||||
g = ok[(ok.window_absorption == w) & (ok.blend_delay_max == d) & (ok.alpha == a)]
|
||||
if g.empty:
|
||||
continue
|
||||
cells, best = [], None
|
||||
for u in CAPS:
|
||||
gu = g[g.max_uncles == u]
|
||||
m = gu.mean_ratio.mean() if len(gu) else np.nan
|
||||
cells.append(f"{m:.3f}")
|
||||
if best is None and m >= BAR:
|
||||
best = u
|
||||
ref = g[g.max_uncles == max(CAPS)]
|
||||
print(f"{d:6.1f} {a:6.2f} | " + " ".join(cells)
|
||||
+ f" | {str(best):>4} {ref.p_ref_honest.mean():7.3f}"
|
||||
+ f" {ref.deep_ref_share.mean():8.3f} {ref.fork_rate.mean():5.3f}")
|
||||
n_col = int(df.collapsed.sum())
|
||||
if n_col:
|
||||
print(f"\n{n_col} of {len(df)} runs collapsed into the §6.2 branch (excluded above)")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
print(f"=== selfish uncle-margin sweep ({len(ALPHAS)*len(DELAYS)*len(CAPS)*len(WINDOWS)*REPS}"
|
||||
f" runs; recovery bar {BAR}) ===")
|
||||
report(sweep())
|
||||
report_decomposition(decompose())
|
||||
print(f"\nwrote {RUNS}/selfish_uncle_margin{{,_decomp}}.parquet")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -1,214 +0,0 @@
|
||||
"""One-time migration: split REPORT-tsi-parameter-selection.md into a thematic 4-part set + index.
|
||||
|
||||
The single report grew dense and heavily cross-referenced; this slices it into four cohesive parts
|
||||
(kept in tsi-sim-pernode/ so all report-figures/ links stay valid) plus a short index that reuses the
|
||||
canonical filename as the entry point. Section NUMBERS (§1-§9, A-C) are preserved as stable identifiers
|
||||
across files; every §ref is rewritten into a clickable link to a portable `<a id="s6-5">` anchor,
|
||||
same-file or cross-file as appropriate. Figure embeds and §9's config/script/run paths are untouched.
|
||||
|
||||
Run: python scripts/split_report.py (reads REPORT-...md, writes the 4 parts + overwrites the index)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# ruff: noqa: E501 (one-time migration; index/nav strings are intentionally long prose)
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent.parent
|
||||
SRC = HERE / "REPORT-tsi-parameter-selection.md"
|
||||
INDEX = "REPORT-tsi-parameter-selection.md"
|
||||
P1 = "tsi-report-1-overview-and-recommendations.md"
|
||||
P2 = "tsi-report-2-accuracy-and-design.md"
|
||||
P3 = "tsi-report-3-robustness-and-incentives.md"
|
||||
P4 = "tsi-report-4-reproducibility-and-appendices.md"
|
||||
|
||||
# top-level section id -> part filename, and the section order within each part
|
||||
PART_SECTIONS = {
|
||||
P1: ["1", "7", "8"],
|
||||
P2: ["2", "3", "4", "5"],
|
||||
P3: ["6"],
|
||||
P4: ["9", "A", "B", "C"],
|
||||
}
|
||||
PART_TITLE = {
|
||||
P1: "Part 1 — Overview and recommendations",
|
||||
P2: "Part 2 — Accuracy and design",
|
||||
P3: "Part 3 — Robustness and incentives",
|
||||
P4: "Part 4 — Reproducibility and appendices",
|
||||
}
|
||||
SEC_TO_FILE = {s: f for f, secs in PART_SECTIONS.items() for s in secs}
|
||||
|
||||
|
||||
def top_id(line: str) -> str | None:
|
||||
m = re.match(r"^##\s+Appendix\s+([A-C])\b", line)
|
||||
if m:
|
||||
return m.group(1)
|
||||
m = re.match(r"^##\s+(\d+)\.", line)
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
def header_anchor(line: str) -> str | None:
|
||||
"""Anchor id for a section/subsection header line, e.g. §6.5 -> s6-5, App B.2 -> sB-2."""
|
||||
m = re.match(r"^##\s+Appendix\s+([A-C])\b", line)
|
||||
if m:
|
||||
return "s" + m.group(1)
|
||||
m = re.match(r"^##\s+(\d+)\.", line)
|
||||
if m:
|
||||
return "s" + m.group(1)
|
||||
m = re.match(r"^###\s+([0-9A-C]+)\.(\d+)", line)
|
||||
if m:
|
||||
return f"s{m.group(1)}-{m.group(2)}"
|
||||
return None
|
||||
|
||||
|
||||
def make_ref_rewriter(current_file: str):
|
||||
"""Rewrite §N/§N.M and 'Appendix X' refs into links to their anchor (same- or cross-file)."""
|
||||
def link(top: str, sub: str | None, label: str) -> str:
|
||||
anchor = "s" + top + (f"-{sub}" if sub else "")
|
||||
tgt = SEC_TO_FILE.get(top)
|
||||
if tgt is None:
|
||||
return label # unknown target: leave as text
|
||||
dest = f"#{anchor}" if tgt == current_file else f"{tgt}#{anchor}"
|
||||
return f"[{label}]({dest})"
|
||||
|
||||
def sec_sub(m: re.Match) -> str:
|
||||
top, sub = m.group(1), m.group(2)
|
||||
return link(top, sub, m.group(0))
|
||||
|
||||
def appendix(m: re.Match) -> str:
|
||||
return link(m.group(1), None, m.group(0))
|
||||
|
||||
sec_re = re.compile(r"§\s?(\d+)(?:\.(\d+))?")
|
||||
app_re = re.compile(r"\bAppendix\s+([A-C])\b")
|
||||
|
||||
def rewrite(text: str) -> str:
|
||||
return app_re.sub(appendix, sec_re.sub(sec_sub, text))
|
||||
|
||||
return rewrite
|
||||
|
||||
|
||||
def render_lines(lines: list[str], current_file: str) -> list[str]:
|
||||
"""Inject anchors before headers and rewrite §refs, skipping fenced code blocks."""
|
||||
rewrite = make_ref_rewriter(current_file)
|
||||
out: list[str] = []
|
||||
in_fence = False
|
||||
for ln in lines:
|
||||
if ln.lstrip().startswith("```"):
|
||||
in_fence = not in_fence
|
||||
out.append(ln)
|
||||
continue
|
||||
if in_fence:
|
||||
out.append(ln) # never touch code (refs there stay plain text)
|
||||
continue
|
||||
aid = header_anchor(ln)
|
||||
if aid is not None:
|
||||
out.append(f'<a id="{aid}"></a>')
|
||||
out.append(ln) # header title kept verbatim (no links inside headers)
|
||||
continue
|
||||
out.append(rewrite(ln))
|
||||
return out
|
||||
|
||||
|
||||
def part_header(fname: str, units_note: str) -> list[str]:
|
||||
nav = (f"*[Part 1 — Overview & recommendations]({P1}) · [Part 2 — Accuracy & design]({P2}) · "
|
||||
f"[Part 3 — Robustness & incentives]({P3}) · [Part 4 — Reproducibility & appendices]({P4}) · "
|
||||
f"[Index]({INDEX})*")
|
||||
where = ("*Sections live across the set: §1/§7/§8 in Part 1, §2–§5 in Part 2, §6 in Part 3, "
|
||||
"§9 and Appendices A–C in Part 4.*")
|
||||
return [
|
||||
f"# Total-Stake-Inference parameter selection — {PART_TITLE[fname].split('— ')[1]}",
|
||||
"",
|
||||
units_note,
|
||||
"",
|
||||
nav,
|
||||
"",
|
||||
where,
|
||||
"",
|
||||
"---",
|
||||
"",
|
||||
]
|
||||
|
||||
|
||||
def build_index(units_note: str) -> str:
|
||||
lines = [
|
||||
"# Total-Stake-Inference parameter selection",
|
||||
"",
|
||||
units_note,
|
||||
"",
|
||||
"This analysis selects and justifies the TSI parameters for Cryptarchia, from a per-node "
|
||||
"network simulation (`tsi-sim-pernode`). It is split into four parts:",
|
||||
"",
|
||||
f"1. **[Overview and recommendations]({P1})** — the executive summary, the per-knob parameter "
|
||||
"reference (§7), and the safest selection with residual risks and the recommendation-vs-spec "
|
||||
"deltas (§8).",
|
||||
f"2. **[Accuracy and design]({P2})** — the model and counting rule (§2), the seven findings and "
|
||||
"their evidence (§3), and the design equations / selection algorithm (§4–§5).",
|
||||
f"3. **[Robustness and incentives]({P3})** — jitter, grinding, withholding, selfish mining, the "
|
||||
"reward design, fork/reorg depth, and organic churn (§6).",
|
||||
f"4. **[Reproducibility and appendices]({P4})** — how to re-run every study (§9), the residual "
|
||||
"f-rounding offset (App A), the per-epoch noise floor (App B), and consensus detail (App C).",
|
||||
"",
|
||||
f"**Headline recommendation** (Cryptarchia baseline f = 1/30): security `k = 2160`, uncle "
|
||||
f"window `W = 300` slots, uncle cap `U = ⌈ρ⌉ + 1` (2 at the Blend target), learning rate "
|
||||
f"`β = 1`, peering degree ≥ 6 at scale, soft uncle rewards with `w_u + w_n < 1`, and operate "
|
||||
f"at load `ρ = f·D_vis < 1`. The full recommended-configuration table and rationale are in "
|
||||
f"**[Part 1 →]({P1})**.",
|
||||
"",
|
||||
]
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
text = SRC.read_text()
|
||||
if "## 6. Robustness" not in text:
|
||||
sys.exit("Source has already been split (no '## 6. Robustness' found in "
|
||||
f"{SRC.name}, which is now the index). This one-time migration is complete; "
|
||||
"re-run against the pre-split backup only.")
|
||||
raw = text.split("\n")
|
||||
|
||||
# frontmatter (title + units note) is everything before the first "## " header
|
||||
first_h = next(i for i, l in enumerate(raw) if l.startswith("## "))
|
||||
units_note = raw[2] # the italic "*Per-node network simulation ... 1 slot = 1 s.*" line
|
||||
|
||||
# slice into top-level sections
|
||||
sections: dict[str, list[str]] = {}
|
||||
cur: str | None = None
|
||||
for l in raw[first_h:]:
|
||||
tid = top_id(l)
|
||||
if tid is not None:
|
||||
cur = tid
|
||||
sections[cur] = []
|
||||
if cur is not None:
|
||||
sections[cur].append(l)
|
||||
|
||||
# update the reading-order note (in §1) to describe the 4-part structure
|
||||
ro_old_prefix = "The rest of the report, in reading order:"
|
||||
ro_new = ("This report is split into four parts (see the [index](" + INDEX + ")): "
|
||||
"**Part 1** — the recommended configuration, the per-knob parameter reference (§7) and "
|
||||
"the safest selection with residual risks and spec deltas (§8); **Part 2** — the model "
|
||||
"and counting rule (§2), the evidence behind each finding (§3), and the design equations "
|
||||
"and selection algorithm (§4–§5); **Part 3** — robustness against noise, attacks and the "
|
||||
"incentive design (§6); **Part 4** — reproducibility (§9) and the appendices (the residual "
|
||||
"~1 % f-rounding offset, the ±0.9 % per-epoch noise floor, and consensus detail).")
|
||||
sections["1"] = [
|
||||
ro_new if l.startswith(ro_old_prefix) else l for l in sections["1"]
|
||||
]
|
||||
|
||||
# assemble each part
|
||||
for fname, sec_ids in PART_SECTIONS.items():
|
||||
body: list[str] = list(part_header(fname, units_note))
|
||||
for sid in sec_ids:
|
||||
body.extend(sections[sid])
|
||||
body.append("") # spacer between sections
|
||||
rendered = render_lines(body, fname)
|
||||
(HERE / fname).write_text("\n".join(rendered).rstrip() + "\n")
|
||||
print(f"wrote {fname} ({len(rendered)} lines)")
|
||||
|
||||
# index last (overwrites the source-name file)
|
||||
(HERE / INDEX).write_text(build_index(units_note))
|
||||
print(f"wrote {INDEX} (index)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -1,9 +1,9 @@
|
||||
"""Relative stake estimate vs network delay (fig16).
|
||||
|
||||
Shows the report's central relationship: the recovered *relative stake* ``D̂/D`` as a function of the
|
||||
mean block-visibility delay ``D_vis`` (seconds), one curve per uncle cap ``U``. Accuracy holds near
|
||||
the ceiling ``c(f)`` while the load ``ρ = f·D_vis`` stays below ``⌈U⌉``, then collapses — so larger
|
||||
delay needs more uncles. Blend transport, f = 1/30 (30 s blocks); delay swept via the
|
||||
mean block-visibility delay ``D_vis`` (seconds), one curve per uncle cap ``U``. Accuracy holds at
|
||||
the 1.0 exact-recovery bound while the load ``ρ = f·D_vis`` stays below ``⌈U⌉``, then collapses — so
|
||||
larger delay needs more uncles. Blend transport, f = 1/30 (30 s blocks); delay swept via the
|
||||
per-hop blending budget ``blend_delay_max``.
|
||||
|
||||
Run: python scripts/stake_vs_delay.py (writes runs/stake_vs_delay.parquet + fig16)
|
||||
|
||||
@ -174,6 +174,76 @@ def _max_span_blocks(active_slots: np.ndarray, counts: np.ndarray, span: float)
|
||||
return best
|
||||
|
||||
|
||||
class _SelfishCoalition:
|
||||
"""Eyal–Sirer SM1 private-chain state, driven from the coalition's shared view.
|
||||
|
||||
The coalition mines one private chain and releases it under the classic SM1 rules, in terms
|
||||
of ``a`` = unreleased private blocks since the fork and ``h`` = public blocks since the fork
|
||||
as the coalition sees them:
|
||||
|
||||
h > a adopt — the public chain won; the private blocks are dead
|
||||
h == a (a > 0) match — release all; the two chains race at equal length
|
||||
h == a - 1 (a>=2) override — release all; the public ``h`` blocks are orphaned
|
||||
h < a - 1 wait — stay hidden and keep the lead
|
||||
|
||||
Only *visibility* is modelled here; the coalition's **mining** needs no special case. A
|
||||
coalition member's fork choice already builds on the private tip whenever the private chain
|
||||
leads, because that tip has the greatest height among the blocks that member can see — and
|
||||
it falls back to the public chain exactly when the public chain overtakes, which is the
|
||||
"adopt" branch. So the private chain forms, extends and is abandoned emergently.
|
||||
|
||||
The coalition is treated as one entity that shares its view instantly: a member learns a
|
||||
public block as soon as the *earliest* member does, and sees every private block at its
|
||||
production slot. Both are best-case-for-the-adversary, which is the direction we want for a
|
||||
bound on the damage.
|
||||
"""
|
||||
|
||||
def __init__(self, coal_idx: np.ndarray, n_blocks: int, E: int) -> None:
|
||||
self.coal = coal_idx
|
||||
self.priv: list[int] = [] # unreleased private blocks, oldest first
|
||||
self.fork_height = 0 # height of the block the private chain forked from
|
||||
self.unreleased = np.zeros(n_blocks, dtype=bool)
|
||||
self.coal_arr = np.full(n_blocks, float(E) + 1.0) # when the coalition learns of a block
|
||||
self.coal_arr[0] = 0.0
|
||||
self.n_released = 0 # blocks made public by a release
|
||||
self.n_abandoned = 0 # private blocks the coalition gave up on
|
||||
self.n_override = 0 # releases that orphaned >=1 honest block
|
||||
|
||||
def note_block(self, b: int, arrival_at_coalition: float) -> None:
|
||||
self.coal_arr[b] = arrival_at_coalition
|
||||
|
||||
def add_private(self, b: int, t: int, parent_height: int) -> None:
|
||||
if not self.priv: # opening a new private chain: record its fork height
|
||||
self.fork_height = parent_height
|
||||
self.priv.append(b)
|
||||
self.unreleased[b] = True
|
||||
self.coal_arr[b] = float(t) # shared inside the coalition immediately
|
||||
|
||||
def public_height(self, t: int, height: np.ndarray, nb: int) -> int:
|
||||
"""Best height the coalition can see on the PUBLIC chain (private blocks excluded)."""
|
||||
vis = (self.coal_arr[:nb] <= t) & (~self.unreleased[:nb])
|
||||
return int(height[:nb][vis].max()) if vis.any() else 0
|
||||
|
||||
def decide(self, t: int, height: np.ndarray, nb: int) -> list[int]:
|
||||
"""Apply the SM1 rule; return the private blocks to release now (possibly empty)."""
|
||||
a = len(self.priv)
|
||||
if a == 0:
|
||||
return []
|
||||
h = self.public_height(t, height, nb) - self.fork_height
|
||||
if h > a: # adopt: the public chain won outright
|
||||
self.n_abandoned += a
|
||||
self.priv.clear()
|
||||
return []
|
||||
if h == a or (h == a - 1 and a >= 2): # match / override: publish the whole chain
|
||||
out = self.priv
|
||||
self.priv = []
|
||||
self.n_released += len(out)
|
||||
if h >= 1:
|
||||
self.n_override += 1
|
||||
return out
|
||||
return [] # wait
|
||||
|
||||
|
||||
def build_tree_pernode(
|
||||
active_slots: np.ndarray,
|
||||
winners_per_slot: list[np.ndarray],
|
||||
@ -237,7 +307,12 @@ def build_tree_pernode(
|
||||
key[0] = np.int64(0) * c1 - np.int64(-1) * c2 - np.int64(0)
|
||||
NEG = np.iinfo(np.int64).min
|
||||
|
||||
windowed = bool(config.windowed_fork_choice)
|
||||
selfish = (adversary_mask is not None and config.adversary_frac > 0.0
|
||||
and config.adversary_strategy == "selfish")
|
||||
# A private chain breaks the windowed horizon's premise: an unreleased block is old enough to
|
||||
# be "fully propagated" while no honest node has it, and it becomes visible LATER (on release),
|
||||
# which the one-way frontier pointer can never revisit. So selfish runs the exact full scan.
|
||||
windowed = bool(config.windowed_fork_choice) and not selfish
|
||||
if not windowed:
|
||||
horizon = float(E) # full scan (gb unused)
|
||||
elif config.topology == "blend":
|
||||
@ -264,6 +339,7 @@ def build_tree_pernode(
|
||||
withholding = (adversary_mask is not None and config.adversary_frac > 0.0
|
||||
and config.adversary_strategy == "withhold")
|
||||
if config.prune_arrival and windowed and config.jitter_mean == 0.0 and not withholding:
|
||||
# (selfish already cleared `windowed`, so it never reaches the pruned path either)
|
||||
return _build_pruned(active_slots, winners_per_slot, path_latency, config, rng,
|
||||
slot, parent, height, leader, uncles, key, c1, c2,
|
||||
float(horizon), n_blocks, E, n, adversary_mask)
|
||||
@ -289,6 +365,8 @@ def build_tree_pernode(
|
||||
gb_id = 0
|
||||
fp_idx = 1 # frontier pointer over fully-propagated blocks
|
||||
|
||||
coalition = _SelfishCoalition(np.nonzero(adversary_mask)[0], n_blocks, E) if selfish else None
|
||||
|
||||
nb = 1
|
||||
for si in range(active_slots.shape[0]):
|
||||
t = int(active_slots[si])
|
||||
@ -338,12 +416,46 @@ def build_tree_pernode(
|
||||
if hide:
|
||||
A[:, b] = float(E) + 1.0 # withheld: never arrives -> orphan
|
||||
withheld[b] = True
|
||||
elif coalition is not None and adv:
|
||||
# Private: visible to the whole coalition at once, invisible to everyone else
|
||||
# until released. Kept off the honest side by the same sentinel `withhold` uses.
|
||||
A[:, b] = float(E) + 1.0
|
||||
A[coalition.coal, b] = max(float(t), float(A[v, p_id]))
|
||||
withheld[b] = True # flipped back on release
|
||||
coalition.add_private(b, t, int(height[p_id]))
|
||||
else:
|
||||
np.maximum(col, A[:, p_id], out=col)
|
||||
A[:, b] = col
|
||||
A[v, b] = max(float(t), float(A[v, p_id])) # producer sees own block at its slot
|
||||
if coalition is not None:
|
||||
coalition.note_block(b, float(A[coalition.coal, b].min()))
|
||||
nb += 1
|
||||
|
||||
if coalition is not None:
|
||||
for rb in coalition.decide(t, height, nb):
|
||||
# Release by DIRECT gossip from the producer, bypassing the Blend cascade: the
|
||||
# adversary has no privacy budget to respect and wants the race won, so this is
|
||||
# its fastest legal publication. Oldest first, so each block's parent arrival is
|
||||
# already final when the no-earlier-than-parent clamp is applied.
|
||||
prod = int(leader[rb])
|
||||
rel = float(t) + path_latency[prod]
|
||||
np.maximum(rel, A[:, int(parent[rb])], out=rel)
|
||||
np.minimum(rel, A[:, rb], out=rel) # coalition already had it privately
|
||||
A[:, rb] = rel
|
||||
withheld[rb] = False
|
||||
coalition.unreleased[rb] = False
|
||||
|
||||
if coalition is not None and coalition.priv:
|
||||
# Private blocks still hidden when the epoch ends are abandoned: the race they were held
|
||||
# for is over, so they can never be cashed in. Hide them from the coalition too, or the
|
||||
# canonical-tip search (which takes the best tip ANY node holds) would crown a chain no
|
||||
# honest node ever saw and credit it phantom blocks.
|
||||
stranded = np.array(coalition.priv, dtype=np.int64)
|
||||
A[:, stranded] = float(E) + 1.0
|
||||
withheld[stranded] = True
|
||||
coalition.n_abandoned += len(coalition.priv)
|
||||
coalition.priv.clear()
|
||||
|
||||
tree = BlockTree(slot=slot, parent=parent, height=height, leader=leader, uncles=uncles)
|
||||
return tree, A
|
||||
|
||||
@ -364,7 +476,9 @@ def _build_pruned(active_slots, winners_per_slot, path_latency, config, rng,
|
||||
from .uncles import select_uncles_at_production
|
||||
|
||||
NEG = np.iinfo(np.int64).min
|
||||
keepspan = max(float(horizon), float(config.uncle_window)) # columns kept within this span
|
||||
# columns kept within this span; the uncle window is model-dependent (derived W/f for
|
||||
# countable, uncle_window slots for --old), so use the effective value.
|
||||
keepspan = max(float(horizon), float(config.effective_uncle_window))
|
||||
counts = np.array([int(g.shape[0]) for g in winners_per_slot], dtype=np.int64)
|
||||
cap = _max_span_blocks(active_slots, counts, keepspan) # max live blocks at once
|
||||
max_slot = int(counts.max()) if counts.size else 0
|
||||
|
||||
@ -10,6 +10,16 @@ from . import constants
|
||||
|
||||
StakeDist = Literal["uniform", "pareto"]
|
||||
UncleStrategy = Literal["oldest", "random"]
|
||||
# Uncle counting/selection model:
|
||||
# "countable" (default) — the spec's counting-only model (cryptarchia-v1-protocol.md):
|
||||
# only the FIRST block of a fork is referenceable/countable (its parent lies on the
|
||||
# referencing chain), the window is derived as w_u = window_absorption / f slots,
|
||||
# selection excludes slots already occupied on the producer's chain and picks at most
|
||||
# one uncle per slot, and counting re-checks every rule per reference.
|
||||
# "old" — the pre-redesign model (run with --old): window = uncle_window slots directly,
|
||||
# any orphan in view is referenceable regardless of fork depth, no occupied-slot or
|
||||
# per-slot exclusion, and every baked reference counts.
|
||||
UncleModel = Literal["countable", "old"]
|
||||
Topology = Literal["full_mesh", "regular", "blend"]
|
||||
LinkLatencyDist = Literal["fixed", "uniform", "exp", "geo"]
|
||||
JitterDist = Literal["exp", "poisson"]
|
||||
@ -20,7 +30,18 @@ InitDest = Literal["common", "heterogeneous"]
|
||||
# "withhold" — never gossips its blocks (they are orphaned, its won slots become gaps in the
|
||||
# canonical chain), so the counted density drops ~adversary_frac and TSI deflates D_est toward
|
||||
# the reduced ACTIVE stake. Stronger, but the withheld blocks earn nothing (griefing/grinding).
|
||||
AdversaryStrategy = Literal["suppress", "withhold"]
|
||||
# "selfish" — mines a PRIVATE chain and releases it to orphan honest blocks (Eyal-Sirer SM1).
|
||||
# Unlike "withhold" (which discards its blocks — abstention, a dead loss), this recovers the
|
||||
# forfeit by displacing honest work, and is the one profitable lever (report §6.6). Its
|
||||
# estimator damage is what the countable uncle rule can only partly repair, because an
|
||||
# override discards a CHAIN of honest blocks and only the first is referenceable (§2.1).
|
||||
AdversaryStrategy = Literal["suppress", "withhold", "selfish"]
|
||||
# WHICH nodes make up that coalition, at the same total stake:
|
||||
# "random" — a uniformly random set grown until its stake reaches adversary_frac (the default; the
|
||||
# block share is then smooth in adversary_frac, which is all the density levers depend on);
|
||||
# "whale" — the LARGEST holders first. Same stake, far fewer nodes, so the coalition's block
|
||||
# production is lumpier — the untested concentration case flagged in report §6.5's scope.
|
||||
AdversarySelection = Literal["random", "whale"]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@ -81,22 +102,48 @@ class SimConfig:
|
||||
jitter_frac: float = 1.0 # fraction of deliveries hit (poisson model; exp uses all)
|
||||
|
||||
# --- uncle references ---
|
||||
uncle_window: int = constants.W_DEFAULT # W
|
||||
uncle_model: UncleModel = "countable" # countable (spec, default) | old (--old)
|
||||
# Countable model: window absorption parameter W; the uncle reference window is DERIVED
|
||||
# as w_u = W / f slots (W expected block-intervals), bounded 1 <= W <= 0.6*k
|
||||
# (constants.W_ABS_MAX_FACTOR). Ignored by the old model.
|
||||
window_absorption: float = constants.W_ABS_DEFAULT
|
||||
# Old model only (--old): the uncle reference window w_u in slots, set directly.
|
||||
# Ignored by the countable model, which derives the window from window_absorption.
|
||||
uncle_window: int = constants.W_DEFAULT
|
||||
# COMMON RANDOM NUMBERS for countable-vs-old comparisons. Off by default, and deliberately
|
||||
# NOT part of key() — with it off every seed is byte-identical to before, so historical runs
|
||||
# and --old bit-reproduction are untouched.
|
||||
#
|
||||
# The two uncle models normally draw independent streams (uncle_model is in the key), so a
|
||||
# comparison pays the full between-run variance TWICE and the arms differ in stake draw,
|
||||
# peering graph and every lottery outcome. With paired_streams=True the RNG root is derived
|
||||
# from the model-independent part of the key instead, so both arms get the SAME stake, the
|
||||
# SAME graph and the SAME lottery draws; the only difference is the uncle rule, and the
|
||||
# per-replicate difference becomes a paired observation with the shared variance cancelled.
|
||||
# Trajectories still diverge legitimately after epoch 0 — a different counted density feeds
|
||||
# back into the next epoch's difficulty — which is the effect being measured, not noise.
|
||||
paired_streams: bool = False
|
||||
max_uncles: int = 0 # U (0 = baseline, no uncles)
|
||||
# "oldest" IS the spec rule: cryptarchia-v1-protocol.md (Uncle Selection) has the proposer
|
||||
# take the oldest candidates first, deterministically, because an uncle expires w_u slots
|
||||
# after its own slot so the oldest are the closest to expiring. Every headline result uses it.
|
||||
uncle_strategy: UncleStrategy = "oldest"
|
||||
# Coin-flip inclusion prob for the "random" strategy. Only 0.5 reproduces the spec's
|
||||
# unbiased coin (cryptarchia-v1-protocol.md); other values are a deliberate, non-spec
|
||||
# sensitivity knob, not protocol behaviour.
|
||||
# "random" is NOT a spec variant — it is the deviation probe: walk the same oldest-first
|
||||
# candidate order but include each candidate with probability uncle_random_p, so a lone
|
||||
# candidate is dropped half the time. Uncle selection is proposer-local and unvalidated, so a
|
||||
# proposer CAN deviate; this measures what that costs the estimate (report §3.4).
|
||||
uncle_random_p: float = 0.5
|
||||
# --- adversary (grinding via D_est deflation) ---
|
||||
# Fraction of TOTAL STAKE controlled by an adversary that suppresses uncle references in its
|
||||
# own blocks (references no uncles), starving the TSI density count so honest nodes under-count
|
||||
# blocks and infer a LOW D_est -> everyone's win probability phi(f, w/D_est) rises, which is the
|
||||
# grinding payoff. 0.0 = fully honest (the studied baseline). The coalition is a RANDOM node set
|
||||
# whose stake sums to adversary_frac (see engine._adversary_mask); block production is
|
||||
# stake-proportional, so the deflation depends only on that summed share, not on whether the
|
||||
# coalition is one whale or many small nodes. Withholding is a separate, stronger lever.
|
||||
# grinding payoff. 0.0 = fully honest (the studied baseline). The coalition is by default a
|
||||
# RANDOM node set whose stake sums to adversary_frac (see engine._adversary_mask); block
|
||||
# production is stake-proportional, so the deflation depends only on that summed share, not on
|
||||
# whether the coalition is one whale or many small nodes. Withholding is a separate, stronger
|
||||
# lever, and adversary_selection controls WHICH nodes are taken at that fixed stake.
|
||||
adversary_frac: float = 0.0
|
||||
adversary_selection: AdversarySelection = "random"
|
||||
# Dynamic (withhold-then-rejoin) schedule for the withholding lever (§6.5). The coalition is
|
||||
# FIXED (identity from adversary_frac); this only gates whether it withholds in a given epoch.
|
||||
# adversary_period == 0 -> STATIC: the coalition attacks (withholds) every epoch (the §6.4
|
||||
@ -115,8 +162,10 @@ class SimConfig:
|
||||
k: int = 64 # scaled by default; full scale = 2160
|
||||
genesis_d_factor: float = 0.5 # genesis D = factor * true total stake
|
||||
epochs: int = 40
|
||||
# If True, mirror the spec's integer fixed-point f-truncation (f_p = int(f*1000)/1000),
|
||||
# which the on-chain estimator uses; this reproduces its ~1% systematic overestimate.
|
||||
# If True, quantise the target rate the way an on-chain integer estimator does:
|
||||
# f_p = int(f*tsi.PRECISION)/tsi.PRECISION. With tsi.PRECISION = 1_000_000 (the report's
|
||||
# recommended 10^-6 f-precision, §8) this gives f_p = 0.033333 and a negligible residual
|
||||
# f/f_p < 1e-5 — not the ~1% overestimate the old 10^-3 truncation produced.
|
||||
# Default False keeps the analysis-faithful exact-f behaviour.
|
||||
fixed_point: bool = False
|
||||
# If True, count uncle references per BLOCK ID (the pre-fix behaviour, which double-counts
|
||||
@ -139,9 +188,10 @@ class SimConfig:
|
||||
churn_amp: float = 0.0
|
||||
churn_period: int = 4
|
||||
churn_mode: ChurnMode = "sine"
|
||||
# Per-node constant slot-clock offset (~Uniform(-clock_skew_max, +clock_skew_max) slots),
|
||||
# applied to each node's measurement-window bounds — tests whether a whole-timeline clock shift
|
||||
# (unlike per-arrival jitter) can split slot-occupancy at the window edges and break consensus.
|
||||
# INERT: nothing reads this. The §6.1 clock-skew study is run stand-alone by
|
||||
# scripts/clock_skew.py, which applies its own per-node offsets — not through this field.
|
||||
# Retained only as a key() seed contributor for run-hash compatibility (like `per_node_dest`);
|
||||
# leave at 0.
|
||||
clock_skew_max: int = 0
|
||||
# Each node updates its OWN D_est from its OWN view — the point of this simulator, and the ONLY
|
||||
# mode implemented here (always True). The global-consensus-D_est baseline (per_node_dest=False)
|
||||
@ -155,8 +205,9 @@ class SimConfig:
|
||||
init_spread: float = 0.0 # relative spread of heterogeneous initial D_est
|
||||
|
||||
# --- performance ---
|
||||
# >1 parallelises the per-slot lottery across slot-chunks (opt-in; must be pinned and
|
||||
# recorded because it changes the RNG stream — see lottery.sample_wins_chunked).
|
||||
# INERT: nothing reads this — `simulate_epoch` never calls `lottery.sample_wins_chunked`,
|
||||
# so it has no modelled effect. Retained as a key() seed contributor for run-hash
|
||||
# compatibility (like `per_node_dest` above); leave at 1.
|
||||
lottery_chunks: int = 1
|
||||
# Windowed fork choice bounds the per-slot candidate scan to a horizon of the max path
|
||||
# latency (plus the fully-propagated best tip), turning O(n_blocks^2) into O(n_blocks*H).
|
||||
@ -183,6 +234,25 @@ class SimConfig:
|
||||
raise ValueError(f"stake_dist must be uniform|pareto, got {self.stake_dist!r}")
|
||||
if self.uncle_strategy not in ("oldest", "random"):
|
||||
raise ValueError(f"uncle_strategy must be oldest|random, got {self.uncle_strategy!r}")
|
||||
if self.uncle_model not in ("countable", "old"):
|
||||
raise ValueError(f"uncle_model must be countable|old, got {self.uncle_model!r}")
|
||||
if self.uncle_model == "countable":
|
||||
if self.window_absorption < 1.0:
|
||||
raise ValueError(
|
||||
f"window_absorption W={self.window_absorption} must be >= 1")
|
||||
if self.window_absorption > constants.W_ABS_MAX_FACTOR * self.k:
|
||||
# The spec bounds W <= 0.6*k (w_u <= 0.6*k/f, inside the finalization
|
||||
# window). Scaled-down research geometries (small k) may violate it on
|
||||
# purpose — warn loudly rather than refuse, but full-scale runs should
|
||||
# never see this.
|
||||
import warnings
|
||||
|
||||
warnings.warn(
|
||||
f"window_absorption W={self.window_absorption} exceeds the spec bound "
|
||||
f"{constants.W_ABS_MAX_FACTOR}*k = "
|
||||
f"{constants.W_ABS_MAX_FACTOR * self.k:g} (k={self.k}); the derived "
|
||||
f"window is outside the finalization window at this geometry",
|
||||
RuntimeWarning, stacklevel=2)
|
||||
if self.topology not in ("full_mesh", "regular", "blend"):
|
||||
raise ValueError(f"topology must be full_mesh|regular|blend, got {self.topology!r}")
|
||||
if self.link_latency_dist not in ("fixed", "uniform", "exp", "geo"):
|
||||
@ -202,8 +272,11 @@ class SimConfig:
|
||||
raise ValueError(f"churn_period must be >= 1, got {self.churn_period}")
|
||||
if self.clock_skew_max < 0:
|
||||
raise ValueError(f"clock_skew_max must be >= 0, got {self.clock_skew_max}")
|
||||
if self.adversary_strategy not in ("suppress", "withhold"):
|
||||
raise ValueError(f"adversary_strategy must be suppress|withhold, got "
|
||||
if self.adversary_selection not in ("random", "whale"):
|
||||
raise ValueError(f"adversary_selection must be random|whale, got "
|
||||
f"{self.adversary_selection!r}")
|
||||
if self.adversary_strategy not in ("suppress", "withhold", "selfish"):
|
||||
raise ValueError(f"adversary_strategy must be suppress|withhold|selfish, got "
|
||||
f"{self.adversary_strategy!r}")
|
||||
checks = {
|
||||
"n_nodes": self.n_nodes >= 1,
|
||||
@ -260,6 +333,17 @@ class SimConfig:
|
||||
return (epoch % self.adversary_period) < self.adversary_withhold_epochs
|
||||
|
||||
# derived geometry -------------------------------------------------------
|
||||
@property
|
||||
def effective_uncle_window(self) -> int:
|
||||
"""The uncle reference window ``w_u`` in slots actually used by this run.
|
||||
|
||||
Countable model (default): derived, ``w_u = round(window_absorption / f)``.
|
||||
Old model (``--old``): ``uncle_window`` taken directly.
|
||||
"""
|
||||
if self.uncle_model == "old":
|
||||
return self.uncle_window
|
||||
return constants.uncle_window_slots(self.window_absorption, self.f)
|
||||
|
||||
@property
|
||||
def epoch_len(self) -> int:
|
||||
return constants.epoch_len(self.k, self.f)
|
||||
@ -268,12 +352,8 @@ class SimConfig:
|
||||
def period_T(self) -> int:
|
||||
return constants.period_T(self.k, self.f)
|
||||
|
||||
def key(self) -> tuple:
|
||||
"""Hashable identity used to seed the RNG deterministically.
|
||||
|
||||
Must include EVERY field that affects the run (guarded by test_rng), otherwise two
|
||||
distinct configs would share an RNG stream.
|
||||
"""
|
||||
def _base_key(self) -> tuple:
|
||||
"""Identity fields shared by both uncle models — see ``key`` and ``seed_key``."""
|
||||
return (
|
||||
self.n_nodes, self.stake_dist, self.pareto_shape, self.uniform_random,
|
||||
self.total_stake, self.latency, self.latency_stochastic, self.uncle_window,
|
||||
@ -292,10 +372,37 @@ class SimConfig:
|
||||
# compute/memory optimisations that consume no RNG and (at jitter_mean == 0) change no
|
||||
# result, so pruned and full-matrix runs must share a seed (see test_pernode parity).
|
||||
|
||||
def key(self) -> tuple:
|
||||
"""Hashable identity used to seed the RNG deterministically.
|
||||
|
||||
Must include EVERY field that affects the run (guarded by test_rng), otherwise two
|
||||
distinct configs would share an RNG stream. ``uncle_model`` /
|
||||
``window_absorption`` are appended ONLY for the countable model: an ``--old`` run's
|
||||
key is then byte-identical to the pre-redesign key, so ``--old`` bit-reproduces
|
||||
historical runs (the two models still get distinct streams from the marker).
|
||||
"""
|
||||
# uncle_model == "old" keeps the historical tuple exactly (--old bit-compat).
|
||||
base = (self._base_key() if self.uncle_model == "old"
|
||||
else self._base_key() + (self.uncle_model, self.window_absorption))
|
||||
# Appended ONLY when non-default, for the same reason the uncle_model marker is: a
|
||||
# "random"-coalition run's key must stay byte-identical to every historical run's.
|
||||
return base if self.adversary_selection == "random" else base + (self.adversary_selection,)
|
||||
|
||||
def seed_key(self) -> tuple:
|
||||
"""The identity the RNG root is actually derived from (see ``rng.seedseq_for``).
|
||||
|
||||
Identical to ``key`` except under ``paired_streams``, where it deliberately drops the
|
||||
uncle-model marker so that a countable run and an ``--old`` run of the SAME cell draw
|
||||
the SAME root seed — common random numbers, which is what makes the two arms a
|
||||
*paired* sample (see ``paired_streams``).
|
||||
"""
|
||||
return self._base_key() if self.paired_streams else self.key()
|
||||
|
||||
|
||||
# Axes that can be swept; every SimConfig field is legal here.
|
||||
_SWEEP_AXES = (
|
||||
"n_nodes", "stake_dist", "latency", "max_uncles", "uncle_strategy", "uncle_window",
|
||||
"window_absorption",
|
||||
"topology", "degree", "link_latency_mean", "link_latency_dist",
|
||||
"blend_hops", "blend_delay_max", "init_dest", "f",
|
||||
)
|
||||
@ -311,6 +418,7 @@ class SweepConfig:
|
||||
max_uncles: list[int] = field(default_factory=lambda: [0, 1, 2, 4])
|
||||
uncle_strategy: list[UncleStrategy] = field(default_factory=lambda: ["oldest"])
|
||||
uncle_window: list[int] = field(default_factory=lambda: [constants.W_DEFAULT])
|
||||
window_absorption: list[float] = field(default_factory=lambda: [constants.W_ABS_DEFAULT])
|
||||
topology: list[Topology] = field(default_factory=lambda: ["regular"])
|
||||
degree: list[int] = field(default_factory=lambda: [8])
|
||||
link_latency_mean: list[float] = field(default_factory=lambda: [1.0])
|
||||
@ -329,11 +437,22 @@ class SweepConfig:
|
||||
axis_values = [getattr(self, ax) for ax in _SWEEP_AXES]
|
||||
for combo in itertools.product(*axis_values):
|
||||
overrides = dict(zip(_SWEEP_AXES, combo, strict=True))
|
||||
# U=0 references no uncles, so it is independent of uncle_strategy AND uncle_window;
|
||||
# keep only the first of each to avoid duplicate (identical) work.
|
||||
# U=0 references no uncles, so it is independent of uncle_strategy AND the window
|
||||
# knobs; keep only the first of each to avoid duplicate (identical) work.
|
||||
if overrides["max_uncles"] == 0 and (
|
||||
overrides["uncle_strategy"] != self.uncle_strategy[0]
|
||||
or overrides["uncle_window"] != self.uncle_window[0]
|
||||
or overrides["window_absorption"] != self.window_absorption[0]
|
||||
):
|
||||
continue
|
||||
# each uncle model reads exactly one window knob — collapse the other axis so a
|
||||
# sweep never emits duplicate cells that differ only in an ignored field.
|
||||
if base.uncle_model == "countable" and (
|
||||
overrides["uncle_window"] != self.uncle_window[0]
|
||||
):
|
||||
continue
|
||||
if base.uncle_model == "old" and (
|
||||
overrides["window_absorption"] != self.window_absorption[0]
|
||||
):
|
||||
continue
|
||||
# full mesh ignores degree / link-latency model; keep only the first to avoid dupes.
|
||||
|
||||
@ -10,10 +10,23 @@ from __future__ import annotations
|
||||
# --- True protocol values (full scale) -------------------------------------
|
||||
K_TRUE = 2160 # security parameter (blocks)
|
||||
F = 1.0 / 30.0 # slot activation coefficient (default; configurable per run)
|
||||
W_DEFAULT = 300 # uncle reference window w_u (slots)
|
||||
W_DEFAULT = 300 # old model: uncle reference window w_u (slots), set directly (--old)
|
||||
BETA_DEFAULT = 1.0 # TSI learning rate
|
||||
SLOT_SECONDS = 1 # slot length (seconds) — so 1 slot == 1 s
|
||||
|
||||
# --- Countable uncle model (cryptarchia-v1-protocol.md, uncle references) ---
|
||||
# The spec derives the uncle reference window from the *window absorption parameter* W:
|
||||
# w_u = W * f^-1 slots, i.e. W expected block-intervals. W is bounded by 1 <= W <= 0.6*k,
|
||||
# equivalently w_u <= 0.6*k/f = s/5, keeping the window strictly inside the finalization
|
||||
# window. The default W = 10 reproduces w_u = 300 slots at f = 1/30.
|
||||
W_ABS_DEFAULT = 10.0 # window absorption parameter W (expected block-intervals)
|
||||
W_ABS_MAX_FACTOR = 0.6 # bound: W <= W_ABS_MAX_FACTOR * k
|
||||
|
||||
|
||||
def uncle_window_slots(w_abs: float, f: float = F) -> int:
|
||||
"""Derived uncle reference window ``w_u = W / f`` in slots (countable model)."""
|
||||
return max(1, int(round(w_abs / f)))
|
||||
|
||||
|
||||
# --- Real-world inter-node network latency (per gossip link) ---------------
|
||||
# A slot is SLOT_SECONDS = 1 s, so measured internet latencies (tens–hundreds of ms) are
|
||||
|
||||
@ -25,24 +25,61 @@ ES_MEASURE = 10 # measurement epochs run after detection
|
||||
|
||||
|
||||
def _adversary_mask(config: SimConfig, stake: np.ndarray) -> np.ndarray | None:
|
||||
"""Nodes controlled by the uncle-suppressing adversary — a random coalition whose stake sums to
|
||||
``adversary_frac`` of the total, giving smooth control of the adversary's block share. (For
|
||||
uncle suppression the deflation depends only on that block share, not on whether the coalition
|
||||
is one whale or many small nodes, so concentration is not modelled here.) ``None`` if honest.
|
||||
"""Nodes controlled by the uncle-suppressing adversary — a coalition whose stake sums to
|
||||
``adversary_frac`` of the total, giving smooth control of the adversary's block share.
|
||||
``None`` if honest.
|
||||
|
||||
``adversary_selection`` picks *which* nodes, at that same total stake:
|
||||
|
||||
* ``"random"`` (default) — a uniformly random set. For uncle suppression the deflation depends
|
||||
only on the summed block share, not on whether the coalition is one whale or many small
|
||||
nodes, so this is the neutral choice and concentration does not enter.
|
||||
* ``"whale"`` — the largest holders first. Same stake in far fewer nodes, so the coalition's
|
||||
block production is lumpier: it is the concentration case report §6.5 flags as untested, and
|
||||
the reason to run it is the *variance* of the coalition's share, not its mean.
|
||||
|
||||
Seeded from a standalone ``SeedSequence([root_seed, replicate, 0xADEADBEEF])`` (independent of
|
||||
the main spawn hierarchy), and drawn only after the ``adversary_frac <= 0`` early return, so an
|
||||
``adversary_frac == 0`` run is bit-identical to the honest baseline.
|
||||
``adversary_frac == 0`` run is bit-identical to the honest baseline. The whale order is
|
||||
deterministic given the stake vector and consumes no randomness, but the seed is still drawn
|
||||
first so that switching selection never perturbs the rest of the stream.
|
||||
"""
|
||||
if config.adversary_frac <= 0.0:
|
||||
return None
|
||||
adv_seed = np.random.SeedSequence([config.root_seed, config.replicate, 0xADEADBEEF])
|
||||
order = np.random.default_rng(adv_seed).permutation(config.n_nodes)
|
||||
rand_order = np.random.default_rng(adv_seed).permutation(config.n_nodes)
|
||||
target = config.adversary_frac * float(stake.sum())
|
||||
cum = np.cumsum(stake[order])
|
||||
take = int(np.searchsorted(cum, target, side="left")) + 1 # smallest coalition >= target
|
||||
mask = np.zeros(config.n_nodes, dtype=bool)
|
||||
mask[order[:take]] = True
|
||||
|
||||
if config.adversary_selection == "whale":
|
||||
# Largest stake first, ties broken by the random order so equal-stake runs stay unbiased.
|
||||
# Taking whales until the cumulative sum first EXCEEDS the target would overshoot badly
|
||||
# under a heavy tail (the top holder alone can be a sixth of the stake), and a coalition
|
||||
# holding visibly more than adversary_frac would confound concentration with stake. So:
|
||||
# walk descending and take every node that still FITS under the target, then close any
|
||||
# remaining gap with the smallest node that can. The coalition is therefore dominated by
|
||||
# the top holders, with small nodes only topping it up onto the target — a handful of
|
||||
# members against the random arm's hundreds, at the same stake.
|
||||
order = rand_order[np.argsort(-stake[rand_order], kind="stable")]
|
||||
st = stake[order]
|
||||
chosen = np.zeros(order.size, dtype=bool)
|
||||
cum = 0.0
|
||||
for j, s in enumerate(st):
|
||||
if cum + s <= target:
|
||||
chosen[j] = True
|
||||
cum += s
|
||||
if cum < target:
|
||||
# `st` is descending, so among the nodes that can close the gap the LAST is the
|
||||
# smallest. Every skipped node exceeds the current gap (the gap only shrinks), so a
|
||||
# candidate always exists while any node remains; the fallback is defensive only.
|
||||
cand = np.nonzero(~chosen & (st >= target - cum))[0]
|
||||
chosen[cand[-1] if cand.size else np.nonzero(~chosen)[0][0]] = True
|
||||
mask[order[chosen]] = True
|
||||
return mask
|
||||
|
||||
cum_r = np.cumsum(stake[rand_order])
|
||||
take = int(np.searchsorted(cum_r, target, side="left")) + 1 # smallest coalition >= target
|
||||
mask[rand_order[:take]] = True
|
||||
return mask
|
||||
|
||||
|
||||
|
||||
@ -30,6 +30,9 @@ class EpochResult:
|
||||
max_reorg_depth: int # deepest maximal orphan branch (blocks a reorg would discard)
|
||||
mean_reorg_depth: float # mean maximal-orphan-branch depth
|
||||
p_ref: float # emergent reference rate: in-window orphans referenced as uncles
|
||||
p_ref_honest: float # ...restricted to orphans produced OUTSIDE the coalition
|
||||
deep_ref_share: float # share of examined references rejected by the parent-on-chain
|
||||
# (first-fork) counting rule; 0 under the old model
|
||||
|
||||
|
||||
def _canonical_producer_split(
|
||||
@ -101,14 +104,19 @@ def simulate_epoch(
|
||||
|
||||
# measurement: each node's own canonical chain, deduped by tip + numba-accelerated
|
||||
ms = measure(tree, A, active_slots, T, cutoff=E,
|
||||
legacy_block_count=config.legacy_block_count)
|
||||
legacy_block_count=config.legacy_block_count,
|
||||
countable=config.uncle_model != "old",
|
||||
w=config.effective_uncle_window)
|
||||
n_active_window = int((active_slots < T).sum())
|
||||
|
||||
d_next = tsi.update_D_vec(d_est, ms.m, T, f, config.beta, config.fixed_point)
|
||||
|
||||
attribution = coalition_mask if coalition_mask is not None else adversary_mask
|
||||
adv_blocks, honest_blocks = _canonical_producer_split(tree, A, attribution, T, E)
|
||||
fork_rate, max_reorg_depth, mean_reorg_depth, p_ref = fork.fork_stats(tree, A, T, cutoff=E)
|
||||
fork_rate, max_reorg_depth, mean_reorg_depth, p_ref, p_ref_honest = fork.fork_stats(
|
||||
tree, A, T, cutoff=E, coalition_mask=attribution)
|
||||
ref_total = int(ms.ref_total.sum())
|
||||
deep_ref_share = (int(ms.ref_deep.sum()) / ref_total) if ref_total else 0.0
|
||||
|
||||
return EpochResult(
|
||||
d_next=d_next, m=ms.m, q=ms.q, q_eff=ms.q_eff, n_blocks=tree.n_blocks - 1,
|
||||
@ -117,5 +125,5 @@ def simulate_epoch(
|
||||
mean_orphan_rate=float(ms.orphan_rate.mean()),
|
||||
adv_blocks=adv_blocks, honest_blocks=honest_blocks,
|
||||
fork_rate=fork_rate, max_reorg_depth=max_reorg_depth, mean_reorg_depth=mean_reorg_depth,
|
||||
p_ref=p_ref,
|
||||
p_ref=p_ref, p_ref_honest=p_ref_honest, deep_ref_share=deep_ref_share,
|
||||
)
|
||||
|
||||
@ -18,17 +18,24 @@ import numpy as np
|
||||
from .blocktree import BlockTree
|
||||
|
||||
|
||||
def fork_stats(tree: BlockTree, A, T: int, cutoff: int) -> tuple[float, int, float, float]:
|
||||
"""Return ``(fork_rate, max_reorg_depth, mean_reorg_depth, p_ref)`` over in-window blocks.
|
||||
def fork_stats(tree: BlockTree, A, T: int, cutoff: int,
|
||||
coalition_mask=None) -> tuple[float, int, float, float, float]:
|
||||
"""Return ``(fork_rate, max_reorg_depth, mean_reorg_depth, p_ref, p_ref_honest)``.
|
||||
|
||||
``p_ref`` is the emergent **reference rate**: the fraction of in-window orphans that some
|
||||
canonical block references as an uncle — the quantity the §6.8 soft-inclusion argument
|
||||
assumes is high. ``A`` is the arrival matrix (full ``np.ndarray`` or pruned): only used to
|
||||
exclude withheld blocks (which reach no node) from canonical-tip selection.
|
||||
|
||||
``p_ref_honest`` restricts that to orphans produced by nodes OUTSIDE ``coalition_mask``.
|
||||
Under a private-chain attack the two diverge and only the honest one measures the repair the
|
||||
report credits to uncle counting: an attacker's own discarded blocks are its loss to bear,
|
||||
and counting them would flatter `p_ref` with orphans nobody is owed. Equal to ``p_ref`` when
|
||||
no mask is given.
|
||||
"""
|
||||
nb = tree.n_blocks
|
||||
if nb <= 1:
|
||||
return 0.0, 0, 0.0, 1.0
|
||||
return 0.0, 0, 0.0, 1.0, 1.0
|
||||
ids = np.arange(nb)
|
||||
if isinstance(A, np.ndarray):
|
||||
arrived = (A <= cutoff).any(axis=0)
|
||||
@ -48,7 +55,7 @@ def fork_stats(tree: BlockTree, A, T: int, cutoff: int) -> tuple[float, int, flo
|
||||
in_win = (tree.slot >= 0) & (tree.slot < T)
|
||||
total = int(in_win.sum())
|
||||
if total == 0:
|
||||
return 0.0, 0, 0.0, 1.0
|
||||
return 0.0, 0, 0.0, 1.0, 1.0
|
||||
|
||||
# depth[b] = length of the non-canonical run ending at b (0 if canonical). Parent-before-child
|
||||
# holds because a block\'s parent has a strictly smaller id (built earlier).
|
||||
@ -75,4 +82,11 @@ def fork_stats(tree: BlockTree, A, T: int, cutoff: int) -> tuple[float, int, flo
|
||||
referenced[u] = True
|
||||
ref_orphans = int((orphan_in_win & referenced).sum())
|
||||
p_ref = ref_orphans / n_orphan if n_orphan else 1.0
|
||||
return fork_rate, max_depth, mean_depth, p_ref
|
||||
|
||||
if coalition_mask is None:
|
||||
p_ref_honest = p_ref
|
||||
else:
|
||||
honest_orphan = orphan_in_win & ~np.asarray(coalition_mask)[tree.leader]
|
||||
n_ho = int(honest_orphan.sum())
|
||||
p_ref_honest = (int((honest_orphan & referenced).sum()) / n_ho) if n_ho else 1.0
|
||||
return fork_rate, max_depth, mean_depth, p_ref, p_ref_honest
|
||||
|
||||
@ -6,6 +6,22 @@ independently — O(N x chain) Python and ~95% of an epoch. Two exact optimisati
|
||||
The counted density ``m`` is SLOT-based (canonical slots + recovered uncle slots — the
|
||||
"one count per slot" invariant; ``legacy_block_count`` reproduces the old per-block count).
|
||||
|
||||
Counting models (``countable`` flag; CLI ``--old`` clears it):
|
||||
|
||||
* **countable** (default) — the spec's counting rules are re-checked per reference
|
||||
(cryptarchia-v1-protocol.md): the reference must be within the window
|
||||
(``0 < slot_B - slot_U <= w``), the uncle must not lie on the counting chain, and its
|
||||
**parent must lie on the counting chain** (only the first block of a fork counts).
|
||||
References failing the parent rule are tallied as ``deep``. On a real run that tally is
|
||||
always ZERO and is a defensive invariant, not a measured rate: countable *selection*
|
||||
already refuses non-first-fork candidates, and for a chain block ``B`` the producer's
|
||||
chain below ``B`` is the same ancestor path as the counting chain below ``B``, so the
|
||||
two rules cannot disagree. A non-zero ``deep`` means selection and counting have drifted
|
||||
(asserted end-to-end by test_countable_counting.py). Only hand-baked references — the
|
||||
ones the unit tests construct — can make it fire.
|
||||
* **old** — every baked reference in the measurement window counts (fork depth ignored),
|
||||
reproducing the pre-redesign behaviour.
|
||||
|
||||
1. **Dedup by tip.** Nodes sharing a current tip share their whole canonical chain and every
|
||||
derived quantity, so we compute once per *distinct* tip and broadcast. High node agreement
|
||||
(the common case) collapses N to a handful of computations.
|
||||
@ -13,7 +29,8 @@ The counted density ``m`` is SLOT-based (canonical slots + recovered uncle slots
|
||||
recovered orphan slots, window-prefix fingerprint) runs as one cached, C-speed routine over
|
||||
flat arrays. A pure-Python fallback keeps the package importable without numba.
|
||||
|
||||
Results are identical to the reference loop (``measure_reference``); see test_measure.
|
||||
Results are identical to the naive per-node loop (``_reference`` in tests/test_measure.py;
|
||||
see ``test_measure_matches_reference``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@ -35,10 +52,16 @@ except ImportError: # pragma: no cover - numba is an optional accelerator
|
||||
|
||||
@dataclass
|
||||
class Measurement:
|
||||
m: np.ndarray # (N,) per-node block count
|
||||
m: np.ndarray # (N,) per-node counted density: canonical + recovered uncle SLOTS
|
||||
# (per-block-id count only under legacy_block_count=True)
|
||||
q: np.ndarray # (N,) honest active-slot fraction
|
||||
q_eff: np.ndarray # (N,) uncle-recovered fraction
|
||||
orphan_rate: np.ndarray # (N,)
|
||||
ref_total: np.ndarray # (N,) distinct referenced uncles examined (slot in [0,T); the
|
||||
# per-reference window check is applied after this tally)
|
||||
ref_deep: np.ndarray # (N,) of those, rejected by the parent-on-chain (first-fork) rule.
|
||||
# ALWAYS 0 on a real run under either model — a selection/
|
||||
# counting drift alarm, not a rate (see the module docstring)
|
||||
agreement_window: float
|
||||
agreement_tip: float
|
||||
|
||||
@ -56,23 +79,26 @@ def _uncles_csr(tree: BlockTree) -> tuple[np.ndarray, np.ndarray]:
|
||||
return flat, ptr
|
||||
|
||||
|
||||
def _measure_tips_py(distinct_tips, parent, slot, uncle_flat, uncle_ptr, T,
|
||||
uncle_stamp, honest_stamp):
|
||||
def _measure_tips_py(distinct_tips, parent, slot, uncle_flat, uncle_ptr, T, w, countable,
|
||||
uncle_stamp, honest_stamp, chain_stamp):
|
||||
"""Pure-Python per-distinct-tip walk (fallback / reference for the kernel)."""
|
||||
K = distinct_tips.shape[0]
|
||||
m = np.empty(K, np.int64)
|
||||
n_honest = np.empty(K, np.int64)
|
||||
n_rec = np.empty(K, np.int64)
|
||||
n_ref = np.empty(K, np.int64)
|
||||
n_deep = np.empty(K, np.int64)
|
||||
chain_len = np.empty(K, np.int64)
|
||||
fp = np.empty(K, np.uint64)
|
||||
for ki in range(K):
|
||||
# pass 1: chain -> honest count, mark honest slots, fingerprint, chain length
|
||||
# pass 1: chain -> honest count, mark honest slots + chain membership, fingerprint
|
||||
honest = 0
|
||||
clen = 0
|
||||
f = np.uint64(0)
|
||||
b = int(distinct_tips[ki])
|
||||
while b > 0:
|
||||
clen += 1
|
||||
chain_stamp[b] = ki # chain membership (any slot, incl. outside T)
|
||||
s = int(slot[b])
|
||||
if 0 <= s < T:
|
||||
honest += 1
|
||||
@ -82,13 +108,27 @@ def _measure_tips_py(distinct_tips, parent, slot, uncle_flat, uncle_ptr, T,
|
||||
# pass 2: deduped referenced uncles in window + recovered orphan slots
|
||||
ucnt = 0
|
||||
rec = 0
|
||||
refs = 0
|
||||
deep = 0
|
||||
b = int(distinct_tips[ki])
|
||||
while b > 0:
|
||||
for j in range(int(uncle_ptr[b]), int(uncle_ptr[b + 1])):
|
||||
u = int(uncle_flat[j])
|
||||
su = int(slot[u])
|
||||
if 0 <= su < T and uncle_stamp[u] != ki:
|
||||
uncle_stamp[u] = ki # dedup uncles by id (m counts blocks)
|
||||
uncle_stamp[u] = ki # dedup uncles by id
|
||||
refs += 1
|
||||
if countable:
|
||||
# spec counting rules, re-checked per reference:
|
||||
d = int(slot[b]) - su
|
||||
if d <= 0 or d > w:
|
||||
continue # outside the reference window
|
||||
if chain_stamp[u] == ki:
|
||||
continue # uncle lies on the counting chain
|
||||
pu = int(parent[u])
|
||||
if pu != 0 and chain_stamp[pu] != ki:
|
||||
deep += 1 # not a first fork block: uncounted
|
||||
continue
|
||||
ucnt += 1
|
||||
if honest_stamp[su] != ki:
|
||||
rec += 1 # recovered slots deduped by slot
|
||||
@ -97,9 +137,11 @@ def _measure_tips_py(distinct_tips, parent, slot, uncle_flat, uncle_ptr, T,
|
||||
m[ki] = honest + ucnt
|
||||
n_honest[ki] = honest
|
||||
n_rec[ki] = rec
|
||||
n_ref[ki] = refs
|
||||
n_deep[ki] = deep
|
||||
chain_len[ki] = clen
|
||||
fp[ki] = f
|
||||
return m, n_honest, n_rec, chain_len, fp
|
||||
return m, n_honest, n_rec, n_ref, n_deep, chain_len, fp
|
||||
|
||||
|
||||
def _mix_py(x: np.uint64) -> np.uint64:
|
||||
@ -117,12 +159,14 @@ if _HAVE_NUMBA:
|
||||
return x ^ (x >> uint64(31))
|
||||
|
||||
@njit(cache=True)
|
||||
def _measure_tips_nb(distinct_tips, parent, slot, uncle_flat, uncle_ptr, T,
|
||||
uncle_stamp, honest_stamp):
|
||||
def _measure_tips_nb(distinct_tips, parent, slot, uncle_flat, uncle_ptr, T, w, countable,
|
||||
uncle_stamp, honest_stamp, chain_stamp):
|
||||
K = distinct_tips.shape[0]
|
||||
m = np.empty(K, np.int64)
|
||||
n_honest = np.empty(K, np.int64)
|
||||
n_rec = np.empty(K, np.int64)
|
||||
n_ref = np.empty(K, np.int64)
|
||||
n_deep = np.empty(K, np.int64)
|
||||
chain_len = np.empty(K, np.int64)
|
||||
fp = np.empty(K, np.uint64)
|
||||
for ki in range(K):
|
||||
@ -132,6 +176,7 @@ if _HAVE_NUMBA:
|
||||
b = distinct_tips[ki]
|
||||
while b > 0:
|
||||
clen += 1
|
||||
chain_stamp[b] = ki # chain membership (any slot, incl. outside T)
|
||||
s = slot[b]
|
||||
if 0 <= s < T:
|
||||
honest += 1
|
||||
@ -140,6 +185,8 @@ if _HAVE_NUMBA:
|
||||
b = parent[b]
|
||||
ucnt = 0
|
||||
rec = 0
|
||||
refs = 0
|
||||
deep = 0
|
||||
b = distinct_tips[ki]
|
||||
while b > 0:
|
||||
for j in range(uncle_ptr[b], uncle_ptr[b + 1]):
|
||||
@ -147,25 +194,45 @@ if _HAVE_NUMBA:
|
||||
su = slot[u]
|
||||
if 0 <= su < T and uncle_stamp[u] != ki:
|
||||
uncle_stamp[u] = ki # dedup uncles by id
|
||||
ucnt += 1
|
||||
if honest_stamp[su] != ki:
|
||||
rec += 1 # recovered slots deduped by slot
|
||||
honest_stamp[su] = ki
|
||||
refs += 1
|
||||
ok = True
|
||||
if countable:
|
||||
d = slot[b] - su
|
||||
if d <= 0 or d > w:
|
||||
ok = False # outside the reference window
|
||||
elif chain_stamp[u] == ki:
|
||||
ok = False # uncle lies on the counting chain
|
||||
else:
|
||||
pu = parent[u]
|
||||
if pu != 0 and chain_stamp[pu] != ki:
|
||||
deep += 1 # not a first fork block: uncounted
|
||||
ok = False
|
||||
if ok:
|
||||
ucnt += 1
|
||||
if honest_stamp[su] != ki:
|
||||
rec += 1 # recovered slots deduped by slot
|
||||
honest_stamp[su] = ki
|
||||
b = parent[b]
|
||||
m[ki] = honest + ucnt
|
||||
n_honest[ki] = honest
|
||||
n_rec[ki] = rec
|
||||
n_ref[ki] = refs
|
||||
n_deep[ki] = deep
|
||||
chain_len[ki] = clen
|
||||
fp[ki] = f
|
||||
return m, n_honest, n_rec, chain_len, fp
|
||||
return m, n_honest, n_rec, n_ref, n_deep, chain_len, fp
|
||||
|
||||
|
||||
def measure(tree: BlockTree, A, active_slots: np.ndarray, T: int, cutoff: int,
|
||||
use_numba: bool = True, legacy_block_count: bool = False) -> Measurement:
|
||||
use_numba: bool = True, legacy_block_count: bool = False,
|
||||
countable: bool = False, w: int = 0) -> Measurement:
|
||||
"""Per-node m/q/q_eff + agreement, deduped by tip and (optionally) numba-accelerated.
|
||||
|
||||
``A`` is the full ``(N, n_blocks)`` arrival matrix or a pruned ``SlidingArrival`` — only
|
||||
``tips_for_all_nodes`` reads it, so ``N`` is taken from the returned per-node tips.
|
||||
``countable`` applies the spec's per-reference counting rules (window ``w``,
|
||||
not-on-chain, parent-on-chain); ``countable=False`` reproduces the old model where every
|
||||
baked reference counts. ``A`` is the full ``(N, n_blocks)`` arrival matrix or a pruned
|
||||
``SlidingArrival`` — only ``tips_for_all_nodes`` reads it, so ``N`` is taken from the
|
||||
returned per-node tips.
|
||||
"""
|
||||
tips = tips_for_all_nodes(tree, A, cutoff)
|
||||
N = tips.shape[0]
|
||||
@ -177,11 +244,13 @@ def measure(tree: BlockTree, A, active_slots: np.ndarray, T: int, cutoff: int,
|
||||
uncle_flat, uncle_ptr = _uncles_csr(tree)
|
||||
uncle_stamp = np.full(tree.n_blocks, -1, np.int64)
|
||||
honest_stamp = np.full(max(T, 1), -1, np.int64)
|
||||
chain_stamp = np.full(tree.n_blocks, -1, np.int64)
|
||||
|
||||
kernel = _measure_tips_nb if (_HAVE_NUMBA and use_numba) else _measure_tips_py
|
||||
m_d, nh_d, nrec_d, clen_d, fp_d = kernel(
|
||||
m_d, nh_d, nrec_d, nref_d, ndeep_d, clen_d, fp_d = kernel(
|
||||
distinct_tips.astype(np.int64), tree.parent, tree.slot,
|
||||
uncle_flat, uncle_ptr, np.int64(T), uncle_stamp, honest_stamp)
|
||||
uncle_flat, uncle_ptr, np.int64(T), np.int64(w), bool(countable),
|
||||
uncle_stamp, honest_stamp, chain_stamp)
|
||||
|
||||
# correct slot counting: canonical slots + recovered (non-canonical, deduped) uncle slots.
|
||||
# legacy_block_count reproduces the earlier per-block-id count (kernel's m = honest + ucnt).
|
||||
@ -198,4 +267,5 @@ def measure(tree: BlockTree, A, active_slots: np.ndarray, T: int, cutoff: int,
|
||||
agreement_window = max(fp_counts.values()) / N
|
||||
|
||||
return Measurement(m=m, q=q, q_eff=q_eff, orphan_rate=orphan_rate,
|
||||
ref_total=nref_d[inverse], ref_deep=ndeep_d[inverse],
|
||||
agreement_window=agreement_window, agreement_tip=agreement_tip)
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
"""Per-epoch per-node divergence rows and equilibrium summaries."""
|
||||
"""Per-epoch per-node divergence rows."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@ -13,10 +13,16 @@ from .epoch import EpochResult
|
||||
_CONFIG_FIELDS = (
|
||||
"n_nodes", "stake_dist", "pareto_shape", "latency", "topology", "degree",
|
||||
"link_latency_mean", "link_latency_dist", "blend_hops", "blend_delay_max",
|
||||
"init_dest", "init_spread", "uncle_window", "max_uncles", "uncle_strategy",
|
||||
"init_dest", "init_spread", "uncle_model", "window_absorption",
|
||||
"uncle_window", "max_uncles", "uncle_strategy",
|
||||
# Recorded so downstream analysis can TELL whether a countable/--old pair actually shared
|
||||
# its RNG streams. The paired test is only valid on paired runs, and without this column
|
||||
# the analysis silently falls back to the much weaker unpaired test.
|
||||
"paired_streams",
|
||||
"f", "beta", "k", "genesis_d_factor", "epochs", "fixed_point", "legacy_block_count",
|
||||
"replicate",
|
||||
"adversary_frac", "adversary_strategy", "adversary_period", "adversary_withhold_epochs",
|
||||
"adversary_frac", "adversary_strategy", "adversary_selection", "adversary_period",
|
||||
"adversary_withhold_epochs",
|
||||
)
|
||||
|
||||
|
||||
@ -58,25 +64,7 @@ def divergence_row(
|
||||
max_reorg_depth=er.max_reorg_depth,
|
||||
mean_reorg_depth=er.mean_reorg_depth,
|
||||
p_ref=er.p_ref,
|
||||
p_ref_honest=er.p_ref_honest,
|
||||
deep_ref_share=er.deep_ref_share,
|
||||
)
|
||||
return row
|
||||
|
||||
|
||||
def equilibrium_stats(values: np.ndarray, burn_in: int) -> dict[str, float]:
|
||||
"""Mean/variance of ``values`` after ``burn_in`` epochs."""
|
||||
values = np.asarray(values, dtype=float)
|
||||
if values.size == 0:
|
||||
return {"mean": float("nan"), "var": float("nan"), "std": float("nan")}
|
||||
tail = values[burn_in:]
|
||||
if tail.size == 0:
|
||||
tail = values[-1:]
|
||||
return {"mean": float(np.mean(tail)), "var": float(np.var(tail)), "std": float(np.std(tail))}
|
||||
|
||||
|
||||
def epochs_to_within(values: np.ndarray, target: float, eps: float) -> int:
|
||||
"""First epoch after which ``|values - target| <= eps`` holds for the rest."""
|
||||
within = np.abs(np.asarray(values, dtype=float) - target) <= eps
|
||||
if within.size == 0:
|
||||
return 0
|
||||
false_idx = np.flatnonzero(~within)
|
||||
return int(false_idx[-1] + 1) if false_idx.size else 0
|
||||
|
||||
@ -13,15 +13,128 @@ import pandas as pd
|
||||
|
||||
from . import style
|
||||
|
||||
CONFIG_COLS = ["n_nodes", "stake_dist", "topology", "degree", "link_latency_mean",
|
||||
"blend_hops", "blend_delay_max", "latency", "max_uncles",
|
||||
"uncle_strategy", "uncle_window", "init_dest", "k"]
|
||||
# A trajectory's identity: every config axis a sweep can vary. Anything swept but ABSENT here is
|
||||
# silently averaged into one cell — the block-rate sweep varies `f`, so it must be listed — so keep
|
||||
# this exhaustive over the recorded config fields (see metrics._CONFIG_FIELDS).
|
||||
CONFIG_COLS = ["n_nodes", "stake_dist", "pareto_shape", "topology", "degree",
|
||||
"link_latency_mean", "link_latency_dist", "blend_hops", "blend_delay_max",
|
||||
"latency", "uncle_model", "window_absorption", "max_uncles", "uncle_strategy",
|
||||
"uncle_window", "init_dest", "init_spread", "genesis_d_factor",
|
||||
"f", "beta", "k", "fixed_point", "legacy_block_count"]
|
||||
|
||||
# Graph topologies (as opposed to the full_mesh baseline) and the dominant latency knob each
|
||||
# is plotted against: regular varies the per-link latency, blend varies the per-hop mix delay.
|
||||
GRAPH_TOPOLOGIES = ("regular", "blend")
|
||||
|
||||
|
||||
def sem(x) -> float:
|
||||
"""Standard error of the mean over replicates (0 for a single replicate).
|
||||
|
||||
Replicate spread is the only uncertainty estimate these sweeps carry, and at high
|
||||
mixing delay it is large enough to swamp the effects being compared — so any figure or
|
||||
table quoting a cell mean should quote this alongside it.
|
||||
"""
|
||||
x = np.asarray(x, dtype=float)
|
||||
return float(x.std(ddof=1) / np.sqrt(len(x))) if len(x) > 1 else 0.0
|
||||
|
||||
|
||||
def recovery_rate(q, q_u):
|
||||
"""Invert ``theory.q_effective``: ``r = (q_u - q) / (1 - q)``.
|
||||
|
||||
The share of wasted active slots that a countable uncle reference puts back into the
|
||||
count. Clamped denominator so a saturated ``q -> 1`` cell stays finite.
|
||||
"""
|
||||
q, q_u = np.asarray(q, dtype=float), np.asarray(q_u, dtype=float)
|
||||
return (q_u - q) / np.maximum(1.0 - q, 1e-12)
|
||||
|
||||
|
||||
def graph_ell_mean(df: pd.DataFrame) -> float:
|
||||
"""``ell_mean`` — the mean shortest-path (gossip) latency of the run's OWN peering graph.
|
||||
|
||||
Measured from the run's recorded ``(n_nodes, degree, link_latency_mean,
|
||||
link_latency_dist)`` rather than hardcoded, so any derived quantity stays correct if
|
||||
those change. A statistical property of the random d-regular geo graph (seed-invariant
|
||||
to <1% at these N), so one representative draw suffices. Post-processing only: this
|
||||
rebuilds the latency *graph* to read off its mean and never re-runs the simulation.
|
||||
"""
|
||||
from ..config import SimConfig
|
||||
from ..topology import build_path_latency
|
||||
|
||||
row = df.iloc[0]
|
||||
cfg = SimConfig(n_nodes=int(row.n_nodes), degree=int(row.degree), topology="blend",
|
||||
link_latency_mean=float(row.link_latency_mean),
|
||||
link_latency_dist=str(row.link_latency_dist), k=int(row.k))
|
||||
pl = build_path_latency(cfg, np.random.default_rng(0))
|
||||
n = pl.shape[0]
|
||||
return float(pl[~np.eye(n, dtype=bool)].mean())
|
||||
|
||||
|
||||
def rho_for(df: pd.DataFrame, delay) -> np.ndarray:
|
||||
"""Load ``rho = f * D_vis`` with ``D_vis = hops*delta_max/2 + (hops+1)*ell_mean``.
|
||||
|
||||
Every report quotation of ``rho`` must come through here. ``ell_mean`` is the MEASURED
|
||||
mean gossip latency (1.21 slots at N=1000/degree=6), not the per-link
|
||||
``link_latency_mean`` parameter (0.5) — using the latter understates ``rho`` by ~0.1,
|
||||
and hand-substituting a guessed value is how wrong axis labels get into a report.
|
||||
"""
|
||||
f = float(df.f.iloc[0])
|
||||
hops = int(df.blend_hops.iloc[0])
|
||||
ell = graph_ell_mean(df)
|
||||
return f * (hops * np.asarray(delay, dtype=float) / 2.0 + (hops + 1) * ell)
|
||||
|
||||
|
||||
DELAY = "blend_delay_max"
|
||||
# Normal approximation: at 20+ replicates the t-quantile is within a few percent of
|
||||
# 1.96, and the replicate spread dominates, so 1.96 is precise enough for these CIs.
|
||||
Z95 = 1.96
|
||||
|
||||
|
||||
def paired_gaps(cnt_raw: pd.DataFrame, old_raw: pd.DataFrame) -> pd.DataFrame | None:
|
||||
"""Per-replicate differences, when both arms were run with ``paired_streams``.
|
||||
|
||||
Under common random numbers replicate *i* of each arm shares the stake draw, the peering
|
||||
graph and the lottery outcomes, so ``d_i = countable_i - unrestricted_i`` is a PAIRED
|
||||
observation and the shared variance cancels. The test is then a one-sample t on the d_i,
|
||||
which is what makes a sub-0.1 % effect reachable per cell instead of only after pooling.
|
||||
|
||||
Returns None when the runs are not paired, so the caller falls back to the unpaired test.
|
||||
"""
|
||||
if not (cnt_raw.get("paired_streams", pd.Series([False])).all()
|
||||
and old_raw.get("paired_streams", pd.Series([False])).all()):
|
||||
return None
|
||||
c, o = equilibrium(cnt_raw), equilibrium(old_raw)
|
||||
keys = ["blend_delay_max", "max_uncles", "replicate"]
|
||||
m = c[[*keys, "mean_ratio"]].merge(o[[*keys, "mean_ratio"]], on=keys,
|
||||
suffixes=("_c", "_o"))
|
||||
m["d"] = m.mean_ratio_c - m.mean_ratio_o
|
||||
rows = []
|
||||
for (dl, u), s in m.groupby(["blend_delay_max", "max_uncles"]):
|
||||
d = s.d.to_numpy()
|
||||
se = sem(d)
|
||||
rows.append({"blend_delay_max": dl, "max_uncles": u, "gap": float(d.mean()), "se": se,
|
||||
"ci95": Z95 * se, "t": abs(d.mean()) / se if se > 0 else np.inf,
|
||||
"n_pair": len(d), "n_zero": int((d == 0.0).sum())})
|
||||
return pd.DataFrame(rows).sort_values(["max_uncles", "blend_delay_max"])
|
||||
|
||||
|
||||
def pooled_by_delay(g: pd.DataFrame) -> pd.DataFrame:
|
||||
"""Inverse-variance pooled gap across the U >= 1 arms, per delay.
|
||||
|
||||
Individual cells are underpowered against a sub-0.1 % effect even at 40 replicates, but
|
||||
the three uncle caps are independent measurements of the same underlying difference, so
|
||||
pooling them buys back a factor of ~sqrt(3) and is what actually resolves the trend.
|
||||
"""
|
||||
u = g[g.max_uncles > 0]
|
||||
rows = []
|
||||
for d, s in u.groupby(DELAY):
|
||||
w = 1.0 / s.se.to_numpy() ** 2
|
||||
p = float((s.gap.to_numpy() * w).sum() / w.sum())
|
||||
e = float(np.sqrt(1.0 / w.sum()))
|
||||
rows.append({DELAY: d, "gap": p, "se": e, "ci95": Z95 * e,
|
||||
"t": abs(p) / e if e > 0 else np.inf})
|
||||
return pd.DataFrame(rows).sort_values(DELAY)
|
||||
|
||||
|
||||
def _lat_axis(topo: str) -> tuple[str, str]:
|
||||
"""(dataframe column, axis label) for the dominant latency knob of a graph topology."""
|
||||
if topo == "blend":
|
||||
@ -36,12 +149,15 @@ def equilibrium(df: pd.DataFrame, burn_frac: float = 0.5) -> pd.DataFrame:
|
||||
``epochs`` — early-stopped runs (config.early_stop) terminate well before the planned
|
||||
``epochs``, so thresholding on the configured value would drop every row.
|
||||
"""
|
||||
max_epoch = df.groupby([*CONFIG_COLS, "replicate"])["epoch"].transform("max")
|
||||
cfg_cols = [c for c in CONFIG_COLS if c in df.columns] # old parquets lack new fields
|
||||
max_epoch = df.groupby([*cfg_cols, "replicate"])["epoch"].transform("max")
|
||||
tail = df[df["epoch"] >= max_epoch * burn_frac]
|
||||
agg = {c: (c, "mean") for c in
|
||||
("mean_ratio", "range_ratio", "iqr_ratio", "agreement_window", "agreement_tip",
|
||||
"mean_q", "mean_q_eff", "mean_orphan_rate", "max_ratio", "min_ratio")}
|
||||
return tail.groupby([*CONFIG_COLS, "replicate"], as_index=False).agg(**agg)
|
||||
"mean_q", "mean_q_eff", "mean_orphan_rate", "max_ratio", "min_ratio",
|
||||
"deep_ref_share", "p_ref", "fork_rate")
|
||||
if c in tail.columns}
|
||||
return tail.groupby([*cfg_cols, "replicate"], as_index=False).agg(**agg)
|
||||
|
||||
|
||||
def _prov(df: pd.DataFrame) -> str:
|
||||
|
||||
@ -20,8 +20,11 @@ per-node loop. Two ingredients tie it to the protocol parameters:
|
||||
``+1`` w.p. ``alpha_eff`` (adversary extends privately), ``-1`` w.p. ``1-alpha_eff`` (honest
|
||||
extends the public chain). Each excursion above 0 is one attack; the deepest reorg it can force
|
||||
is the maximum lead reached (release the private chain at its peak, displacing that many public
|
||||
confirmations). The tail is the Nakamoto/Rosenfeld gambler's-ruin first passage
|
||||
``P(depth >= d) = (alpha_eff/(1-alpha_eff))**d`` for ``alpha_eff < 1/2``.
|
||||
confirmations). ``P(depth >= d) = (alpha_eff/(1-alpha_eff))**d`` (``alpha_eff < 1/2``) is the
|
||||
Nakamoto/Rosenfeld gambler's-ruin *catch-up* probability — the chance an adversary ``d`` blocks
|
||||
behind ever draws level — an upper bound on that per-attack maximum, not its exact distribution
|
||||
(which is smaller). ``fig27`` plots the occupancy Monte-Carlo; §6.10's 4/8/17 and 8/15/34 are the
|
||||
engine's realised reorg depths.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@ -71,3 +74,20 @@ def simulate_deepest_reorg(alpha_eff: float, n_events: int,
|
||||
peak = 0
|
||||
# honest block at lead 0 -> canonical growth, no attack in progress
|
||||
return np.asarray(depths, dtype=np.int64)
|
||||
|
||||
|
||||
def countable_recovery_from_depths(depths: np.ndarray) -> float:
|
||||
"""Share of reorg-discarded public blocks the countable model can reference back (§6.6).
|
||||
|
||||
A depth-``d`` reorg displaces ``d`` *consecutive* public blocks. They form one chain rooted at
|
||||
the fork point, and the countable model can reference only the block whose parent lies on the
|
||||
referencing chain — so one of the ``d`` is recoverable into the density and ``d-1`` are not.
|
||||
Over a run of attacks that ceiling is ``attacks / blocks = len(depths) / sum(depths)``.
|
||||
|
||||
This is the depth-*maximising* adversary's value; the revenue-maximising one is
|
||||
:func:`tsi_sim.selfish_mdp.optimal_policy_stats`. Neither is a worst case for the estimator —
|
||||
an adversary optimising deflation directly would trade revenue for depth.
|
||||
"""
|
||||
d = np.asarray(depths, dtype=np.int64)
|
||||
total = int(d.sum())
|
||||
return (d.size / total) if total else 1.0
|
||||
|
||||
@ -17,7 +17,9 @@ from .config import SimConfig
|
||||
|
||||
|
||||
def _entropy(config: SimConfig) -> int:
|
||||
payload = repr((config.root_seed, config.key())).encode()
|
||||
# seed_key() == key() unless paired_streams is set, in which case the uncle-model marker is
|
||||
# dropped so both arms of a comparison share one stream (common random numbers).
|
||||
payload = repr((config.root_seed, config.seed_key())).encode()
|
||||
digest = hashlib.blake2b(payload, digest_size=16).digest()
|
||||
return int.from_bytes(digest, "big")
|
||||
|
||||
|
||||
@ -63,6 +63,29 @@ class RaceResult:
|
||||
"""Counted canonical blocks / all mined blocks — the factor TSI deflates ``D̂`` by."""
|
||||
return (self.adv + self.hon) / self.events if self.events else 1.0
|
||||
|
||||
@property
|
||||
def orphan_hon_runs(self) -> int:
|
||||
"""Orphaned honest *chains* — the number of them the countable model can reference.
|
||||
|
||||
The countable uncle model (§2.1) can reference only the first block of a fork, so a
|
||||
discarded chain of ``h`` honest blocks yields one countable uncle, not ``h``. **Under SM1
|
||||
this equals** ``orphan_hon``: the rule acts the moment the honest branch reaches length 1
|
||||
(match at a 1-lead, override at a 2-lead, publish-one above it), so it never buries a
|
||||
second block behind the first and every orphan it makes is the first block of its fork.
|
||||
The optimal policy *waits* and does bury them — see
|
||||
:func:`tsi_sim.selfish_mdp.optimal_policy_stats`, where the ratio drops well below 1.
|
||||
"""
|
||||
return self.orphan_hon
|
||||
|
||||
@property
|
||||
def countable_recovery(self) -> float:
|
||||
"""Ceiling on the uncle-recovery fraction ``eta`` under the first-fork rule: 1.0 for SM1.
|
||||
|
||||
Stated as a property rather than a constant so the SM1 claim is checkable in one place
|
||||
(``tests/test_countable_selfish.py``) against the same quantity computed for the optimum.
|
||||
"""
|
||||
return self.orphan_hon_runs / self.orphan_hon if self.orphan_hon else 1.0
|
||||
|
||||
|
||||
def simulate_selfish(is_adv: np.ndarray, gamma: float, rng: np.random.Generator) -> RaceResult:
|
||||
"""Eyal–Sirer SM1 selfish-mining race over a stream of block-finding events.
|
||||
@ -217,6 +240,12 @@ def tsi_dhat_ratio(race: RaceResult, uncle_recovery: float) -> float:
|
||||
TSI drives the *counted* density to ``f``; the counted density is the canonical fraction plus a
|
||||
recovered fraction ``uncle_recovery ∈ [0,1]`` of the orphaned HONEST blocks (referenced back as
|
||||
uncles — adversary blocks reference none, §6.4). ``D̂/D* = (adv + hon + u*orphan_hon)/events``.
|
||||
|
||||
``uncle_recovery`` is a free knob. Against **SM1** the deployed counting rules admit its full
|
||||
range — every orphan SM1 makes is countable (``race.countable_recovery == 1``). Against the
|
||||
**optimal** policy they do not: use
|
||||
:meth:`tsi_sim.selfish_mdp.OptimalPolicyStats.dhat_ratio`, which caps recovery at one uncle per
|
||||
orphaned chain (§6.6).
|
||||
"""
|
||||
u = float(np.clip(uncle_recovery, 0.0, 1.0))
|
||||
counted = race.adv + race.hon + u * race.orphan_hon
|
||||
|
||||
@ -13,10 +13,19 @@ bisection, solving each inner MDP by relative value iteration.
|
||||
|
||||
The optimal revenue is an upper bound on any selfish adversary's take; it lower-bounds the honest
|
||||
stake threshold above which deviating pays. Used in §6.6 to bracket the real profit frontier.
|
||||
|
||||
:func:`optimal_policy_stats` reads a second quantity off the same solution: how the orphaned
|
||||
blocks are *shaped*. The countable uncle model (§2.1) can reference only the **first block of a
|
||||
fork**, so an override that discards ``h`` honest blocks — one chain — yields one countable
|
||||
uncle, not ``h``. SM1 never lets the honest branch grow past 1 before acting, so under SM1 every
|
||||
orphan is countable; the optimum *waits*, and that is what the first-fork restriction cannot
|
||||
recover (§6.6).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
import numpy as np
|
||||
|
||||
# fork labels
|
||||
@ -34,46 +43,53 @@ def _build_states(cap: int):
|
||||
|
||||
|
||||
def _transitions(a, h, f, action, alpha, gamma, cap):
|
||||
"""Legal-action transition list: [(prob, (a',h',f'), adv_reward, hon_reward)].
|
||||
"""Legal-action transition list: [(prob, (a',h',f'), adv_reward, hon_reward, orph_hon,
|
||||
orph_adv)].
|
||||
|
||||
Returns None if the action is illegal in this state. At the cap, only adopt/override remain so
|
||||
the chain stays bounded (the optimal policy resolves long before the cap in the tested range).
|
||||
|
||||
``orph_hon`` / ``orph_adv`` are the honest / adversary blocks *discarded* on that branch. They
|
||||
are carried here rather than re-derived downstream so the orphan accounting cannot drift from
|
||||
the race logic. Each is a **contiguous chain** rooted at the fork point, which is what makes the
|
||||
countable model's first-fork rule recover exactly one of them (:func:`optimal_policy_stats`).
|
||||
"""
|
||||
beta = 1.0 - alpha
|
||||
at_cap = a >= cap or h >= cap
|
||||
|
||||
if action == ADOPT:
|
||||
# abandon the secret chain; the h honest blocks are confirmed, then one block is mined
|
||||
return [(alpha, (1, 0, IRRELEVANT), 0, h),
|
||||
(beta, (0, 1, IRRELEVANT), 0, h)]
|
||||
# abandon the secret chain; the h honest blocks are confirmed, then one block is mined.
|
||||
# The a secret blocks are discarded as one chain off the fork point.
|
||||
return [(alpha, (1, 0, IRRELEVANT), 0, h, 0, a),
|
||||
(beta, (0, 1, IRRELEVANT), 0, h, 0, a)]
|
||||
|
||||
if action == OVERRIDE:
|
||||
if a <= h:
|
||||
return None # need a strictly longer chain to override
|
||||
# publish h+1 blocks -> they override the public h; a-h-1 stay secret; then one block mined
|
||||
return [(alpha, (a - h, 0, IRRELEVANT), h + 1, 0),
|
||||
(beta, (a - h - 1, 1, RELEVANT), h + 1, 0)]
|
||||
return [(alpha, (a - h, 0, IRRELEVANT), h + 1, 0, h, 0),
|
||||
(beta, (a - h - 1, 1, RELEVANT), h + 1, 0, h, 0)]
|
||||
|
||||
if at_cap:
|
||||
return None # only adopt/override allowed at the boundary
|
||||
|
||||
if action == WAIT:
|
||||
if f != ACTIVE:
|
||||
return [(alpha, (a + 1, h, IRRELEVANT), 0, 0),
|
||||
(beta, (a, h + 1, RELEVANT), 0, 0)]
|
||||
return [(alpha, (a + 1, h, IRRELEVANT), 0, 0, 0, 0),
|
||||
(beta, (a, h + 1, RELEVANT), 0, 0, 0, 0)]
|
||||
if a < h:
|
||||
return None # inconsistent (unreachable) active state: only adopt is valid
|
||||
# waiting while a fork is active: the same race dynamics as match
|
||||
return [(alpha, (a + 1, h, ACTIVE), 0, 0),
|
||||
(gamma * beta, (a - h, 1, RELEVANT), h, 0), # adv's matched branch wins h
|
||||
((1 - gamma) * beta, (a, h + 1, RELEVANT), 0, 0)]
|
||||
return [(alpha, (a + 1, h, ACTIVE), 0, 0, 0, 0),
|
||||
(gamma * beta, (a - h, 1, RELEVANT), h, 0, h, 0), # adv's matched branch wins h
|
||||
((1 - gamma) * beta, (a, h + 1, RELEVANT), 0, 0, 0, 0)]
|
||||
|
||||
if action == MATCH:
|
||||
if not (f == RELEVANT and a >= h):
|
||||
return None # match needs equal-or-longer chain on a fresh tip
|
||||
return [(alpha, (a + 1, h, ACTIVE), 0, 0),
|
||||
(gamma * beta, (a - h, 1, RELEVANT), h, 0),
|
||||
((1 - gamma) * beta, (a, h + 1, RELEVANT), 0, 0)]
|
||||
return [(alpha, (a + 1, h, ACTIVE), 0, 0, 0, 0),
|
||||
(gamma * beta, (a - h, 1, RELEVANT), h, 0, h, 0),
|
||||
((1 - gamma) * beta, (a, h + 1, RELEVANT), 0, 0, 0, 0)]
|
||||
|
||||
return None
|
||||
|
||||
@ -89,6 +105,8 @@ def _precompute(alpha, gamma, states, index, cap):
|
||||
nxt = np.zeros((4, n, _K), dtype=np.int64)
|
||||
radv = np.zeros((4, n, _K))
|
||||
rhon = np.zeros((4, n, _K))
|
||||
ohon = np.zeros((4, n, _K)) # honest blocks discarded on the branch (one chain)
|
||||
oadv = np.zeros((4, n, _K)) # adversary blocks discarded on the branch (one chain)
|
||||
legal = np.zeros((4, n), dtype=bool)
|
||||
for i, (a, h, f) in enumerate(states):
|
||||
for action in (ADOPT, OVERRIDE, MATCH, WAIT):
|
||||
@ -96,24 +114,26 @@ def _precompute(alpha, gamma, states, index, cap):
|
||||
if tr is None:
|
||||
continue
|
||||
legal[action, i] = True
|
||||
for b, (p, s2, ra, rh) in enumerate(tr):
|
||||
for b, (p, s2, ra, rh, oh, oa) in enumerate(tr):
|
||||
probs[action, i, b] = p
|
||||
nxt[action, i, b] = index[s2]
|
||||
radv[action, i, b] = ra
|
||||
rhon[action, i, b] = rh
|
||||
return probs, nxt, radv, rhon, legal
|
||||
ohon[action, i, b] = oh
|
||||
oadv[action, i, b] = oa
|
||||
return probs, nxt, radv, rhon, ohon, oadv, legal
|
||||
|
||||
|
||||
def _solve_mdp(pc, rho, ref, iters, tol):
|
||||
"""Optimal average gain for the rho-parametrised reward, by *damped* relative value iteration.
|
||||
def _solve_reward(pc, reward, ref, iters, tol):
|
||||
"""Optimal average gain for an arbitrary per-branch reward, by *damped* relative value
|
||||
iteration. Returns ``(gain, V)``.
|
||||
|
||||
The chain is periodic, so undamped VI oscillates and a naive |Δgain| stop can false-trigger as
|
||||
the gain crosses zero. We damp (``V ← V + τ(TV − V)``) to break periodicity and stop on the
|
||||
textbook span criterion: at the average-reward fixed point ``TV − V = g·1`` (span → 0), and the
|
||||
gain ``g`` is that uniform increment. Returns the span-centre of the final Bellman increment.
|
||||
"""
|
||||
probs, nxt, radv, rhon, legal = pc
|
||||
reward = (1.0 - rho) * radv - rho * rhon # (4, n, K), constant across iterations
|
||||
probs, nxt, _radv, _rhon, _ohon, _oadv, legal = pc
|
||||
V = np.zeros(probs.shape[1])
|
||||
tau = 0.5
|
||||
d = np.zeros(1)
|
||||
@ -125,7 +145,61 @@ def _solve_mdp(pc, rho, ref, iters, tol):
|
||||
break
|
||||
V = V + tau * d
|
||||
V -= V[ref] # anchor to keep values bounded
|
||||
return 0.5 * (d.max() + d.min())
|
||||
return 0.5 * (d.max() + d.min()), V
|
||||
|
||||
|
||||
def _solve_mdp(pc, rho, ref, iters, tol):
|
||||
"""Optimal average gain for the rho-parametrised *revenue* reward (the ratio transform)."""
|
||||
_probs, _nxt, radv, rhon, _ohon, _oadv, _legal = pc
|
||||
return _solve_reward(pc, (1.0 - rho) * radv - rho * rhon, ref, iters, tol)[0]
|
||||
|
||||
|
||||
def _greedy_policy(pc, reward, V):
|
||||
"""Greedy action per state, tie-broken toward the lowest index (ADOPT first, i.e. the
|
||||
least-deviating action) so near-ties resolve deterministically rather than arbitrarily."""
|
||||
probs, nxt, _radv, _rhon, _ohon, _oadv, legal = pc
|
||||
q = (probs * (reward + V[nxt])).sum(axis=2)
|
||||
q[~legal] = -1e18
|
||||
best = q.max(axis=0)
|
||||
return np.where(q >= best[None, :] - 1e-9, np.arange(4)[:, None], 99).min(axis=0)
|
||||
|
||||
|
||||
def _stationary(pc, pol):
|
||||
"""Stationary distribution of the policy-induced chain. The chain is periodic (see
|
||||
:func:`_solve_reward`), so iterate the LAZY chain — same stationary vector, no oscillation."""
|
||||
probs, nxt, *_ = pc
|
||||
n = probs.shape[1]
|
||||
rows = np.arange(n)
|
||||
p_s, n_s = probs[pol, rows], nxt[pol, rows]
|
||||
pi = np.full(n, 1.0 / n)
|
||||
for _ in range(500_000):
|
||||
new = 0.5 * pi + 0.5 * np.bincount(n_s.ravel(), weights=(pi[:, None] * p_s).ravel(),
|
||||
minlength=n)
|
||||
new /= new.sum()
|
||||
if np.abs(new - pi).max() < 1e-15:
|
||||
return new
|
||||
pi = new
|
||||
return pi
|
||||
|
||||
|
||||
def _policy_rates(pc, pol, pi):
|
||||
"""Per-block-finding-event rates under a policy's stationary distribution.
|
||||
|
||||
Every MDP transition consumes exactly one block-finding event (the alpha/beta branch), so
|
||||
stationary per-step rates *are* per-event rates.
|
||||
"""
|
||||
probs, nxt, radv, rhon, ohon, oadv, _legal = pc
|
||||
rows = np.arange(probs.shape[1])
|
||||
w = pi[:, None] * probs[pol, rows]
|
||||
oh, oa = ohon[pol, rows], oadv[pol, rows]
|
||||
return dict(
|
||||
adv_rate=float((w * radv[pol, rows]).sum()),
|
||||
hon_rate=float((w * rhon[pol, rows]).sum()),
|
||||
orphan_hon_blocks=float((w * oh).sum()),
|
||||
orphan_hon_runs=float((w * (oh > 0)).sum()),
|
||||
orphan_adv_blocks=float((w * oa).sum()),
|
||||
orphan_adv_runs=float((w * (oa > 0)).sum()),
|
||||
)
|
||||
|
||||
|
||||
def optimal_selfish_revenue(alpha: float, gamma: float, cap: int = 60,
|
||||
@ -142,6 +216,11 @@ def optimal_selfish_revenue(alpha: float, gamma: float, cap: int = 60,
|
||||
states, index = _build_states(cap)
|
||||
pc = _precompute(alpha, gamma, states, index, cap)
|
||||
ref = index[(1, 0, IRRELEVANT)]
|
||||
return _bisect_revenue(pc, ref, alpha, iters, tol)
|
||||
|
||||
|
||||
def _bisect_revenue(pc, ref, alpha, iters, tol):
|
||||
"""Bisection on the ratio objective — the shared inner loop of the revenue solvers."""
|
||||
lo, hi = alpha - 1e-9, 1.0 # relative revenue in [alpha, 1)
|
||||
for _ in range(44):
|
||||
mid = 0.5 * (lo + hi)
|
||||
@ -151,3 +230,188 @@ def optimal_selfish_revenue(alpha: float, gamma: float, cap: int = 60,
|
||||
else:
|
||||
hi = mid
|
||||
return 0.5 * (lo + hi)
|
||||
|
||||
|
||||
@dataclass
|
||||
class OptimalPolicyStats:
|
||||
"""Per-block-finding-event rates under the optimal policy's stationary distribution.
|
||||
|
||||
Every MDP transition consumes exactly one block-finding event (the alpha/beta branch), so
|
||||
stationary per-step rates *are* per-event rates. ``deviates`` is False below the profitability
|
||||
threshold, where the optimum is honest mining and the MDP is indifferent across policies (the
|
||||
value-iteration policy is then arbitrary and its orphan structure meaningless).
|
||||
"""
|
||||
alpha: float
|
||||
gamma: float
|
||||
revenue: float
|
||||
deviates: bool
|
||||
density_fraction: float # canonical blocks per event — the raw TSI deflation factor
|
||||
orphan_hon_blocks: float # honest blocks orphaned per event
|
||||
orphan_hon_runs: float # honest orphan *chains* per event (1 countable uncle each)
|
||||
orphan_adv_blocks: float # adversary blocks discarded per event
|
||||
orphan_adv_runs: float
|
||||
|
||||
@property
|
||||
def countable_recovery(self) -> float:
|
||||
"""Ceiling on the uncle-recovery fraction ``eta`` under the first-fork rule (§2.1).
|
||||
|
||||
``runs / blocks``: an override discards a *chain* of honest blocks and only its first is
|
||||
referenceable, so this is the largest ``eta`` the deployed counting rules admit — 1.0 under
|
||||
SM1 (which never buries a second block), below 1 whenever the policy waits.
|
||||
"""
|
||||
if self.orphan_hon_blocks <= 0:
|
||||
return 1.0
|
||||
return self.orphan_hon_runs / self.orphan_hon_blocks
|
||||
|
||||
@property
|
||||
def countable_recovery_adv(self) -> float:
|
||||
"""The same ceiling on the attacker *self-uncling* its own abandoned chain (§6.7(a))."""
|
||||
if self.orphan_adv_blocks <= 0:
|
||||
return 1.0
|
||||
return self.orphan_adv_runs / self.orphan_adv_blocks
|
||||
|
||||
def dhat_ratio(self, p_ref: float = 1.0, countable: bool = True) -> float:
|
||||
"""Equilibrium ``D̂/D*`` = canonical density + the referenced share of honest orphans.
|
||||
|
||||
``countable=True`` counts one uncle per orphaned *chain* (the deployed rule);
|
||||
``countable=False`` is the unrestricted baseline that counts every orphaned block.
|
||||
"""
|
||||
rec = self.orphan_hon_runs if countable else self.orphan_hon_blocks
|
||||
return self.density_fraction + float(np.clip(p_ref, 0.0, 1.0)) * rec
|
||||
|
||||
|
||||
def deflation_optimal_stats(alpha: float, gamma: float, p_ref: float = 1.0, cap: int = 64,
|
||||
iters: int = 4000, tol: float = 1e-10) -> OptimalPolicyStats:
|
||||
"""The policy that MINIMISES the estimate, rather than the one that maximises revenue.
|
||||
|
||||
Both ceilings in §6.6 come from adversaries optimising something else — revenue (the SSZ
|
||||
objective) and reorg depth — so they bound ``eta`` from above without bounding the damage
|
||||
from below. This closes that gap by optimising the estimator directly.
|
||||
|
||||
No ratio transform is needed, unlike the revenue objective. Every transition consumes exactly
|
||||
one block-finding event, and the estimate is
|
||||
|
||||
D̂/D = (canonical blocks + p_ref · countable uncles) / events
|
||||
|
||||
with one countable uncle per discarded honest *run* (§2.1). So the per-event contribution is
|
||||
``radv + rhon + p_ref·[orphaned honest run]``, and minimising its long-run average is a plain
|
||||
average-reward MDP — solved by maximising the negated reward in one value-iteration pass.
|
||||
|
||||
``p_ref`` is the share of countable orphans that honest referencers actually pick up; at the
|
||||
default 1 the adversary faces the most effective possible repair, so the resulting deflation
|
||||
is the worst case it can force against a fully-cooperative honest network.
|
||||
|
||||
**The unconstrained optimum is degenerate, and usefully so.** It is pure abstention: mine
|
||||
privately, publish nothing, adopt when overtaken. That drives ``D̂`` to exactly ``1 - alpha``
|
||||
and revenue to zero. But §6.4 already establishes that this is *correct* measurement rather
|
||||
than mis-measurement — a coalition that publishes nothing genuinely is not participating, and
|
||||
``1 - alpha`` is the right answer for the stake that is. So the unconstrained objective asks
|
||||
the wrong question; the one that matters is how far ``D̂`` can be pushed by an adversary that
|
||||
stays profitable, which :func:`deflation_frontier` traces.
|
||||
"""
|
||||
states, index = _build_states(cap)
|
||||
pc = _precompute(alpha, gamma, states, index, cap)
|
||||
probs, nxt, radv, rhon, ohon, oadv, legal = pc
|
||||
ref = index[(1, 0, IRRELEVANT)]
|
||||
|
||||
dhat_step = radv + rhon + float(p_ref) * (ohon > 0)
|
||||
gain, V = _solve_reward(pc, -dhat_step, ref, iters, tol)
|
||||
pol = _greedy_policy(pc, -dhat_step, V)
|
||||
pi = _stationary(pc, pol)
|
||||
rates = _policy_rates(pc, pol, pi)
|
||||
|
||||
# The gain IS the negated minimum estimate; cross-check it against the stationary rates so a
|
||||
# silent mismatch between the solver and the accounting cannot pass unnoticed.
|
||||
dhat = rates["adv_rate"] + rates["hon_rate"] + float(p_ref) * rates["orphan_hon_runs"]
|
||||
if abs(-gain - dhat) > 1e-6:
|
||||
raise AssertionError(f"deflation MDP gain {-gain:.9f} != stationary D-hat {dhat:.9f}")
|
||||
|
||||
revenue = (rates["adv_rate"] / (rates["adv_rate"] + rates["hon_rate"])
|
||||
if rates["adv_rate"] + rates["hon_rate"] > 0 else 0.0)
|
||||
return OptimalPolicyStats(alpha=alpha, gamma=gamma, revenue=revenue, deviates=True,
|
||||
density_fraction=rates["adv_rate"] + rates["hon_rate"],
|
||||
orphan_hon_blocks=rates["orphan_hon_blocks"],
|
||||
orphan_hon_runs=rates["orphan_hon_runs"],
|
||||
orphan_adv_blocks=rates["orphan_adv_blocks"],
|
||||
orphan_adv_runs=rates["orphan_adv_runs"])
|
||||
|
||||
|
||||
def deflation_frontier(alpha: float, gamma: float, lam: float, p_ref: float = 1.0,
|
||||
cap: int = 64, iters: int = 4000, tol: float = 1e-10) -> dict:
|
||||
"""One point on the profit/deflation trade-off: the policy optimal for a mixed objective.
|
||||
|
||||
Neither pure objective answers item 16. Maximising revenue ignores the estimator; minimising
|
||||
the estimate degenerates to abstention, which forfeits every block reward and is correctly
|
||||
measured anyway (:func:`deflation_optimal_stats`). What the report needs to know is how much
|
||||
deflation an adversary can force *while still being paid* — i.e. the Pareto frontier between
|
||||
the two.
|
||||
|
||||
Sweeping ``lam`` from 0 upward traces it: the per-event reward is
|
||||
``lam · (adversary blocks) − (contribution to D̂)``, so ``lam = 0`` is the deflation optimum
|
||||
and large ``lam`` approaches the revenue optimum. The point of interest is where the revenue
|
||||
*share* crosses ``alpha`` — an adversary doing at least as well as honest mining — because
|
||||
below that the attack is self-punishing griefing already bounded by §6.5.
|
||||
"""
|
||||
states, index = _build_states(cap)
|
||||
pc = _precompute(alpha, gamma, states, index, cap)
|
||||
_probs, _nxt, radv, rhon, ohon, _oadv, _legal = pc
|
||||
ref = index[(1, 0, IRRELEVANT)]
|
||||
|
||||
reward = float(lam) * radv - (radv + rhon + float(p_ref) * (ohon > 0))
|
||||
_gain, V = _solve_reward(pc, reward, ref, iters, tol)
|
||||
pol = _greedy_policy(pc, reward, V)
|
||||
rates = _policy_rates(pc, pol, _stationary(pc, pol))
|
||||
canonical = rates["adv_rate"] + rates["hon_rate"]
|
||||
blocks, runs = rates["orphan_hon_blocks"], rates["orphan_hon_runs"]
|
||||
revenue = (rates["adv_rate"] / canonical) if canonical > 0 else 0.0
|
||||
return dict(
|
||||
alpha=alpha, gamma=gamma, lam=lam,
|
||||
revenue=revenue,
|
||||
reward_per_stake=(revenue / alpha) if alpha else 0.0,
|
||||
density_fraction=canonical,
|
||||
dhat_countable=canonical + float(p_ref) * runs,
|
||||
dhat_unrestricted=canonical + float(p_ref) * blocks,
|
||||
eta=(runs / blocks) if blocks > 0 else 1.0,
|
||||
orphan_hon_blocks=blocks,
|
||||
)
|
||||
|
||||
|
||||
def optimal_policy_stats(alpha: float, gamma: float, cap: int = 64,
|
||||
iters: int = 4000, tol: float = 1e-10) -> OptimalPolicyStats:
|
||||
"""Orphan structure of the *optimal* selfish policy — the input the countable model needs.
|
||||
|
||||
Solves the same MDP as :func:`optimal_selfish_revenue`, then reads the greedy policy off the
|
||||
value function at the optimal ``rho``, finds its stationary distribution, and accumulates the
|
||||
per-event canonical / orphan rates carried on the transition table.
|
||||
|
||||
``cap`` must be larger here than for the revenue alone: the revenue converges once long leads
|
||||
are rare, but the orphan *shape* keeps changing while the policy still waits near the cap
|
||||
(measured drift at alpha = 0.45 is ~0.005 in eta from cap 48 to 64, ~0.0003 at alpha = 0.4).
|
||||
"""
|
||||
states, index = _build_states(cap)
|
||||
pc = _precompute(alpha, gamma, states, index, cap)
|
||||
probs, nxt, radv, rhon, ohon, oadv, legal = pc
|
||||
ref = index[(1, 0, IRRELEVANT)]
|
||||
revenue = _bisect_revenue(pc, ref, alpha, iters, tol)
|
||||
|
||||
# Below the profitability threshold the optimum is honest mining (revenue == alpha) and the MDP
|
||||
# is indifferent among many policies; report the honest outcome rather than an arbitrary one.
|
||||
if revenue <= alpha * (1.0 + 1e-6):
|
||||
return OptimalPolicyStats(alpha=alpha, gamma=gamma, revenue=revenue, deviates=False,
|
||||
density_fraction=1.0, orphan_hon_blocks=0.0,
|
||||
orphan_hon_runs=0.0, orphan_adv_blocks=0.0,
|
||||
orphan_adv_runs=0.0)
|
||||
|
||||
# Recover V at the optimal rho, then read off the greedy policy and its stationary rates.
|
||||
reward = (1.0 - revenue) * radv - revenue * rhon
|
||||
_gain, V = _solve_reward(pc, reward, ref, iters, tol)
|
||||
pol = _greedy_policy(pc, reward, V)
|
||||
rates = _policy_rates(pc, pol, _stationary(pc, pol))
|
||||
return OptimalPolicyStats(
|
||||
alpha=alpha, gamma=gamma, revenue=revenue, deviates=True,
|
||||
density_fraction=rates["adv_rate"] + rates["hon_rate"],
|
||||
orphan_hon_blocks=rates["orphan_hon_blocks"],
|
||||
orphan_hon_runs=rates["orphan_hon_runs"],
|
||||
orphan_adv_blocks=rates["orphan_adv_blocks"],
|
||||
orphan_adv_runs=rates["orphan_adv_runs"],
|
||||
)
|
||||
|
||||
@ -77,7 +77,7 @@ def _arrival_columns(config: SimConfig, peak_blocks: int) -> int:
|
||||
if not (config.prune_arrival and config.windowed_fork_choice):
|
||||
return peak_blocks
|
||||
per_slot = peak_blocks / config.epoch_len if config.epoch_len else peak_blocks
|
||||
keepspan = float(config.uncle_window)
|
||||
keepspan = float(config.effective_uncle_window)
|
||||
if config.topology == "blend":
|
||||
lat = max(config.link_latency_mean, 0.1)
|
||||
keepspan = max(keepspan, (config.blend_hops + 1) * lat * 4
|
||||
@ -314,19 +314,26 @@ def main(argv: list[str] | None = None) -> None:
|
||||
"(default) probes when N>2000, 'always', or 'never' (estimate only)")
|
||||
parser.add_argument("--no-figures", action="store_true",
|
||||
help="skip auto figure generation")
|
||||
parser.add_argument("--old", action="store_true",
|
||||
help="run the old (pre countable redesign) uncle model: window = "
|
||||
"uncle_window slots, any-depth orphans referenceable, every "
|
||||
"baked reference counts; bit-reproduces historical runs")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
batch_size = int(args.batch_size) if args.batch_size != "auto" else "auto"
|
||||
label = args.label or Path(args.config).stem
|
||||
label = args.label or (Path(args.config).stem + ("-old" if args.old else ""))
|
||||
run_dir = new_run_dir(args.outdir, label)
|
||||
|
||||
sweep = load_sweep_yaml(args.config)
|
||||
if args.old:
|
||||
sweep.base["uncle_model"] = "old"
|
||||
df = run_sweep(sweep, n_jobs=args.n_jobs, batch_size=batch_size, mem_frac=args.mem_frac,
|
||||
calibrate=args.calibrate)
|
||||
results_path = run_dir / "results.parquet"
|
||||
persist(df, results_path)
|
||||
key_cols = ["n_nodes", "stake_dist", "topology", "degree", "link_latency_mean",
|
||||
"latency", "max_uncles", "uncle_strategy", "init_dest", "replicate"]
|
||||
"latency", "uncle_model", "max_uncles", "uncle_strategy", "init_dest",
|
||||
"replicate"]
|
||||
n_cfg = len(df[key_cols].drop_duplicates())
|
||||
print(f"wrote {len(df)} rows ({n_cfg} configs) -> {results_path}")
|
||||
|
||||
|
||||
@ -18,14 +18,40 @@ def expected_ratio(f: float, q: ArrayLike) -> ArrayLike:
|
||||
return np.log(1.0 - f) / np.log(1.0 - f / q)
|
||||
|
||||
|
||||
def block_count_ceiling(f: float) -> float:
|
||||
"""Equilibrium ratio at *full* uncle recovery for the equal-stake limit.
|
||||
def q_effective(q: ArrayLike, r: ArrayLike) -> ArrayLike:
|
||||
"""Effective slot utilisation with uncle recovery: ``q_u = q + (1 - q) r``.
|
||||
|
||||
TSI counts blocks (all lottery wins, rate ``-ln(1-f)`` per slot in the small-stake
|
||||
limit), whereas ``f`` is the *active-slot* rate. So even with every orphan recovered the
|
||||
estimate equilibrates at ``-ln(1-f)/f`` (~1.017 for f=1/30), not 1.0. This is a
|
||||
deterministic overshoot floor, not noise; concentrated (Pareto) stake gives a smaller
|
||||
value because of the concavity of ``phi``.
|
||||
``r`` is the recovery rate — the probability that a wasted active slot is recovered by a
|
||||
countable referenced uncle. Recovery is a binomial thinning of the waste
|
||||
(``n ~ Bin(p, A)`` wasted, ``u | n ~ Bin(r, n)`` recovered, so the residual waste is
|
||||
``Bin(p(1-r), A)``), hence every closed-form result above holds verbatim with ``q``
|
||||
replaced by ``q_u``. Full recovery (``r = 1``) gives ``q_u = 1`` and an unbiased
|
||||
equilibrium; ``r = 0`` reduces to the chain-only ``q``.
|
||||
"""
|
||||
q = np.asarray(q, dtype=float)
|
||||
r = np.asarray(r, dtype=float)
|
||||
return q + (1.0 - q) * r
|
||||
|
||||
|
||||
def window_miss_prob(f: float, w_abs: ArrayLike) -> ArrayLike:
|
||||
"""P(no canonical block appears within the uncle window ``w_u = W/f``) — the window's
|
||||
contribution to non-recovery: ``(1-f)^(W/f) ~ e^-W`` (4.5e-5 at the default W = 10).
|
||||
The absorption parameter W therefore controls the miss probability directly.
|
||||
"""
|
||||
w_abs = np.asarray(w_abs, dtype=float)
|
||||
return (1.0 - f) ** (w_abs / f)
|
||||
|
||||
|
||||
def block_count_ceiling(f: float) -> float:
|
||||
"""LEGACY-mode ceiling: the equilibrium ratio under ``legacy_block_count=True``.
|
||||
|
||||
Applies only to the superseded per-*block* counting (``tsi.density_m(...,
|
||||
legacy_block_count=True)``), which counted every lottery win (rate ``-ln(1-f)`` per slot
|
||||
in the small-stake limit) while ``f`` is the *active-slot* rate. In that mode, even with
|
||||
every orphan recovered the estimate equilibrated at ``-ln(1-f)/f`` (~1.017 for f=1/30) —
|
||||
a deterministic overshoot, not noise; concentrated (Pareto) stake gave a smaller value
|
||||
from the concavity of ``phi``. The default slot-counting engine has no such ceiling: the
|
||||
equilibrium is bounded by 1 (report §2.1/§2.2).
|
||||
"""
|
||||
return -np.log(1.0 - f) / f
|
||||
|
||||
|
||||
@ -19,30 +19,61 @@ from .blocktree import BlockTree
|
||||
|
||||
|
||||
def referenced_uncle_ids(tree: BlockTree, canonical_ids: list[int]) -> set[int]:
|
||||
"""Deduplicated set of uncle ids referenced by the canonical chain."""
|
||||
"""Deduplicated set of uncle ids referenced by the canonical chain (old model: all)."""
|
||||
ref: set[int] = set()
|
||||
for b in canonical_ids:
|
||||
ref.update(tree.uncles[b])
|
||||
return ref
|
||||
|
||||
|
||||
def countable_refs(tree: BlockTree, canonical_ids: list[int], w: int) -> set[int]:
|
||||
"""Deduplicated set of COUNTABLE referenced uncles (spec counting rules).
|
||||
|
||||
A reference ``u`` of canonical block ``b`` is countable iff ``u`` is not itself
|
||||
canonical, ``0 < slot_b - slot_u <= w``, and ``u``'s parent lies on the canonical chain
|
||||
(only the first block of a fork counts) — the per-reference re-check of
|
||||
cryptarchia-v1-protocol.md's counting rules. Reference implementation for the
|
||||
measurement kernel (see test_measure / test_tsi_counting).
|
||||
"""
|
||||
canon = set(canonical_ids)
|
||||
out: set[int] = set()
|
||||
for b in canonical_ids:
|
||||
sb = int(tree.slot[b])
|
||||
for u in tree.uncles[b]:
|
||||
if u in canon:
|
||||
continue # uncle lies on the counting chain
|
||||
du = sb - int(tree.slot[u])
|
||||
if not 0 < du <= w:
|
||||
continue # outside the reference window
|
||||
p = int(tree.parent[u])
|
||||
if p != 0 and p not in canon:
|
||||
continue # not a first fork block (deep): uncounted
|
||||
out.add(u)
|
||||
return out
|
||||
|
||||
|
||||
def _in_window(slot: int, T: int) -> bool:
|
||||
return 0 <= slot < T
|
||||
|
||||
|
||||
def density_m(tree: BlockTree, canonical_ids: list[int], T: int,
|
||||
legacy_block_count: bool = False) -> int:
|
||||
legacy_block_count: bool = False,
|
||||
countable: bool = False, w: int = 0) -> int:
|
||||
"""Slot count ``m`` for the TSI update: canonical slots + recovered uncle slots.
|
||||
|
||||
A slot counts at most once: canonical blocks occupy distinct slots by construction, and
|
||||
a referenced uncle contributes only if its slot is not already canonical-occupied (and
|
||||
only once per slot, however many same-slot uncles are referenced). ``legacy_block_count``
|
||||
reproduces the earlier per-block-id counting (double-counts multi-winner slots).
|
||||
only once per slot, however many same-slot uncles are referenced). ``countable`` applies
|
||||
the spec's per-reference counting rules via :func:`countable_refs` (window ``w``);
|
||||
``countable=False`` is the old model where every baked reference counts.
|
||||
``legacy_block_count`` reproduces the earlier per-block-id counting (double-counts
|
||||
multi-winner slots).
|
||||
"""
|
||||
s = tree.slot[canonical_ids]
|
||||
in_win = (s >= 0) & (s < T)
|
||||
honest = int(in_win.sum())
|
||||
ref = referenced_uncle_ids(tree, canonical_ids)
|
||||
ref = (countable_refs(tree, canonical_ids, w) if countable
|
||||
else referenced_uncle_ids(tree, canonical_ids))
|
||||
if legacy_block_count:
|
||||
return honest + sum(1 for u in ref if _in_window(int(tree.slot[u]), T))
|
||||
canon_slots = set(int(x) for x in s[in_win])
|
||||
|
||||
@ -1,70 +1,58 @@
|
||||
"""Proposer-local uncle selection.
|
||||
|
||||
For each canonical block ``B`` (processed oldest-first so ancestors' references are
|
||||
known), candidates are orphan (non-canonical) blocks ``U`` with
|
||||
``0 < slot_B - slot_U <= W`` that have not already been referenced by an ancestor of
|
||||
``B``. Two strategies match the spec: deterministic oldest-first, and random (oldest-first
|
||||
order, a coin of probability ``uncle_random_p`` per candidate, capped at ``U``). The spec's
|
||||
coin is unbiased (``uncle_random_p = 0.5``, the default); other values are a non-spec
|
||||
sensitivity knob. Dedup across ancestors is enforced by threading a ``referenced`` set down
|
||||
the canonical chain.
|
||||
Two models, switched by ``config.uncle_model`` (CLI: ``--old``):
|
||||
|
||||
**countable** (default) — the spec's counting-only model (cryptarchia-v1-protocol.md,
|
||||
Uncle Selection): candidates are orphan blocks in the producer's view within the DERIVED
|
||||
window ``w_u = window_absorption / f`` whose **parent lies on the producer's chain** (only
|
||||
the first block of a fork is countable), excluding candidates whose slot is already
|
||||
occupied on that chain (by a canonical block or an already-referenced uncle), and picking
|
||||
at most one uncle per slot, oldest-first (or the ``random`` sensitivity knob).
|
||||
|
||||
**old** (pre-redesign; kept verbatim for ``--old`` reproduction) — candidates are ANY
|
||||
orphan blocks in view with ``0 < slot_B - slot_U <= uncle_window``, regardless of fork
|
||||
depth, that are not on the producer's chain and not already referenced by it; dedup is by
|
||||
block id only (no slot exclusion).
|
||||
|
||||
Selected refs are baked at production and immutable once adopted, so density counting
|
||||
stays view-independent under both models.
|
||||
|
||||
``select_uncles_at_production`` is the ONLY selection entry point — blocktree.py calls it
|
||||
once per produced block. Tests drive it through the same signature (see
|
||||
tests/test_uncles.py ``annotate_via_production``) rather than through a parallel offline
|
||||
implementation, so there is no second copy of the rules to drift out of sync.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
|
||||
from .blocktree import GENESIS, BlockTree
|
||||
from .blocktree import GENESIS
|
||||
from .config import SimConfig
|
||||
|
||||
|
||||
def _orphans_sorted(tree: BlockTree, canonical_ids: list[int]) -> tuple[np.ndarray, np.ndarray]:
|
||||
"""Return orphan block ids sorted by (slot, id) and their slots."""
|
||||
canonical = np.zeros(tree.n_blocks, dtype=bool)
|
||||
canonical[canonical_ids] = True
|
||||
all_real = np.arange(1, tree.n_blocks)
|
||||
orphan_ids = all_real[~canonical[1:]]
|
||||
orphan_slots = tree.slot[orphan_ids]
|
||||
order = np.lexsort((orphan_ids, orphan_slots)) # by slot, then id
|
||||
return orphan_ids[order], orphan_slots[order]
|
||||
|
||||
|
||||
def annotate_uncles(
|
||||
tree: BlockTree, canonical_ids: list[int], config: SimConfig, rng: np.random.Generator
|
||||
) -> None:
|
||||
"""Fill ``tree.uncles[B]`` for every canonical block ``B`` per the selection rule."""
|
||||
u_max = config.max_uncles
|
||||
if u_max <= 0:
|
||||
return
|
||||
w = config.uncle_window
|
||||
orphan_ids, orphan_slots = _orphans_sorted(tree, canonical_ids)
|
||||
if orphan_ids.size == 0:
|
||||
return
|
||||
|
||||
referenced: set[int] = set()
|
||||
# oldest canonical block first
|
||||
for b in reversed(canonical_ids):
|
||||
sb = int(tree.slot[b])
|
||||
lo = int(np.searchsorted(orphan_slots, sb - w, side="left")) # slot_U >= sb - W
|
||||
hi = int(np.searchsorted(orphan_slots, sb, side="left")) # slot_U < sb
|
||||
if hi <= lo:
|
||||
continue
|
||||
window_ids = orphan_ids[lo:hi] # already oldest-first
|
||||
selected = _select(window_ids, referenced, config, rng)
|
||||
if selected:
|
||||
tree.uncles[b] = tuple(selected)
|
||||
referenced.update(selected)
|
||||
|
||||
|
||||
def _select(
|
||||
window_ids: np.ndarray, referenced: set[int], config: SimConfig, rng: np.random.Generator
|
||||
window_ids: np.ndarray,
|
||||
referenced: set[int],
|
||||
config: SimConfig,
|
||||
rng: np.random.Generator,
|
||||
slot: np.ndarray | None = None,
|
||||
one_per_slot: bool = False,
|
||||
) -> list[int]:
|
||||
"""Pick up to ``max_uncles`` candidates. ``one_per_slot`` adds the countable model's
|
||||
per-slot dedup (a second same-slot candidate adds no occupied slot, so it is skipped)."""
|
||||
u_max = config.max_uncles
|
||||
out: list[int] = []
|
||||
slots_taken: set[int] = set()
|
||||
if config.uncle_strategy == "oldest":
|
||||
for bid in window_ids.tolist():
|
||||
if bid in referenced:
|
||||
continue
|
||||
if one_per_slot:
|
||||
s = int(slot[bid])
|
||||
if s in slots_taken:
|
||||
continue
|
||||
slots_taken.add(s)
|
||||
out.append(bid)
|
||||
if len(out) >= u_max:
|
||||
break
|
||||
@ -73,7 +61,11 @@ def _select(
|
||||
for bid in window_ids.tolist():
|
||||
if bid in referenced:
|
||||
continue
|
||||
if one_per_slot and int(slot[bid]) in slots_taken:
|
||||
continue
|
||||
if rng.random() < p:
|
||||
if one_per_slot:
|
||||
slots_taken.add(int(slot[bid]))
|
||||
out.append(bid)
|
||||
if len(out) >= u_max:
|
||||
break
|
||||
@ -96,21 +88,24 @@ def select_uncles_at_production(
|
||||
) -> tuple[int, ...]:
|
||||
"""Uncles a block gets when produced by node ``v`` (arrival row ``arrival_v``) at slot ``t``.
|
||||
|
||||
Candidates are blocks in ``v``'s view (``arrival_v[b] <= t``) with slot in ``[t-W, t)``
|
||||
that are NOT on the chain ``v`` extends (ancestors of ``parent_id``) and not already
|
||||
referenced by that chain. Selected once and baked globally (same for everyone who adopts
|
||||
the block), so density counting stays view-independent.
|
||||
Candidates are blocks in ``v``'s view (``arrival_v[b] <= t``) with slot in
|
||||
``[t-w_u, t)`` that are NOT on the chain ``v`` extends (ancestors of ``parent_id``) and
|
||||
not already referenced by that chain; the countable model (default) additionally
|
||||
requires the candidate's **parent to lie on that chain** (first block of its fork),
|
||||
excludes candidates whose slot is already occupied on the chain, and picks at most one
|
||||
per slot. Selected once and baked globally (same for everyone who adopts the block), so
|
||||
density counting stays view-independent.
|
||||
|
||||
``arrival_v`` is indexed by *block id minus ``arr_base``* — ``arr_base=0`` for the full arrival
|
||||
matrix row ``A[v]``, or the sliding-window buffer's base offset when pruning (every uncle-window
|
||||
block ``[t-W, t)`` is inside the kept span, so the buffer row covers all candidates).
|
||||
block ``[t-w_u, t)`` is inside the kept span, so the buffer row covers all candidates).
|
||||
"""
|
||||
u_max = config.max_uncles
|
||||
if u_max <= 0:
|
||||
return ()
|
||||
w = config.uncle_window
|
||||
w = config.effective_uncle_window
|
||||
slot_view = slot[:nb]
|
||||
lo = int(np.searchsorted(slot_view, t - w, side="left")) # slot >= t-W
|
||||
lo = int(np.searchsorted(slot_view, t - w, side="left")) # slot >= t-w_u
|
||||
hi = int(np.searchsorted(slot_view, t, side="left")) # slot < t
|
||||
if hi <= lo:
|
||||
return ()
|
||||
@ -118,18 +113,61 @@ def select_uncles_at_production(
|
||||
arrived = np.nonzero(arrival_v[lo - arr_base:hi - arr_base] <= t)[0] + lo
|
||||
if arrived.size == 0:
|
||||
return ()
|
||||
# v's own chain within the window + the uncles it already references (for dedup)
|
||||
on_chain: set[int] = set()
|
||||
referenced: set[int] = set()
|
||||
|
||||
if config.uncle_model == "old":
|
||||
# --- old model (pre countable redesign; kept verbatim for --old) ----------------
|
||||
# v's own chain within the window + the uncles it already references (for dedup)
|
||||
on_chain: set[int] = set()
|
||||
referenced: set[int] = set()
|
||||
a = int(parent_id)
|
||||
while a > GENESIS and int(slot[a]) >= t - w:
|
||||
on_chain.add(a)
|
||||
referenced.update(uncles[a])
|
||||
a = int(parent[a])
|
||||
cands = np.array(
|
||||
[b for b in arrived.tolist()
|
||||
if b > GENESIS and b not in on_chain and b not in referenced],
|
||||
dtype=np.int64,
|
||||
)
|
||||
if cands.size == 0:
|
||||
return ()
|
||||
return tuple(_select(cands, set(), config, rng)) # cands already oldest-first (slot,id)
|
||||
|
||||
# --- countable model (spec counting rules; the default) -----------------------------
|
||||
# Window walk over v's chain: chain blocks, their referenced uncles, and the slots both
|
||||
# occupy (the spec's occupied-slot exclusion in Uncle Selection).
|
||||
on_chain = set()
|
||||
referenced = set()
|
||||
occupied: set[int] = set()
|
||||
a = int(parent_id)
|
||||
while a > GENESIS and int(slot[a]) >= t - w:
|
||||
on_chain.add(a)
|
||||
referenced.update(uncles[a])
|
||||
occupied.add(int(slot[a]))
|
||||
for u in uncles[a]:
|
||||
referenced.add(u)
|
||||
su = int(slot[u])
|
||||
if su >= t - w:
|
||||
occupied.add(su)
|
||||
a = int(parent[a])
|
||||
pre = [b for b in arrived.tolist()
|
||||
if b > GENESIS and b not in on_chain and b not in referenced
|
||||
and int(slot[b]) not in occupied]
|
||||
if not pre:
|
||||
return ()
|
||||
# Parent-on-chain (only the first block of a fork is countable): the window walk covers
|
||||
# parents inside the window; extend chain membership exactly far enough below it to
|
||||
# decide the oldest candidate parent (cheap — parents are typically recent).
|
||||
pmin = min(int(slot[int(parent[b])]) for b in pre)
|
||||
below: set[int] = set()
|
||||
while a > GENESIS and int(slot[a]) >= pmin:
|
||||
below.add(a)
|
||||
a = int(parent[a])
|
||||
chain_ids = on_chain | below
|
||||
cands = np.array(
|
||||
[b for b in arrived.tolist() if b > GENESIS and b not in on_chain and b not in referenced],
|
||||
[b for b in pre if int(parent[b]) == GENESIS or int(parent[b]) in chain_ids],
|
||||
dtype=np.int64,
|
||||
)
|
||||
if cands.size == 0:
|
||||
return ()
|
||||
return tuple(_select(cands, set(), config, rng)) # cands already oldest-first (slot,id)
|
||||
# cands already oldest-first (slot, id); one uncle per slot per the spec's selection.
|
||||
return tuple(_select(cands, set(), config, rng, slot=slot, one_per_slot=True))
|
||||
|
||||
@ -38,9 +38,18 @@ def check(name: str, ok: bool, detail: str) -> bool:
|
||||
return ok
|
||||
|
||||
|
||||
def main() -> int:
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
import argparse
|
||||
|
||||
ap = argparse.ArgumentParser(description="Per-node TSI analytic checks")
|
||||
ap.add_argument("--old", action="store_true",
|
||||
help="run the old (pre countable redesign) uncle model")
|
||||
args = ap.parse_args(argv)
|
||||
uncle_model = "old" if args.old else "countable"
|
||||
|
||||
results = []
|
||||
common = dict(n_nodes=300, stake_dist="uniform", k=K, epochs=EPOCHS, genesis_d_factor=0.5)
|
||||
common = dict(n_nodes=300, stake_dist="uniform", k=K, epochs=EPOCHS, genesis_d_factor=0.5,
|
||||
uncle_model=uncle_model)
|
||||
|
||||
# 1. Full-mesh baseline: zero per-node divergence, full window agreement.
|
||||
fm = SimConfig(topology="full_mesh", latency=4, max_uncles=0, **common)
|
||||
|
||||
@ -85,13 +85,15 @@ def test_blend_dedup_does_not_multiply_non_blend_configs():
|
||||
|
||||
|
||||
def test_uncle_window_sweeps_and_collapses_for_u0():
|
||||
# uncle_window is a live axis for U>0, but U=0 references no uncles so it must collapse.
|
||||
# OLD model: uncle_window is a live axis for U>0, but U=0 references no uncles so it
|
||||
# must collapse. (The countable model ignores uncle_window entirely — see the twin
|
||||
# test below.)
|
||||
sweep = SweepConfig(
|
||||
n_nodes=[100], stake_dist=["uniform"], topology=["blend"], degree=[6],
|
||||
link_latency_mean=[0.5], link_latency_dist=["geo"], blend_hops=[3],
|
||||
blend_delay_max=[4.0], uncle_window=[10, 100], max_uncles=[0, 1],
|
||||
uncle_strategy=["oldest"], init_dest=["common"], replicates=1,
|
||||
base={"k": 8, "epochs": 3},
|
||||
base={"k": 8, "epochs": 3, "uncle_model": "old"},
|
||||
)
|
||||
configs = sweep.expand()
|
||||
u0 = [c for c in configs if c.max_uncles == 0]
|
||||
@ -100,6 +102,24 @@ def test_uncle_window_sweeps_and_collapses_for_u0():
|
||||
assert {c.uncle_window for c in u1} == {10, 100} # both W kept for U=1
|
||||
|
||||
|
||||
def test_window_absorption_sweeps_and_ignored_axis_collapses():
|
||||
# COUNTABLE model: window_absorption is the live window axis; uncle_window is ignored
|
||||
# and must collapse. And vice versa for the old model (guarded above).
|
||||
sweep = SweepConfig(
|
||||
n_nodes=[100], stake_dist=["uniform"], topology=["blend"], degree=[6],
|
||||
link_latency_mean=[0.5], link_latency_dist=["geo"], blend_hops=[3],
|
||||
blend_delay_max=[4.0], uncle_window=[10, 100], window_absorption=[2.0, 4.0],
|
||||
max_uncles=[0, 1], uncle_strategy=["oldest"], init_dest=["common"], replicates=1,
|
||||
base={"k": 8, "epochs": 3},
|
||||
)
|
||||
configs = sweep.expand()
|
||||
u0 = [c for c in configs if c.max_uncles == 0]
|
||||
u1 = [c for c in configs if c.max_uncles == 1]
|
||||
assert len(u0) == 1 # all window knobs collapse
|
||||
assert {c.window_absorption for c in u1} == {2.0, 4.0} # live axis kept for U=1
|
||||
assert {c.uncle_window for c in u1} == {10} # ignored axis collapsed
|
||||
|
||||
|
||||
def test_unknown_sweep_key_rejected():
|
||||
with pytest.raises(ValueError, match="unknown sweep keys"):
|
||||
SweepConfig.from_dict({"latencies": [0, 1], "base": {}}) # typo: latencies vs latency
|
||||
@ -117,7 +137,14 @@ def test_key_covers_every_field():
|
||||
# optimisation (no RNG, identical results) so it is intentionally not in key().
|
||||
# early_stop is truncation-only (per-epoch RNG streams are pre-spawned, so the epochs
|
||||
# that DO run are bit-identical to a full run's prefix) — intentionally excluded from key().
|
||||
ignored = {"root_seed", "windowed_fork_choice", "prune_arrival", "early_stop"}
|
||||
# uncle_window is read ONLY by the old model; under the (default) countable model it is
|
||||
# an ignored field, deliberately left in the base tuple at its old position so that an
|
||||
# --old run's key stays byte-identical to historical keys.
|
||||
# paired_streams must NOT be in key(): it selects WHICH key the RNG root is derived from
|
||||
# (see seed_key), so putting it in key() would perturb every historical seed and break
|
||||
# --old bit-reproduction. Its own behaviour is pinned in test_rng.py.
|
||||
ignored = {"root_seed", "windowed_fork_choice", "prune_arrival", "early_stop",
|
||||
"uncle_window", "paired_streams"}
|
||||
names = {f.name for f in dataclasses.fields(SimConfig)} - ignored
|
||||
a = SimConfig()
|
||||
for name in names:
|
||||
@ -125,6 +152,46 @@ def test_key_covers_every_field():
|
||||
alt = _perturb(cur)
|
||||
b = dataclasses.replace(a, **{name: alt})
|
||||
assert a.key() != b.key(), f"key() does not distinguish field {name!r}"
|
||||
# ... and uncle_window IS distinguished under the old model, where it is live.
|
||||
old = SimConfig(uncle_model="old")
|
||||
assert old.key() != dataclasses.replace(old, uncle_window=old.uncle_window + 1).key()
|
||||
# paired_streams leaves key() untouched but DOES change the seed derived from it.
|
||||
a_paired = dataclasses.replace(a, paired_streams=True)
|
||||
assert a.key() == a_paired.key()
|
||||
assert a.seed_key() != a_paired.seed_key()
|
||||
|
||||
|
||||
def test_old_model_key_is_historical():
|
||||
# --old must bit-reproduce historical runs: its key is exactly the pre-uncle_model
|
||||
# tuple (no uncle_model / window_absorption entries), and the countable key extends it.
|
||||
old = SimConfig(uncle_model="old")
|
||||
new = SimConfig()
|
||||
assert new.key()[: len(old.key())] == old.key()
|
||||
assert new.key()[len(old.key()):] == ("countable", new.window_absorption)
|
||||
# window_absorption is ignored (and absent from key) under the old model...
|
||||
assert dataclasses.replace(old, window_absorption=2.0).key() == old.key()
|
||||
# ...and live under the countable model.
|
||||
assert dataclasses.replace(new, window_absorption=2.0).key() != new.key()
|
||||
|
||||
|
||||
def test_effective_uncle_window():
|
||||
# countable: derived w_u = round(W / f); old: uncle_window taken directly.
|
||||
assert SimConfig(k=2160).effective_uncle_window == 300 # W=10, f=1/30
|
||||
assert SimConfig(k=2160, window_absorption=5.0).effective_uncle_window == 150
|
||||
assert SimConfig(uncle_model="old", uncle_window=42).effective_uncle_window == 42
|
||||
|
||||
|
||||
def test_window_absorption_bound():
|
||||
import warnings
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
SimConfig(window_absorption=0.5) # W < 1 rejected
|
||||
with pytest.warns(RuntimeWarning, match="exceeds the spec bound"):
|
||||
SimConfig(k=8, window_absorption=10.0) # W > 0.6*k warns
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("error")
|
||||
SimConfig(k=2160, window_absorption=10.0) # full scale: silent
|
||||
SimConfig(k=8, uncle_model="old", uncle_window=300) # old model: no bound
|
||||
|
||||
|
||||
def _perturb(v):
|
||||
@ -140,6 +207,10 @@ def _perturb(v):
|
||||
"fixed": "exp", "common": "heterogeneous", "suppress": "withhold",
|
||||
"exp": "poisson", # jitter_dist
|
||||
"sine": "ramp", # churn_mode
|
||||
"countable": "old", # uncle_model
|
||||
# adversary_selection. Keyed by VALUE, so this only fires on fields that are currently
|
||||
# "random" — uncle_strategy defaults to "oldest" and keeps its own flip above.
|
||||
"random": "whale",
|
||||
}
|
||||
if isinstance(v, str) and v in flips:
|
||||
return flips[v]
|
||||
|
||||
@ -0,0 +1,109 @@
|
||||
"""Countable-model counting: measurement kernels vs the tsi.py reference oracle.
|
||||
|
||||
Hand-built tree exercising every counting rule on baked references:
|
||||
canonical 1(s0) -> 2(s2) -> 3(s5) -> 4(s10, tip); orphans 5(s1, parent 1, first fork),
|
||||
6(s6, parent 5, DEEP), 7(s7, parent 1, first fork), 8(s7, parent 2, first fork, same slot
|
||||
as 7). References: block2 -> (5,), block3 -> (1,) [a canonical block], block4 -> (6, 7, 8).
|
||||
|
||||
With w = 5 and T = 20 the countable verdicts are: 5 counted (d=1); 1 skipped (on chain);
|
||||
6 deep-rejected (parent is an orphan); 7 counted (d=3); 8 counted by id but its slot is
|
||||
already recovered by 7 (slot dedup). Old model counts every reference.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from tsi_sim.blocktree import BlockTree
|
||||
from tsi_sim.measure import measure
|
||||
from tsi_sim.tsi import countable_refs, density_m
|
||||
|
||||
W = 5
|
||||
T = 20
|
||||
|
||||
|
||||
def _tree() -> BlockTree:
|
||||
tree = BlockTree(
|
||||
slot=np.array([-1, 0, 2, 5, 10, 1, 6, 7, 7], np.int64),
|
||||
parent=np.array([-1, 0, 1, 2, 3, 1, 5, 1, 2], np.int64),
|
||||
height=np.array([0, 1, 2, 3, 4, 2, 3, 2, 3], np.int64),
|
||||
leader=np.array([-1, 0, 1, 2, 3, 4, 5, 6, 7], np.int64),
|
||||
uncles=[() for _ in range(9)],
|
||||
)
|
||||
tree.uncles[2] = (5,)
|
||||
tree.uncles[3] = (1,)
|
||||
tree.uncles[4] = (6, 7, 8)
|
||||
return tree
|
||||
|
||||
|
||||
CANONICAL = [4, 3, 2, 1]
|
||||
ACTIVE = np.array([0, 1, 2, 5, 6, 7, 10], np.int64) # 7 distinct active slots in T
|
||||
|
||||
|
||||
def test_countable_refs_oracle():
|
||||
assert countable_refs(_tree(), CANONICAL, W) == {5, 7, 8}
|
||||
|
||||
|
||||
def test_density_m_countable_and_old():
|
||||
tree = _tree()
|
||||
# countable: honest slots {0,2,5,10} + recovered slots {1, 7} -> 6
|
||||
assert density_m(tree, CANONICAL, T, countable=True, w=W) == 6
|
||||
# old model: every reference counts -> recovered slots {1, 6, 7} -> 7
|
||||
assert density_m(tree, CANONICAL, T) == 7
|
||||
|
||||
|
||||
@pytest.mark.parametrize("use_numba", [False, True])
|
||||
def test_measure_countable_matches_oracle(use_numba):
|
||||
tree = _tree()
|
||||
A = np.zeros((2, tree.n_blocks)) # both nodes received everything
|
||||
ms = measure(tree, A, ACTIVE, T, cutoff=15, use_numba=use_numba,
|
||||
countable=True, w=W)
|
||||
np.testing.assert_array_equal(ms.m, [6, 6]) # = density_m countable
|
||||
np.testing.assert_allclose(ms.q, 4 / 7) # canonical slots / active
|
||||
np.testing.assert_allclose(ms.q_eff, 6 / 7) # + recovered slots
|
||||
np.testing.assert_array_equal(ms.ref_total, [5, 5]) # ids 5,1,6,7,8 examined
|
||||
np.testing.assert_array_equal(ms.ref_deep, [1, 1]) # id 6 rejected as deep
|
||||
|
||||
|
||||
@pytest.mark.parametrize("use_numba", [False, True])
|
||||
def test_measure_old_counts_all_refs(use_numba):
|
||||
tree = _tree()
|
||||
A = np.zeros((2, tree.n_blocks))
|
||||
ms = measure(tree, A, ACTIVE, T, cutoff=15, use_numba=use_numba)
|
||||
np.testing.assert_array_equal(ms.m, [7, 7]) # = density_m old
|
||||
np.testing.assert_allclose(ms.q_eff, 7 / 7)
|
||||
np.testing.assert_array_equal(ms.ref_deep, [0, 0]) # no rule to reject on
|
||||
|
||||
|
||||
def test_window_recheck_rejects_stale_reference():
|
||||
# A baked reference outside the counting window is uncounted under countable
|
||||
# (the old model still counts it): shrink w below block4 -> uncle 7 distance (d=3).
|
||||
tree = _tree()
|
||||
assert countable_refs(tree, CANONICAL, 2) == {5} # 7, 8 now out of window (d=3)
|
||||
assert density_m(tree, CANONICAL, T, countable=True, w=2) == 5
|
||||
|
||||
|
||||
def test_deep_ref_share_is_zero_end_to_end():
|
||||
"""The counting-side parent-on-chain re-check must never fire on a real countable run.
|
||||
|
||||
Countable SELECTION already refuses to reference a non-first-fork block, and for a
|
||||
block ``b`` on the counting chain the producer's chain below ``b`` IS the counting
|
||||
chain below ``b`` (they are the same ancestor path). So ``ref_deep`` is a defensive
|
||||
invariant, not a measured rate: any non-zero value means selection and counting have
|
||||
drifted apart. The hand-built trees above are the only way to make it fire — they bake
|
||||
references selection would never emit.
|
||||
"""
|
||||
import pandas as pd
|
||||
|
||||
from tsi_sim.config import SimConfig
|
||||
from tsi_sim.engine import run_trajectory
|
||||
|
||||
# Small but fork-rich: Blend delay spreads proposals over many slots.
|
||||
cfg = SimConfig(n_nodes=60, topology="blend", blend_hops=2, blend_delay_max=8.0,
|
||||
degree=4, max_uncles=2, k=32, epochs=3, f=0.1,
|
||||
stake_dist="pareto", init_dest="common")
|
||||
df = pd.DataFrame(run_trajectory(cfg))
|
||||
assert df.n_blocks.sum() > 0 # the run actually produced blocks
|
||||
assert (df.deep_ref_share == 0.0).all(), (
|
||||
f"counting rejected references selection emitted: {df.deep_ref_share.tolist()}")
|
||||
@ -0,0 +1,149 @@
|
||||
"""Countable (first-fork) uncle recovery under a selfish adversary (§6.6).
|
||||
|
||||
The countable model can reference only the first block of a fork, so a discarded *chain* of
|
||||
honest blocks yields one countable uncle however long it is. These tests pin the two ends of
|
||||
that: SM1 never buries a second block (so the restriction costs nothing), while the optimal
|
||||
policy waits and does (so it costs a factor of ~2 in recoverable orphans).
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from tsi_sim.selfish import race_from_alpha, selfish_threshold
|
||||
from tsi_sim.selfish_mdp import optimal_policy_stats
|
||||
|
||||
FAST = dict(cap=16, iters=1500)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("gamma", [0.0, 0.5, 1.0])
|
||||
@pytest.mark.parametrize("alpha", [0.2, 1 / 3, 0.4, 0.45])
|
||||
def test_sm1_orphans_are_all_countable(alpha, gamma):
|
||||
# SM1 acts as soon as the honest branch reaches length 1, so every orphan it makes is the
|
||||
# first block of its fork: the first-fork restriction costs SM1 exactly nothing.
|
||||
r = race_from_alpha(alpha, 200_000, gamma, np.random.default_rng(3))
|
||||
assert r.orphan_hon_runs == r.orphan_hon
|
||||
assert r.countable_recovery == 1.0
|
||||
|
||||
|
||||
@pytest.mark.parametrize("gamma", [0.0, 0.5])
|
||||
def test_optimal_policy_block_conservation(gamma):
|
||||
# Every block-finding event yields exactly one block, which ends up canonical or orphaned.
|
||||
# Per-event rates must therefore sum to 1 — the same invariant test_selfish asserts for SM1.
|
||||
s = optimal_policy_stats(0.4, gamma, **FAST)
|
||||
total = s.density_fraction + s.orphan_hon_blocks + s.orphan_adv_blocks
|
||||
assert abs(total - 1.0) < 1e-9
|
||||
|
||||
|
||||
@pytest.mark.parametrize("gamma", [0.0, 0.5])
|
||||
def test_optimal_policy_buries_orphans(gamma):
|
||||
# Above the profitability threshold the optimum waits before overriding, so it discards
|
||||
# multi-block honest chains that the first-fork rule cannot recover.
|
||||
s = optimal_policy_stats(0.4, gamma, **FAST)
|
||||
assert s.deviates
|
||||
assert s.orphan_hon_runs < s.orphan_hon_blocks
|
||||
assert s.countable_recovery < 0.7 # measured ~0.44 (gamma=0) / ~0.55 (gamma=0.5)
|
||||
|
||||
|
||||
def test_below_threshold_does_not_deviate():
|
||||
# Below the threshold the optimum is honest mining; the MDP is indifferent across policies
|
||||
# there, so the orphan structure of an arbitrary greedy tie-break must not be reported.
|
||||
alpha = 0.25
|
||||
assert alpha < selfish_threshold(0.0)
|
||||
s = optimal_policy_stats(alpha, 0.0, **FAST)
|
||||
assert not s.deviates
|
||||
assert s.orphan_hon_blocks == 0.0
|
||||
assert s.density_fraction == 1.0
|
||||
|
||||
|
||||
def test_countable_dhat_is_below_unrestricted():
|
||||
s = optimal_policy_stats(0.4, 0.0, **FAST)
|
||||
# With no references the two models agree; with them, countable recovers strictly less.
|
||||
assert s.dhat_ratio(p_ref=0.0, countable=True) == s.dhat_ratio(p_ref=0.0, countable=False)
|
||||
assert s.dhat_ratio(p_ref=1.0, countable=True) < s.dhat_ratio(p_ref=1.0, countable=False)
|
||||
# and both are bounded by the no-attack value
|
||||
assert s.dhat_ratio(p_ref=1.0, countable=False) <= 1.0
|
||||
# monotone in the reference rate
|
||||
assert (s.dhat_ratio(p_ref=0.0, countable=True)
|
||||
< s.dhat_ratio(p_ref=0.5, countable=True)
|
||||
< s.dhat_ratio(p_ref=1.0, countable=True))
|
||||
|
||||
|
||||
def test_attacker_self_uncle_is_capped_too():
|
||||
# The attacker's abandoned secret chain is also one chain, so it can self-uncle only its
|
||||
# first block — the §6.7(a) farming channel is narrower than the block count suggests.
|
||||
s = optimal_policy_stats(0.4, 0.0, **FAST)
|
||||
assert s.orphan_adv_runs < s.orphan_adv_blocks
|
||||
assert 0.5 < s.countable_recovery_adv < 1.0
|
||||
|
||||
|
||||
def test_unconstrained_deflation_optimum_is_abstention():
|
||||
# Minimising the estimate with no constraint degenerates: publish nothing, and D-hat lands on
|
||||
# exactly 1 - alpha with zero revenue. §6.4 already covers that case and shows it is CORRECT
|
||||
# measurement rather than mis-measurement, which is why item 16 needs the paid frontier.
|
||||
from tsi_sim.selfish_mdp import deflation_optimal_stats
|
||||
|
||||
for alpha in (0.2, 0.4):
|
||||
s = deflation_optimal_stats(alpha, 0.0, cap=16)
|
||||
assert abs(s.dhat_ratio(1.0, True) - (1.0 - alpha)) < 1e-6
|
||||
assert s.revenue < 1e-9
|
||||
assert s.orphan_hon_blocks < 1e-9 # it orphans no honest work at all
|
||||
|
||||
|
||||
def test_deflation_solver_gain_matches_its_stationary_accounting():
|
||||
# deflation_optimal_stats raises if the MDP's average gain disagrees with the estimate
|
||||
# recomputed from the stationary distribution -- an independent check that the solver and the
|
||||
# orphan accounting describe the same policy. Exercise it across a spread of inputs.
|
||||
from tsi_sim.selfish_mdp import deflation_optimal_stats
|
||||
|
||||
for alpha in (0.25, 0.35, 0.45):
|
||||
for p_ref in (0.0, 0.85, 1.0):
|
||||
deflation_optimal_stats(alpha, 0.0, p_ref=p_ref, cap=16) # no AssertionError
|
||||
|
||||
|
||||
def test_frontier_endpoints_bracket_the_two_pure_objectives():
|
||||
from tsi_sim.selfish_mdp import deflation_frontier, deflation_optimal_stats
|
||||
|
||||
alpha = 0.4
|
||||
zero = deflation_frontier(alpha, 0.0, 0.0, cap=16)
|
||||
pure = deflation_optimal_stats(alpha, 0.0, cap=16)
|
||||
assert abs(zero["dhat_countable"] - pure.dhat_ratio(1.0, True)) < 1e-6 # lam=0 is that optimum
|
||||
# Selfish mining takes a bigger share of a SMALLER pie, so maximising raw adversary block
|
||||
# rate returns to honest mining -- the frontier is not monotone in revenue, by construction.
|
||||
far = deflation_frontier(alpha, 0.0, 50.0, cap=16)
|
||||
assert abs(far["revenue"] - alpha) < 1e-3
|
||||
assert abs(far["dhat_countable"] - 1.0) < 1e-3
|
||||
|
||||
|
||||
def test_a_paid_policy_deflates_further_than_the_revenue_optimum():
|
||||
# Item 16's answer: the revenue-optimal adversary is not the estimator's worst case. At
|
||||
# alpha = 0.4 a policy exists that pays at least as well as honest mining yet deflates
|
||||
# substantially further than the revenue optimum does.
|
||||
from tsi_sim.selfish_mdp import deflation_frontier, optimal_policy_stats
|
||||
|
||||
alpha, cap = 0.4, 32
|
||||
ro = optimal_policy_stats(alpha, 0.0, cap=cap)
|
||||
paid = [deflation_frontier(alpha, 0.0, lam, cap=cap) for lam in (0.4, 0.6, 0.8, 1.0)]
|
||||
paid = [p for p in paid if p["reward_per_stake"] >= 1.0 - 1e-9]
|
||||
assert paid, "expected at least one break-even-or-better frontier point"
|
||||
assert min(p["dhat_countable"] for p in paid) < ro.dhat_ratio(1.0, True) - 0.05
|
||||
|
||||
|
||||
def test_reorg_countable_recovery_from_depths():
|
||||
# A depth-d reorg discards one chain of d blocks -> 1 countable uncle: runs / blocks.
|
||||
from tsi_sim.reorg import countable_recovery_from_depths
|
||||
|
||||
assert countable_recovery_from_depths(np.array([], dtype=np.int64)) == 1.0
|
||||
assert countable_recovery_from_depths(np.array([1, 1, 1])) == 1.0 # SM1-like: all depth-1
|
||||
assert countable_recovery_from_depths(np.array([3, 1, 2])) == 0.5 # 3 runs / 6 blocks
|
||||
# and it is the depth-weighted harmonic sense of "share": deeper reorgs drag it down
|
||||
assert countable_recovery_from_depths(np.array([10])) == 0.1
|
||||
|
||||
|
||||
@pytest.mark.slow
|
||||
def test_cap_convergence():
|
||||
# The orphan shape converges more slowly in cap than the revenue does; check the drift is
|
||||
# small where the report quotes numbers.
|
||||
a = optimal_policy_stats(0.4, 0.0, cap=48)
|
||||
b = optimal_policy_stats(0.4, 0.0, cap=64)
|
||||
assert abs(a.countable_recovery - b.countable_recovery) < 2e-3
|
||||
assert abs(a.revenue - b.revenue) < 1e-3
|
||||
@ -23,14 +23,14 @@ def make_tree(slots, parents, heights):
|
||||
def test_no_forks():
|
||||
# a straight chain 1->2->3, no orphans
|
||||
tree = make_tree([-1, 0, 1, 2], [-1, 0, 1, 2], [0, 1, 2, 3])
|
||||
fr, mx, mn, pr = fork_stats(tree, None, T=10, cutoff=100)
|
||||
fr, mx, mn, pr, _ = fork_stats(tree, None, T=10, cutoff=100)
|
||||
assert fr == 0.0 and mx == 0 and mn == 0.0
|
||||
|
||||
|
||||
def test_single_orphan_depth_one():
|
||||
# canonical 1(s0),2(s1),4(s3); orphan 3(s2) hangs off block1 -> branch depth 1
|
||||
tree = make_tree([-1, 0, 1, 2, 3], [-1, 0, 1, 1, 2], [0, 1, 2, 2, 3])
|
||||
fr, mx, mn, pr = fork_stats(tree, None, T=10, cutoff=100)
|
||||
fr, mx, mn, pr, _ = fork_stats(tree, None, T=10, cutoff=100)
|
||||
assert mx == 1
|
||||
assert abs(fr - 1 / 4) < 1e-9 # 1 orphan of 4 in-window blocks
|
||||
|
||||
@ -39,7 +39,7 @@ def test_deep_orphan_branch():
|
||||
# canonical spine 1..3 (heights 1,2,3); a 2-deep orphan branch 4->5 off block1
|
||||
# blocks: 0 gen; 1(s0,h1),2(s1,h2),3(s2,h3) canonical; 4(s1,h2)->1, 5(s2,h3)->4 orphan
|
||||
tree = make_tree([-1, 0, 1, 2, 1, 2], [-1, 0, 1, 2, 1, 4], [0, 1, 2, 3, 2, 3])
|
||||
fr, mx, mn, pr = fork_stats(tree, None, T=10, cutoff=100)
|
||||
fr, mx, mn, pr, _ = fork_stats(tree, None, T=10, cutoff=100)
|
||||
assert mx == 2 # branch 4->5 is 2 deep
|
||||
assert abs(fr - 2 / 5) < 1e-9 # 2 orphans of 5
|
||||
|
||||
|
||||
@ -32,3 +32,121 @@ def test_replicates_differ():
|
||||
b = run_trajectory(SimConfig(n_nodes=300, topology="regular", k=8, epochs=6,
|
||||
link_latency_mean=2.0, replicate=1))
|
||||
assert a[-1]["mean_ratio"] != b[-1]["mean_ratio"]
|
||||
|
||||
|
||||
def test_paired_streams_shares_the_root_across_uncle_models():
|
||||
"""Common random numbers: with paired_streams the two arms draw the SAME root seed."""
|
||||
from tsi_sim.rng import seedseq_for
|
||||
|
||||
kw = dict(n_nodes=50, max_uncles=2, blend_delay_max=5.0, topology="blend",
|
||||
k=32, epochs=2, replicate=3, paired_streams=True)
|
||||
c = SimConfig(uncle_model="countable", **kw)
|
||||
o = SimConfig(uncle_model="old", **kw)
|
||||
assert c.seed_key() == o.seed_key() # the marker is dropped
|
||||
assert c.key() != o.key() # ...but identity still distinguishes them
|
||||
assert seedseq_for(c).entropy == seedseq_for(o).entropy
|
||||
|
||||
|
||||
def test_unpaired_is_the_default_and_separates_the_models():
|
||||
kw = dict(n_nodes=50, max_uncles=2, blend_delay_max=5.0, topology="blend",
|
||||
k=32, epochs=2, replicate=3)
|
||||
c, o = SimConfig(uncle_model="countable", **kw), SimConfig(uncle_model="old", **kw)
|
||||
assert c.paired_streams is False and o.paired_streams is False
|
||||
assert c.seed_key() == c.key() and o.seed_key() == o.key()
|
||||
assert seedseq_for(c).entropy != seedseq_for(o).entropy
|
||||
|
||||
|
||||
def test_paired_streams_does_not_perturb_unpaired_seeds():
|
||||
"""The flag must not enter key(): every historical seed stays byte-identical.
|
||||
|
||||
This is what protects --old bit-reproduction of the pre-redesign runs (report §9).
|
||||
"""
|
||||
from tsi_sim.rng import seedseq_for
|
||||
|
||||
for model in ("countable", "old"):
|
||||
base = SimConfig(uncle_model=model, n_nodes=50, max_uncles=2, k=32, epochs=2)
|
||||
flagged = SimConfig(uncle_model=model, n_nodes=50, max_uncles=2, k=32, epochs=2,
|
||||
paired_streams=False)
|
||||
assert base.key() == flagged.key()
|
||||
assert seedseq_for(base).entropy == seedseq_for(flagged).entropy
|
||||
# and the old model's key is still exactly the base tuple (no marker appended)
|
||||
o = SimConfig(uncle_model="old", n_nodes=50, k=32, epochs=2)
|
||||
assert o.key() == o._base_key()
|
||||
|
||||
|
||||
def test_paired_streams_gives_both_arms_the_same_stake_and_graph():
|
||||
"""Pairing must reach the actual shared inputs, not just the root seed."""
|
||||
import numpy as np
|
||||
|
||||
from tsi_sim import topology
|
||||
from tsi_sim.rng import seedseq_for
|
||||
from tsi_sim.stake import make_stake
|
||||
|
||||
kw = dict(n_nodes=60, degree=4, topology="blend", max_uncles=2, k=32, epochs=2,
|
||||
blend_delay_max=5.0, replicate=1)
|
||||
c = SimConfig(uncle_model="countable", paired_streams=True, **kw)
|
||||
o = SimConfig(uncle_model="old", paired_streams=True, **kw)
|
||||
kids = {n: seedseq_for(cfg).spawn(cfg.epochs + 3)
|
||||
for n, cfg in (("c", c), ("o", o))}
|
||||
s_c = make_stake(c, np.random.default_rng(kids["c"][0]))
|
||||
s_o = make_stake(o, np.random.default_rng(kids["o"][0]))
|
||||
np.testing.assert_array_equal(s_c, s_o) # same stake draw
|
||||
g_c = topology.build_path_latency(c, np.random.default_rng(kids["c"][1]))
|
||||
g_o = topology.build_path_latency(o, np.random.default_rng(kids["o"][1]))
|
||||
np.testing.assert_array_equal(g_c, g_o) # same peering graph
|
||||
|
||||
|
||||
def test_paired_streams_is_recorded_in_the_output_row():
|
||||
"""A paired run must be identifiable from its parquet alone.
|
||||
|
||||
scripts/plot_fine_delay.py picks the paired test only when both arms report
|
||||
paired_streams; if the flag were missing from the recorded config it would silently fall
|
||||
back to the unpaired test and quietly discard the whole point of the paired sweep.
|
||||
"""
|
||||
from tsi_sim.metrics import _CONFIG_FIELDS
|
||||
|
||||
assert "paired_streams" in _CONFIG_FIELDS
|
||||
|
||||
|
||||
def test_whale_selection_does_not_perturb_random_coalition_seeds():
|
||||
"""A default ("random") coalition's key must stay byte-identical to every historical run's.
|
||||
|
||||
Same protection as test_paired_streams_does_not_perturb_unpaired_seeds gives the uncle model:
|
||||
the marker is appended only when non-default, so adding the knob rewrites no existing seed.
|
||||
"""
|
||||
from tsi_sim.rng import seedseq_for
|
||||
|
||||
for model in ("countable", "old"):
|
||||
base = SimConfig(uncle_model=model, n_nodes=50, k=32, epochs=2, adversary_frac=0.3)
|
||||
explicit = SimConfig(uncle_model=model, n_nodes=50, k=32, epochs=2, adversary_frac=0.3,
|
||||
adversary_selection="random")
|
||||
assert base.key() == explicit.key()
|
||||
assert seedseq_for(base).entropy == seedseq_for(explicit).entropy
|
||||
o = SimConfig(uncle_model="old", n_nodes=50, k=32, epochs=2, adversary_frac=0.3)
|
||||
assert o.key() == o._base_key() # still exactly the historical tuple
|
||||
whale = SimConfig(uncle_model="old", n_nodes=50, k=32, epochs=2, adversary_frac=0.3,
|
||||
adversary_selection="whale")
|
||||
assert whale.key() != o.key() # ...but the whale arm is its own stream
|
||||
|
||||
|
||||
def test_whale_coalition_takes_the_largest_holders_at_the_same_stake():
|
||||
from tsi_sim.engine import _adversary_mask
|
||||
from tsi_sim.stake import make_stake
|
||||
|
||||
cfg_r = SimConfig(n_nodes=400, stake_dist="pareto", k=32, epochs=2, adversary_frac=0.3)
|
||||
cfg_w = SimConfig(n_nodes=400, stake_dist="pareto", k=32, epochs=2, adversary_frac=0.3,
|
||||
adversary_selection="whale")
|
||||
stake = make_stake(cfg_r, rng_for(cfg_r))
|
||||
m_r, m_w = _adversary_mask(cfg_r, stake), _adversary_mask(cfg_w, stake)
|
||||
total = stake.sum()
|
||||
# The point of the knob is concentration at MATCHED stake, so the realised shares must agree
|
||||
# closely — a whale arm holding visibly more stake would confound the two.
|
||||
assert abs(stake[m_r].sum() / total - 0.3) < 0.01
|
||||
assert abs(stake[m_w].sum() / total - 0.3) < 0.01
|
||||
# ...and the whales get there with far fewer nodes: that is the variable under test.
|
||||
assert m_w.sum() * 5 < m_r.sum()
|
||||
# The coalition is drawn from the top of the distribution and dominated by it: the biggest
|
||||
# holders are all in, and the small top-up nodes contribute almost none of its stake.
|
||||
top3 = np.argsort(-stake)[:3]
|
||||
assert m_w[top3].all()
|
||||
assert stake[top3].sum() / stake[m_w].sum() > 0.9
|
||||
|
||||
@ -0,0 +1,174 @@
|
||||
"""The private-chain (SM1) adversary inside the per-node engine (§6.6, open item 5).
|
||||
|
||||
§6.8 recorded that "the per-node engine has no private-chain strategy", which is why the
|
||||
selfish results came from the global race model with uncle recovery as a free knob. These
|
||||
tests pin the engine version: that it leaves every honest result untouched, that its blocks
|
||||
are conserved, and that it actually orphans honest work rather than merely hiding its own.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from tsi_sim.blocktree import build_tree_pernode
|
||||
from tsi_sim.config import SimConfig
|
||||
from tsi_sim.engine import _adversary_mask, run_trajectory
|
||||
from tsi_sim.rng import rng_for
|
||||
from tsi_sim.stake import make_stake
|
||||
|
||||
BASE = dict(n_nodes=200, stake_dist="pareto", topology="blend", degree=6,
|
||||
link_latency_mean=0.5, link_latency_dist="geo", blend_hops=3, blend_delay_max=8.0,
|
||||
max_uncles=2, k=32, epochs=4, genesis_d_factor=0.5, early_stop=False)
|
||||
|
||||
|
||||
def _traj(**over):
|
||||
return run_trajectory(SimConfig(**{**BASE, **over}))
|
||||
|
||||
|
||||
def test_selfish_at_zero_stake_keeps_the_honest_fast_paths():
|
||||
"""With no coalition, `selfish` must not disturb the honest engine at all.
|
||||
|
||||
Note it is NOT bit-identical to `suppress` at frac = 0: adversary_strategy sits in the base
|
||||
RNG key, so switching it reseeds the run even though the field is inert without a coalition.
|
||||
That is pre-existing and harmless (both are valid honest runs), so the invariant worth
|
||||
pinning is the one that protects committed results — that the windowed fork choice and the
|
||||
arrival prune, which `selfish` disables when it IS active, stay enabled and stay exact here.
|
||||
"""
|
||||
exact = _traj(adversary_strategy="selfish", adversary_frac=0.0,
|
||||
windowed_fork_choice=False, prune_arrival=False)
|
||||
fast = _traj(adversary_strategy="selfish", adversary_frac=0.0)
|
||||
assert [r["mean_ratio"] for r in fast] == [r["mean_ratio"] for r in exact]
|
||||
assert max(r["range_ratio"] for r in fast) == 0.0 # honest run: nodes agree exactly
|
||||
|
||||
|
||||
def test_selfish_key_is_distinct_from_the_other_strategies():
|
||||
# adversary_strategy already sits in the base key, so no historical seed moves; this just
|
||||
# pins that the new value is not silently aliased onto an existing stream.
|
||||
keys = {s: SimConfig(**BASE, adversary_frac=0.3, adversary_strategy=s).key()
|
||||
for s in ("suppress", "withhold", "selfish")}
|
||||
assert len(set(keys.values())) == 3
|
||||
|
||||
|
||||
def test_selfish_is_deterministic():
|
||||
a = _traj(adversary_frac=0.3, adversary_strategy="selfish")
|
||||
b = _traj(adversary_frac=0.3, adversary_strategy="selfish")
|
||||
assert [r["mean_ratio"] for r in a] == [r["mean_ratio"] for r in b]
|
||||
|
||||
|
||||
def _tree(**over):
|
||||
cfg = SimConfig(**{**BASE, **over})
|
||||
stake = make_stake(cfg, rng_for(cfg))
|
||||
mask = _adversary_mask(cfg, stake)
|
||||
from tsi_sim import lottery, topology
|
||||
root = __import__("tsi_sim.rng", fromlist=["seedseq_for"]).seedseq_for(cfg)
|
||||
kids = root.spawn(cfg.epochs + 3)
|
||||
pl = topology.build_path_latency(cfg, np.random.default_rng(kids[1]))
|
||||
d_est = np.full(cfg.n_nodes, cfg.genesis_d_factor * stake.sum())
|
||||
p = lottery.win_probs(stake, d_est, cfg.f)
|
||||
ws, wn = lottery.sample_wins(p, cfg.epoch_len, np.random.default_rng(kids[3]))
|
||||
slots, groups = lottery.group_by_slot(ws, wn)
|
||||
tree, A = build_tree_pernode(slots, groups, pl, cfg, np.random.default_rng(kids[4]),
|
||||
adversary_mask=mask)
|
||||
return cfg, tree, A, mask
|
||||
|
||||
|
||||
def test_private_blocks_are_invisible_to_honest_nodes_while_hidden():
|
||||
cfg, tree, A, mask = _tree(adversary_frac=0.3, adversary_strategy="selfish")
|
||||
E = cfg.epoch_len
|
||||
honest = ~mask
|
||||
# Every block is either public (some honest node has it) or hidden from ALL honest nodes.
|
||||
reaches_honest = (A[honest] <= E).any(axis=0)
|
||||
hidden = ~reaches_honest
|
||||
hidden[0] = False
|
||||
# a hidden block is never a partial leak: no honest node holds it
|
||||
assert not (A[honest][:, hidden] <= E).any()
|
||||
# and every hidden block was produced by the coalition, never by an honest node
|
||||
assert mask[tree.leader[hidden]].all()
|
||||
|
||||
|
||||
def test_released_blocks_never_precede_their_parent():
|
||||
# The release path applies its own no-earlier-than-parent clamp; a violation would let a
|
||||
# node build on a child before its parent and corrupt the tree.
|
||||
cfg, tree, A, mask = _tree(adversary_frac=0.3, adversary_strategy="selfish")
|
||||
for b in range(1, tree.n_blocks):
|
||||
p = int(tree.parent[b])
|
||||
assert (A[:, b] >= A[:, p] - 1e-9).all(), f"block {b} precedes parent {p}"
|
||||
|
||||
|
||||
def _honest_orphans_in_window(cfg, tree, A, mask) -> int:
|
||||
"""In-window orphans produced by NON-coalition nodes — the displaced honest work."""
|
||||
E, T = cfg.epoch_len, cfg.period_T
|
||||
nb = tree.n_blocks
|
||||
ids = np.arange(nb)
|
||||
arrived = (A <= E).any(axis=0)
|
||||
arrived[0] = True
|
||||
h = np.where(arrived, tree.height, np.iinfo(np.int64).min)
|
||||
best = int(np.lexsort((-ids, -tree.slot, h))[-1])
|
||||
canonical = np.zeros(nb, dtype=bool)
|
||||
b = best
|
||||
while b > 0:
|
||||
canonical[b] = True
|
||||
b = int(tree.parent[b])
|
||||
in_win = (tree.slot >= 0) & (tree.slot < T)
|
||||
return int((in_win & ~canonical & ~mask[tree.leader]).sum())
|
||||
|
||||
|
||||
def test_selfish_displaces_honest_work_where_withholding_only_hides_its_own():
|
||||
# This is the distinction between the two levers, and the reason only one of them is
|
||||
# profitable: withholding discards the coalition's OWN blocks (a dead loss, and honest
|
||||
# blocks keep their places), while a private chain overrides HONEST blocks off the chain.
|
||||
# Compare the honest orphan count at matched stake -- not fork_rate, which counts the
|
||||
# withholder's own vanished blocks as orphans too and so runs high for the wrong reason.
|
||||
kw = dict(adversary_frac=0.4, max_uncles=0)
|
||||
cfg_s, tree_s, A_s, mask_s = _tree(adversary_strategy="selfish", **kw)
|
||||
cfg_w, tree_w, A_w, mask_w = _tree(adversary_strategy="withhold", **kw)
|
||||
assert (_honest_orphans_in_window(cfg_s, tree_s, A_s, mask_s)
|
||||
> _honest_orphans_in_window(cfg_w, tree_w, A_w, mask_w))
|
||||
|
||||
|
||||
def test_selfish_deflates_the_estimate_below_the_honest_baseline():
|
||||
tail = slice(2, None)
|
||||
honest = np.mean([r["mean_ratio"] for r in _traj(max_uncles=0)[tail]])
|
||||
selfish = np.mean([r["mean_ratio"] for r in
|
||||
_traj(adversary_frac=0.35, adversary_strategy="selfish",
|
||||
max_uncles=0)[tail]])
|
||||
assert selfish < honest
|
||||
|
||||
|
||||
def test_uncle_counting_repairs_part_of_the_selfish_deflation():
|
||||
# The §6.6 claim, now measurable in the engine rather than through the free knob eta:
|
||||
# uncles recover some of the loss, and (per §6.6/fig36) not all of it.
|
||||
tail = slice(2, None)
|
||||
d0 = np.mean([r["mean_ratio"] for r in
|
||||
_traj(adversary_frac=0.35, adversary_strategy="selfish", max_uncles=0)[tail]])
|
||||
d2 = np.mean([r["mean_ratio"] for r in
|
||||
_traj(adversary_frac=0.35, adversary_strategy="selfish", max_uncles=2)[tail]])
|
||||
assert d2 > d0
|
||||
|
||||
|
||||
def test_p_ref_honest_defaults_to_p_ref_without_a_coalition():
|
||||
from tsi_sim.fork import fork_stats
|
||||
cfg, tree, A, _ = _tree()
|
||||
*_, p_ref, p_ref_h = fork_stats(tree, A, cfg.period_T, cutoff=cfg.epoch_len)
|
||||
assert p_ref == p_ref_h
|
||||
|
||||
|
||||
@pytest.mark.parametrize("strategy", ["selfish", "withhold"])
|
||||
def test_hidden_blocks_are_excluded_from_the_canonical_chain(strategy):
|
||||
# A chain no honest node ever saw must not be crowned canonical, or it would collect
|
||||
# phantom rewards and phantom density.
|
||||
from tsi_sim.epoch import _canonical_producer_split
|
||||
cfg, tree, A, mask = _tree(adversary_frac=0.4, adversary_strategy=strategy)
|
||||
E, T = cfg.epoch_len, cfg.period_T
|
||||
adv, hon = _canonical_producer_split(tree, A, mask, T, E)
|
||||
reaches_honest = (A[~mask] <= E).any(axis=0)
|
||||
# walk the chosen canonical tip: every block on it is public
|
||||
ids = np.arange(tree.n_blocks)
|
||||
arrived = (A <= E).any(axis=0)
|
||||
arrived[0] = True
|
||||
h = np.where(arrived, tree.height, np.iinfo(np.int64).min)
|
||||
best = int(np.lexsort((-ids, -tree.slot, h))[-1])
|
||||
b = best
|
||||
while b > 0:
|
||||
assert reaches_honest[b], f"canonical block {b} was never public"
|
||||
b = int(tree.parent[b])
|
||||
assert adv + hon > 0
|
||||
@ -64,19 +64,31 @@ def test_distinct_slot_uncles_still_counted():
|
||||
|
||||
|
||||
def test_zero_delay_equilibrium_is_one_not_ceiling():
|
||||
"""The c(f) ceiling was the bug: corrected counting equilibrates at 1.0 with uncles."""
|
||||
"""The c(f) ceiling was the bug: corrected counting equilibrates at 1.0 with uncles.
|
||||
|
||||
Holds under the (default) countable model too: at zero delay the only orphans are
|
||||
same-slot co-winners, which countable selection never references (occupied slot) and
|
||||
which add nothing to the slot count anyway. 5 replicates / 0.02 tolerance because the
|
||||
countable model's key() draws a different RNG stream than the historical runs the old
|
||||
3-rep/0.015 margin was tuned on.
|
||||
"""
|
||||
base = dict(n_nodes=300, stake_dist="uniform", topology="full_mesh", latency=0,
|
||||
max_uncles=2, uncle_window=300, k=64, epochs=24, genesis_d_factor=1.0)
|
||||
tails = []
|
||||
for rep in range(3):
|
||||
for rep in range(5):
|
||||
df = pd.DataFrame(run_trajectory(SimConfig(**base, replicate=rep)))
|
||||
tails.append(df[df.epoch >= 8].mean_ratio.mean())
|
||||
assert abs(np.mean(tails) - 1.0) < 0.015
|
||||
assert abs(np.mean(tails) - 1.0) < 0.02
|
||||
|
||||
|
||||
def test_legacy_flag_reproduces_the_ceiling():
|
||||
# OLD model on purpose: the c(f) ceiling arises from referencing same-slot co-winners
|
||||
# and counting them per block id. The countable model never references a same-slot
|
||||
# co-winner (its slot is already occupied on the chain), so under it the legacy flag
|
||||
# has nothing to double-count and this historical bug cannot be reproduced.
|
||||
base = dict(n_nodes=300, stake_dist="uniform", topology="full_mesh", latency=0,
|
||||
max_uncles=2, uncle_window=300, k=64, epochs=24, genesis_d_factor=1.0)
|
||||
max_uncles=2, uncle_window=300, k=64, epochs=24, genesis_d_factor=1.0,
|
||||
uncle_model="old")
|
||||
tails = []
|
||||
for rep in range(3):
|
||||
df = pd.DataFrame(run_trajectory(
|
||||
|
||||
@ -38,3 +38,46 @@ def test_fixed_point_bias_about_one_percent():
|
||||
b = theory.fixed_point_bias(F)
|
||||
assert abs(b - (F / (33 / 1000))) < 1e-12
|
||||
assert 1.005 < b < 1.02
|
||||
|
||||
|
||||
def test_q_effective_interpolates_between_q_and_one():
|
||||
# r = 0 -> no recovery (chain-only q); r = 1 -> every wasted slot recovered (q_u = 1).
|
||||
for q in (0.3, 0.65, 0.9):
|
||||
assert abs(float(theory.q_effective(q, 0.0)) - q) < 1e-15
|
||||
assert abs(float(theory.q_effective(q, 1.0)) - 1.0) < 1e-15
|
||||
# monotone and strictly between for partial recovery
|
||||
half = float(theory.q_effective(q, 0.5))
|
||||
assert q < half < 1.0
|
||||
|
||||
|
||||
def test_q_effective_round_trips_the_measured_recovery_rate():
|
||||
# The identity the report quotes: given measured q and q_u, r = (q_u - q)/(1 - q)
|
||||
# reconstructs q_u exactly. This is how plot_countable_vs_old.py derives its overlay.
|
||||
for q, q_u in ((0.3129, 0.6147), (0.5380, 0.9874), (0.7485, 0.9992)):
|
||||
r = (q_u - q) / (1.0 - q)
|
||||
assert abs(float(theory.q_effective(q, r)) - q_u) < 1e-12
|
||||
|
||||
|
||||
def test_q_effective_recovers_unbiased_equilibrium():
|
||||
# Full recovery must lift the equilibrium to exactly 1 for any starting q.
|
||||
for q in (0.3, 0.65, 0.9):
|
||||
assert abs(float(theory.expected_ratio(F, theory.q_effective(q, 1.0))) - 1.0) < 1e-12
|
||||
|
||||
|
||||
def test_window_miss_prob_decays_as_exp_minus_w():
|
||||
# P(no canonical block in w_u = W/f slots) = (1-f)^(W/f) = exp(W * ln(1-f)/f).
|
||||
# The docstring's "~ e^-W" is the f -> 0 limit: ln(1-f)/f = -(1 + f/2 + ...) = -1.0170
|
||||
# at f = 1/30, so the true decay is slightly FASTER than e^-W, by a factor that grows
|
||||
# with W (16% low by W = 10). Assert the exact form, and bracket the heuristic.
|
||||
rate = np.log(1.0 - F) / F
|
||||
assert -1.02 < rate < -1.0
|
||||
for w_abs in (1.0, 3.0, 10.0):
|
||||
p = float(theory.window_miss_prob(F, w_abs))
|
||||
assert abs(p - (1.0 - F) ** (w_abs / F)) < 1e-15
|
||||
assert abs(p - np.exp(rate * w_abs)) < 1e-15 # exact closed form
|
||||
assert np.exp(-1.02 * w_abs) < p < np.exp(-w_abs) # brackets the e^-W heuristic
|
||||
# the spec default W = 10 makes the window a negligible loss channel
|
||||
assert float(theory.window_miss_prob(F, 10.0)) < 1e-4
|
||||
# strictly decreasing in W
|
||||
ps = [float(theory.window_miss_prob(F, w)) for w in (1, 2, 3, 5, 7, 10)]
|
||||
assert all(a > b for a, b in zip(ps[:-1], ps[1:], strict=True))
|
||||
|
||||
@ -2,7 +2,7 @@ import numpy as np
|
||||
|
||||
from tsi_sim.blocktree import BlockTree
|
||||
from tsi_sim.config import SimConfig
|
||||
from tsi_sim.uncles import annotate_uncles
|
||||
from tsi_sim.uncles import select_uncles_at_production
|
||||
|
||||
|
||||
def make_tree(slots, parents, heights, leaders):
|
||||
@ -16,6 +16,23 @@ def make_tree(slots, parents, heights, leaders):
|
||||
)
|
||||
|
||||
|
||||
def annotate_via_production(tree, canonical_ids, config, rng):
|
||||
"""Fill ``tree.uncles`` by replaying the PRODUCTION selection for each canonical block.
|
||||
|
||||
The simulator has exactly one selection implementation
|
||||
(``select_uncles_at_production``, called from blocktree.py per produced block); this
|
||||
replays it offline — oldest canonical block first, each block selecting as its producer
|
||||
would have at ``t = slot[b]`` extending ``parent[b]`` — so the tests below exercise the
|
||||
live code path rather than a parallel offline copy. Requires ``tree.slot`` ascending by
|
||||
block id (true for every tree these tests build, and for real runs).
|
||||
"""
|
||||
arrival = np.zeros(tree.n_blocks) # every block already in the producer's view
|
||||
for b in reversed(canonical_ids): # oldest first, so earlier refs are visible
|
||||
tree.uncles[b] = select_uncles_at_production(
|
||||
tree.slot, tree.parent, tree.uncles, arrival, nb=tree.n_blocks,
|
||||
parent_id=int(tree.parent[b]), t=int(tree.slot[b]), config=config, rng=rng)
|
||||
|
||||
|
||||
def _canonical_and_orphan_tree():
|
||||
# genesis(0); canonical chain 1(slot0)->3(slot3)->4(slot5); orphan 2(slot1)
|
||||
tree = make_tree(
|
||||
@ -31,7 +48,7 @@ def _canonical_and_orphan_tree():
|
||||
def test_oldest_selection_and_window():
|
||||
tree, canonical = _canonical_and_orphan_tree()
|
||||
cfg = SimConfig(max_uncles=1, uncle_window=300, uncle_strategy="oldest")
|
||||
annotate_uncles(tree, canonical, cfg, np.random.default_rng(0))
|
||||
annotate_via_production(tree, canonical, cfg, np.random.default_rng(0))
|
||||
# orphan 2 (slot1) is within window of block 3 (slot3) -> referenced there
|
||||
referenced = {u for b in canonical for u in tree.uncles[b]}
|
||||
assert referenced == {2}
|
||||
@ -40,19 +57,38 @@ def test_oldest_selection_and_window():
|
||||
def test_no_uncles_when_u_zero():
|
||||
tree, canonical = _canonical_and_orphan_tree()
|
||||
cfg = SimConfig(max_uncles=0)
|
||||
annotate_uncles(tree, canonical, cfg, np.random.default_rng(0))
|
||||
annotate_via_production(tree, canonical, cfg, np.random.default_rng(0))
|
||||
assert all(tree.uncles[b] == () for b in canonical)
|
||||
|
||||
|
||||
def test_window_excludes_out_of_range_orphan():
|
||||
# OLD model: uncle_window is read directly. (The countable model ignores uncle_window
|
||||
# and derives the window from window_absorption — see the countable twin below.)
|
||||
tree, canonical = _canonical_and_orphan_tree()
|
||||
cfg = SimConfig(max_uncles=1, uncle_window=1, uncle_strategy="oldest")
|
||||
annotate_uncles(tree, canonical, cfg, np.random.default_rng(0))
|
||||
cfg = SimConfig(max_uncles=1, uncle_window=1, uncle_strategy="oldest", uncle_model="old")
|
||||
annotate_via_production(tree, canonical, cfg, np.random.default_rng(0))
|
||||
# orphan 2 at slot1; nearest canonical after it is block3 at slot3 -> gap 2 > W=1
|
||||
referenced = {u for b in canonical for u in tree.uncles[b]}
|
||||
assert referenced == set()
|
||||
|
||||
|
||||
def test_countable_window_is_derived_from_absorption():
|
||||
# countable: w_u = round(W / f). With f=0.5 and W=1, w_u = 2 slots: the orphan at slot1
|
||||
# is out of range of the canonical block at slot5 (gap 4) and of slot3 (gap 2 <= 2 OK).
|
||||
tree, canonical = _canonical_and_orphan_tree()
|
||||
cfg = SimConfig(max_uncles=1, f=0.5, window_absorption=1.0)
|
||||
assert cfg.effective_uncle_window == 2
|
||||
annotate_via_production(tree, canonical, cfg, np.random.default_rng(0))
|
||||
referenced = {u for b in canonical for u in tree.uncles[b]}
|
||||
assert referenced == {2} # block3 (slot3) still reaches it
|
||||
# shrink f so the derived window rounds to 1 slot: gap 2 > 1 -> excluded
|
||||
tree2, canonical2 = _canonical_and_orphan_tree()
|
||||
cfg2 = SimConfig(max_uncles=1, f=0.9, window_absorption=1.0)
|
||||
assert cfg2.effective_uncle_window == 1
|
||||
annotate_via_production(tree2, canonical2, cfg2, np.random.default_rng(0))
|
||||
assert {u for b in canonical2 for u in tree2.uncles[b]} == set()
|
||||
|
||||
|
||||
def _wide_orphan_tree():
|
||||
# canonical 1(0)->6(6); orphans 2,3,4,5 at slots 1,2,3,4 (all within window of block6)
|
||||
tree = make_tree(
|
||||
@ -69,8 +105,8 @@ def test_random_strategy_deterministic_and_capped():
|
||||
tree_a, canon = _wide_orphan_tree()
|
||||
tree_b, _ = _wide_orphan_tree()
|
||||
cfg = SimConfig(max_uncles=2, uncle_window=300, uncle_strategy="random", uncle_random_p=0.5)
|
||||
annotate_uncles(tree_a, canon, cfg, np.random.default_rng(7))
|
||||
annotate_uncles(tree_b, canon, cfg, np.random.default_rng(7))
|
||||
annotate_via_production(tree_a, canon, cfg, np.random.default_rng(7))
|
||||
annotate_via_production(tree_b, canon, cfg, np.random.default_rng(7))
|
||||
assert tree_a.uncles == tree_b.uncles # same seed -> identical
|
||||
total = sum(len(tree_a.uncles[b]) for b in canon)
|
||||
assert total <= cfg.max_uncles # capped
|
||||
@ -80,18 +116,20 @@ def test_random_p_one_matches_oldest():
|
||||
import numpy as np
|
||||
tree_r, canon = _wide_orphan_tree()
|
||||
tree_o, _ = _wide_orphan_tree()
|
||||
annotate_uncles(tree_r, canon, SimConfig(max_uncles=2, uncle_strategy="random",
|
||||
uncle_random_p=1.0), np.random.default_rng(1))
|
||||
annotate_uncles(tree_o, canon, SimConfig(max_uncles=2, uncle_strategy="oldest"),
|
||||
np.random.default_rng(1))
|
||||
annotate_via_production(tree_r, canon, SimConfig(max_uncles=2, uncle_strategy="random",
|
||||
uncle_random_p=1.0),
|
||||
np.random.default_rng(1))
|
||||
annotate_via_production(tree_o, canon, SimConfig(max_uncles=2, uncle_strategy="oldest"),
|
||||
np.random.default_rng(1))
|
||||
assert tree_r.uncles == tree_o.uncles # p=1 deterministically takes oldest-first
|
||||
|
||||
|
||||
def test_random_p_zero_selects_nothing():
|
||||
import numpy as np
|
||||
tree, canon = _wide_orphan_tree()
|
||||
annotate_uncles(tree, canon, SimConfig(max_uncles=4, uncle_strategy="random",
|
||||
uncle_random_p=0.0), np.random.default_rng(1))
|
||||
annotate_via_production(tree, canon, SimConfig(max_uncles=4, uncle_strategy="random",
|
||||
uncle_random_p=0.0),
|
||||
np.random.default_rng(1))
|
||||
assert all(tree.uncles[b] == () for b in canon)
|
||||
|
||||
|
||||
@ -105,6 +143,125 @@ def test_dedup_across_ancestors():
|
||||
)
|
||||
canonical = [4, 3, 1]
|
||||
cfg = SimConfig(max_uncles=4, uncle_window=300, uncle_strategy="oldest")
|
||||
annotate_uncles(tree, canonical, cfg, np.random.default_rng(0))
|
||||
annotate_via_production(tree, canonical, cfg, np.random.default_rng(0))
|
||||
counts = sum(len(tree.uncles[b]) for b in canonical)
|
||||
assert counts == 1 # orphan 2 referenced exactly once despite two eligible blocks
|
||||
|
||||
|
||||
# --- countable model (spec counting rules) ---------------------------------------------
|
||||
|
||||
|
||||
def _deep_fork_tree():
|
||||
# canonical 1(slot0)->5(slot5); orphan branch 2(slot1,parent=1)->3(slot2,parent=2);
|
||||
# orphan 4(slot3, parent=1). Blocks 2,4 are FIRST fork blocks; 3 is deep.
|
||||
tree = make_tree(
|
||||
slots=[-1, 0, 1, 2, 3, 5],
|
||||
parents=[-1, 0, 1, 2, 1, 1],
|
||||
heights=[0, 1, 2, 3, 2, 2],
|
||||
leaders=[-1, 0, 1, 2, 3, 4],
|
||||
)
|
||||
return tree, [5, 1] # tip-first
|
||||
|
||||
|
||||
def test_countable_excludes_deep_fork_blocks():
|
||||
tree, canonical = _deep_fork_tree()
|
||||
cfg = SimConfig(max_uncles=4)
|
||||
annotate_via_production(tree, canonical, cfg, np.random.default_rng(0))
|
||||
referenced = {u for b in canonical for u in tree.uncles[b]}
|
||||
assert referenced == {2, 4} # deep block 3 (parent is an orphan) excluded
|
||||
|
||||
|
||||
def test_old_model_still_references_deep_fork_blocks():
|
||||
tree, canonical = _deep_fork_tree()
|
||||
cfg = SimConfig(max_uncles=4, uncle_model="old", uncle_window=300)
|
||||
annotate_via_production(tree, canonical, cfg, np.random.default_rng(0))
|
||||
referenced = {u for b in canonical for u in tree.uncles[b]}
|
||||
assert referenced == {2, 3, 4} # --old: fork depth ignored
|
||||
|
||||
|
||||
def test_countable_excludes_occupied_slots_and_dedups_per_slot():
|
||||
# canonical 1(slot0)->5(slot4); orphans: 2 at slot0 (canonical-occupied), 3/4 at slot2.
|
||||
tree = make_tree(
|
||||
slots=[-1, 0, 0, 2, 2, 4],
|
||||
parents=[-1, 0, 0, 1, 1, 1],
|
||||
heights=[0, 1, 1, 2, 2, 2],
|
||||
leaders=[-1, 0, 1, 2, 3, 4],
|
||||
)
|
||||
canonical = [5, 1]
|
||||
cfg = SimConfig(max_uncles=4)
|
||||
annotate_via_production(tree, canonical, cfg, np.random.default_rng(0))
|
||||
referenced = {u for b in canonical for u in tree.uncles[b]}
|
||||
# slot0 is canonical-occupied -> orphan 2 excluded; slot2 pair -> exactly one picked
|
||||
assert referenced == {3}
|
||||
|
||||
|
||||
def test_production_selection_countable_rules():
|
||||
from tsi_sim.uncles import select_uncles_at_production
|
||||
|
||||
# 0 genesis; 1 canonical slot0; 2 first-fork slot1 (parent 1); 3 deep slot2 (parent 2);
|
||||
# 4/5 same-slot first-forks at slot3 (parent 1).
|
||||
slot = np.array([-1, 0, 1, 2, 3, 3], np.int64)
|
||||
parent = np.array([-1, 0, 1, 2, 1, 1], np.int64)
|
||||
uncles: list = [() for _ in range(6)]
|
||||
arrival = np.zeros(6) # everything arrived immediately
|
||||
cfg = SimConfig(max_uncles=4)
|
||||
sel = select_uncles_at_production(
|
||||
slot, parent, uncles, arrival, nb=6, parent_id=1, t=5, config=cfg,
|
||||
rng=np.random.default_rng(0))
|
||||
assert sel == (2, 4) # deep 3 excluded; one per slot at slot3
|
||||
|
||||
old = SimConfig(max_uncles=4, uncle_model="old", uncle_window=300)
|
||||
sel_old = select_uncles_at_production(
|
||||
slot, parent, uncles, arrival, nb=6, parent_id=1, t=5, config=old,
|
||||
rng=np.random.default_rng(0))
|
||||
assert sel_old == (2, 3, 4, 5) # --old: depth and slot-dedup ignored
|
||||
|
||||
|
||||
def test_production_parent_below_window_resolves_chain_membership():
|
||||
"""Candidate parents can sit BELOW the reference window (uncles.py `pmin`/`below` walk).
|
||||
|
||||
The window walk only establishes chain membership for ancestors with slot >= t-w, so a
|
||||
candidate whose parent is older than that needs the extra walk down to `pmin`. With a
|
||||
narrow window this is the branch that decides first-fork eligibility.
|
||||
|
||||
chain 1(s0) -> 2(s1) -> 6(s8); off-chain 7(s2, parent 3) where 3(s2) is NOT on the chain.
|
||||
Candidates at t=9 with w=4 are the blocks in slots [5,9): 4(s5, parent 2 -> ON chain,
|
||||
countable) and 5(s6, parent 3 -> OFF chain, deep). Both parents sit below the window,
|
||||
so only the `below` walk can tell them apart.
|
||||
"""
|
||||
from tsi_sim.uncles import select_uncles_at_production
|
||||
|
||||
# id: 0 1 2 3 4 5 6
|
||||
slot = np.array([-1, 0, 1, 2, 5, 6, 8], np.int64)
|
||||
parent = np.array([-1, 0, 1, 1, 2, 3, 2], np.int64)
|
||||
uncles: list = [() for _ in range(7)]
|
||||
arrival = np.zeros(7)
|
||||
# w = round(1/0.25) = 4 slots; window at t=9 is [5, 9)
|
||||
cfg = SimConfig(max_uncles=4, f=0.25, window_absorption=1.0)
|
||||
assert cfg.effective_uncle_window == 4
|
||||
sel = select_uncles_at_production(
|
||||
slot, parent, uncles, arrival, nb=7, parent_id=6, t=9, config=cfg,
|
||||
rng=np.random.default_rng(0))
|
||||
assert sel == (4,) # 5 rejected: parent 3 is off-chain (deep) and below the window
|
||||
|
||||
|
||||
def test_production_excludes_slots_occupied_by_chain_and_prior_uncles():
|
||||
"""The occupied-slot exclusion is built from the producer's chain walk (uncles.py).
|
||||
|
||||
chain 1(s0) -> 2(s1) -> 5(s4); block 2 already references uncle 3(s2). Candidates at
|
||||
t=5 are 3 (already referenced), 4(s2, same slot as the referenced 3 -> slot occupied)
|
||||
and 6(s3, free slot). Only 6 survives.
|
||||
"""
|
||||
from tsi_sim.uncles import select_uncles_at_production
|
||||
|
||||
# id: 0 1 2 3 4 5 6
|
||||
slot = np.array([-1, 0, 1, 2, 2, 4, 3], np.int64)
|
||||
parent = np.array([-1, 0, 1, 1, 1, 2, 1], np.int64)
|
||||
uncles: list = [() for _ in range(7)]
|
||||
uncles[2] = (3,) # block 2 already referenced uncle 3
|
||||
arrival = np.zeros(7)
|
||||
cfg = SimConfig(max_uncles=4) # w = 300, whole tree in window
|
||||
sel = select_uncles_at_production(
|
||||
slot, parent, uncles, arrival, nb=7, parent_id=5, t=5, config=cfg,
|
||||
rng=np.random.default_rng(0))
|
||||
assert sel == (6,) # 3 already referenced; 4 shares slot 2 with it; 6 is free
|
||||
|
||||