The sec 8.4 capstone was still reporting pre-countable numbers

Re-running the capstone under both anchors turned up something worse than the
anchor question. The uncle-anchored arm should have bit-reproduced the
committed capstone, since its key is unchanged -- and it did not. Bisecting
against the pre-session source showed my changes are clean (identical
trajectory, epochs=21, tail 0.994358 either way); the committed parquet is
simply stale. It is dated 24 July and has no uncle_model column at all, so it
predates the countable redesign. sec 8.4's headline table has been reporting
UNRESTRICTED-model numbers ever since that landed. sec 9's "all studies re-run"
note covers the slot-counting fix of 23/24 July, not the countable redesign of
4 August, and the capstone was missed.

Corrected, countable model, delta_max = 8, U = 2, W = 10, k = 2160, 8 reps:
  honest    D-hat/D 0.996 (was 1.001)   p_ref 0.944 (was 1.000)
  30% suppress  0.994 (was 0.998)       p_ref 0.936 (was 0.990)
Lower, and p_ref materially lower, because the first-fork restriction makes
some orphans unreferenceable that the earlier model counted. No recommendation
changes: accuracy sits inside the +-0.9% per-epoch noise floor below the hard
ceiling of 1, consensus is exact, p_ref stays far above the ~0.3 the soft rule
needs. sec 6.8's p_ref quotes came from the same stale run and are corrected too.

The anchor result the re-run was for: parent-anchored gives 0.996 honest --
unchanged -- but 0.974 under the 30% suppression adversary, with p_ref 0.875.
The effects compound: a tighter effective window leaves an orphan fewer chances
to be referenced before its PARENT ages out, and a suppressing adversary
removes some of those chances. Neither isolated sweep shows it, because sec 6.12's
honest arms hold the adversary at zero and its adversarial arms use a slack
cap. That is what the capstone is for, and it is the strongest argument for
pairing the anchor change with W = 12 rather than adopting it at W = 10.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Marcin Pawlowski 2026-08-10 11:06:39 +02:00
parent a16284ebc5
commit de618d7ea8
No known key found for this signature in database
2 changed files with 20 additions and 11 deletions

View File

@ -687,7 +687,7 @@ The backfire is *unconditional* below the `⅓` threshold and holds at higher `
[§6.7](#s6-7)(3) shows a purely *voluntary* uncle reward can backfire on selfish mining. The tempting fix — a *mandatory* inclusion rule that makes a block **invalid** if it omits an owed uncle — cannot be encoded safely: "owed" means "the producer could have seen it," which depends on per-node gossip timing that no node can prove. An **objective** encoding (slot-gap `s_block s_uncle ≥ g`) lets an adversary *withhold-then-late-release* an uncle to retroactively **invalidate honest blocks** (censorship); a **visibility-based** encoding is *subjective*, so the same block is valid to some nodes and invalid to others — a **consensus fork** with no adversary at all. **We therefore make inclusion a soft rule:** *a block is always valid; a producer that includes an available in-window uncle earns the nephew reward `w_n`, and the orphan producer earns `w_u` — omission simply forfeits those rewards.* No validity edge, so no fork and no censorship weapon.
**The emergent `p_ref` is high — now measured, not assumed.** The per-node engine records `p_ref` directly (the fraction of in-window orphans referenced as uncles by some canonical block): at the recommended `W = 300` it is **≈ 1.0 honestly** and **0.990 under a 30 %-stake coalition that produces its share of the (straddling) canonical blocks and suppresses references on them** ([§8](#s8) capstone). That suppression case *is* the reference-suppression channel a selfish attacker would use — an honest orphan loses its reference only if *every* one of its in-window straddlers is a suppressing coalition block — so even with 30 % of the straddlers hostile, barely 1 % of honest orphans go unreferenced, comfortably above the `p_ref ≳ 0.3` the soft rule needs for `α ≥ 0.4` (`fig15`). That, however, is the *suppression* channel only. The stronger one — the attacker *orphaning extra* honest blocks via a private-chain override — is measured separately below, and it is where the structural argument ("published honest orphans are straddled by *honest* blocks within `W`, which the attacker does not control") turns out to be too optimistic: it holds for the orphans that are *referenceable*, but a private-chain override buries most of them where no proposer may reach. The earlier analytic sweeps used `p_ref ≈ 0.85` as a conservative stand-in; against suppression that is pessimistic, against a private chain it is not conservative enough.
**The emergent `p_ref` is high — now measured, not assumed.** The per-node engine records `p_ref` directly (the fraction of in-window orphans referenced as uncles by some canonical block): at the recommended `W = 300` it is **0.944 honestly** and **0.936 under a 30 %-stake coalition that produces its share of the (straddling) canonical blocks and suppresses references on them** ([§8](#s8) capstone). That suppression case *is* the reference-suppression channel a selfish attacker would use — an honest orphan loses its reference only if *every* one of its in-window straddlers is a suppressing coalition block — so even with 30 % of the straddlers hostile the reference rate falls by less than a point, and stays far above the `p_ref ≳ 0.3` the soft rule needs for `α ≥ 0.4` (`fig15`). (The honest rate is 0.944 rather than ≈ 1 because the first-fork restriction leaves some orphans unreferenceable outright — an earlier edition quoted ≈ 1.0 from a capstone run that predated the countable model, [§8.4](#s8-4).) That, however, is the *suppression* channel only. The stronger one — the attacker *orphaning extra* honest blocks via a private-chain override — is measured separately below, and it is where the structural argument ("published honest orphans are straddled by *honest* blocks within `W`, which the attacker does not control") turns out to be too optimistic: it holds for the orphans that are *referenceable*, but a private-chain override buries most of them where no proposer may reach. The earlier analytic sweeps used `p_ref ≈ 0.85` as a conservative stand-in; against suppression that is pessimistic, against a private chain it is not conservative enough.
**Why a soft rule still works — the reference rate is emergent and high (`fig15`).** Under a soft rule `p_ref` is not the attacker's to set to 0: an honest orphan was **published** (honest miners publish immediately), so *any honest canonical block that sees it within `W`* references it for `w_n`. The attacker only withholds references on **its own** canonical blocks; the honest blocks that follow do the referencing. So `p_ref` sits **high** in practice, and `fig15` reads off the payoff along it — at a realistic honest-referencer rate `p_ref ≈ 0.85`, the selfish attacker's reward share is already **below block-only and near stake** (`0.420` at `α = 0.4`, vs `0.520` at the suppressed `p_ref = 0` and `0.484` block-only) with honest recovery `0.830.93`. For `α ≥ 0.4` the crossover below block-only is at `p_ref ≈ 0.3` (the grey band); for a near-threshold `α ≈ 0.35` a small residual backfire (`≤ +0.03` for `p_ref ≳ 0.3`, rising to `+0.05` at the unreachable `p_ref = 0`; share still near stake) persists until `p_ref ≈ 0.75`. So the soft rule buys **most** of the (unattainable) hard-mandate benefit, plus the full fairness and hiding-disincentive of [§6.7](#s6-7)(12), with **none** of the fork/censorship hazard.
@ -942,11 +942,15 @@ Every finding above comes from a different sweep. As a final check that the reco
| metric | honest | 30 % suppression adversary |
|---|---|---|
| accuracy `D̂/D` | 1.001 | 0.998 |
| accuracy `D̂/D` | 0.996 | 0.994 |
| consensus `range_ratio` / `agreement` | 0 / 1.000 | 0 / 1.000 |
| fork rate | 0.342 ± 0.009 | 0.343 ± 0.005 |
| fork rate | 0.328 ± 0.014 | 0.350 ± 0.004 |
| max reorg depth | 3 | 3 |
| emergent `p_ref` | 1.000 | 0.990 |
| emergent `p_ref` | 0.944 | 0.936 |
*These numbers are the **countable** model's. An earlier edition of this table reported 1.001 / 0.998 with `p_ref` 1.000 / 0.990, measured before the countable redesign and never re-run against it — the capstone was missed when the other studies were brought over ([§9](#s9)). The corrected values are lower and the reference rate is materially lower (0.944 against 1.000), because the first-fork restriction makes some orphans unreferenceable that the earlier model counted. Nothing in the recommendation changes: accuracy still sits within the ±0.9 % per-epoch noise floor of [Appendix B](#sB) below the hard ceiling of 1, consensus is still exact, and `p_ref` remains far above the `≳ 0.3` the soft rule needs.*
**Under the [§6.12](#s6-12) parent-anchored window** the same run gives 0.996 honest — no change — but **0.974 under the 30 % suppression adversary**, with `p_ref` falling to 0.875. The two effects compound: a tighter effective window leaves an orphan fewer chances to be referenced before its *parent* ages out, and an adversary suppressing references on its own blocks removes some of those chances. Neither isolated sweep shows this, because [§6.12](#s6-12)'s honest arms hold the adversary at zero and its adversarial arms use a slack cap. It is the reason the capstone exists, and it is the strongest argument for pairing the anchor change with `W = 12` rather than adopting it at `W = 10`.
The whole recipe coheres: accurate to the true stake (the ~1 % rounding removed), unanimous, shallow-forking, and its soft-inclusion premise (`p_ref` high) confirmed — and all of it survives a 30 % adversary actively trying to deflate the estimate — the honest and adversarial fork rates are statistically indistinguishable (0.342 ± 0.009 vs 0.343 ± 0.005), i.e. the suppression adversary does not measurably change the fork structure it exploits. (The honest `D̂/D` = 1.001 sits within the ±0.9 % per-epoch noise floor of [Appendix B](#sB), consistent with the equilibrium's hard ceiling of 1.) This also resolves the finding-2 ([§3.2](#s3-2)) vs row-4 tension in the recommendation's favour: at this operating point `⌈ρ⌉ = 1`, and the recommended `U = 2` (the `+1` margin) holds every metric at target with room to spare.
@ -1029,7 +1033,7 @@ Sweep studies are committed configs, run with `make <name>` (writes a dated `run
Because the two models draw independent RNG streams, every countable-vs-unrestricted comparison is **unpaired**, and its resolution is set by the replicate spread rather than by the effect size. Each comparison sweep therefore includes a `U = 0` arm as a **negative control**: with no uncles the models are identical by construction, so the measured `U = 0` gap is a direct reading of the noise floor at that delay and replicate count. At `δ_max = 32` with 5 replicates that floor is ≈ 0.23 in `D̂/D` — larger than several real effects elsewhere in the grid — which is why [§3.2](#s3-2) reports a `t` statistic per cell and why the design regime is measured separately at 40 replicates ([§3.2a](#s3-2a)).
All studies were **re-run on 2026-07-23/24 with the corrected slot-counting mechanism** ([§2.1](#s2-1)) and the early-stop optimisation; the resilient batch is `scripts/run_all_reruns.sh` (per-step log in `runs/rerun_status.log`). Canonical run directories (latest): fullscale N=5000/10000 = `2026-07-24_094519_fullscale`; fullscale N=1000/2000 = `2026-07-23_171803_fullscale-small`; uncle-window = `2026-07-24_001456`; window-uncles = `2026-07-24_014240`; block-rate = `2026-07-24_043943`; blend-hops-delay = `2026-07-24_064052`; window-scale = `2026-07-24_085234`; latency-shape = `2026-07-24_090014_expdist`; stake-tail = `2026-07-24_090044_pareto133`; heterogeneous-start = `2026-07-24_090114_default`; N-scaling = the `nscaling-{a,b}` + `nscaling32-{a,b}` runs; adversary grids = `runs/adversary_grid/`; jitter = `runs/jitter_grid/`; bootstrap = `runs/bootstrap_fullscale/`; fluctuation = `runs/fluctuation_u0.parquet`; fork-rate = `runs/fork_rate_vs_delay.parquet`; ρ-boundary = `2026-07-27_195627_rho-boundary`. Referencing-model studies: countable-vs-unrestricted (paired, primary) = `2026-08-05_120804_cvo-paired-countable` / `2026-08-05_123403_cvo-paired-old`; the superseded unpaired pair = `2026-08-04_103536_cvo-countable` / `2026-08-04_104010_cvo-old`; window absorption = `2026-08-04_104633_absorption-window`; fine delay band (unpaired) = `2026-08-04_191441_fine-countable` / `2026-08-04_195353_fine-old`; fine delay band (paired, primary) = `2026-08-05_111109_fine-paired-countable` / `2026-08-05_113218_fine-paired-old`. Adversarial-countability studies: uncle-selection deviation = `2026-08-05_152617_uncle-selection`; countable-selfish MDP = `runs/countable_selfish.parquet` + `runs/countable_selfish_reorg.parquet`; §6.5 scope variants and the withhold-load sweep = `runs/adversary_variants_{whale,jitter,beta,withhold_load}.parquet`.
The capstone was re-run under the countable model (and both window anchors) after it was found still carrying pre-redesign numbers; its parquet now carries an `anchor` column. All other studies were **re-run on 2026-07-23/24 with the corrected slot-counting mechanism** ([§2.1](#s2-1)) and the early-stop optimisation; the resilient batch is `scripts/run_all_reruns.sh` (per-step log in `runs/rerun_status.log`). Canonical run directories (latest): fullscale N=5000/10000 = `2026-07-24_094519_fullscale`; fullscale N=1000/2000 = `2026-07-23_171803_fullscale-small`; uncle-window = `2026-07-24_001456`; window-uncles = `2026-07-24_014240`; block-rate = `2026-07-24_043943`; blend-hops-delay = `2026-07-24_064052`; window-scale = `2026-07-24_085234`; latency-shape = `2026-07-24_090014_expdist`; stake-tail = `2026-07-24_090044_pareto133`; heterogeneous-start = `2026-07-24_090114_default`; N-scaling = the `nscaling-{a,b}` + `nscaling32-{a,b}` runs; adversary grids = `runs/adversary_grid/`; jitter = `runs/jitter_grid/`; bootstrap = `runs/bootstrap_fullscale/`; fluctuation = `runs/fluctuation_u0.parquet`; fork-rate = `runs/fork_rate_vs_delay.parquet`; ρ-boundary = `2026-07-27_195627_rho-boundary`. Referencing-model studies: countable-vs-unrestricted (paired, primary) = `2026-08-05_120804_cvo-paired-countable` / `2026-08-05_123403_cvo-paired-old`; the superseded unpaired pair = `2026-08-04_103536_cvo-countable` / `2026-08-04_104010_cvo-old`; window absorption = `2026-08-04_104633_absorption-window`; fine delay band (unpaired) = `2026-08-04_191441_fine-countable` / `2026-08-04_195353_fine-old`; fine delay band (paired, primary) = `2026-08-05_111109_fine-paired-countable` / `2026-08-05_113218_fine-paired-old`. Adversarial-countability studies: uncle-selection deviation = `2026-08-05_152617_uncle-selection`; countable-selfish MDP = `runs/countable_selfish.parquet` + `runs/countable_selfish_reorg.parquet`; §6.5 scope variants and the withhold-load sweep = `runs/adversary_variants_{whale,jitter,beta,withhold_load}.parquet`.
Figures are embedded from [`report-figures/`](report-figures) and versioned here alongside the report; the simulator folder does **not** commit its own generated figures, so the copies checked in here are the figures of record. Figures are in `report-figures/` (`fig1``fig29`, plus [Appendix B](#sB)'s `figB1``figB2`; numbering is generation order, not order of appearance). Committed generators: `fig1` (bootstrap, k=2160) by `scripts/bootstrap_dynamics.py`; `fig2`,`fig4`,`fig5`,`fig17``fig22` by `scripts/regenerate_extra_figs.py` from the latest sweeps (`fig3` hops×delay×U grid by `scripts/hops_delay_grid.py`, `fig6` (block-rate `U_min` grid + ρ-collapse) rendered ad hoc from `runs/2026-07-24_043943_block-rate` with no committed generator; `fig26` deficit-vs-ρ by `scripts/rho_boundary_analysis.py`) (fullscale-derived `fig17``fig20` pool both sizes in that run, N = 5 000 and N = 10 000 — the generators filter on stake_dist/topology/degree/init_dest only, never on `n_nodes`); `fig8`,`fig9` by `scripts/adversary_figs.py` from `runs/adversary_grid/`; `fig10``fig12` by `scripts/dynamic_withhold.py`; `fig13``fig15` by `scripts/selfish_mining.py`/`selfish_rewards.py`/`reward_mandate.py`; `fig16` by `scripts/stake_vs_delay.py`; `fig23``fig24` by `scripts/nscaling_analysis.py`; `fig25` by `scripts/window_scale_analysis.py`; `fig27``fig28` by `scripts/reorg_depth.py` (fork rates via `--measure`; private-chain model `src/tsi_sim/reorg.py`); `fig29` by `scripts/churn.py`; `figB1``figB2` by `scripts/appendix_fluct.py`. `fig7` (feedback fixed-point) is an analytic overlay; `fig30``fig33` (countable-vs-unrestricted accuracy, `q_u`-prediction check, recovery rate, absorption-window sweep) by `scripts/plot_countable_vs_old.py` from the `cvo-countable`/`cvo-old`/`absorption-window` runs; `fig34``fig35` (design-regime accuracy and the model gap with 95 % CIs) by `scripts/plot_fine_delay.py` from the `fine-countable`/`fine-old` runs; `fig36` (the countable recovery ceiling under a selfish adversary) by `scripts/countable_selfish.py`, which solves the MDP directly and writes `runs/countable_selfish.parquet` + `runs/countable_selfish_reorg.parquet` (no sweep input); `fig37` (the profit/deflation frontier) by `scripts/deflation_frontier.py` from `runs/deflation_frontier.parquet`, likewise solver-only; `fig38` (the reference-window anchor) by `scripts/uncle_parent_window.py`. Every figure type the per-node simulator generates appears in this report, and the fork-rate/reorg-depth study closes the previous reproducibility gap for the adversarial figures (`fig8`,`fig9` now have committed generators from `runs/adversary_grid/`).

View File

@ -24,23 +24,28 @@ REC = dict(n_nodes=1000, stake_dist="pareto", topology="blend", degree=6,
genesis_d_factor=0.5, early_stop=True)
def _one(adv: float, rep: int) -> list[dict]:
cfg = SimConfig(**REC, adversary_frac=adv, adversary_strategy="suppress", replicate=rep)
def _one(adv: float, rep: int, anchor: str = "uncle") -> list[dict]:
cfg = SimConfig(**REC, adversary_frac=adv, adversary_strategy="suppress",
uncle_window_anchor=anchor, replicate=rep)
rows = run_trajectory(cfg)
for r in rows:
r["adv"] = adv
r["anchor"] = anchor
return rows
def main() -> None:
out = Path(__file__).resolve().parents[1] / "runs"
jobs = [(a, r) for a in (0.0, 0.3) for r in range(8)]
# Both window anchors: the spec's uncle-anchored rule, and the §6.12 proposal. The capstone
# is the "whole recipe together" check, so a change to any rule in the recipe has to be run
# through it rather than argued from the isolated sweeps.
jobs = [(a, r, w) for w in ("uncle", "parent") for a in (0.0, 0.3) for r in range(8)]
res = Parallel(n_jobs=4, backend="loky", inner_max_num_threads=1)(
delayed(_one)(a, r) for a, r in jobs)
delayed(_one)(a, r, w) for a, r, w in jobs)
df = pd.DataFrame([row for traj in res for row in traj])
df.to_parquet(out / "capstone.parquet", index=False)
print("=== Capstone: recommended config, all metrics together (equilibrium tail) ===")
for adv, g in df.groupby("adv"):
for (anchor, adv), g in df.groupby(["anchor", "adv"]):
# Per-REPLICATE tail: early_stop ends replicates at different epochs, so a per-arm cut
# (epoch >= arm_max//2) would silently drop any replicate that stopped before the cut
# and skew the tail toward the slow-converging ones. The report's §8.4 numbers are the
@ -48,7 +53,7 @@ def main() -> None:
t = pd.concat([r[r.epoch >= r.epoch.max() // 2] for _, r in g.groupby("replicate")])
per_rep = t.groupby("replicate").fork_rate.mean()
sem = per_rep.std(ddof=1) / (len(per_rep) ** 0.5)
print(f"adversary {adv:.0%}: D̂/D {t.mean_ratio.mean():.4f} "
print(f"{anchor:>6}-anchored, adversary {adv:.0%}: D̂/D {t.mean_ratio.mean():.4f} "
f"range_ratio {t.range_ratio.max():.4f} agreement {t.agreement_window.min():.4f} "
f"fork_rate {per_rep.mean():.3f}+-{sem:.3f}(SEM over {len(per_rep)} reps) "
f"max_reorg_depth {t.max_reorg_depth.max()} p_ref {t.p_ref.mean():.3f}")