From 5a8cc4437aa6d5a4d1ad2ae9f0c67b88d3fc770b Mon Sep 17 00:00:00 2001 From: Marcin Pawlowski Date: Thu, 6 Aug 2026 12:56:23 +0200 Subject: [PATCH] Item 5 resolved: the uncle cap is not the binding constraint under a private chain With the SM1 adversary in the engine, the question item 5 could not ask is now a measurement. It asked whether the honest-load cap needs margin when the attack inflates orphaning, on the theory that owed uncles would defer past W. They do not. Sweeping U x W x alpha against the attack (512 runs, N=1000, k=256, 8 reps): at the design point and alpha = 0.3, D-hat/D reads 0.729/0.755/0.738/0.758 for U = 1/2/3/4 -- flat within noise -- and no attacked cell reaches the 0.98 bar at any cap or either window. The honest baseline in the same sweep reproduces sec 3.4 exactly (U=1 clears at delta=8; delta=16 needs U=2 at W=10 or W=20 at U=1), which is a useful check that the engine adversary has not disturbed the honest regime. Splitting the honest orphans by WHY they went unreferenced explains it. Neither existing metric separates the two causes -- p_ref mixes them, and deep_ref_share is 0 by construction here because the proposer's candidate filter drops deep-fork blocks before any reference to one is proposed -- so the script walks the tree. Countable share (first block of its fork): 97% honest, 76-81% at alpha=0.2, 59-72% at alpha=0.3. Referenced OF those: 90-93% honest, 84-93% and 80-88% under attack. The queue drains at essentially the honest rate whatever the cap; what collapses is eligibility. An override discards a CHAIN and only its first block has a parent on the surviving chain, so 20-40% of the honest work destroyed is unreferenceable by construction. U governs drain capacity for candidates that exist; it cannot manufacture eligibility. So U = ceil(rho) + 1 stands unchanged and needs no adversarial margin -- and the one place the cap does matter is the honest-load reason it was sized for (U=1 -> 2 lifts the referenced-of-countable rate from 84% to 93% at alpha = 0.2, then U=4 adds nothing). This is the fig36 first-fork ceiling reached from an independent direction: a per-node network simulation with real delays and a real queue, versus a stationary MDP. Two models sharing no code, agreeing on direction and rough size, is the strongest available evidence that the ceiling is a property of the counting rule rather than of either model. Recorded in sec 6.6 and sec 6.8, with the sec 6.8 structural argument corrected: it holds for orphans that are referenceable, but a private chain buries most of them out of reach. Co-Authored-By: Claude Fable 5 --- reports/tsi/README.md | 24 +++++- .../scripts/selfish_uncle_margin.py | 81 ++++++++++++++++++- 2 files changed, 99 insertions(+), 6 deletions(-) diff --git a/reports/tsi/README.md b/reports/tsi/README.md index 17b0c90..b2a7899 100644 --- a/reports/tsi/README.md +++ b/reports/tsi/README.md @@ -619,11 +619,13 @@ The curve is **within `0.09` for `β = 0.5` and `β = 1.0`** at every duty — p So **SM1 is a faithful proxy for selfish-mining revenue but not for TSI's estimator damage**: at `α = 0.4, γ = 0` its revenue share (0.484) sits within 1 % of the optimum's (0.489), while its recoverable-orphan fraction (1.00) is more than double (0.44). The repair is real but roughly **half** of what the unrestricted model shows — `0.81`, not `0.94`, at `α = 0.4` — and it moves the *wrong way* with `α`: the unrestricted recovery improves as the attacker grows (0.926 → 0.955) because a stronger attacker orphans a larger share of a shrinking honest chain, while the countable recovery **degrades** (0.856 → 0.619) because a stronger attacker waits longer and buries more blocks per fork. (Read the `α ≥ 0.42` cells as indicative: the orphan *shape* converges more slowly in the MDP's lead cap than the revenue does — `η` at `α = 0.45` still moves by ~3 % relative between cap 48 and 64, against ~0.1 % at `α = 0.4`.) The depth-maximising adversary of [§6.10](#s6-10) agrees on the magnitude from the other direction — `η = 0.52` at `α = 0.30` with the measured honest fork rate `o = 0.35` (`reorg.countable_recovery_from_depths`). Neither figure is a worst case: both adversaries optimise something else (revenue, depth), and one optimising *deflation* directly would trade revenue for fork depth. The attacker's own channel is capped by the same rule — its abandoned secret chain is also one chain, so it can self-uncle only the first block of it (`η_adv = 0.750` at `α = 0.4`), which narrows the [§6.7](#s6-7)(a) farming payoff below what the block count suggests. +**The per-node engine reaches the same ceiling independently.** Everything above is a stationary MDP: one global chain, no network, recovery inferred from the policy's orphan structure. Running a private-chain coalition inside the per-node simulator instead — real propagation delays, per-node views, an actual uncle queue draining against `U` and `W` — reproduces the mechanism from the other side: of the honest blocks a private-chain attack destroys, only 59–81 % are countable at all (against 97 % of ordinary latency orphans), and the shortfall is precisely the blocks buried behind the first of each discarded run ([§6.8](#s6-8)). Two models that share no code and make different idealisations agree on both the direction and the rough size of the loss, which is the strongest evidence available here that the first-fork ceiling is a property of the counting rule rather than of either model. + ![Fig 36 — the countable recovery ceiling under a selfish adversary: SM1 leaves every orphan referenceable (flat 1.0) while the optimal policy buries them (left); the resulting D̂ is about half the repair the unrestricted model shows, and degrades with α (right).](report-figures/fig36_countable_selfish.png) **Issuance model.** Does deflating `D̂` inflate the reward pie? No, for a per-canonical-block schedule: TSI targets *counted* density `= f`, so the canonical block rate is held at `≈ f` regardless of the attack (the extra lottery wins a deflated `D̂` produces are orphans that earn nothing). Absolute reward per unit stake therefore equals the revenue *share* metric — **[§6.5](#s6-5)'s "abstention is unprofitable" is robust under per-block issuance**, and [§6.6](#s6-6)'s selfish premium is the genuine profit channel. MEV (value extracted from transaction ordering) or a per-*raw*-block schedule would reopen this — an issuance-design question, not a TSI-estimation one. *(This clean equivalence assumes the block-only schedule; once [§6.7](#s6-7) pays and **counts** uncles, the counted density held at `f` includes recovered uncles, so the canonical rate sits slightly below `f` and the pie's size and split both shift. [§6.7](#s6-7)'s profitability is then read directly off the reward *share*, not off a fixed pie — the two subsections use different issuance premises, reconciled only in the `w_u = 0` limit.)* -**Scope.** Modelled: SM1 *and* the optimal-MDP selfish strategy with a network tie-break `γ`, a single coalition, uncle density-recovery as the `[0,1]` knob `η`, and — for the optimal policy — the ceiling the countable rule puts on `η`. Multi-coalition races are discussed (not solved) in [§6.9](#s6-9); their orphan *shape*, and hence their own `η` ceiling, is unmodelled for the same reason. The reward analysis of [§6.7](#s6-7)–[§6.8](#s6-8) runs on the SM1 race, where every orphan is countable, so its numbers are unaffected by the restriction — but they inherit SM1's optimism about *which* orphans exist to be compensated. The headline: **private-chain withholding is the real profit threat, bounded by the classic stake threshold; its *estimator* deflation is roughly halved by uncle-counting rather than undone, while its *profit* premium is only conditionally reduced by rewarding uncles ([§6.7](#s6-7)–[§6.8](#s6-8)).** +**Scope.** Modelled: SM1 *and* the optimal-MDP selfish strategy with a network tie-break `γ`, a single coalition, uncle density-recovery as the `[0,1]` knob `η`, the ceiling the countable rule puts on `η` for the optimal policy, and — in the per-node engine, where the knob becomes a real uncle queue — an SM1 private chain under explicit propagation delay ([§6.8](#s6-8)). Multi-coalition races are discussed (not solved) in [§6.9](#s6-9); their orphan *shape*, and hence their own `η` ceiling, is unmodelled for the same reason. The reward analysis of [§6.7](#s6-7)–[§6.8](#s6-8) runs on the SM1 race, where every orphan is countable, so its numbers are unaffected by the restriction — but they inherit SM1's optimism about *which* orphans exist to be compensated. The headline: **private-chain withholding is the real profit threat, bounded by the classic stake threshold; its *estimator* deflation is roughly halved by uncle-counting rather than undone, while its *profit* premium is only conditionally reduced by rewarding uncles ([§6.7](#s6-7)–[§6.8](#s6-8)).** ### 6.7 Block/uncle rewards: robust for fairness and anti-hiding, but a *voluntary* reward can backfire on selfish mining (`fig14`, `fig15`) @@ -655,13 +657,24 @@ The backfire is *unconditional* below the `⅓` threshold and holds at higher ` [§6.7](#s6-7)(3) shows a purely *voluntary* uncle reward can backfire on selfish mining. The tempting fix — a *mandatory* inclusion rule that makes a block **invalid** if it omits an owed uncle — cannot be encoded safely: "owed" means "the producer could have seen it," which depends on per-node gossip timing that no node can prove. An **objective** encoding (slot-gap `s_block − s_uncle ≥ g`) lets an adversary *withhold-then-late-release* an uncle to retroactively **invalidate honest blocks** (censorship); a **visibility-based** encoding is *subjective*, so the same block is valid to some nodes and invalid to others — a **consensus fork** with no adversary at all. **We therefore make inclusion a soft rule:** *a block is always valid; a producer that includes an available in-window uncle earns the nephew reward `w_n`, and the orphan producer earns `w_u` — omission simply forfeits those rewards.* No validity edge, so no fork and no censorship weapon. -**The emergent `p_ref` is high — now measured, not assumed.** The per-node engine records `p_ref` directly (the fraction of in-window orphans referenced as uncles by some canonical block): at the recommended `W = 300` it is **≈ 1.0 honestly** and **0.990 under a 30 %-stake coalition that produces its share of the (straddling) canonical blocks and suppresses references on them** ([§8](#s8) capstone). That suppression case *is* the reference-suppression channel a selfish attacker would use — an honest orphan loses its reference only if *every* one of its in-window straddlers is a suppressing coalition block — so even with 30 % of the straddlers hostile, barely 1 % of honest orphans go unreferenced, comfortably above the `p_ref ≳ 0.3` the soft rule needs for `α ≥ 0.4` (`fig15`). The one selfish channel this measurement does *not* exercise is the attacker *orphaning extra* honest blocks via a private-chain override (the per-node engine has no private-chain strategy): there we rely on the structural argument below — published honest orphans are straddled by *honest* blocks within `W`, which the attacker does not control — and read the payoff off the `fig15` `p_ref` sweep rather than a single measured point. The earlier analytic sweeps used `p_ref ≈ 0.85` as a conservative stand-in; the suppression measurement confirms it is, if anything, pessimistic. +**The emergent `p_ref` is high — now measured, not assumed.** The per-node engine records `p_ref` directly (the fraction of in-window orphans referenced as uncles by some canonical block): at the recommended `W = 300` it is **≈ 1.0 honestly** and **0.990 under a 30 %-stake coalition that produces its share of the (straddling) canonical blocks and suppresses references on them** ([§8](#s8) capstone). That suppression case *is* the reference-suppression channel a selfish attacker would use — an honest orphan loses its reference only if *every* one of its in-window straddlers is a suppressing coalition block — so even with 30 % of the straddlers hostile, barely 1 % of honest orphans go unreferenced, comfortably above the `p_ref ≳ 0.3` the soft rule needs for `α ≥ 0.4` (`fig15`). That, however, is the *suppression* channel only. The stronger one — the attacker *orphaning extra* honest blocks via a private-chain override — is measured separately below, and it is where the structural argument ("published honest orphans are straddled by *honest* blocks within `W`, which the attacker does not control") turns out to be too optimistic: it holds for the orphans that are *referenceable*, but a private-chain override buries most of them where no proposer may reach. The earlier analytic sweeps used `p_ref ≈ 0.85` as a conservative stand-in; against suppression that is pessimistic, against a private chain it is not conservative enough. **Why a soft rule still works — the reference rate is emergent and high (`fig15`).** Under a soft rule `p_ref` is not the attacker's to set to 0: an honest orphan was **published** (honest miners publish immediately), so *any honest canonical block that sees it within `W`* references it for `w_n`. The attacker only withholds references on **its own** canonical blocks; the honest blocks that follow do the referencing. So `p_ref` sits **high** in practice, and `fig15` reads off the payoff along it — at a realistic honest-referencer rate `p_ref ≈ 0.85`, the selfish attacker's reward share is already **below block-only and near stake** (`0.420` at `α = 0.4`, vs `0.520` at the suppressed `p_ref = 0` and `0.484` block-only) with honest recovery `0.83–0.93`. For `α ≥ 0.4` the crossover below block-only is at `p_ref ≈ 0.3` (the grey band); for a near-threshold `α ≈ 0.35` a small residual backfire (`≤ +0.03` for `p_ref ≳ 0.3`, rising to `+0.05` at the unreachable `p_ref = 0`; share still near stake) persists until `p_ref ≈ 0.75`. So the soft rule buys **most** of the (unattainable) hard-mandate benefit, plus the full fairness and hiding-disincentive of [§6.7](#s6-7)(1–2), with **none** of the fork/censorship hazard. ![Fig 15 — soft rule: as the emergent reference rate p_ref rises (with W), the selfish attacker's reward share falls below block-only (dotted) toward stake (left); honest-orphan compensation rises (right). The backfire is confined to the small-W corner.](report-figures/fig15_mandate.png) -**What sets `p_ref` — the window `W` and visibility (the residual).** `p_ref` rises toward 1 as `W` grows (more following honest blocks get a chance to reference a given orphan) and falls only when an orphan **ages out of `W` before any honest referencer sees it** — i.e. under a *deep reorg* longer than `W`, or genuine propagation loss. This is exactly the "can't guarantee a node sees every fork in the window" limit, and it is `W`-tunable: the [§3.4](#s3-4) sizing `W ≈ 10/f` is already generous, and the residual adversarial lever (drive `p_ref` down via reorgs deeper than `W`) is bounded — deep reorgs need large `α`, are rare, and risk the attacker's private chain. Two provisos remain, both about *degree* not *direction*: the counted-density (`D̂`) recovery is likewise `p_ref`-limited, so under attack-inflated orphaning (`orphan/blocks ≈ 0.35` at `α = 0.46`) the cap `U` — sized for the *honest* load ([§3.3](#s3-3)) — may need margin, since owed uncles beyond `U` per block defer and can age out; and even at `p_ref = 1` the selfish share is `≈ α` with a small residual premium that **grows with `α`** — `≈ 0` near the `⅓` threshold, `+0.006` at `α = 0.4`, `+0.014` at `α = 0.46` — not exact break-even. +**What sets `p_ref` — the window `W` and visibility (the residual).** `p_ref` rises toward 1 as `W` grows (more following honest blocks get a chance to reference a given orphan) and falls only when an orphan **ages out of `W` before any honest referencer sees it** — i.e. under a *deep reorg* longer than `W`, or genuine propagation loss. This is exactly the "can't guarantee a node sees every fork in the window" limit, and it is `W`-tunable: the [§3.4](#s3-4) sizing `W ≈ 10/f` is already generous, and the residual adversarial lever (drive `p_ref` down via reorgs deeper than `W`) is bounded — deep reorgs need large `α`, are rare, and risk the attacker's private chain. One proviso remains, about *degree* not *direction*: even at `p_ref = 1` the selfish share is `≈ α` with a small residual premium that **grows with `α`** — `≈ 0` near the `⅓` threshold, `+0.006` at `α = 0.4`, `+0.014` at `α = 0.46` — not exact break-even. + +**Under a private chain the uncle cap is not the binding constraint — the fork structure is.** The engine carries a private-chain coalition (`adversary_strategy: selfish`, SM1 release rules), so the whole loop is present: the attack orphans honest blocks in runs, the survivors queue for the `U` uncle slots of each canonical block, and whatever does not drain within `W` ages out. Sweeping the cap against the attack (`scripts/selfish_uncle_margin.py`; N = 1 000, k = 256, 8 replicates, `U ∈ {1,2,3,4}` × `W ∈ {10, 20}`) settles the question the honest-load rule left open. **Raising `U` does not buy the estimate back.** At the design point (`δ_max` = 8 s) and `α` = 0.3, `D̂/D` reads 0.729 / 0.755 / 0.738 / 0.758 for `U` = 1 / 2 / 3 / 4 — flat within replicate noise — and no attacked cell reaches the 0.98 recovery bar at any cap or either window, while the honest baseline in the same sweep reproduces [§3.4](#s3-4) exactly (`U` = 1 clears at `δ_max` = 8; `δ_max` = 16 needs `U` = 2 at `W` = 10, or `W` = 20 at `U` = 1). + +The reason is visible once the honest orphans are split by *why* they went unreferenced — structurally unreachable, or eligible but starved of a slot. (Neither `p_ref` nor `deep_ref_share` separates these: the first mixes them, and the second is 0 by construction here, because the proposer's candidate filter drops deep-fork blocks before any reference to one is ever proposed.) Walking the tree directly (same script, 6 replicates): + +| | honest | `α` = 0.2 | `α` = 0.3 | +|---|---|---|---| +| honest orphans that are **countable** (first block of their fork) | 97 % | 76–81 % | 59–72 % | +| of those, **referenced** within `W` | 90–93 % | 84–93 % | 80–88 % | + +The queue is draining. Countable orphans are referenced at essentially the honest rate whatever the attack, and the one place the cap matters is the honest-load reason it was sized for — going `U` = 1 → 2 lifts that rate from 84 % to 93 % at `α` = 0.2, then `U` = 4 adds nothing. What collapses is the *countable* share: an override discards a **chain**, and only its first block has a parent on the surviving chain ([§2.1](#s2-1)), so 20–40 % of the honest work the attacker destroys is unreferenceable by construction. `U` governs drain capacity for candidates that exist; it cannot manufacture eligibility. This is the [§6.6](#s6-6) first-fork ceiling arriving from a completely independent direction — a per-node network simulation rather than a stationary MDP — and it is why `U = ⌈ρ⌉ + 1` needs no adversarial margin: the margin would be spent on a queue that is not the bottleneck. **Reward-side attack surface (bounded, self-defeating).** *(a) Farming* — orphaning a real VRF win to self-uncle it — pays `w_u + w_n` against `1`, so it is loss-making under the [§6.7](#s6-7)(a) invariant **`w_u + w_n < 1`**; in the model the attacker's share *falls* as it self-uncles more (`0.406 → 0.394`). *(b)* Because honest referencers count the orphans, a flood *inflates* `D̂` rather than deflating it — the opposite of what a deflation attack needs. *(c) Nephew leakage* to the attacker is bounded by `w_n` and dominated by the `w_u` that flows to the honest producer. @@ -771,6 +784,8 @@ Every parameter a protocol designer or operator can set, in three groups: **prot | knob | probes | result | where | |---|---|---|---| +| `adversary_strategy: selfish` | private-chain (SM1) override in the per-node engine | the profitable lever, now measurable end-to-end: revenue tracks Eyal–Sirer at sub-slot latency (0.0356 vs 0.0356 at α = 0.1) and exceeds it as natural forking lifts `α_eff`; deflates `D̂` to 0.76 at α = 0.3 (δ = 8) with **no** cap able to recover it ([§6.8](#s6-8)) | [§6.6](#s6-6), [§6.8](#s6-8) | +| `p_ref_honest` (engine output) | reference rate over orphans produced *outside* the coalition | equals `p_ref` under suppression; diverges sharply under a private chain (0.45–0.69 vs ≈ 0.99), which is the quantity the estimator repair actually depends on | [§6.8](#s6-8) | | `adversary_frac` (β_adv), `adversary_strategy` | uncle suppression / abstention withholding | suppression weak at ρ < 1, a cheap lever only at ρ > 1 (1.43× at ρ = 1.36, β_adv = 0.5); withholding → `D̂ ≈ (1−β_adv)` — the *correct* active-stake answer, topology-independent | [§6.3](#s6-3)/6.4, figs 8, 9 | | `adversary_period`, `adversary_withhold_epochs` | dynamic withhold-rejoin schedules | unprofitable at every duty ψ > 0; griefing bounded (`deflation = β_adv·ψ`); no persistence | [§6.5](#s6-5), figs 10–12 | | private-chain reorg (`α` = 0/10/20/30 %) | deepest-reorg longest-chain attack | `α` = the attacking coalition's stake fraction — the selfish-mining notation for `β_adv` above, and distinct from the TSI learning rate `β` ([§6.3](#s6-3)); reorg-depth tail `(α_eff/(1−α_eff))^d` with the *effective* share `α_eff=α/(α+(1−α)(1−o))` (`o` = honest fork rate); shallow at ρ<1, unbounded at 30 % once forks push `α_eff>½` | [§6.10](#s6-10), figs 27, 28 | @@ -827,7 +842,7 @@ Everything below is assembled from this report's own measurements; each row cite 2. **Multi-coalition selfish mining is open ([§6.9](#s6-9)).** The literature threshold can fall below ⅓ and total orphaning can exceed the single-coalition value, so `D̂ ≈ 0.70` at α = 0.4 is not an upper bound. Flagged, not solved. The [§6.6](#s6-6) countable ceiling adds a second unmodelled dimension: competing coalitions orphan each other in *runs* as well as in volume, and the recoverable share depends on that shape, not just on the total (item 16). 3. **Near-threshold reward backfire ([§6.7](#s6-7)–[§6.8](#s6-8)).** At α ≈ 0.35 a small backfire (≤ +0.03 above block-only for `p_ref ≳ 0.3`, rising to +0.05 at the unreachable `p_ref = 0`) persists until `p_ref ≳ 0.75`; even `p_ref = 1` leaves a premium growing with α. 4. **No re-convergence ([Appendix C](#sC)).** Any mechanism that could inject persistent per-node `D̂` disagreement (genesis/clock exploit) would never be self-corrected — safety rests entirely on row 7. -5. **Adversarial `U` margin unquantified ([§6.8](#s6-8)).** Under attack-inflated orphaning (orphan/blocks ≈ 0.35 at α = 0.46) the honest-load cap may need extra margin (owed uncles beyond `U` defer and can age out of `W`); this report does not size it. +5. **Adversarial `U` margin — resolved: none is needed, and none would help ([§6.8](#s6-8)).** The concern was that under attack-inflated orphaning the honest-load cap would leave owed uncles deferring past `W`. Measured with a private-chain coalition in the engine, the queue is not the bottleneck: countable orphans are referenced at the honest rate (84–93 %) whatever the cap, and raising `U` past the recommended 2 moves `D̂` not at all. The loss is structural — an override buries 20–40 % of the honest blocks it destroys behind the first block of the discarded run, where the counting rule cannot reach them. `U = ⌈ρ⌉ + 1` therefore stands unchanged, and what is *not* solved is the deflation itself, tracked as items 2 and 16 rather than as a sizing question. 6. **Clock skew — bounded, not a break ([§6.1](#s6-1)).** A per-node whole-timeline slot-clock shift moves each node's window bounds, giving a bounded inter-node spread `≤ 2·skew/T` (`< 10⁻⁴` at the production window). Small and quantified, but — unlike per-arrival jitter — not identically zero; large skew is still an untested extreme. 7. **Semi-empirical constants ([§5](#s5)).** `a ≈ 2` and the 7/f floor are fits; eq 4′ is an N ≈ 1 000 fit that worsens with N — for production sizing use the margined rules (rows 3–4) and the fully-recovered accuracy 1.0, never eq 4′. 8. **Residual ~1 % offset — resolved here, pending in the spec ([§2.2](#s2-2), row 14, [Appendix A](#sA)).** After the slot-counting fix the only systematic offset is the 10⁻³ rounding of `f` (≈ 1 % `f/f_p` over-estimate); this report's estimator applies the row-14 precision bump, but the **spec still uses 10⁻³** and should adopt it. @@ -923,6 +938,7 @@ Sweep studies are committed configs, run with `make ` (writes a dated `run | **window absorption sweep** (`W` in expected block-intervals, `w_u = W/f` derived) | `configs/absorption-window.yaml`; figure by `scripts/plot_countable_vs_old.py` | [§3.4](#s3-4) | | **uncle-selection deviation** (the spec's deterministic oldest-first vs a coin, over `W` × delay) | `configs/uncle-selection.yaml` (`uncle_strategy`, `uncle_random_p`) | [§3.4](#s3-4) | | **residual §6.5-scope variants** (whale coalitions, `jitter > 0`, very slow `β`, and the static-withholding load sweep that finds the collapsed-branch event) | `scripts/adversary_variants.py` (`adversary_selection`); `runs/adversary_variants_{whale,jitter,beta,withhold_load}.parquet` | [§6.2](#s6-2), [§6.5](#s6-5) | +| **uncle cap under a private chain** (`U` × `W` × `α` against the SM1 engine adversary, plus the structural/queue decomposition of the honest orphans) | `scripts/selfish_uncle_margin.py` (`adversary_strategy: selfish`); `runs/selfish_uncle_margin{,_decomp}.parquet` | [§6.8](#s6-8) | **Uncle-model convention.** The simulator's default is the **countable** model — first-fork candidates only, derived window `w_u = W/f`, occupied-slot exclusion, per-reference counting rules ([§2.1](#s2-1)). The **unrestricted** baseline is preserved in the code and selected with `--old` on `tsi-sweep`/`tsi-verify`. Its RNG key is byte-identical to the pre-restriction key, so `--old` **bit-reproduces the earlier runs**: a `rho-boundary` cell (δ_max = 8, U = 2, k = 256, N = 1 000) re-run under `--old` matches the committed `2026-07-27_195627_rho-boundary` parquet with `max |Δ| = 0` on every epoch and every metric. Studies in the table above that predate the countable default were produced under the unrestricted model and reproduce exactly under `--old`; the comparison rows quantify where the two models differ, and in the design regime (`ρ < 1`) no difference is resolvable, so those findings carry over unchanged. diff --git a/tools/simulators/tsi/tsi-sim-pernode/scripts/selfish_uncle_margin.py b/tools/simulators/tsi/tsi-sim-pernode/scripts/selfish_uncle_margin.py index a398f4d..36cc317 100644 --- a/tools/simulators/tsi/tsi-sim-pernode/scripts/selfish_uncle_margin.py +++ b/tools/simulators/tsi/tsi-sim-pernode/scripts/selfish_uncle_margin.py @@ -33,9 +33,13 @@ import numpy as np import pandas as pd from joblib import Parallel, delayed +from tsi_sim import lottery, topology +from tsi_sim.blocktree import build_tree_pernode from tsi_sim.config import SimConfig -from tsi_sim.engine import run_trajectory +from tsi_sim.engine import _adversary_mask, run_trajectory from tsi_sim.memguard import ArrivalMatrixTooLarge +from tsi_sim.rng import rng_for, seedseq_for +from tsi_sim.stake import make_stake HERE = Path(__file__).resolve().parent.parent RUNS = HERE / "runs" @@ -86,6 +90,78 @@ def sweep() -> pd.DataFrame: return df +def _decompose_cell(alpha: float, u: int, rep: int, delay: float = 8.0, w: int = 10) -> dict: + """Split the honest orphans into "unreferenceable" and "eligible but unreferenced". + + ``p_ref_honest`` alone cannot answer item 5, because it falls for two unrelated reasons: a + block can be *structurally* uncountable (buried behind the first block of an override, so no + proposer may reference it — §2.1) or countable but starved of an uncle slot (the queue the + cap `U` drains). Only the second is a cap-sizing problem. ``deep_ref_share`` does not + separate them either: the proposer's candidate filter drops deep-fork blocks before they are + ever proposed, so no deep reference is examined and the metric is 0 by construction here. + This walks the tree and measures both directly. + """ + cfg = SimConfig(**{**BASE, "epochs": 4}, blend_delay_max=delay, max_uncles=u, + window_absorption=w, adversary_frac=alpha, replicate=rep, + prune_arrival=False, windowed_fork_choice=False) + stake = make_stake(cfg, rng_for(cfg)) + mask = _adversary_mask(cfg, stake) + flat = np.zeros(cfg.n_nodes, dtype=bool) if mask is None else mask + kids = seedseq_for(cfg).spawn(cfg.epochs + 3) + pl = topology.build_path_latency(cfg, np.random.default_rng(kids[1])) + d_est = np.full(cfg.n_nodes, cfg.genesis_d_factor * float(stake.sum())) + p = lottery.win_probs(stake, d_est, cfg.f) + ws, wn = lottery.sample_wins(p, cfg.epoch_len, np.random.default_rng(kids[3])) + slots, groups = lottery.group_by_slot(ws, wn) + tree, A = build_tree_pernode(slots, groups, pl, cfg, np.random.default_rng(kids[4]), + adversary_mask=mask) + + E, T, nb = cfg.epoch_len, cfg.period_T, tree.n_blocks + ids = np.arange(nb) + arrived = (A <= E).any(axis=0) + arrived[0] = True + h = np.where(arrived, tree.height, np.iinfo(np.int64).min) + canon = np.zeros(nb, dtype=bool) + b = int(np.lexsort((-ids, -tree.slot, h))[-1]) + while b > 0: + canon[b] = True + b = int(tree.parent[b]) + canon[0] = True + in_win = (tree.slot >= 0) & (tree.slot < T) + hon_orph = in_win & ~canon & ~flat[tree.leader] + countable = hon_orph & canon[tree.parent] # first block of its fork + referenced = np.zeros(nb, dtype=bool) + for cb in np.nonzero(canon)[0]: + for un in tree.uncles[cb]: + referenced[un] = True + n, nc = int(hon_orph.sum()), int(countable.sum()) + return dict(alpha=alpha, max_uncles=u, rep=rep, honest_orphans=n, + countable_share=(nc / n) if n else np.nan, + referenced_of_countable=(int((countable & referenced).sum()) / nc) + if nc else np.nan, + referenced_of_all=(int((hon_orph & referenced).sum()) / n) if n else np.nan) + + +def decompose(reps: int = 6) -> pd.DataFrame: + jobs = [(a, u, r) for a in (0.0, 0.2, 0.3) for u in (1, 2, 4) for r in range(reps)] + df = pd.DataFrame(Parallel(n_jobs=N_JOBS, backend="loky", inner_max_num_threads=1)( + delayed(_decompose_cell)(a, u, r) for a, u, r in jobs)) + df.to_parquet(RUNS / "selfish_uncle_margin_decomp.parquet", index=False) + return df + + +def report_decomposition(df: pd.DataFrame) -> None: + print("\n=== why p_ref_honest falls: structure vs queue (delta = 8, W = 10) ===") + print(f"{'alpha':>6} {'U':>2} | {'countable share':>16} {'referenced OF those':>20}" + f" {'referenced of all':>18}") + for a in sorted(df.alpha.unique()): + for u in sorted(df.max_uncles.unique()): + g = df[(df.alpha == a) & (df.max_uncles == u)] + print(f"{a:6.2f} {u:2d} | {g.countable_share.mean() * 100:14.1f}%" + f" {g.referenced_of_countable.mean() * 100:18.1f}%" + f" {g.referenced_of_all.mean() * 100:16.1f}%") + + BAR = 0.98 # the §3.6 recovery bar, as a fraction of the true stake @@ -120,7 +196,8 @@ def main() -> None: print(f"=== selfish uncle-margin sweep ({len(ALPHAS)*len(DELAYS)*len(CAPS)*len(WINDOWS)*REPS}" f" runs; recovery bar {BAR}) ===") report(sweep()) - print(f"\nwrote {RUNS}/selfish_uncle_margin.parquet") + report_decomposition(decompose()) + print(f"\nwrote {RUNS}/selfish_uncle_margin{{,_decomp}}.parquet") if __name__ == "__main__":