logos-blockchain/research
1
0
Fork 0
mirror of https://github.com/logos-blockchain/research.git synced 2026-08-08 20:23:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add deep_orphan_share: the structural observable behind the first-fork cost

Browse Source 
E5 needs to watch whether per-recipient delay variance manufactures the
depth->=2 forks the countable rule cannot reach, and no recorded metric measured
that. p_ref conflates "unreachable by construction" with "eligible but never
picked up" -- the distinction that turned out to be the whole answer to item 5 --
and deep_ref_share is 0 by construction under the countable model, since the
proposer's candidate filter drops deep-fork blocks before any reference to one
is proposed. deep_orphan_share is the fraction of in-window orphans sitting
below the first block of their fork, computed from the depth array fork_stats
already builds.

Also fixes a splat-unpack in test_selfish_engine that silently re-bound to the
wrong quantities when fork_stats grew this field (it read deep_orphan_share as
p_ref_honest). fork_stats has now gained a field twice; both call sites unpack
by position explicitly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Marcin Pawlowski 2026-08-07 11:15:55 +02:00
parent e5a89004f2
commit c202ad9c17
No known key found for this signature in database
7 changed files with 181 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

47
tools/simulators/tsi/tsi-sim-pernode/configs/spec-point-jitter.yaml Normal file
View File

@ -0,0 +1,47 @@
# Handoff E5 — the diagnostic: does per-recipient delay variance reproduce the standalone result?
#
# This is the only experiment in the handoff that could invalidate the REPORT rather than the spec
# section. The primary hypothesis for the original discrepancy is modelling, not measurement: the
# standalone simulation drew an INDEPENDENT propagation delay per (block, recipient), while this
# simulator's blend topology runs a cascade of relays and then floods network-wide from the LAST
# relay, so nodes receive a block at nearly the same time and their views stay synchronised.
# Independent per-recipient draws maximise view divergence, which is exactly what manufactures the
# depth->=2 forks the first-fork rule cannot recover.
#
# jitter_mean adds per-(block, node) arrival noise on top of the cascade, so sweeping it
# interpolates between the two models: 0 is the report's cascade, large values approach the
# standalone's independent-delay regime. The observable that decides it is deep_orphan_share --
# the fraction of in-window orphans sitting deeper than the first block of their fork, which is
# precisely the structural quantity behind claim C2.
#
# This is also the report's open item 15 (correlated/heterogeneous latency untested).
#
# EXACT ORACLE REQUIRED: the windowed fork choice and the arrival prune are only bit-exact at
# jitter_mean == 0 (a jittered arrival can cross the deterministic horizon), so both are off. That
# forces the full (N x n_blocks) matrix and makes each run ~17 s at k = 2160.
#
# Run TWICE — default (countable) and with --old. `--old` is NOT a candidate design: under the
# current spec a block carrying a deep-fork reference is REJECTED, so it is an unreachable upper
# bound on what any counting rule could recover, and the countable-vs-ceiling gap is the
# first-fork cost. Latency is in SLOTS (1 slot = 1 s).
n_nodes: [1000] # network size
stake_dist: [pareto] # heavy-tailed (realistic) stake distribution
topology: [blend] # Blend mixnet — the deployment transport
degree: [6] # peering degree of the d-regular graph
link_latency_mean: [0.5] # natural geographic transport (sub-slot)
link_latency_dist: [geo] # real-world geographic band mixture
blend_hops: [3] # the spec's Blend cascade length
blend_delay_max: [4.0] # E1: the spec's operating point (rho ~ 0.27)
max_uncles: [0, 1, 2, 4] # 0 = negative control; 4 = the spec's MAX_UNCLES
uncle_strategy: [oldest] # spec Uncle Selection
init_dest: [common] # per-node initial D_est from agreement
replicates: 12 # exact-oracle runs are ~17 s each; 12 x 40 cells
base: # per-run settings shared by every cell
k: 2160 # true security parameter
epochs: 20 # equilibrium within ~2 epochs; burn 50%
f: 0.03333333333333333 # slot activation coefficient (1/30)
genesis_d_factor: 0.5 # start near true stake (cheap epoch 0)
early_stop: true
windowed_fork_choice: false # exact oracle: required once jitter > 0
prune_arrival: false # ...and the prune needs the same horizon
jitter_mean: 0.0 # OVERRIDDEN per run by scripts/spec_jitter.py

106
tools/simulators/tsi/tsi-sim-pernode/scripts/spec_jitter.py Normal file
View File

@ -0,0 +1,106 @@
"""E5 — does per-recipient delay variance reproduce the standalone result? (the diagnostic).
The only experiment in the fork-loss handoff that could invalidate the REPORT rather than the
spec section. The hypothesis for the original discrepancy is a modelling difference, not a
measurement one: the standalone simulation drew an independent propagation delay per
(block, recipient), whereas this simulator's Blend cascade floods network-wide from the last
relay, so nodes receive a block at nearly the same time and their views stay synchronised.
Independent per-recipient draws maximise view divergence, which is what manufactures the
depth->=2 forks the first-fork rule cannot recover.
`jitter_mean` adds per-(block, node) arrival noise on top of the cascade, so sweeping it
interpolates between the two models. The deciding observable is `deep_orphan_share`: the fraction
of in-window orphans sitting deeper than the first block of their fork — precisely the structural
quantity behind claim C2, and the thing `p_ref` conflates with "never picked up".
Pass / fail, as the handoff sets it:
* D-hat/D holds at ~1.000 and deep orphans stay negligible as jitter rises -> the standalone
model was simply wrong; C1/C2 are artefacts and the report is robust to this failure mode.
* accuracy degrades toward 0.986 and deep orphans reach ~1 % of blocks at some jitter level
-> record that level and compare it to what Blend plausibly delivers; per-recipient variance
then becomes a parameter the report must carry, and the spec section's number is defensible
under a stated assumption.
Exact oracle throughout: the windowed fork choice and the arrival prune are bit-exact only at
jitter_mean == 0, so both are disabled and the full arrival matrix is used.
Run: python scripts/spec_jitter.py (writes runs/spec_jitter.parquet)
"""
from __future__ import annotations
from pathlib import Path
import pandas as pd
from joblib import Parallel, delayed
from tsi_sim.config import SimConfig
from tsi_sim.engine import run_trajectory
HERE = Path(__file__).resolve().parent.parent
RUNS = HERE / "runs"
RUNS.mkdir(exist_ok=True)
REPS = 12
N_JOBS = 12
JITTERS = [0.0, 1.0, 2.0, 4.0, 8.0]
CAPS = [0, 1, 2, 4]
SPEC_POINT = dict(n_nodes=1000, stake_dist="pareto", topology="blend", degree=6,
link_latency_mean=0.5, link_latency_dist="geo", blend_hops=3,
blend_delay_max=4.0, uncle_strategy="oldest", window_absorption=10.0,
k=2160, epochs=20, genesis_d_factor=0.5, early_stop=True,
windowed_fork_choice=False, prune_arrival=False)
def _cell(model: str, jitter: float, u: int, rep: int) -> dict:
cfg = SimConfig(**SPEC_POINT, uncle_model=model, jitter_mean=jitter,
max_uncles=u, replicate=rep)
t = pd.DataFrame(run_trajectory(cfg))
t = t[t.epoch >= t.epoch.max() // 2]
return dict(model=model, jitter_mean=jitter, max_uncles=u, rep=rep,
mean_ratio=float(t.mean_ratio.mean()),
fork_rate=float(t.fork_rate.mean()),
deep_orphan_share=float(t.deep_orphan_share.mean()),
p_ref=float(t.p_ref.mean()),
max_reorg_depth=int(t.max_reorg_depth.max()),
range_ratio=float(t.range_ratio.max()),
agreement_window=float(t.agreement_window.min()))
def sweep() -> pd.DataFrame:
jobs = [(m, j, u, r) for m in ("countable", "old") for j in JITTERS
for u in CAPS for r in range(REPS)]
df = pd.DataFrame(Parallel(n_jobs=N_JOBS, backend="loky", inner_max_num_threads=1)(
delayed(_cell)(m, j, u, r) for m, j, u, r in jobs))
df.to_parquet(RUNS / "spec_jitter.parquet", index=False)
return df
def report(df: pd.DataFrame) -> None:
print("\n=== accuracy vs per-(block,node) jitter at the spec point (delta_max = 4) ===")
print(f"{'jitter':>7} | " + " ".join(f"U={u}" for u in CAPS)
+ f" | {'ceiling U=1':>11} {'gap':>8} {'deep orph':>10} {'fork':>6} {'consensus':>10}")
for j in JITTERS:
c = df[(df.model == "countable") & (df.jitter_mean == j)]
o = df[(df.model == "old") & (df.jitter_mean == j)]
cells = [f"{c[c.max_uncles == u].mean_ratio.mean():.4f}" for u in CAPS]
c1 = c[c.max_uncles == 1].mean_ratio.mean()
o1 = o[o.max_uncles == 1].mean_ratio.mean()
deep = c[c.max_uncles == 1].deep_orphan_share.mean()
fork = c[c.max_uncles == 1].fork_rate.mean()
ok = "exact" if c.range_ratio.max() == 0 else "SPREAD"
print(f"{j:7.1f} | " + " ".join(cells)
+ f" | {o1:11.4f} {o1 - c1:+8.4f} {deep:10.4f} {fork:6.3f} {ok:>10}")
print("\ndeep orph = share of in-window orphans below their fork's first block "
"(uncountable by construction); gap = ceiling - countable at U=1")
def main() -> None:
print(f"=== E5: jitter sweep, exact oracle, {len(JITTERS)*len(CAPS)*REPS*2} runs ===")
report(sweep())
print(f"\nwrote {RUNS}/spec_jitter.parquet")
if __name__ == "__main__":
main()

8
tools/simulators/tsi/tsi-sim-pernode/src/tsi_sim/epoch.py
View File

@ -31,6 +31,8 @@ class EpochResult:
mean_reorg_depth: float # mean maximal-orphan-branch depth
p_ref: float # emergent reference rate: in-window orphans referenced as uncles
p_ref_honest: float # ...restricted to orphans produced OUTSIDE the coalition
deep_orphan_share: float # in-window orphans deeper than their fork's first block
# (uncountable by construction, §2.1)
deep_ref_share: float # share of examined references rejected by the parent-on-chain
# (first-fork) counting rule; 0 under the old model
@ -114,7 +116,8 @@ def simulate_epoch(
attribution = coalition_mask if coalition_mask is not None else adversary_mask
adv_blocks, honest_blocks = _canonical_producer_split(tree, A, attribution, T, E)
fork_rate, max_reorg_depth, mean_reorg_depth, p_ref, p_ref_honest = fork.fork_stats(
(fork_rate, max_reorg_depth, mean_reorg_depth, p_ref, p_ref_honest,
deep_orphan_share) = fork.fork_stats(
tree, A, T, cutoff=E, coalition_mask=attribution)
ref_total = int(ms.ref_total.sum())
deep_ref_share = (int(ms.ref_deep.sum()) / ref_total) if ref_total else 0.0
@ -126,5 +129,6 @@ def simulate_epoch(
mean_orphan_rate=float(ms.orphan_rate.mean()),
adv_blocks=adv_blocks, honest_blocks=honest_blocks,
fork_rate=fork_rate, max_reorg_depth=max_reorg_depth, mean_reorg_depth=mean_reorg_depth,
p_ref=p_ref, p_ref_honest=p_ref_honest, deep_ref_share=deep_ref_share,
p_ref=p_ref, p_ref_honest=p_ref_honest, deep_orphan_share=deep_orphan_share,
deep_ref_share=deep_ref_share,
)

17
tools/simulators/tsi/tsi-sim-pernode/src/tsi_sim/fork.py
View File

@ -19,8 +19,9 @@ from .blocktree import BlockTree
def fork_stats(tree: BlockTree, A, T: int, cutoff: int,
coalition_mask=None) -> tuple[float, int, float, float, float]:
"""Return ``(fork_rate, max_reorg_depth, mean_reorg_depth, p_ref, p_ref_honest)``.
coalition_mask=None) -> tuple[float, int, float, float, float, float]:
"""Return ``(fork_rate, max_reorg_depth, mean_reorg_depth, p_ref, p_ref_honest,
deep_orphan_share)``.
``p_ref`` is the emergent **reference rate**: the fraction of in-window orphans that some
canonical block references as an uncle — the quantity the §6.8 soft-inclusion argument
@ -35,7 +36,7 @@ def fork_stats(tree: BlockTree, A, T: int, cutoff: int,
"""
nb = tree.n_blocks
if nb <= 1:
return 0.0, 0, 0.0, 1.0, 1.0
return 0.0, 0, 0.0, 1.0, 1.0, 0.0
ids = np.arange(nb)
if isinstance(A, np.ndarray):
arrived = (A <= cutoff).any(axis=0)
@ -55,7 +56,7 @@ def fork_stats(tree: BlockTree, A, T: int, cutoff: int,
in_win = (tree.slot >= 0) & (tree.slot < T)
total = int(in_win.sum())
if total == 0:
return 0.0, 0, 0.0, 1.0, 1.0
return 0.0, 0, 0.0, 1.0, 1.0, 0.0
# depth[b] = length of the non-canonical run ending at b (0 if canonical). Parent-before-child
# holds because a block\'s parent has a strictly smaller id (built earlier).
@ -89,4 +90,10 @@ def fork_stats(tree: BlockTree, A, T: int, cutoff: int,
honest_orphan = orphan_in_win & ~np.asarray(coalition_mask)[tree.leader]
n_ho = int(honest_orphan.sum())
p_ref_honest = (int((honest_orphan & referenced).sum()) / n_ho) if n_ho else 1.0
return fork_rate, max_depth, mean_depth, p_ref, p_ref_honest
# Share of in-window orphans that sit DEEPER than the first block of their fork, i.e. whose
# parent is itself off-chain. These are exactly the blocks the countable rule can never
# reference (§2.1), so this is the direct structural observable behind the first-fork cost —
# p_ref conflates it with orphans that were merely never picked up.
deep_orphan_share = (float((depth[orphan_in_win] >= 2).sum()) / n_orphan) if n_orphan else 0.0
return fork_rate, max_depth, mean_depth, p_ref, p_ref_honest, deep_orphan_share

1
tools/simulators/tsi/tsi-sim-pernode/src/tsi_sim/metrics.py
View File

@ -66,6 +66,7 @@ def divergence_row(
mean_reorg_depth=er.mean_reorg_depth,
p_ref=er.p_ref,
p_ref_honest=er.p_ref_honest,
deep_orphan_share=er.deep_orphan_share,
deep_ref_share=er.deep_ref_share,
)
return row

6
tools/simulators/tsi/tsi-sim-pernode/tests/test_fork.py
View File

@ -23,14 +23,14 @@ def make_tree(slots, parents, heights):
def test_no_forks():
# a straight chain 1->2->3, no orphans
tree = make_tree([-1, 0, 1, 2], [-1, 0, 1, 2], [0, 1, 2, 3])
fr, mx, mn, pr, _ = fork_stats(tree, None, T=10, cutoff=100)
fr, mx, mn, pr, _, _ = fork_stats(tree, None, T=10, cutoff=100)
assert fr == 0.0 and mx == 0 and mn == 0.0
def test_single_orphan_depth_one():
# canonical 1(s0),2(s1),4(s3); orphan 3(s2) hangs off block1 -> branch depth 1
tree = make_tree([-1, 0, 1, 2, 3], [-1, 0, 1, 1, 2], [0, 1, 2, 2, 3])
fr, mx, mn, pr, _ = fork_stats(tree, None, T=10, cutoff=100)
fr, mx, mn, pr, _, _ = fork_stats(tree, None, T=10, cutoff=100)
assert mx == 1
assert abs(fr - 1 / 4) < 1e-9 # 1 orphan of 4 in-window blocks
@ -39,7 +39,7 @@ def test_deep_orphan_branch():
# canonical spine 1..3 (heights 1,2,3); a 2-deep orphan branch 4->5 off block1
# blocks: 0 gen; 1(s0,h1),2(s1,h2),3(s2,h3) canonical; 4(s1,h2)->1, 5(s2,h3)->4 orphan
tree = make_tree([-1, 0, 1, 2, 1, 2], [-1, 0, 1, 2, 1, 4], [0, 1, 2, 3, 2, 3])
fr, mx, mn, pr, _ = fork_stats(tree, None, T=10, cutoff=100)
fr, mx, mn, pr, _, _ = fork_stats(tree, None, T=10, cutoff=100)
assert mx == 2 # branch 4->5 is 2 deep
assert abs(fr - 2 / 5) < 1e-9 # 2 orphans of 5

8
tools/simulators/tsi/tsi-sim-pernode/tests/test_selfish_engine.py
View File

@ -148,8 +148,12 @@ def test_uncle_counting_repairs_part_of_the_selfish_deflation():
def test_p_ref_honest_defaults_to_p_ref_without_a_coalition():
from tsi_sim.fork import fork_stats
cfg, tree, A, _ = _tree()
*_, p_ref, p_ref_h = fork_stats(tree, A, cfg.period_T, cutoff=cfg.epoch_len)
assert p_ref == p_ref_h
# Unpack by position, not with a splat: fork_stats has grown a field twice now, and a
# trailing `*_, a, b` silently re-binds to different quantities each time it does.
(_fork_rate, _max_d, _mean_d, p_ref, p_ref_honest,
deep_orphan_share) = fork_stats(tree, A, cfg.period_T, cutoff=cfg.epoch_len)
assert p_ref == p_ref_honest # no coalition -> the two coincide
assert 0.0 <= deep_orphan_share <= 1.0
@pytest.mark.parametrize("strategy", ["selfish", "withhold"])