test: add sequencer vs replayer target

.github/workflows/fuzz.yml

@@ -30,6 +30,12 @@ jobs:
           - fuzz_replay_prevention
           - fuzz_state_diff_computation
           - fuzz_validate_execute_consistency
+          - fuzz_state_serialization
+          - fuzz_witness_set_verification
+          - fuzz_program_deployment_lifecycle
+          - fuzz_apply_state_diff_split_path
+          - fuzz_multi_block_state_sequence
+          - fuzz_sequencer_vs_replayer
     steps:
       - uses: actions/checkout@v4
 
@@ -96,6 +102,12 @@ jobs:
           - fuzz_replay_prevention
           - fuzz_state_diff_computation
           - fuzz_validate_execute_consistency
+          - fuzz_state_serialization
+          - fuzz_witness_set_verification
+          - fuzz_program_deployment_lifecycle
+          - fuzz_apply_state_diff_split_path
+          - fuzz_multi_block_state_sequence
+          - fuzz_sequencer_vs_replayer
     steps:
       - uses: actions/checkout@v4
       - name: Checkout logos-execution-zone alongside lez-fuzzing
@@ -171,7 +183,13 @@ jobs:
             fuzz_signature_verification \
             fuzz_replay_prevention \
             fuzz_state_diff_computation \
-            fuzz_validate_execute_consistency; do
+            fuzz_validate_execute_consistency \
+            fuzz_state_serialization \
+            fuzz_witness_set_verification \
+            fuzz_program_deployment_lifecycle \
+            fuzz_apply_state_diff_split_path \
+            fuzz_multi_block_state_sequence \
+            fuzz_sequencer_vs_replayer; do
             echo "=== $target ===" | tee -a perf_baseline.txt
             cargo fuzz run "$target" -- -max_total_time=30 2>&1 \
               | grep -E "exec/s|execs_per_sec" | tail -1 | tee -a perf_baseline.txt

current_vs_alternative_approach.md

@@ -11,7 +11,7 @@ The `lez-fuzzing` repository is a **coverage-guided, structured mutation fuzzing
 | Rich generators | [`fuzz_props::generators`](fuzz_props/src/generators.rs) adds `proptest` strategies for pathological sequences, phantom-account attacks, overflow amounts, replay sequences |
 | Protocol invariants | [`fuzz_props::invariants`](fuzz_props/src/invariants.rs) expresses zero-mutation-on-rejection and replay-rejection as reusable `ProtocolInvariant` objects |
 | ZK-awareness | `RISC0_DEV_MODE=1` stubs out `risc0-zkvm` proofs, enabling ~5 000–200 000 exec/sec depending on target |
-| 14 dedicated targets | Covers encoding, signature verification, stateless checks, state transitions, state diffs, replay prevention, validate/execute consistency, block verification, state serialization, witness-set verification, program deployment lifecycle, split-path equivalence, multi-block sequences |
+| 15 dedicated targets | Covers encoding, signature verification, stateless checks, state transitions, state diffs, replay prevention, validate/execute consistency, block verification, state serialization, witness-set verification, program deployment lifecycle, split-path equivalence, multi-block sequences, sequencer-vs-replayer differential |
 | CI integration | GitHub Actions smoke, regression, and performance-baseline jobs run on every PR |
 | Pre-seeded corpus | Hundreds of minimised seed files in [`fuzz/corpus/`](fuzz/corpus/) ensure regressions are caught instantly |
 
@@ -82,7 +82,7 @@ The extension noted in [`docs/fuzzing.md`](docs/fuzzing.md:356) is:
 | Implementation cost | High (replayer in scope) | Low |
 | Value for protocol correctness | Very high | High |
 
-**Decision-maker view**: This is the **highest-value extension** to the current project. The `fuzz_validate_execute_consistency` target proves the pattern works. A sequencer-vs-replayer target would catch consensus-breaking state root divergence — a class of bug no single-oracle target can detect. Estimated cost: 1–2 engineer-weeks.
+**Decision-maker view**: ✅ **Implemented** as [`fuzz_sequencer_vs_replayer`](fuzz/fuzz_targets/fuzz_sequencer_vs_replayer.rs). The target feeds up to 8 transactions through the sequencer path (`validate_on_state` → `apply_state_diff`) and the replayer path (`execute_check_on_state`) with the same initial state and block context, then asserts **SequencerReplayerEquivalence** (identical balance, nonce, data, and program_owner for all known accounts), **ReplayerAcceptsAllSequencerTxs** (replayer must accept every transaction the sequencer accepted), and **ClockConsistency** (mandatory clock invocation must succeed and leave both states identical). This catches the consensus-breaking divergence class — a state root difference between sequencer and replayer — that no single-oracle target can detect.
 
 ---
 
@@ -123,7 +123,7 @@ The extension noted in [`docs/fuzzing.md`](docs/fuzzing.md:356) is:
 | AFL++ | High (different bugs) | Medium | Low | ✅ Yes | Add `just fuzz-afl` (already planned) |
 | Honggfuzz | High on Linux | Medium | Medium | ✅ Yes | Add for Linux CI only |
 | proptest-only | Low–medium | Low | ✅ Done | Already present | Keep as unit-test layer |
-| Differential (sequencer/replayer) | Very high (new bug class) | Medium | Medium–high | ✅ Yes | **Priority extension** |
+| Differential (sequencer/replayer) | Very high (new bug class) | Medium | ✅ Done | ✅ Yes | ✅ Implemented (`fuzz_sequencer_vs_replayer`) |
 | Formal verification | Exhaustive (selected invariants) | Very high | Very high | ✅ Yes | Long-term supplement |
 | Mutation testing (`cargo-mutants`) | Measures assertion quality | High | Low | ✅ Yes | Pre-audit quality gate |
 
@@ -135,9 +135,9 @@ The current implementation is **well-architected and production-ready** for a pr
 
 **Highest-ROI next steps, in priority order:**
 
-1. **The invariant framework is complete for the current target set** — three invariants are fully implemented and auto-run by [`assert_invariants()`](fuzz_props/src/invariants.rs:325): [`StateIsolationOnFailure`](fuzz_props/src/invariants.rs:60), [`BalanceConservation`](fuzz_props/src/invariants.rs:94), and [`FailedTxNonceStability`](fuzz_props/src/invariants.rs:130). Two further invariants ([`ReplayRejection`](fuzz_props/src/invariants.rs:169) and [`NonceIncrementCorrectness`](fuzz_props/src/invariants.rs:196)) are registered stubs; callers use the dedicated `assert_replay_rejection` and `assert_nonce_increment_correctness` helpers directly. The next step is to audit all 14 targets to confirm every applicable invariant is wired up, then add mutation tests via `cargo-mutants`.
+1. **The invariant framework is complete for the current target set** — three invariants are fully implemented and auto-run by [`assert_invariants()`](fuzz_props/src/invariants.rs:325): [`StateIsolationOnFailure`](fuzz_props/src/invariants.rs:60), [`BalanceConservation`](fuzz_props/src/invariants.rs:94), and [`FailedTxNonceStability`](fuzz_props/src/invariants.rs:130). Two further invariants ([`ReplayRejection`](fuzz_props/src/invariants.rs:169) and [`NonceIncrementCorrectness`](fuzz_props/src/invariants.rs:196)) are registered stubs; callers use the dedicated `assert_replay_rejection` and `assert_nonce_increment_correctness` helpers directly. The next step is to audit all 15 targets to confirm every applicable invariant is wired up, then add mutation tests via `cargo-mutants`.
 
-2. **Add the sequencer-vs-replayer differential target** — highest new bug-finding value, unique to this protocol's architecture, already identified in [`docs/fuzzing.md`](docs/fuzzing.md:356).
+2. ✅ **The sequencer-vs-replayer differential target is implemented** — [`fuzz_sequencer_vs_replayer`](fuzz/fuzz_targets/fuzz_sequencer_vs_replayer.rs) catches consensus-breaking state root divergence between the sequencer and replayer pipelines, unique to this protocol's architecture.
 
 3. **Add AFL++ as a parallel fuzzing lane** (`just fuzz-afl`) — zero corpus migration cost, discovers different mutation paths through the same targets as libFuzzer.
 

docs/fuzzing.md

@@ -77,6 +77,7 @@ just fuzz-regression
 | `fuzz_program_deployment_lifecycle` | `V03State::transition_from_program_deployment_transaction` no-panic on arbitrary WASM bytecode (**NoPanic**); **BalanceIsolation** (successful deployment must not move tokens); **StateIsolationOnFailure** (failed deployment must not change any genesis account balance or nonce) | `fuzz/fuzz_targets/fuzz_program_deployment_lifecycle.rs` |
 | `fuzz_apply_state_diff_split_path` | **SplitPathEquivalence**: for every known account, `validate_on_state` + `apply_state_diff` must produce exactly the same balance, nonce, data, and program_owner as `execute_check_on_state`; **NonceIncrementCorrectness**: nonce after the split path equals nonce after the direct path for all signer accounts (catches bugs in the two-step `apply_state_diff` nonce-increment logic) | `fuzz/fuzz_targets/fuzz_apply_state_diff_split_path.rs` |
 | `fuzz_multi_block_state_sequence` | **LongRangeBalanceConservation**: total genesis-account balance identical before and after all N (≤ 16) blocks; **FailedTxNonceStability**: every genesis-account nonce unchanged after a rejected transaction; **PerBlockReplayRejection**: every transaction accepted in block B is rejected in block B+1 (cumulative nonce-interaction coverage) | `fuzz/fuzz_targets/fuzz_multi_block_state_sequence.rs` |
+| `fuzz_sequencer_vs_replayer` | **SequencerReplayerEquivalence**: for every known account (genesis ∪ diff-declared), the sequencer path (`validate_on_state` → `apply_state_diff`) and the replayer path (`execute_check_on_state`) must produce identical balance, nonce, data, and program_owner after applying a full block of up to 8 transactions plus the mandatory clock invocation; **ReplayerAcceptsAllSequencerTxs**: every transaction accepted by `validate_on_state` must also be accepted by `execute_check_on_state`; **ClockConsistency**: the mandatory clock invocation must succeed on both paths and leave both states identical | `fuzz/fuzz_targets/fuzz_sequencer_vs_replayer.rs` |
 
 ---
 
@@ -317,6 +318,7 @@ Measured on a 4-core x86_64 Linux runner with `RISC0_DEV_MODE=1`:
 | `fuzz_program_deployment_lifecycle` | ~4 000 exec/sec *(estimate)* |
 | `fuzz_apply_state_diff_split_path` | ~5 000 exec/sec *(estimate)* |
 | `fuzz_multi_block_state_sequence` | ~1 000 exec/sec *(estimate)* |
+| `fuzz_sequencer_vs_replayer` | ~2 000 exec/sec *(estimate)* |
 
 > Throughput figures for the five new targets are rough estimates; run `just perf-baseline`
 > locally or check the `perf-baseline` CI artifact for up-to-date measurements.
@@ -353,5 +355,5 @@ flag stubs out ZK proof generation and replaces it with a fast mock implementati
 | `PrivacyPreservingTransaction` coverage | Excluded from `fuzz_encoding_roundtrip` because its ZK receipt cannot be reconstructed in a fuzzing loop. A dedicated slow target with `RISC0_DEV_MODE=1` and `proptest` should be added after the current targets are stable |
 | `fuzz_validate_execute_consistency` new-account detection | If `execute_check_on_state` creates a brand-new account absent from both the genesis set and the diff, that state-widening will not be detected — full detection requires iterating all accounts in `V03State`, which the API does not currently expose |
 | AFL++ integration | A `just fuzz-afl` recipe can be added later; the same corpus is compatible |
-| Differential testing (sequencer vs replayer) | Add a target that feeds the same block to `SequencerCore` and `indexer_core` and asserts identical state roots |
+| Differential testing (sequencer vs replayer) | ✅ Implemented — `fuzz_sequencer_vs_replayer` feeds the same block through the sequencer path (`validate_on_state` → `apply_state_diff`) and the replayer path (`execute_check_on_state`) and asserts identical state for all known accounts |
 | LEZ version tracking | There is no submodule pin — `lez-fuzzing` reads `../logos-execution-zone` as checked out. Update that repo to a release tag or a tested commit, then run `just update-lez` (which does `git pull --ff-only`) and open a PR to bump it |

fuzz/Cargo.toml

@@ -107,3 +107,9 @@ name = "fuzz_multi_block_state_sequence"
 path = "fuzz_targets/fuzz_multi_block_state_sequence.rs"
 test = false
 bench = false
+
+[[bin]]
+name = "fuzz_sequencer_vs_replayer"
+path = "fuzz_targets/fuzz_sequencer_vs_replayer.rs"
+test = false
+bench = false

fuzz/fuzz_targets/fuzz_sequencer_vs_replayer.rs

@@ -0,0 +1,222 @@
+#![no_main]
+//! Fuzz target: sequencer vs replayer differential state-root equivalence.
+//!
+//! Feeds the same block of transactions through two independent state-transition
+//! pipelines and asserts that the resulting state is bit-for-bit identical:
+//!
+//! - **Sequencer path** — mirrors `SequencerCore::build_block_from_mempool`:
+//!   for each user transaction call `validate_on_state` and, on success, call
+//!   `apply_state_diff`; skip rejected transactions.  Append the mandatory clock
+//!   invocation last via `transition_from_public_transaction`.
+//!
+//! - **Replayer path** — mirrors `IndexerStore::put_block`:
+//!   for each transaction that the sequencer accepted, call
+//!   `transaction_stateless_check` followed by `execute_check_on_state`.
+//!   Apply the clock invocation last via `transition_from_public_transaction`.
+//!
+//! Both pipelines start from the **same** fuzz-generated initial state and
+//! process the **same** set of accepted transactions with the same block context
+//! (block_id, timestamp).  Any difference in the resulting account states is a
+//! consensus-breaking bug: a replaying node would derive a different state root
+//! from the sequencer, which would invalidate all subsequent blocks.
+//!
+//! # Invariants
+//!
+//! 1. **SequencerReplayerEquivalence** — for every known account (genesis ∪
+//!    accounts declared in any accepted transaction's diff), the sequencer state
+//!    and the replayer state must agree on balance, nonce, data, and
+//!    program_owner after applying the full block.
+//!
+//! 2. **ReplayerAcceptsAllSequencerTxs** — every transaction accepted by the
+//!    sequencer (`validate_on_state` returned `Ok`) must also be accepted by the
+//!    replayer (`execute_check_on_state` returned `Ok`).  A replayer rejection of
+//!    a sequencer-accepted transaction is a validity-rule divergence bug.
+//!
+//! 3. **ClockConsistency** — the mandatory clock invocation appended at the end
+//!    of every block must succeed on both paths and leave both states identical.
+
+use std::collections::HashSet;
+
+use arbitrary::{Arbitrary, Unstructured};
+use common::transaction::{NSSATransaction, clock_invocation};
+use fuzz_props::generators::{arb_fuzz_native_transfer, arbitrary_fuzz_state, arbitrary_transaction};
+use libfuzzer_sys::fuzz_target;
+use nssa::V03State;
+
+fuzz_target!(|data: &[u8]| {
+    let mut u = Unstructured::new(data);
+
+    // ── Initial state ─────────────────────────────────────────────────────────
+    // Generate a fuzz-driven initial state so that state-dependent bugs
+    // (e.g. zero balance, u128::MAX nonce) are reachable by the fuzzer.
+    let fuzz_accs = match arbitrary_fuzz_state(&mut u) {
+        Ok(accs) => accs,
+        Err(_) => return,
+    };
+    let init_accs: Vec<(nssa::AccountId, u128)> = fuzz_accs
+        .iter()
+        .map(|a| (a.account_id, a.balance))
+        .collect();
+
+    // Fixed block context — both pipelines use identical block_id and timestamp
+    // so the only variable is the code path (sequencer vs replayer).
+    let block_id: u64 = 2; // block 1 is genesis; this is the first "real" block
+    let timestamp: u64 = 1_000;
+
+    // Shared base state — cloned once for each pipeline.
+    let base_state = V03State::new_with_genesis_accounts(&init_accs, vec![], 0);
+
+    // Track all account IDs touched by accepted transactions so we can compare
+    // them across both pipelines after the full block is applied.
+    let mut touched_ids: HashSet<nssa::AccountId> =
+        init_accs.iter().map(|&(id, _)| id).collect();
+
+    // ── Phase 1: Sequencer path ───────────────────────────────────────────────
+    // Mirrors `SequencerCore::build_block_from_mempool`:
+    //   for each mempool transaction: try validate_on_state; on success apply_state_diff.
+    let mut seq_state = base_state.clone();
+
+    // Accepted transaction list — populated here, consumed by the replayer phase
+    // so that both pipelines process exactly the same set of transactions.
+    let mut accepted_txs: Vec<NSSATransaction> = Vec::new();
+
+    let n_txs: u8 = u8::arbitrary(&mut u).unwrap_or(0) % 8;
+
+    for _ in 0..n_txs {
+        // Mix correctly-signed fuzz transfers (likely to succeed) with
+        // random structured transactions (likely to fail — stress the skip path).
+        let tx_raw = if bool::arbitrary(&mut u).unwrap_or(false) {
+            match arb_fuzz_native_transfer(&mut u, &fuzz_accs) {
+                Ok(tx) => tx,
+                Err(_) => break,
+            }
+        } else {
+            match arbitrary_transaction(&mut u) {
+                Ok(tx) => tx,
+                Err(_) => break,
+            }
+        };
+
+        // Stateless gate — both sequencer and replayer reject malformed transactions
+        // before they ever touch state.
+        let Ok(tx) = tx_raw.transaction_stateless_check() else {
+            continue;
+        };
+
+        // Sequencer: validate_on_state borrows `tx` (does not consume it).
+        let Ok(diff) = tx.validate_on_state(&seq_state, block_id, timestamp) else {
+            // Sequencer skips failed transactions; they do not appear in the block.
+            continue;
+        };
+
+        // Record the account IDs declared by this diff so they are included in
+        // the invariant check after both pipelines finish.
+        for acc_id in diff.public_diff().keys().copied() {
+            touched_ids.insert(acc_id);
+        }
+
+        // Sequencer: apply_state_diff consumes the diff and mutates seq_state.
+        seq_state.apply_state_diff(diff);
+
+        // Save the accepted transaction for the replayer phase.
+        accepted_txs.push(tx);
+    }
+
+    // Sequencer: append the mandatory clock invocation as the last transaction
+    // in the block.  If the clock fails here (e.g. corrupted initial state),
+    // the block cannot be produced — abort without a panic.
+    let clock_tx = clock_invocation(timestamp);
+    if seq_state
+        .transition_from_public_transaction(&clock_tx, block_id, timestamp)
+        .is_err()
+    {
+        return;
+    }
+
+    // ── Phase 2: Replayer path ────────────────────────────────────────────────
+    // Mirrors `IndexerStore::put_block`:
+    //   for each transaction in the block: stateless_check → execute_check_on_state.
+    let mut rep_state = base_state.clone();
+
+    for tx in &accepted_txs {
+        // Replayer: stateless check.  This must succeed because the sequencer
+        // already passed the same check above (deterministic, no state involved).
+        let Ok(checked_tx) = tx.clone().transaction_stateless_check() else {
+            // INVARIANT 2: sequencer accepted this tx but stateless check fails
+            // on the replayer.  Stateless validity is deterministic — this is a bug.
+            panic!(
+                "INVARIANT VIOLATION [ReplayerAcceptsAllSequencerTxs]: \
+                 transaction_stateless_check failed on the replayer for a \
+                 sequencer-accepted transaction (stateless check is deterministic)"
+            );
+        };
+
+        // Replayer: execute_check_on_state must succeed for every transaction
+        // the sequencer accepted (INVARIANT 2).
+        checked_tx
+            .execute_check_on_state(&mut rep_state, block_id, timestamp)
+            .unwrap_or_else(|e| {
+                panic!(
+                    "INVARIANT VIOLATION [ReplayerAcceptsAllSequencerTxs]: \
+                     execute_check_on_state rejected a sequencer-accepted \
+                     transaction on the replayer: {e:?}"
+                )
+            });
+    }
+
+    // Replayer: apply the same clock invocation (INVARIANT 3).
+    rep_state
+        .transition_from_public_transaction(&clock_tx, block_id, timestamp)
+        .unwrap_or_else(|e| {
+            panic!(
+                "INVARIANT VIOLATION [ClockConsistency]: \
+                 clock invocation succeeded on the sequencer state but failed \
+                 on the replayer state: {e:?}"
+            )
+        });
+
+    // ── Invariant 1: SequencerReplayerEquivalence ─────────────────────────────
+    // Compare every known account (genesis ∪ diff-declared) across both states.
+    // Any mismatch means the two pipelines derived a different state root —
+    // a consensus-breaking bug.
+    for acc_id in &touched_ids {
+        let seq_acc = seq_state.get_account_by_id(*acc_id);
+        let rep_acc = rep_state.get_account_by_id(*acc_id);
+
+        assert_eq!(
+            seq_acc.balance,
+            rep_acc.balance,
+            "INVARIANT VIOLATION [SequencerReplayerEquivalence]: balance diverges \
+             for account {:?} — sequencer={} replayer={}",
+            acc_id,
+            seq_acc.balance,
+            rep_acc.balance,
+        );
+
+        assert_eq!(
+            seq_acc.nonce,
+            rep_acc.nonce,
+            "INVARIANT VIOLATION [SequencerReplayerEquivalence]: nonce diverges \
+             for account {:?} — sequencer={:?} replayer={:?}",
+            acc_id,
+            seq_acc.nonce,
+            rep_acc.nonce,
+        );
+
+        assert_eq!(
+            seq_acc.data,
+            rep_acc.data,
+            "INVARIANT VIOLATION [SequencerReplayerEquivalence]: data field \
+             diverges for account {:?}",
+            acc_id,
+        );
+
+        assert_eq!(
+            seq_acc.program_owner,
+            rep_acc.program_owner,
+            "INVARIANT VIOLATION [SequencerReplayerEquivalence]: program_owner \
+             diverges for account {:?}",
+            acc_id,
+        );
+    }
+});