logos-blockchain/logos-blockchain-testing
1
0
Fork 0
mirror of https://github.com/logos-blockchain/logos-blockchain-testing.git synced 2026-01-03 05:43:09 +00:00
Code Issues Packages Projects Releases Wiki Activity
logos-blockchain-testing/docs/print.html
andrussal b44a96f392 Publish book
2025-12-20 09:51:51 +01:00

6761 lines
371 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE HTML>
<html lang="en" class="light" dir="ltr">
<head>
<!-- Book generated using mdBook -->
<meta charset="UTF-8">
<title>Logos Blockchain Testing Framework Book</title>
<meta name="robots" content="noindex">
<!-- Custom HTML head -->
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#ffffff">
<link rel="icon" href="favicon.svg">
<link rel="shortcut icon" href="favicon.png">
<link rel="stylesheet" href="css/variables.css">
<link rel="stylesheet" href="css/general.css">
<link rel="stylesheet" href="css/chrome.css">
<link rel="stylesheet" href="css/print.css" media="print">
<!-- Fonts -->
<link rel="stylesheet" href="FontAwesome/css/font-awesome.css">
<link rel="stylesheet" href="fonts/fonts.css">
<!-- Highlight.js Stylesheets -->
<link rel="stylesheet" href="highlight.css">
<link rel="stylesheet" href="tomorrow-night.css">
<link rel="stylesheet" href="ayu-highlight.css">
<!-- Custom theme stylesheets -->
</head>
<body class="sidebar-visible no-js">
<div id="body-container">
<!-- Provide site root to javascript -->
<script>
var path_to_root = "";
var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "navy" : "light";
</script>
<!-- Work around some values being stored in localStorage wrapped in quotes -->
<script>
try {
var theme = localStorage.getItem('mdbook-theme');
var sidebar = localStorage.getItem('mdbook-sidebar');
if (theme.startsWith('"') && theme.endsWith('"')) {
localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));
}
if (sidebar.startsWith('"') && sidebar.endsWith('"')) {
localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));
}
} catch (e) { }
</script>
<!-- Set the theme before any content is loaded, prevents flash -->
<script>
var theme;
try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }
if (theme === null || theme === undefined) { theme = default_theme; }
var html = document.querySelector('html');
html.classList.remove('light')
html.classList.add(theme);
var body = document.querySelector('body');
body.classList.remove('no-js')
body.classList.add('js');
</script>
<input type="checkbox" id="sidebar-toggle-anchor" class="hidden">
<!-- Hide / unhide sidebar before it is displayed -->
<script>
var body = document.querySelector('body');
var sidebar = null;
var sidebar_toggle = document.getElementById("sidebar-toggle-anchor");
if (document.body.clientWidth >= 1080) {
try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }
sidebar = sidebar || 'visible';
} else {
sidebar = 'hidden';
}
sidebar_toggle.checked = sidebar === 'visible';
body.classList.remove('sidebar-visible');
body.classList.add("sidebar-" + sidebar);
</script>
<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<div class="sidebar-scrollbox">
<ol class="chapter"><li class="chapter-item expanded "><a href="project-context-primer.html"><strong aria-hidden="true">1.</strong> Project Context Primer</a></li><li class="chapter-item expanded "><a href="what-you-will-learn.html"><strong aria-hidden="true">2.</strong> What You Will Learn</a></li><li class="chapter-item expanded "><a href="quickstart.html"><strong aria-hidden="true">3.</strong> Quickstart</a></li><li class="chapter-item expanded "><a href="part-i.html"><strong aria-hidden="true">4.</strong> Part I — Foundations</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="introduction.html"><strong aria-hidden="true">4.1.</strong> Introduction</a></li><li class="chapter-item expanded "><a href="architecture-overview.html"><strong aria-hidden="true">4.2.</strong> Architecture Overview</a></li><li class="chapter-item expanded "><a href="testing-philosophy.html"><strong aria-hidden="true">4.3.</strong> Testing Philosophy</a></li><li class="chapter-item expanded "><a href="scenario-lifecycle.html"><strong aria-hidden="true">4.4.</strong> Scenario Lifecycle</a></li><li class="chapter-item expanded "><a href="design-rationale.html"><strong aria-hidden="true">4.5.</strong> Design Rationale</a></li></ol></li><li class="chapter-item expanded "><a href="part-ii.html"><strong aria-hidden="true">5.</strong> Part II — User Guide</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="workspace-layout.html"><strong aria-hidden="true">5.1.</strong> Workspace Layout</a></li><li class="chapter-item expanded "><a href="annotated-tree.html"><strong aria-hidden="true">5.2.</strong> Annotated Tree</a></li><li class="chapter-item expanded "><a href="authoring-scenarios.html"><strong aria-hidden="true">5.3.</strong> Authoring Scenarios</a></li><li class="chapter-item expanded "><a href="workloads.html"><strong aria-hidden="true">5.4.</strong> Core Content: Workloads & Expectations</a></li><li class="chapter-item expanded "><a href="scenario-builder-ext-patterns.html"><strong aria-hidden="true">5.5.</strong> Core Content: ScenarioBuilderExt Patterns</a></li><li class="chapter-item expanded "><a href="best-practices.html"><strong aria-hidden="true">5.6.</strong> Best Practices</a></li><li class="chapter-item expanded "><a href="usage-patterns.html"><strong aria-hidden="true">5.7.</strong> Usage Patterns</a></li><li class="chapter-item expanded "><a href="examples.html"><strong aria-hidden="true">5.8.</strong> Examples</a></li><li class="chapter-item expanded "><a href="examples-advanced.html"><strong aria-hidden="true">5.9.</strong> Advanced & Artificial Examples</a></li><li class="chapter-item expanded "><a href="cucumber-bdd.html"><strong aria-hidden="true">5.10.</strong> Cucumber/BDD Interface</a></li><li class="chapter-item expanded "><a href="running-scenarios.html"><strong aria-hidden="true">5.11.</strong> Running Scenarios</a></li><li class="chapter-item expanded "><a href="runners.html"><strong aria-hidden="true">5.12.</strong> Runners</a></li><li class="chapter-item expanded "><a href="node-control.html"><strong aria-hidden="true">5.13.</strong> RunContext: BlockFeed & Node Control</a></li><li class="chapter-item expanded "><a href="chaos.html"><strong aria-hidden="true">5.14.</strong> Chaos Workloads</a></li><li class="chapter-item expanded "><a href="topology-chaos.html"><strong aria-hidden="true">5.15.</strong> Topology & Chaos Patterns</a></li></ol></li><li class="chapter-item expanded "><a href="part-iii.html"><strong aria-hidden="true">6.</strong> Part III — Developer Reference</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="scenario-model.html"><strong aria-hidden="true">6.1.</strong> Scenario Model (Developer Level)</a></li><li class="chapter-item expanded "><a href="api-levels.html"><strong aria-hidden="true">6.2.</strong> API Levels: Builder DSL vs. Direct</a></li><li class="chapter-item expanded "><a href="extending.html"><strong aria-hidden="true">6.3.</strong> Extending the Framework</a></li><li class="chapter-item expanded "><a href="custom-workload-example.html"><strong aria-hidden="true">6.4.</strong> Example: New Workload & Expectation (Rust)</a></li><li class="chapter-item expanded "><a href="internal-crate-reference.html"><strong aria-hidden="true">6.5.</strong> Internal Crate Reference</a></li></ol></li><li class="chapter-item expanded "><a href="part-iv.html"><strong aria-hidden="true">7.</strong> Part IV — Operations & Deployment</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="operations-overview.html"><strong aria-hidden="true">7.1.</strong> Overview</a></li><li class="chapter-item expanded "><a href="prerequisites.html"><strong aria-hidden="true">7.2.</strong> Prerequisites & Setup</a></li><li class="chapter-item expanded "><a href="running-examples.html"><strong aria-hidden="true">7.3.</strong> Running Examples</a></li><li class="chapter-item expanded "><a href="ci-integration.html"><strong aria-hidden="true">7.4.</strong> CI Integration</a></li><li class="chapter-item expanded "><a href="environment-variables.html"><strong aria-hidden="true">7.5.</strong> Environment Variables</a></li><li class="chapter-item expanded "><a href="logging-observability.html"><strong aria-hidden="true">7.6.</strong> Logging & Observability</a></li></ol></li><li class="chapter-item expanded "><a href="part-v.html"><strong aria-hidden="true">8.</strong> Part V — Appendix</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="dsl-cheat-sheet.html"><strong aria-hidden="true">8.1.</strong> Builder API Quick Reference</a></li><li class="chapter-item expanded "><a href="troubleshooting.html"><strong aria-hidden="true">8.2.</strong> Troubleshooting Scenarios</a></li><li class="chapter-item expanded "><a href="faq.html"><strong aria-hidden="true">8.3.</strong> FAQ</a></li><li class="chapter-item expanded "><a href="glossary.html"><strong aria-hidden="true">8.4.</strong> Glossary</a></li></ol></li></ol>
</div>
<div id="sidebar-resize-handle" class="sidebar-resize-handle">
<div class="sidebar-resize-indicator"></div>
</div>
</nav>
<!-- Track and set sidebar scroll position -->
<script>
var sidebarScrollbox = document.querySelector('#sidebar .sidebar-scrollbox');
sidebarScrollbox.addEventListener('click', function(e) {
if (e.target.tagName === 'A') {
sessionStorage.setItem('sidebar-scroll', sidebarScrollbox.scrollTop);
}
}, { passive: true });
var sidebarScrollTop = sessionStorage.getItem('sidebar-scroll');
sessionStorage.removeItem('sidebar-scroll');
if (sidebarScrollTop) {
// preserve sidebar scroll position when navigating via links within sidebar
sidebarScrollbox.scrollTop = sidebarScrollTop;
} else {
// scroll sidebar to current active section when navigating via "next/previous chapter" buttons
var activeSection = document.querySelector('#sidebar .active');
if (activeSection) {
activeSection.scrollIntoView({ block: 'center' });
}
}
</script>
<div id="page-wrapper" class="page-wrapper">
<div class="page">
<div id="menu-bar-hover-placeholder"></div>
<div id="menu-bar" class="menu-bar sticky">
<div class="left-buttons">
<label id="sidebar-toggle" class="icon-button" for="sidebar-toggle-anchor" title="Toggle Table of Contents" aria-label="Toggle Table of Contents" aria-controls="sidebar">
<i class="fa fa-bars"></i>
</label>
<button id="theme-toggle" class="icon-button" type="button" title="Change theme" aria-label="Change theme" aria-haspopup="true" aria-expanded="false" aria-controls="theme-list">
<i class="fa fa-paint-brush"></i>
</button>
<ul id="theme-list" class="theme-popup" aria-label="Themes" role="menu">
<li role="none"><button role="menuitem" class="theme" id="light">Light</button></li>
<li role="none"><button role="menuitem" class="theme" id="rust">Rust</button></li>
<li role="none"><button role="menuitem" class="theme" id="coal">Coal</button></li>
<li role="none"><button role="menuitem" class="theme" id="navy">Navy</button></li>
<li role="none"><button role="menuitem" class="theme" id="ayu">Ayu</button></li>
</ul>
<button id="search-toggle" class="icon-button" type="button" title="Search. (Shortkey: s)" aria-label="Toggle Searchbar" aria-expanded="false" aria-keyshortcuts="S" aria-controls="searchbar">
<i class="fa fa-search"></i>
</button>
</div>
<h1 class="menu-title">Logos Blockchain Testing Framework Book</h1>
<div class="right-buttons">
<a href="print.html" title="Print this book" aria-label="Print this book">
<i id="print-button" class="fa fa-print"></i>
</a>
</div>
</div>
<div id="search-wrapper" class="hidden">
<form id="searchbar-outer" class="searchbar-outer">
<input type="search" id="searchbar" name="searchbar" placeholder="Search this book ..." aria-controls="searchresults-outer" aria-describedby="searchresults-header">
</form>
<div id="searchresults-outer" class="searchresults-outer hidden">
<div id="searchresults-header" class="searchresults-header"></div>
<ul id="searchresults">
</ul>
</div>
</div>
<!-- Apply ARIA attributes after the sidebar and the sidebar toggle button are added to the DOM -->
<script>
document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');
document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');
Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {
link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);
});
</script>
<div id="content" class="content">
<main>
<h1 id="logos-testing-framework"><a class="header" href="#logos-testing-framework">Logos Testing Framework</a></h1>
<p><strong>Declarative, multi-node blockchain testing for the Logos network</strong></p>
<p>The Logos Testing Framework enables you to test consensus, data availability, and transaction workloads across local processes, Docker Compose, and Kubernetes deployments—all with a unified scenario API.</p>
<p><a href="quickstart.html"><strong>Get Started</strong></a></p>
<hr />
<h2 id="core-concept"><a class="header" href="#core-concept">Core Concept</a></h2>
<p><strong>Everything in this framework is a Scenario.</strong></p>
<p>A Scenario is a controlled experiment over time, composed of:</p>
<ul>
<li><strong>Topology</strong> — The cluster shape (validators, executors, network layout)</li>
<li><strong>Workloads</strong> — Traffic and conditions that exercise the system (transactions, DA, chaos)</li>
<li><strong>Expectations</strong> — Success criteria verified after execution (liveness, inclusion, recovery)</li>
<li><strong>Duration</strong> — The time window for the experiment</li>
</ul>
<p>This single abstraction makes tests declarative, portable, and composable.</p>
<hr />
<h2 id="how-it-works"><a class="header" href="#how-it-works">How It Works</a></h2>
<pre><code class="language-mermaid">flowchart LR
Build[Define Scenario] --&gt; Deploy[Deploy Topology]
Deploy --&gt; Execute[Run Workloads]
Execute --&gt; Evaluate[Check Expectations]
style Build fill:#e1f5ff
style Deploy fill:#fff4e1
style Execute fill:#ffe1f5
style Evaluate fill:#e1ffe1
</code></pre>
<ol>
<li><strong>Define Scenario</strong> — Describe your test: topology, workloads, and success criteria</li>
<li><strong>Deploy Topology</strong> — Launch validators and executors using host, compose, or k8s runners</li>
<li><strong>Run Workloads</strong> — Drive transactions, DA traffic, and chaos operations</li>
<li><strong>Check Expectations</strong> — Verify consensus liveness, inclusion, and system health</li>
</ol>
<hr />
<h2 id="key-features"><a class="header" href="#key-features">Key Features</a></h2>
<p><strong>Declarative API</strong></p>
<ul>
<li>Express scenarios as topology + workloads + expectations</li>
<li>Reuse the same test definition across different deployment targets</li>
<li>Compose complex tests from modular components</li>
</ul>
<p><strong>Multiple Deployment Modes</strong></p>
<ul>
<li><strong>Host Runner</strong>: Local processes for fast iteration</li>
<li><strong>Compose Runner</strong>: Containerized environments with node control</li>
<li><strong>Kubernetes Runner</strong>: Production-like cluster testing</li>
</ul>
<p><strong>Built-in Workloads</strong></p>
<ul>
<li>Transaction submission with configurable rates</li>
<li>Data availability (DA) blob dispersal and sampling</li>
<li>Chaos testing with controlled node restarts</li>
</ul>
<p><strong>Comprehensive Observability</strong></p>
<ul>
<li>Real-time block feed for monitoring consensus progress</li>
<li>Prometheus/Grafana integration for metrics</li>
<li>Per-node log collection and debugging</li>
</ul>
<hr />
<h2 id="quick-example"><a class="header" href="#quick-example">Quick Example</a></h2>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_core::scenario::Deployer as _;
use testing_framework_runner_local::LocalDeployer;
use testing_framework_workflows::ScenarioBuilderExt;
#[tokio::main]
async fn main() -&gt; anyhow::Result&lt;()&gt; {
let mut scenario = ScenarioBuilder::topology_with(|t| {
t.network_star()
.validators(3)
.executors(1)
})
.transactions_with(|tx| tx.rate(10).users(5))
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(60))
.build();
let deployer = LocalDeployer::default();
let runner = deployer.deploy(&amp;scenario).await?;
runner.run(&amp;mut scenario).await?;
Ok(())
}</code></pre>
<p><a href="examples.html">View complete examples</a></p>
<hr />
<h2 id="choose-your-path"><a class="header" href="#choose-your-path">Choose Your Path</a></h2>
<h3 id="new-to-the-framework"><a class="header" href="#new-to-the-framework">New to the Framework?</a></h3>
<p>Start with the <strong><a href="quickstart.html">Quickstart Guide</a></strong> for a hands-on introduction that gets you running tests in minutes.</p>
<h3 id="ready-to-write-tests"><a class="header" href="#ready-to-write-tests">Ready to Write Tests?</a></h3>
<p>Explore the <strong><a href="part-ii.html">User Guide</a></strong> to learn about authoring scenarios, workloads, expectations, and deployment strategies.</p>
<h3 id="setting-up-cicd"><a class="header" href="#setting-up-cicd">Setting Up CI/CD?</a></h3>
<p>Jump to <strong><a href="part-v.html">Operations &amp; Deployment</a></strong> for prerequisites, environment configuration, and continuous integration patterns.</p>
<h3 id="extending-the-framework"><a class="header" href="#extending-the-framework">Extending the Framework?</a></h3>
<p>Check the <strong><a href="part-iii.html">Developer Reference</a></strong> to implement custom workloads, expectations, and runners.</p>
<hr />
<h2 id="project-context"><a class="header" href="#project-context">Project Context</a></h2>
<p><strong>Logos</strong> is a modular blockchain protocol composed of validators, executors, and a data-availability (DA) subsystem:</p>
<ul>
<li><strong>Validators</strong> participate in consensus and produce blocks</li>
<li><strong>Executors</strong> are validators with the DA dispersal service enabled. They perform all validator functions plus submit blob data to the DA network</li>
<li><strong>Data Availability (DA)</strong> ensures that blob data submitted via channel operations in transactions is published and retrievable by the network</li>
</ul>
<p>These roles interact tightly, which is why meaningful testing must be performed in multi-node environments that include real networking, timing, and DA interaction.</p>
<p>The Logos Testing Framework provides the infrastructure to orchestrate these multi-node scenarios reliably across development, CI, and production-like environments.</p>
<p><strong>Learn more about the protocol:</strong> <a href="https://nomos-tech.notion.site/project">Logos Project Documentation</a></p>
<hr />
<h2 id="documentation-structure"><a class="header" href="#documentation-structure">Documentation Structure</a></h2>
<div class="table-wrapper"><table><thead><tr><th>Section</th><th>Description</th></tr></thead><tbody>
<tr><td><strong><a href="part-i.html">Foundations</a></strong></td><td>Architecture, philosophy, and design principles</td></tr>
<tr><td><strong><a href="part-ii.html">User Guide</a></strong></td><td>Writing and running scenarios, workloads, and expectations</td></tr>
<tr><td><strong><a href="part-iii.html">Developer Reference</a></strong></td><td>Extending the framework with custom components</td></tr>
<tr><td><strong><a href="part-iv.html">Operations &amp; Deployment</a></strong></td><td>Setup, CI integration, and environment configuration</td></tr>
<tr><td><strong><a href="part-v.html">Appendix</a></strong></td><td>Quick reference, troubleshooting, FAQ, and glossary</td></tr>
</tbody></table>
</div>
<hr />
<h2 id="quick-links"><a class="header" href="#quick-links">Quick Links</a></h2>
<ul>
<li><strong><a href="what-you-will-learn.html">What You Will Learn</a></strong> — Overview of book contents and learning path</li>
<li><strong><a href="quickstart.html">Quickstart</a></strong> — Get up and running in 10 minutes</li>
<li><strong><a href="examples.html">Examples</a></strong> — Concrete scenario patterns</li>
<li><strong><a href="troubleshooting.html">Troubleshooting</a></strong> — Common issues and solutions</li>
<li><strong><a href="environment-variables.html">Environment Variables</a></strong> — Complete configuration reference</li>
</ul>
<hr />
<p><strong>Ready to start?</strong> Head to the <strong><a href="quickstart.html">Quickstart</a></strong></p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="what-you-will-learn"><a class="header" href="#what-you-will-learn">What You Will Learn</a></h1>
<p>This book gives you a clear mental model for Logos multi-node testing, shows how
to author scenarios that pair realistic workloads with explicit expectations,
and guides you to run them across local, containerized, and cluster environments
without changing the plan.</p>
<h2 id="by-the-end-of-this-book-you-will-be-able-to"><a class="header" href="#by-the-end-of-this-book-you-will-be-able-to">By the End of This Book, You Will Be Able To:</a></h2>
<p><strong>Understand the Framework</strong></p>
<ul>
<li>Explain the six-phase scenario lifecycle (Build, Deploy, Capture, Execute, Evaluate, Cleanup)</li>
<li>Describe how Deployers, Runners, Workloads, and Expectations work together</li>
<li>Navigate the crate architecture and identify extension points</li>
<li>Understand when to use each runner (Host, Compose, Kubernetes)</li>
</ul>
<p><strong>Author and Run Scenarios</strong></p>
<ul>
<li>Define multi-node topologies with validators and executors</li>
<li>Configure transaction and DA workloads with appropriate rates</li>
<li>Add consensus liveness and inclusion expectations</li>
<li>Run scenarios across all three deployment modes</li>
<li>Use BlockFeed to monitor block production in real-time</li>
<li>Implement chaos testing with node restarts</li>
</ul>
<p><strong>Operate in Production</strong></p>
<ul>
<li>Set up prerequisites and dependencies correctly</li>
<li>Configure environment variables for different runners</li>
<li>Integrate tests into CI/CD pipelines (GitHub Actions)</li>
<li>Troubleshoot common failure scenarios</li>
<li>Collect and analyze logs from multi-node runs</li>
<li>Optimize test durations and resource usage</li>
</ul>
<p><strong>Extend the Framework</strong></p>
<ul>
<li>Implement custom Workload traits for new traffic patterns</li>
<li>Create custom Expectation traits for domain-specific checks</li>
<li>Add new Deployer implementations for different backends</li>
<li>Contribute topology helpers and DSL extensions</li>
</ul>
<h2 id="learning-path"><a class="header" href="#learning-path">Learning Path</a></h2>
<p><strong>Beginner</strong> (0-2 hours)</p>
<ul>
<li>Read <a href="quickstart.html">Quickstart</a> and run your first scenario</li>
<li>Review <a href="examples.html">Examples</a> to see common patterns</li>
<li>Understand <a href="scenario-lifecycle.html">Scenario Lifecycle</a> phases</li>
</ul>
<p><strong>Intermediate</strong> (2-8 hours)</p>
<ul>
<li>Study <a href="runners.html">Runners</a> comparison and choose appropriate mode</li>
<li>Learn <a href="workloads.html">Workloads &amp; Expectations</a> in depth</li>
<li>Review <a href="prerequisites.html">Prerequisites &amp; Setup</a> for your environment</li>
<li>Practice with <a href="examples-advanced.html">Advanced Examples</a></li>
</ul>
<p><strong>Advanced</strong> (8+ hours)</p>
<ul>
<li>Master <a href="environment-variables.html">Environment Variables</a> configuration</li>
<li>Implement <a href="extending.html">Custom Workloads</a> for your use cases</li>
<li>Set up <a href="ci-integration.html">CI Integration</a> for automated testing</li>
<li>Explore <a href="internal-crate-reference.html">Internal Crate Reference</a> for deep dives</li>
</ul>
<h2 id="what-this-book-does-not-cover"><a class="header" href="#what-this-book-does-not-cover">What This Book Does NOT Cover</a></h2>
<ul>
<li><strong>Logos node internals</strong> — This book focuses on testing infrastructure, not the blockchain protocol implementation. See the Logos node repository (<code>nomos-node</code>) for protocol documentation.</li>
<li><strong>Consensus algorithm theory</strong> — We assume familiarity with basic blockchain concepts (validators, blocks, transactions, data availability).</li>
<li><strong>Rust language basics</strong> — Examples use Rust, but we don't teach the language. See <a href="https://doc.rust-lang.org/book/">The Rust Book</a> if you're new to Rust.</li>
<li><strong>Kubernetes administration</strong> — We show how to use the K8s runner, but don't cover cluster setup, networking, or operations.</li>
<li><strong>Docker fundamentals</strong> — We assume basic Docker/Compose knowledge for the Compose runner.</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="quickstart"><a class="header" href="#quickstart">Quickstart</a></h1>
<p>Get a working example running quickly.</p>
<h2 id="prerequisites"><a class="header" href="#prerequisites">Prerequisites</a></h2>
<ul>
<li>Rust toolchain (nightly)</li>
<li>This repository cloned</li>
<li>Unix-like system (tested on Linux and macOS)</li>
<li>For Docker Compose examples: Docker daemon running</li>
<li>For Docker Desktop on Apple silicon (compose/k8s): set <code>NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64</code> to avoid slow/fragile amd64 emulation builds</li>
<li><strong><code>versions.env</code> file</strong> at repository root (defines VERSION, NOMOS_NODE_REV, NOMOS_BUNDLE_VERSION)</li>
</ul>
<p><strong>Note:</strong> <code>nomos-node</code> binaries are built automatically on demand or can be provided via prebuilt bundles.</p>
<p><strong>Important:</strong> The <code>versions.env</code> file is required by helper scripts. If missing, the scripts will fail with an error. The file should already exist in the repository root.</p>
<h2 id="your-first-test"><a class="header" href="#your-first-test">Your First Test</a></h2>
<p>The framework ships with runnable example binaries in <code>examples/src/bin/</code>.</p>
<p><strong>Recommended:</strong> Use the convenience script:</p>
<pre><code class="language-bash"># From the logos-blockchain-testing directory
scripts/run/run-examples.sh -t 60 -v 1 -e 1 host
</code></pre>
<p>This handles circuit setup, binary building, and runs a complete scenario: 1 validator + 1 executor, mixed transaction + DA workload (5 tx/block + 1 channel + 1 blob), 60s duration.</p>
<p><strong>Alternative:</strong> Direct cargo run (requires manual setup):</p>
<pre><code class="language-bash"># Requires circuits in place and NOMOS_NODE_BIN/NOMOS_EXECUTOR_BIN set
POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
</code></pre>
<p><strong>Core API Pattern</strong> (simplified example):</p>
<pre><code class="language-rust ignore">use std::time::Duration;
use anyhow::Result;
use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
use testing_framework_runner_local::LocalDeployer;
use testing_framework_workflows::ScenarioBuilderExt;
pub async fn run_local_demo() -&gt; Result&lt;()&gt; {
// Define the scenario (1 validator + 1 executor, tx + DA workload)
let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(1))
.wallets(1_000)
.transactions_with(|txs| {
txs.rate(5) // 5 transactions per block
.users(500) // use 500 of the seeded wallets
})
.da_with(|da| {
da.channel_rate(1) // 1 channel
.blob_rate(1) // target 1 blob per block
.headroom_percent(20) // default headroom when sizing channels
})
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(60))
.build();
// Deploy and run
let deployer = LocalDeployer::default();
let runner = deployer.deploy(&amp;plan).await?;
let _handle = runner.run(&amp;mut plan).await?;
Ok(())
}</code></pre>
<p><strong>Note:</strong> The examples are binaries with <code>#[tokio::main]</code>, not test functions. If you want to write integration tests, wrap this pattern in <code>#[tokio::test]</code> functions in your own test suite.</p>
<p><strong>Important:</strong> <code>POL_PROOF_DEV_MODE=true</code> disables expensive Groth16 zero-knowledge proof generation for leader election. Without it, proof generation is CPU-intensive and tests will timeout. <strong>This is required for all runners</strong> (local, compose, k8s) for practical testing. Never use in production.</p>
<p><strong>What you should see:</strong></p>
<ul>
<li>Nodes spawn as local processes</li>
<li>Consensus starts producing blocks</li>
<li>Scenario runs for the configured duration</li>
<li>Node state/logs written under a temporary per-run directory in the current working directory (removed after the run unless <code>NOMOS_TESTS_KEEP_LOGS=1</code>)</li>
<li>To write per-node log files to a stable location: set <code>NOMOS_LOG_DIR=/path/to/logs</code> (files will have prefix like <code>nomos-node-0*</code>, may include timestamps)</li>
</ul>
<h2 id="what-just-happened"><a class="header" href="#what-just-happened">What Just Happened?</a></h2>
<p>Let's unpack the code:</p>
<h3 id="1-topology-configuration"><a class="header" href="#1-topology-configuration">1. Topology Configuration</a></h3>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::ScenarioBuilder;
pub fn step_1_topology() -&gt; testing_framework_core::scenario::Builder&lt;()&gt; {
ScenarioBuilder::topology_with(|t| {
t.network_star() // Star topology: all nodes connect to seed
.validators(1) // 1 validator node
.executors(1) // 1 executor node (validator + DA dispersal)
})
}</code></pre>
<p>This defines <strong>what</strong> your test network looks like.</p>
<h3 id="2-wallet-seeding"><a class="header" href="#2-wallet-seeding">2. Wallet Seeding</a></h3>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn step_2_wallets() -&gt; testing_framework_core::scenario::Builder&lt;()&gt; {
ScenarioBuilder::with_node_counts(1, 1).wallets(1_000) // Seed 1,000 funded wallet accounts
}</code></pre>
<p>Provides funded accounts for transaction submission.</p>
<h3 id="3-workloads"><a class="header" href="#3-workloads">3. Workloads</a></h3>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn step_3_workloads() -&gt; testing_framework_core::scenario::Builder&lt;()&gt; {
ScenarioBuilder::with_node_counts(1, 1)
.wallets(1_000)
.transactions_with(|txs| {
txs.rate(5) // 5 transactions per block
.users(500) // Use 500 of the 1,000 wallets
})
.da_with(|da| {
da.channel_rate(1) // 1 DA channel (more spawned with headroom)
.blob_rate(1) // target 1 blob per block
.headroom_percent(20) // default headroom when sizing channels
})
}</code></pre>
<p>Generates both transaction and DA traffic to stress both subsystems.</p>
<h3 id="4-expectation"><a class="header" href="#4-expectation">4. Expectation</a></h3>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn step_4_expectation() -&gt; testing_framework_core::scenario::Builder&lt;()&gt; {
ScenarioBuilder::with_node_counts(1, 1).expect_consensus_liveness() // This says what success means: blocks must be produced continuously.
}</code></pre>
<p>This says <strong>what success means</strong>: blocks must be produced continuously.</p>
<h3 id="5-run-duration"><a class="header" href="#5-run-duration">5. Run Duration</a></h3>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::ScenarioBuilder;
pub fn step_5_run_duration() -&gt; testing_framework_core::scenario::Builder&lt;()&gt; {
ScenarioBuilder::with_node_counts(1, 1).with_run_duration(Duration::from_secs(60))
}</code></pre>
<p>Run for 60 seconds (~27 blocks with default 2s slots, 0.9 coefficient). Framework ensures this is at least 2× the consensus slot duration. Adjust consensus timing via <code>CONSENSUS_SLOT_TIME</code> and <code>CONSENSUS_ACTIVE_SLOT_COEFF</code>.</p>
<h3 id="6-deploy-and-execute"><a class="header" href="#6-deploy-and-execute">6. Deploy and Execute</a></h3>
<pre><code class="language-rust ignore">use anyhow::Result;
use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
use testing_framework_runner_local::LocalDeployer;
pub async fn step_6_deploy_and_execute() -&gt; Result&lt;()&gt; {
let mut plan = ScenarioBuilder::with_node_counts(1, 1).build();
let deployer = LocalDeployer::default(); // Use local process deployer
let runner = deployer.deploy(&amp;plan).await?; // Provision infrastructure
let _handle = runner.run(&amp;mut plan).await?; // Execute workloads &amp; expectations
Ok(())
}</code></pre>
<p><strong>Deployer</strong> provisions the infrastructure. <strong>Runner</strong> orchestrates execution.</p>
<h2 id="adjust-the-topology"><a class="header" href="#adjust-the-topology">Adjust the Topology</a></h2>
<p><strong>With run-examples.sh</strong> (recommended):</p>
<pre><code class="language-bash"># Scale up to 3 validators + 2 executors, run for 2 minutes
scripts/run/run-examples.sh -t 120 -v 3 -e 2 host
</code></pre>
<p><strong>With direct cargo run:</strong></p>
<pre><code class="language-bash"># Uses NOMOS_DEMO_* env vars (or legacy *_DEMO_* vars)
NOMOS_DEMO_VALIDATORS=3 \
NOMOS_DEMO_EXECUTORS=2 \
NOMOS_DEMO_RUN_SECS=120 \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin local_runner
</code></pre>
<h2 id="try-docker-compose"><a class="header" href="#try-docker-compose">Try Docker Compose</a></h2>
<p>Use the same API with a different deployer for reproducible containerized environment.</p>
<p><strong>Recommended:</strong> Use the convenience script (handles everything):</p>
<pre><code class="language-bash">scripts/run/run-examples.sh -t 60 -v 1 -e 1 compose
</code></pre>
<p>This automatically:</p>
<ul>
<li>Fetches circuit assets (to <code>testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params</code>)</li>
<li>Builds/uses prebuilt binaries (via <code>NOMOS_BINARIES_TAR</code> if available)</li>
<li>Builds the Docker image</li>
<li>Runs the compose scenario</li>
</ul>
<p><strong>Alternative:</strong> Direct cargo run with manual setup:</p>
<pre><code class="language-bash"># Option 1: Use prebuilt bundle (recommended for compose/k8s)
scripts/build/build-bundle.sh --platform linux # Creates .tmp/nomos-binaries-linux-v0.3.1.tar.gz
export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
# Option 2: Manual circuit/image setup (rebuilds during image build)
scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits
cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/
scripts/build/build_test_image.sh
# Run with Compose
NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin compose_runner
</code></pre>
<p><strong>Benefit:</strong> Reproducible containerized environment (Dockerized nodes, repeatable deployments).</p>
<p><strong>Optional: Prometheus + Grafana</strong></p>
<p>The runner can integrate with external observability endpoints. For a ready-to-run local stack:</p>
<pre><code class="language-bash">scripts/setup/setup-observability.sh compose up
eval "$(scripts/setup/setup-observability.sh compose env)"
</code></pre>
<p>Then run your compose scenario as usual (the environment variables enable PromQL querying and node OTLP metrics export).</p>
<p><strong>Note:</strong> Compose expects KZG parameters at <code>/kzgrs_test_params/kzgrs_test_params</code> inside containers (the directory name is repeated as the filename).</p>
<p><strong>In code:</strong> Just swap the deployer:</p>
<pre><code class="language-rust ignore">use anyhow::Result;
use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
use testing_framework_runner_compose::ComposeDeployer;
pub async fn run_with_compose_deployer() -&gt; Result&lt;()&gt; {
// ... same scenario definition ...
let mut plan = ScenarioBuilder::with_node_counts(1, 1).build();
let deployer = ComposeDeployer::default(); // Use Docker Compose
let runner = deployer.deploy(&amp;plan).await?;
let _handle = runner.run(&amp;mut plan).await?;
Ok(())
}</code></pre>
<h2 id="next-steps"><a class="header" href="#next-steps">Next Steps</a></h2>
<p>Now that you have a working test:</p>
<ul>
<li><strong>Understand the philosophy</strong>: <a href="testing-philosophy.html">Testing Philosophy</a></li>
<li><strong>Learn the architecture</strong>: <a href="architecture-overview.html">Architecture Overview</a></li>
<li><strong>See more examples</strong>: <a href="examples.html">Examples</a></li>
<li><strong>API reference</strong>: <a href="dsl-cheat-sheet.html">Builder API Quick Reference</a></li>
<li><strong>Debug failures</strong>: <a href="troubleshooting.html">Troubleshooting</a></li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="part-i--foundations"><a class="header" href="#part-i--foundations">Part I — Foundations</a></h1>
<p>Conceptual chapters that establish the mental model for the framework and how
it approaches multi-node testing.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="introduction"><a class="header" href="#introduction">Introduction</a></h1>
<p>The Logos Testing Framework is a purpose-built toolkit for exercising Logos in
realistic, multi-node environments. It solves the gap between small, isolated
tests and full-system validation by letting teams describe a cluster layout,
drive meaningful traffic, and assert the outcomes in one coherent plan.</p>
<p>It is for protocol engineers, infrastructure operators, and QA teams who need
repeatable confidence that validators, executors, and data-availability
components work together under network and timing constraints.</p>
<p>Multi-node integration testing is required because many Logos behaviors—block
progress, data availability, liveness under churn—only emerge when several
roles interact over real networking and time. This framework makes those checks
declarative, observable, and portable across environments.</p>
<h2 id="a-scenario-in-20-lines"><a class="header" href="#a-scenario-in-20-lines">A Scenario in 20 Lines</a></h2>
<p>Here's the conceptual shape of every test you'll write:</p>
<pre><code class="language-rust ignore">// 1. Define the cluster
let scenario = ScenarioBuilder::topology_with(|t| {
t.network_star()
.validators(3)
.executors(2)
})
// 2. Add workloads (traffic)
.transactions_with(|tx| tx.rate(10).users(5))
.da_with(|da| da.channel_rate(2).blob_rate(2))
// 3. Define success criteria
.expect_consensus_liveness()
// 4. Set experiment duration
.with_run_duration(Duration::from_secs(60))
.build();
// 5. Deploy and run
let runner = deployer.deploy(&amp;scenario).await?;
runner.run(&amp;mut scenario).await?;</code></pre>
<p>This pattern—topology, workloads, expectations, duration—repeats across all scenarios in this book.</p>
<p><strong>Learn more:</strong> For protocol-level documentation and node internals, see the <a href="https://nomos-tech.notion.site/project">Logos Project Documentation</a>.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="architecture-overview"><a class="header" href="#architecture-overview">Architecture Overview</a></h1>
<p>The framework follows a clear flow: <strong>Topology → Scenario → Deployer → Runner → Workloads → Expectations</strong>.</p>
<h2 id="core-flow"><a class="header" href="#core-flow">Core Flow</a></h2>
<pre><code class="language-mermaid">flowchart LR
A(Topology&lt;br/&gt;shape cluster) --&gt; B(Scenario&lt;br/&gt;plan)
B --&gt; C(Deployer&lt;br/&gt;provision &amp; readiness)
C --&gt; D(Runner&lt;br/&gt;orchestrate execution)
D --&gt; E(Workloads&lt;br/&gt;drive traffic)
E --&gt; F(Expectations&lt;br/&gt;verify outcomes)
</code></pre>
<h2 id="crate-architecture"><a class="header" href="#crate-architecture">Crate Architecture</a></h2>
<pre><code class="language-mermaid">flowchart TB
subgraph Examples["Runner Examples"]
LocalBin[local_runner.rs]
ComposeBin[compose_runner.rs]
K8sBin[k8s_runner.rs]
CucumberBin[cucumber_*.rs]
end
subgraph Workflows["Workflows (Batteries Included)"]
DSL[ScenarioBuilderExt&lt;br/&gt;Fluent API]
TxWorkload[Transaction Workload]
DAWorkload[DA Workload]
ChaosWorkload[Chaos Workload]
Expectations[Built-in Expectations]
end
subgraph Core["Core Framework"]
ScenarioModel[Scenario Model]
Traits[Deployer + Runner Traits]
BlockFeed[BlockFeed]
NodeClients[Node Clients]
Topology[Topology Generation]
end
subgraph Deployers["Runner Implementations"]
LocalDeployer[LocalDeployer]
ComposeDeployer[ComposeDeployer]
K8sDeployer[K8sDeployer]
end
subgraph Support["Supporting Crates"]
Configs[Configs &amp; Topology]
Nodes[Node API Clients]
Cucumber[Cucumber Extensions]
end
Examples --&gt; Workflows
Examples --&gt; Deployers
Workflows --&gt; Core
Deployers --&gt; Core
Deployers --&gt; Support
Core --&gt; Support
Workflows --&gt; Support
style Examples fill:#e1f5ff
style Workflows fill:#e1ffe1
style Core fill:#fff4e1
style Deployers fill:#ffe1f5
style Support fill:#f0f0f0
</code></pre>
<h3 id="layer-responsibilities"><a class="header" href="#layer-responsibilities">Layer Responsibilities</a></h3>
<p><strong>Runner Examples (Entry Points)</strong></p>
<ul>
<li>Executable binaries that demonstrate framework usage</li>
<li>Wire together deployers, scenarios, and execution</li>
<li>Provide CLI interfaces for different modes</li>
</ul>
<p><strong>Workflows (High-Level API)</strong></p>
<ul>
<li><code>ScenarioBuilderExt</code> trait provides fluent DSL</li>
<li>Built-in workloads (transactions, DA, chaos)</li>
<li>Common expectations (liveness, inclusion)</li>
<li>Simplifies scenario authoring</li>
</ul>
<p><strong>Core Framework (Foundation)</strong></p>
<ul>
<li><code>Scenario</code> model and lifecycle orchestration</li>
<li><code>Deployer</code> and <code>Runner</code> traits (extension points)</li>
<li><code>BlockFeed</code> for real-time block observation</li>
<li><code>RunContext</code> providing node clients and metrics</li>
<li>Topology generation and validation</li>
</ul>
<p><strong>Runner Implementations</strong></p>
<ul>
<li><code>LocalDeployer</code> - spawns processes on host</li>
<li><code>ComposeDeployer</code> - orchestrates Docker Compose</li>
<li><code>K8sDeployer</code> - deploys to Kubernetes cluster</li>
<li>Each implements <code>Deployer</code> trait</li>
</ul>
<p><strong>Supporting Crates</strong></p>
<ul>
<li><code>configs</code> - Topology configuration and generation</li>
<li><code>nodes</code> - HTTP/RPC client for node APIs</li>
<li><code>cucumber</code> - BDD/Gherkin integration</li>
</ul>
<h3 id="extension-points"><a class="header" href="#extension-points">Extension Points</a></h3>
<pre><code class="language-mermaid">flowchart LR
Custom[Your Code] -.implements.-&gt; Workload[Workload Trait]
Custom -.implements.-&gt; Expectation[Expectation Trait]
Custom -.implements.-&gt; Deployer[Deployer Trait]
Workload --&gt; Core[Core Framework]
Expectation --&gt; Core
Deployer --&gt; Core
style Custom fill:#ffe1f5
style Core fill:#fff4e1
</code></pre>
<p><strong>Extend by implementing:</strong></p>
<ul>
<li><code>Workload</code> - Custom traffic generation patterns</li>
<li><code>Expectation</code> - Custom success criteria</li>
<li><code>Deployer</code> - Support for new deployment targets</li>
</ul>
<p>See <a href="extending.html">Extending the Framework</a> for details.</p>
<h3 id="components"><a class="header" href="#components">Components</a></h3>
<ul>
<li><strong>Topology</strong> describes the cluster: how many nodes, their roles, and the high-level network and data-availability parameters they should follow.</li>
<li><strong>Scenario</strong> combines that topology with the activities to run and the checks to perform, forming a single plan.</li>
<li><strong>Deployer</strong> provisions infrastructure on the chosen backend (local processes, Docker Compose, or Kubernetes), waits for readiness, and returns a Runner.</li>
<li><strong>Runner</strong> orchestrates scenario execution: starts workloads, observes signals, evaluates expectations, and triggers cleanup.</li>
<li><strong>Workloads</strong> generate traffic and conditions that exercise the system.</li>
<li><strong>Expectations</strong> observe the run and judge success or failure once activity completes.</li>
</ul>
<p>Each layer has a narrow responsibility so that cluster shape, deployment choice,
traffic generation, and health checks can evolve independently while fitting
together predictably.</p>
<h2 id="entry-points"><a class="header" href="#entry-points">Entry Points</a></h2>
<p>The framework is consumed via <strong>runnable example binaries</strong> in <code>examples/src/bin/</code>:</p>
<ul>
<li><code>local_runner.rs</code> — Spawns nodes as host processes</li>
<li><code>compose_runner.rs</code> — Deploys via Docker Compose (requires <code>NOMOS_TESTNET_IMAGE</code> built)</li>
<li><code>k8s_runner.rs</code> — Deploys via Kubernetes Helm (requires cluster + image)</li>
</ul>
<p><strong>Recommended:</strong> Use the convenience script:</p>
<pre><code class="language-bash">scripts/run/run-examples.sh -t &lt;duration&gt; -v &lt;validators&gt; -e &lt;executors&gt; &lt;mode&gt;
# mode: host, compose, or k8s
</code></pre>
<p>This handles circuit setup, binary building/bundling, image building, and execution.</p>
<p><strong>Alternative:</strong> Direct cargo run (requires manual setup):</p>
<pre><code class="language-bash">POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin &lt;name&gt;
</code></pre>
<p><strong>Important:</strong> All runners require <code>POL_PROOF_DEV_MODE=true</code> to avoid expensive Groth16 proof generation that causes timeouts.</p>
<p>These binaries use the framework API (<code>ScenarioBuilder</code>) to construct and execute scenarios.</p>
<h2 id="builder-api"><a class="header" href="#builder-api">Builder API</a></h2>
<p>Scenarios are defined using a fluent builder pattern:</p>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn scenario_plan() -&gt; testing_framework_core::scenario::Scenario&lt;()&gt; {
ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
.wallets(50)
.transactions_with(|txs| txs.rate(5).users(20))
.da_with(|da| da.channel_rate(1).blob_rate(2))
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(90))
.build()
}</code></pre>
<p><strong>Key API Points:</strong></p>
<ul>
<li>Topology uses <code>.topology_with(|t| { t.validators(N).executors(M) })</code> closure pattern</li>
<li>Workloads are configured via <code>_with</code> closures (<code>transactions_with</code>, <code>da_with</code>, <code>chaos_with</code>)</li>
<li>Chaos workloads require <code>.enable_node_control()</code> and a compatible runner</li>
</ul>
<h2 id="deployers"><a class="header" href="#deployers">Deployers</a></h2>
<p>Three deployer implementations:</p>
<div class="table-wrapper"><table><thead><tr><th>Deployer</th><th>Backend</th><th>Prerequisites</th><th>Node Control</th></tr></thead><tbody>
<tr><td><code>LocalDeployer</code></td><td>Host processes</td><td>Binaries (built on demand or via bundle)</td><td>No</td></tr>
<tr><td><code>ComposeDeployer</code></td><td>Docker Compose</td><td>Image with embedded assets/binaries</td><td>Yes</td></tr>
<tr><td><code>K8sDeployer</code></td><td>Kubernetes Helm</td><td>Cluster + image loaded</td><td>Not yet</td></tr>
</tbody></table>
</div>
<p><strong>Compose-specific features:</strong></p>
<ul>
<li>Observability is external (set <code>NOMOS_METRICS_QUERY_URL</code> / <code>NOMOS_METRICS_OTLP_INGEST_URL</code> / <code>NOMOS_GRAFANA_URL</code> as needed)</li>
<li>Optional OTLP trace/metrics endpoints (<code>NOMOS_OTLP_ENDPOINT</code>, <code>NOMOS_OTLP_METRICS_ENDPOINT</code>)</li>
<li>Node control for chaos testing (restart validators/executors)</li>
</ul>
<h2 id="assets-and-images"><a class="header" href="#assets-and-images">Assets and Images</a></h2>
<h3 id="docker-image"><a class="header" href="#docker-image">Docker Image</a></h3>
<p>Built via <code>scripts/build/build_test_image.sh</code>:</p>
<ul>
<li>Embeds KZG circuit parameters and binaries from <code>testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params</code></li>
<li>Includes runner scripts: <code>run_nomos_node.sh</code>, <code>run_nomos_executor.sh</code></li>
<li>Tagged as <code>NOMOS_TESTNET_IMAGE</code> (default: <code>logos-blockchain-testing:local</code>)</li>
<li><strong>Recommended:</strong> Use prebuilt bundle via <code>scripts/build/build-bundle.sh --platform linux</code> and set <code>NOMOS_BINARIES_TAR</code> before building image</li>
</ul>
<h3 id="circuit-assets"><a class="header" href="#circuit-assets">Circuit Assets</a></h3>
<p>KZG parameters required for DA workloads:</p>
<ul>
<li><strong>Host path:</strong> <code>testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params</code> (note repeated filename—directory contains file <code>kzgrs_test_params</code>)</li>
<li><strong>Container path:</strong> <code>/kzgrs_test_params/kzgrs_test_params</code> (for compose/k8s)</li>
<li><strong>Override:</strong> <code>NOMOS_KZGRS_PARAMS_PATH=/custom/path/to/file</code> (must point to file)</li>
<li><strong>Fetch via:</strong> <code>scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/circuits</code> or use <code>scripts/run/run-examples.sh</code></li>
</ul>
<h3 id="compose-stack"><a class="header" href="#compose-stack">Compose Stack</a></h3>
<p>Templates and configs in <code>testing-framework/runners/compose/assets/</code>:</p>
<ul>
<li><code>docker-compose.yml.tera</code> — Stack template (validators, executors)</li>
<li>Cfgsync config: <code>testing-framework/assets/stack/cfgsync.yaml</code></li>
<li>Monitoring assets (not deployed by the framework): <code>testing-framework/assets/stack/monitoring/</code></li>
</ul>
<h2 id="logging-architecture"><a class="header" href="#logging-architecture">Logging Architecture</a></h2>
<p><strong>Two separate logging pipelines:</strong></p>
<div class="table-wrapper"><table><thead><tr><th>Component</th><th>Configuration</th><th>Output</th></tr></thead><tbody>
<tr><td><strong>Runner binaries</strong></td><td><code>RUST_LOG</code></td><td>Framework orchestration logs</td></tr>
<tr><td><strong>Node processes</strong></td><td><code>NOMOS_LOG_LEVEL</code>, <code>NOMOS_LOG_FILTER</code> (+ <code>NOMOS_LOG_DIR</code> on host runner)</td><td>Consensus, DA, mempool logs</td></tr>
</tbody></table>
</div>
<p><strong>Node logging:</strong></p>
<ul>
<li><strong>Local runner:</strong> Writes to temporary directories by default (cleaned up). Set <code>NOMOS_TESTS_TRACING=true</code> + <code>NOMOS_LOG_DIR</code> for persistent files.</li>
<li><strong>Compose runner:</strong> Default logs to container stdout/stderr (<code>docker logs</code>). To write per-node files, set <code>tracing_settings.logger: !File</code> in <code>testing-framework/assets/stack/cfgsync.yaml</code> (and mount a writable directory).</li>
<li><strong>K8s runner:</strong> Logs to pod stdout/stderr (<code>kubectl logs</code>). To write per-node files, set <code>tracing_settings.logger: !File</code> in <code>testing-framework/assets/stack/cfgsync.yaml</code> (and mount a writable directory).</li>
</ul>
<p><strong>File naming:</strong> Per-node files use prefix <code>nomos-node-{index}</code> or <code>nomos-executor-{index}</code> (may include timestamps).</p>
<h2 id="observability"><a class="header" href="#observability">Observability</a></h2>
<p><strong>Prometheus-compatible metrics querying (optional):</strong></p>
<ul>
<li>The framework does <strong>not</strong> deploy Prometheus/Grafana.</li>
<li>Provide a Prometheus-compatible base URL (PromQL API) via <code>NOMOS_METRICS_QUERY_URL</code>.</li>
<li>Accessible in expectations when configured: <code>ctx.telemetry().prometheus().map(|p| p.base_url())</code></li>
</ul>
<p><strong>Grafana dashboards (optional):</strong></p>
<ul>
<li>Dashboards live in <code>testing-framework/assets/stack/monitoring/grafana/dashboards/</code> and can be imported into your Grafana.</li>
<li>If you set <code>NOMOS_GRAFANA_URL</code>, the deployer prints it in <code>TESTNET_ENDPOINTS</code>.</li>
</ul>
<p><strong>Node APIs:</strong></p>
<ul>
<li>HTTP endpoints per node for consensus info, network status, DA membership</li>
<li>Accessible in expectations: <code>ctx.node_clients().validator_clients().get(0)</code></li>
</ul>
<p><strong>OTLP (optional):</strong></p>
<ul>
<li>Trace endpoint: <code>NOMOS_OTLP_ENDPOINT=http://localhost:4317</code></li>
<li>Metrics endpoint: <code>NOMOS_OTLP_METRICS_ENDPOINT=http://localhost:4318</code></li>
<li>Disabled by default (no noise if unset)</li>
</ul>
<p>For detailed logging configuration, see <a href="logging-observability.html">Logging &amp; Observability</a>.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="testing-philosophy"><a class="header" href="#testing-philosophy">Testing Philosophy</a></h1>
<p>This framework embodies specific principles that shape how you author and run
scenarios. Understanding these principles helps you write effective tests and
interpret results correctly.</p>
<h2 id="declarative-over-imperative"><a class="header" href="#declarative-over-imperative">Declarative over Imperative</a></h2>
<p>Describe <strong>what</strong> you want to test, not <strong>how</strong> to orchestrate it:</p>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn declarative_over_imperative() {
// Good: declarative
let _plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(2).executors(1))
.transactions_with(|txs| {
txs.rate(5) // 5 transactions per block
})
.expect_consensus_liveness()
.build();
// Bad: imperative (framework doesn't work this way)
// spawn_validator(); spawn_executor();
// loop { submit_tx(); check_block(); }
}</code></pre>
<p><strong>Why it matters:</strong> The framework handles deployment, readiness, and cleanup.
You focus on test intent, not infrastructure orchestration.</p>
<h2 id="protocol-time-not-wall-time"><a class="header" href="#protocol-time-not-wall-time">Protocol Time, Not Wall Time</a></h2>
<p>Reason in <strong>blocks</strong> and <strong>consensus intervals</strong>, not wall-clock seconds.</p>
<p><strong>Consensus defaults:</strong></p>
<ul>
<li>Slot duration: 2 seconds (NTP-synchronized, configurable via <code>CONSENSUS_SLOT_TIME</code>)</li>
<li>Active slot coefficient: 0.9 (90% block probability per slot, configurable via <code>CONSENSUS_ACTIVE_SLOT_COEFF</code>)</li>
<li>Expected rate: ~27 blocks per minute</li>
</ul>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn protocol_time_not_wall_time() {
// Good: protocol-oriented thinking
let _plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(2).executors(1))
.transactions_with(|txs| {
txs.rate(5) // 5 transactions per block
})
.with_run_duration(Duration::from_secs(60)) // Let framework calculate expected blocks
.expect_consensus_liveness() // "Did we produce the expected blocks?"
.build();
// Bad: wall-clock assumptions
// "I expect exactly 30 blocks in 60 seconds"
// This breaks on slow CI where slot timing might drift
}</code></pre>
<p><strong>Why it matters:</strong> Slot timing is fixed (2s by default, NTP-synchronized), so the
expected number of blocks is predictable: ~27 blocks in 60s with the default
0.9 active slot coefficient. The framework calculates expected blocks from slot
duration and run window, making assertions protocol-based rather than tied to
specific wall-clock expectations. Assert on "blocks produced relative to slots"
not "blocks produced in exact wall-clock seconds".</p>
<h2 id="determinism-first-chaos-when-needed"><a class="header" href="#determinism-first-chaos-when-needed">Determinism First, Chaos When Needed</a></h2>
<p><strong>Default scenarios are repeatable:</strong></p>
<ul>
<li>Fixed topology</li>
<li>Predictable traffic rates</li>
<li>Deterministic checks</li>
</ul>
<p><strong>Chaos is opt-in:</strong></p>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt};
pub fn determinism_first() {
// Separate: functional test (deterministic)
let _plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(2).executors(1))
.transactions_with(|txs| {
txs.rate(5) // 5 transactions per block
})
.expect_consensus_liveness()
.build();
// Separate: chaos test (introduces randomness)
let _chaos_plan =
ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
.enable_node_control()
.chaos_with(|c| {
c.restart()
.min_delay(Duration::from_secs(30))
.max_delay(Duration::from_secs(60))
.target_cooldown(Duration::from_secs(45))
.apply()
})
.transactions_with(|txs| {
txs.rate(5) // 5 transactions per block
})
.expect_consensus_liveness()
.build();
}</code></pre>
<p><strong>Why it matters:</strong> Mixing determinism with chaos creates noisy, hard-to-debug
failures. Separate concerns make failures actionable.</p>
<h2 id="observable-health-signals"><a class="header" href="#observable-health-signals">Observable Health Signals</a></h2>
<p>Prefer <strong>user-facing signals</strong> over internal state:</p>
<p><strong>Good checks:</strong></p>
<ul>
<li>Blocks progressing at expected rate (liveness)</li>
<li>Transactions included within N blocks (inclusion)</li>
<li>DA blobs retrievable (availability)</li>
</ul>
<p><strong>Avoid internal checks:</strong></p>
<ul>
<li>Memory pool size</li>
<li>Internal service state</li>
<li>Cache hit rates</li>
</ul>
<p><strong>Why it matters:</strong> User-facing signals reflect actual system health.
Internal state can be "healthy" while the system is broken from a user
perspective.</p>
<h2 id="minimum-run-windows"><a class="header" href="#minimum-run-windows">Minimum Run Windows</a></h2>
<p>Always run long enough for <strong>meaningful block production</strong>:</p>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn minimum_run_windows() {
// Bad: too short (~2 blocks with default 2s slots, 0.9 coeff)
let _too_short = ScenarioBuilder::with_node_counts(1, 0)
.with_run_duration(Duration::from_secs(5))
.expect_consensus_liveness()
.build();
// Good: enough blocks for assertions (~27 blocks with default 2s slots, 0.9
// coeff)
let _good = ScenarioBuilder::with_node_counts(1, 0)
.with_run_duration(Duration::from_secs(60))
.expect_consensus_liveness()
.build();
}</code></pre>
<p><strong>Note:</strong> Block counts assume default consensus parameters:</p>
<ul>
<li>Slot duration: 2 seconds (configurable via <code>CONSENSUS_SLOT_TIME</code>)</li>
<li>Active slot coefficient: 0.9 (90% block probability per slot, configurable via <code>CONSENSUS_ACTIVE_SLOT_COEFF</code>)</li>
<li>Formula: <code>blocks ≈ (duration / slot_duration) × active_slot_coeff</code></li>
</ul>
<p>If upstream changes these parameters, adjust your duration expectations accordingly.</p>
<p>The framework enforces minimum durations (at least 2× slot duration), but be explicit. Very short runs risk false confidence—one lucky block doesn't prove liveness.</p>
<h2 id="summary"><a class="header" href="#summary">Summary</a></h2>
<p>These principles keep scenarios:</p>
<ul>
<li><strong>Portable</strong> across environments (protocol time, declarative)</li>
<li><strong>Debuggable</strong> (determinism, separation of concerns)</li>
<li><strong>Meaningful</strong> (observable signals, sufficient duration)</li>
</ul>
<p>When authoring scenarios, ask: "Does this test the protocol behavior or
my local environment quirks?"</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="scenario-lifecycle"><a class="header" href="#scenario-lifecycle">Scenario Lifecycle</a></h1>
<p>A scenario progresses through six distinct phases, each with a specific responsibility:</p>
<pre><code class="language-mermaid">flowchart TB
subgraph Phase1["1. Build Phase"]
Build[Define Scenario]
BuildDetails["• Declare topology&lt;br/&gt;• Attach workloads&lt;br/&gt;• Add expectations&lt;br/&gt;• Set run duration"]
Build --&gt; BuildDetails
end
subgraph Phase2["2. Deploy Phase"]
Deploy[Provision Environment]
DeployDetails["• Launch nodes&lt;br/&gt;• Wait for readiness&lt;br/&gt;• Establish connectivity&lt;br/&gt;• Return Runner"]
Deploy --&gt; DeployDetails
end
subgraph Phase3["3. Capture Phase"]
Capture[Baseline Metrics]
CaptureDetails["• Snapshot initial state&lt;br/&gt;• Start BlockFeed&lt;br/&gt;• Initialize expectations"]
Capture --&gt; CaptureDetails
end
subgraph Phase4["4. Execution Phase"]
Execute[Drive Workloads]
ExecuteDetails["• Submit transactions&lt;br/&gt;• Disperse DA blobs&lt;br/&gt;• Trigger chaos events&lt;br/&gt;• Run for duration"]
Execute --&gt; ExecuteDetails
end
subgraph Phase5["5. Evaluation Phase"]
Evaluate[Check Expectations]
EvaluateDetails["• Verify liveness&lt;br/&gt;• Check inclusion&lt;br/&gt;• Validate outcomes&lt;br/&gt;• Aggregate results"]
Evaluate --&gt; EvaluateDetails
end
subgraph Phase6["6. Cleanup Phase"]
Cleanup[Teardown]
CleanupDetails["• Stop nodes&lt;br/&gt;• Remove containers&lt;br/&gt;• Collect logs&lt;br/&gt;• Release resources"]
Cleanup --&gt; CleanupDetails
end
Phase1 --&gt; Phase2
Phase2 --&gt; Phase3
Phase3 --&gt; Phase4
Phase4 --&gt; Phase5
Phase5 --&gt; Phase6
style Phase1 fill:#e1f5ff
style Phase2 fill:#fff4e1
style Phase3 fill:#f0ffe1
style Phase4 fill:#ffe1f5
style Phase5 fill:#e1ffe1
style Phase6 fill:#ffe1e1
</code></pre>
<h2 id="phase-details"><a class="header" href="#phase-details">Phase Details</a></h2>
<h3 id="1-build-the-plan"><a class="header" href="#1-build-the-plan">1. Build the Plan</a></h3>
<p>Declare a topology, attach workloads and expectations, and set the run window. The plan is the single source of truth for what will happen.</p>
<p><strong>Key actions:</strong></p>
<ul>
<li>Define cluster shape (validators, executors, network topology)</li>
<li>Configure workloads (transaction rate, DA traffic, chaos patterns)</li>
<li>Attach expectations (liveness, inclusion, custom checks)</li>
<li>Set timing parameters (run duration, cooldown period)</li>
</ul>
<p><strong>Output:</strong> Immutable <code>Scenario</code> plan</p>
<h3 id="2-deploy"><a class="header" href="#2-deploy">2. Deploy</a></h3>
<p>Hand the plan to a deployer. It provisions the environment on the chosen backend, waits for nodes to signal readiness, and returns a runner.</p>
<p><strong>Key actions:</strong></p>
<ul>
<li>Provision infrastructure (processes, containers, or pods)</li>
<li>Launch validator and executor nodes</li>
<li>Wait for readiness probes (HTTP endpoints respond)</li>
<li>Establish node connectivity and metrics endpoints</li>
<li>Spawn BlockFeed for real-time block observation</li>
</ul>
<p><strong>Output:</strong> <code>Runner</code> + <code>RunContext</code> (with node clients, metrics, control handles)</p>
<h3 id="3-capture-baseline"><a class="header" href="#3-capture-baseline">3. Capture Baseline</a></h3>
<p>Expectations snapshot initial state before workloads begin.</p>
<p><strong>Key actions:</strong></p>
<ul>
<li>Record starting block height</li>
<li>Initialize counters and trackers</li>
<li>Subscribe to BlockFeed</li>
<li>Capture baseline metrics</li>
</ul>
<p><strong>Output:</strong> Captured state for later comparison</p>
<h3 id="4-drive-workloads"><a class="header" href="#4-drive-workloads">4. Drive Workloads</a></h3>
<p>The runner starts traffic and behaviors for the planned duration.</p>
<p><strong>Key actions:</strong></p>
<ul>
<li>Submit transactions at configured rates</li>
<li>Disperse and sample DA blobs</li>
<li>Trigger chaos events (node restarts, network partitions)</li>
<li>Run concurrently for the specified duration</li>
<li>Observe blocks and metrics in real-time</li>
</ul>
<p><strong>Duration:</strong> Controlled by <code>with_run_duration()</code></p>
<h3 id="5-evaluate-expectations"><a class="header" href="#5-evaluate-expectations">5. Evaluate Expectations</a></h3>
<p>Once activity stops (and optional cooldown completes), the runner checks liveness and workload-specific outcomes.</p>
<p><strong>Key actions:</strong></p>
<ul>
<li>Verify consensus liveness (minimum block production)</li>
<li>Check transaction inclusion rates</li>
<li>Validate DA dispersal and sampling</li>
<li>Assess system recovery after chaos events</li>
<li>Aggregate pass/fail results</li>
</ul>
<p><strong>Output:</strong> Success or detailed failure report</p>
<h3 id="6-cleanup"><a class="header" href="#6-cleanup">6. Cleanup</a></h3>
<p>Tear down resources so successive runs start fresh and do not inherit leaked state.</p>
<p><strong>Key actions:</strong></p>
<ul>
<li>Stop all node processes/containers/pods</li>
<li>Remove temporary directories and volumes</li>
<li>Collect and archive logs (if <code>NOMOS_TESTS_KEEP_LOGS=1</code>)</li>
<li>Release ports and network resources</li>
<li>Cleanup observability stack (if spawned)</li>
</ul>
<p><strong>Guarantee:</strong> Runs even on panic via <code>CleanupGuard</code></p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="design-rationale"><a class="header" href="#design-rationale">Design Rationale</a></h1>
<ul>
<li><strong>Modular crates</strong> keep configuration, orchestration, workloads, and runners decoupled so each can evolve without breaking the others.</li>
<li><strong>Pluggable runners</strong> let the same scenario run on a laptop, a Docker host, or a Kubernetes cluster, making validation portable across environments.</li>
<li><strong>Separated workloads and expectations</strong> clarify intent: what traffic to generate versus how to judge success. This simplifies review and reuse.</li>
<li><strong>Declarative topology</strong> makes cluster shape explicit and repeatable, reducing surprise when moving between CI and developer machines.</li>
<li><strong>Maintainability through predictability</strong>: a clear flow from plan to deployment to verification lowers the cost of extending the framework and interpreting failures.</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="part-ii--user-guide"><a class="header" href="#part-ii--user-guide">Part II — User Guide</a></h1>
<p>Practical guidance for shaping scenarios, combining workloads and expectations,
and running them across different environments.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="workspace-layout"><a class="header" href="#workspace-layout">Workspace Layout</a></h1>
<p>The workspace focuses on multi-node integration testing and sits alongside a
<code>nomos-node</code> checkout. Its crates separate concerns to keep scenarios
repeatable and portable:</p>
<ul>
<li><strong>Configs</strong>: prepares high-level node, network, tracing, and wallet settings
used across test environments.</li>
<li><strong>Core scenario orchestration</strong>: the engine that holds topology descriptions,
scenario plans, runtimes, workloads, and expectations.</li>
<li><strong>Workflows</strong>: ready-made workloads (transactions, data-availability, chaos)
and reusable expectations assembled into a user-facing DSL.</li>
<li><strong>Runners</strong>: deployment backends for local processes, Docker Compose, and
Kubernetes, all consuming the same scenario plan.</li>
<li><strong>Runner Examples</strong> (crate name: <code>runner-examples</code>, path: <code>examples/</code>):
runnable binaries (<code>examples/src/bin/local_runner.rs</code>,
<code>examples/src/bin/compose_runner.rs</code>, <code>examples/src/bin/k8s_runner.rs</code>) that
demonstrate complete scenario execution with each deployer.</li>
</ul>
<p>This split keeps configuration, orchestration, reusable traffic patterns, and
deployment adapters loosely coupled while sharing one mental model for tests.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="annotated-tree"><a class="header" href="#annotated-tree">Annotated Tree</a></h1>
<p>Directory structure with key paths annotated:</p>
<pre><code class="language-text">logos-blockchain-testing/
├─ testing-framework/ # Core library crates
│ ├─ configs/ # Node config builders, topology generation, tracing/logging config
│ ├─ core/ # Scenario model (ScenarioBuilder), runtime (Runner, Deployer), topology, node spawning
│ ├─ workflows/ # Workloads (transactions, DA, chaos), expectations (liveness), builder DSL extensions
│ ├─ runners/ # Deployment backends
│ │ ├─ local/ # LocalDeployer (spawns local processes)
│ │ ├─ compose/ # ComposeDeployer (Docker Compose + Prometheus)
│ │ └─ k8s/ # K8sDeployer (Kubernetes Helm)
│ └─ assets/ # Docker/K8s stack assets
│ └─ stack/
│ ├─ kzgrs_test_params/ # KZG circuit parameters directory
│ │ └─ kzgrs_test_params # Actual proving key file (note repeated name)
│ ├─ monitoring/ # Prometheus config
│ ├─ scripts/ # Container entrypoints
│ └─ cfgsync.yaml # Config sync server template
│
├─ examples/ # PRIMARY ENTRY POINT: runnable binaries
│ └─ src/bin/
│ ├─ local_runner.rs # Host processes demo (LocalDeployer)
│ ├─ compose_runner.rs # Docker Compose demo (ComposeDeployer)
│ └─ k8s_runner.rs # Kubernetes demo (K8sDeployer)
│
├─ scripts/ # Helper utilities
│ ├─ run-examples.sh # Convenience script (handles setup + runs examples)
│ ├─ build-bundle.sh # Build prebuilt binaries+circuits bundle
│ ├─ setup-circuits-stack.sh # Fetch KZG parameters (Linux + host)
│ └─ setup-nomos-circuits.sh # Legacy circuit fetcher
│
└─ book/ # This documentation (mdBook)
</code></pre>
<h2 id="key-directories-explained"><a class="header" href="#key-directories-explained">Key Directories Explained</a></h2>
<h3 id="testing-framework"><a class="header" href="#testing-framework"><code>testing-framework/</code></a></h3>
<p>Core library crates providing the testing API.</p>
<div class="table-wrapper"><table><thead><tr><th>Crate</th><th>Purpose</th><th>Key Exports</th></tr></thead><tbody>
<tr><td><code>configs</code></td><td>Node configuration builders</td><td>Topology generation, tracing config</td></tr>
<tr><td><code>core</code></td><td>Scenario model &amp; runtime</td><td><code>ScenarioBuilder</code>, <code>Deployer</code>, <code>Runner</code></td></tr>
<tr><td><code>workflows</code></td><td>Workloads &amp; expectations</td><td><code>ScenarioBuilderExt</code>, <code>ChaosBuilderExt</code></td></tr>
<tr><td><code>runners/local</code></td><td>Local process deployer</td><td><code>LocalDeployer</code></td></tr>
<tr><td><code>runners/compose</code></td><td>Docker Compose deployer</td><td><code>ComposeDeployer</code></td></tr>
<tr><td><code>runners/k8s</code></td><td>Kubernetes deployer</td><td><code>K8sDeployer</code></td></tr>
</tbody></table>
</div>
<h3 id="testing-frameworkassetsstack"><a class="header" href="#testing-frameworkassetsstack"><code>testing-framework/assets/stack/</code></a></h3>
<p>Docker/K8s deployment assets:</p>
<ul>
<li><strong><code>kzgrs_test_params/kzgrs_test_params</code></strong>: Circuit parameters file (note repeated name; override via <code>NOMOS_KZGRS_PARAMS_PATH</code>)</li>
<li><strong><code>monitoring/</code></strong>: Prometheus config</li>
<li><strong><code>scripts/</code></strong>: Container entrypoints</li>
</ul>
<h3 id="scripts"><a class="header" href="#scripts"><code>scripts/</code></a></h3>
<p>Convenience utilities:</p>
<ul>
<li><strong><code>run-examples.sh</code></strong>: All-in-one script for host/compose/k8s modes (recommended)</li>
<li><strong><code>build-bundle.sh</code></strong>: Create prebuilt binaries+circuits bundle for compose/k8s</li>
<li><strong><code>build_test_image.sh</code></strong>: Build the compose/k8s Docker image (bakes in assets)</li>
<li><strong><code>setup-circuits-stack.sh</code></strong>: Fetch KZG parameters for both Linux and host</li>
<li><strong><code>cfgsync.yaml</code></strong>: Configuration sync server template</li>
</ul>
<h3 id="examples-start-here"><a class="header" href="#examples-start-here"><code>examples/</code> (Start Here!)</a></h3>
<p><strong>Runnable binaries</strong> demonstrating framework usage:</p>
<ul>
<li><code>local_runner.rs</code> — Local processes</li>
<li><code>compose_runner.rs</code> — Docker Compose (requires <code>NOMOS_TESTNET_IMAGE</code> built)</li>
<li><code>k8s_runner.rs</code> — Kubernetes (requires cluster + image)</li>
</ul>
<p><strong>Run with:</strong> <code>POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin &lt;name&gt;</code></p>
<p><strong>All runners require <code>POL_PROOF_DEV_MODE=true</code></strong> to avoid expensive proof generation.</p>
<h3 id="scripts-1"><a class="header" href="#scripts-1"><code>scripts/</code></a></h3>
<p>Helper utilities:</p>
<ul>
<li><strong><code>setup-nomos-circuits.sh</code></strong>: Fetch KZG parameters from releases</li>
</ul>
<h2 id="observability-1"><a class="header" href="#observability-1">Observability</a></h2>
<p><strong>Compose runner</strong> includes:</p>
<ul>
<li><strong>Prometheus</strong> at <code>http://localhost:9090</code> (metrics scraping)</li>
<li>Node metrics exposed per validator/executor</li>
<li>Access in expectations: <code>ctx.telemetry().prometheus().map(|p| p.base_url())</code></li>
</ul>
<p><strong>Logging</strong> controlled by:</p>
<ul>
<li><code>NOMOS_LOG_DIR</code> — Write per-node log files</li>
<li><code>NOMOS_LOG_LEVEL</code> — Global log level (error/warn/info/debug/trace)</li>
<li><code>NOMOS_LOG_FILTER</code> — Target-specific filtering (e.g., <code>cryptarchia=trace,nomos_da_sampling=debug</code>)</li>
<li><code>NOMOS_TESTS_TRACING</code> — Enable file logging for local runner</li>
</ul>
<p>See <a href="logging-observability.html">Logging &amp; Observability</a> for details.</p>
<h2 id="navigation-guide"><a class="header" href="#navigation-guide">Navigation Guide</a></h2>
<div class="table-wrapper"><table><thead><tr><th>To Do This</th><th>Go Here</th></tr></thead><tbody>
<tr><td><strong>Run an example</strong></td><td><code>examples/src/bin/</code> → <code>cargo run -p runner-examples --bin &lt;name&gt;</code></td></tr>
<tr><td><strong>Write a custom scenario</strong></td><td><code>testing-framework/core/</code> → Implement using <code>ScenarioBuilder</code></td></tr>
<tr><td><strong>Add a new workload</strong></td><td><code>testing-framework/workflows/src/workloads/</code> → Implement <code>Workload</code> trait</td></tr>
<tr><td><strong>Add a new expectation</strong></td><td><code>testing-framework/workflows/src/expectations/</code> → Implement <code>Expectation</code> trait</td></tr>
<tr><td><strong>Modify node configs</strong></td><td><code>testing-framework/configs/src/topology/configs/</code></td></tr>
<tr><td><strong>Extend builder DSL</strong></td><td><code>testing-framework/workflows/src/builder/</code> → Add trait methods</td></tr>
<tr><td><strong>Add a new deployer</strong></td><td><code>testing-framework/runners/</code> → Implement <code>Deployer</code> trait</td></tr>
</tbody></table>
</div>
<p>For detailed guidance, see <a href="internal-crate-reference.html">Internal Crate Reference</a>.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="authoring-scenarios"><a class="header" href="#authoring-scenarios">Authoring Scenarios</a></h1>
<p>Creating a scenario is a declarative exercise. This page walks you through the core authoring loop with concrete examples, explains the units and timing model, and shows how to structure scenarios in Rust test suites.</p>
<hr />
<h2 id="the-core-authoring-loop"><a class="header" href="#the-core-authoring-loop">The Core Authoring Loop</a></h2>
<p>Every scenario follows the same pattern:</p>
<pre><code class="language-mermaid">flowchart LR
A[1. Topology] --&gt; B[2. Workloads]
B --&gt; C[3. Expectations]
C --&gt; D[4. Duration]
D --&gt; E[5. Deploy &amp; Run]
</code></pre>
<ol>
<li><strong>Shape the topology</strong> — How many nodes, what roles, what network shape</li>
<li><strong>Attach workloads</strong> — What traffic to generate (transactions, blobs, chaos)</li>
<li><strong>Define expectations</strong> — What success looks like (liveness, inclusion, recovery)</li>
<li><strong>Set duration</strong> — How long to run the experiment</li>
<li><strong>Choose a runner</strong> — Where to execute (local, compose, k8s)</li>
</ol>
<hr />
<h2 id="hello-scenario-your-first-test"><a class="header" href="#hello-scenario-your-first-test">Hello Scenario: Your First Test</a></h2>
<p>Let's build a minimal consensus liveness test step-by-step.</p>
<h3 id="step-1-shape-the-topology"><a class="header" href="#step-1-shape-the-topology">Step 1: Shape the Topology</a></h3>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
let scenario = ScenarioBuilder::topology_with(|t| {
t.network_star() // Star network (one gateway + nodes)
.validators(3) // 3 validator nodes
.executors(1) // 1 executor node
})</code></pre>
<p><strong>What goes in topology?</strong></p>
<ul>
<li>Node counts (validators, executors)</li>
<li>Network shape (<code>network_star()</code> is currently the only built-in layout)</li>
<li>Role split (validators vs. executors)</li>
</ul>
<p><strong>What does NOT go in topology?</strong></p>
<ul>
<li>Traffic rates (that's workloads)</li>
<li>Success criteria (that's expectations)</li>
<li>Runtime configuration (that's duration/runner)</li>
</ul>
<h3 id="step-2-attach-workloads"><a class="header" href="#step-2-attach-workloads">Step 2: Attach Workloads</a></h3>
<pre><code class="language-rust ignore">.wallets(20) // Seed funded wallet accounts for transaction workloads
.transactions_with(|tx| {
tx.rate(10) // 10 transactions per block
.users(5) // distributed across 5 wallets
})</code></pre>
<p><strong>What goes in workloads?</strong></p>
<ul>
<li>Transaction traffic (rate, users)</li>
<li>DA traffic (channels, blobs)</li>
<li>Chaos injection (restarts, delays)</li>
</ul>
<p><strong>Units explained:</strong></p>
<ul>
<li><code>.rate(10)</code> = <strong>10 transactions per block</strong> (not per second!)</li>
<li><code>.users(5)</code> = use 5 distinct wallet accounts</li>
<li>The framework adapts to block time automatically</li>
</ul>
<h3 id="step-3-define-expectations"><a class="header" href="#step-3-define-expectations">Step 3: Define Expectations</a></h3>
<pre><code class="language-rust ignore">.expect_consensus_liveness()</code></pre>
<p><strong>What goes in expectations?</strong></p>
<ul>
<li>Health checks that run after the scenario completes</li>
<li>Liveness (blocks produced)</li>
<li>Inclusion (workload activity landed on-chain)</li>
<li>Recovery (system survived chaos)</li>
</ul>
<p><strong>When do expectations run?</strong>
After the duration window ends, during the <strong>evaluation phase</strong> of the scenario lifecycle.</p>
<h3 id="step-4-set-duration"><a class="header" href="#step-4-set-duration">Step 4: Set Duration</a></h3>
<pre><code class="language-rust ignore">use std::time::Duration;
.with_run_duration(Duration::from_secs(60))</code></pre>
<p><strong>How long is enough?</strong></p>
<ul>
<li>Minimum: 2× the expected block time × number of blocks you want</li>
<li>For consensus liveness: 30-60 seconds</li>
<li>For transaction inclusion: 60-120 seconds</li>
<li>For chaos recovery: 2-5 minutes</li>
</ul>
<p><strong>What happens during this window?</strong></p>
<ul>
<li>Nodes are running</li>
<li>Workloads generate traffic</li>
<li>Metrics/logs are collected</li>
<li>BlockFeed broadcasts observations in real-time</li>
</ul>
<h3 id="step-5-build-and-deploy"><a class="header" href="#step-5-build-and-deploy">Step 5: Build and Deploy</a></h3>
<pre><code class="language-rust ignore">.build();
// Choose a runner
use testing_framework_core::scenario::Deployer;
use testing_framework_runner_local::LocalDeployer;
let deployer = LocalDeployer::default();
let runner = deployer.deploy(&amp;scenario).await?;
let _result = runner.run(&amp;mut scenario).await?;</code></pre>
<hr />
<h2 id="complete-hello-scenario"><a class="header" href="#complete-hello-scenario">Complete "Hello Scenario"</a></h2>
<p>Putting it all together:</p>
<pre><code class="language-rust ignore">use std::time::Duration;
use anyhow::Result;
use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
use testing_framework_runner_local::LocalDeployer;
use testing_framework_workflows::ScenarioBuilderExt;
#[tokio::test]
async fn hello_consensus_liveness() -&gt; Result&lt;()&gt; {
let mut scenario = ScenarioBuilder::topology_with(|t| {
t.network_star()
.validators(3)
.executors(1)
})
.wallets(20)
.transactions_with(|tx| tx.rate(10).users(5))
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(60))
.build();
let deployer = LocalDeployer::default();
let runner = deployer.deploy(&amp;scenario).await?;
runner.run(&amp;mut scenario).await?;
Ok(())
}</code></pre>
<p><strong>Run it:</strong></p>
<pre><code class="language-bash">POL_PROOF_DEV_MODE=true cargo test hello_consensus_liveness
</code></pre>
<hr />
<h2 id="understanding-units--timing"><a class="header" href="#understanding-units--timing">Understanding Units &amp; Timing</a></h2>
<h3 id="transaction-rate-per-block-not-per-second"><a class="header" href="#transaction-rate-per-block-not-per-second">Transaction Rate: Per-Block, Not Per-Second</a></h3>
<p><strong>Wrong mental model:</strong> <code>.rate(10)</code> = 10 tx/second</p>
<p><strong>Correct mental model:</strong> <code>.rate(10)</code> = 10 tx/block</p>
<p><strong>Why?</strong> The blockchain produces blocks at variable rates depending on consensus timing. The framework submits the configured rate <strong>per block</strong> to ensure predictable load regardless of block time.</p>
<p><strong>Example:</strong></p>
<ul>
<li>Block time = 2 seconds</li>
<li><code>.rate(10)</code> → 10 tx/block → 5 tx/second average</li>
<li>Block time = 5 seconds</li>
<li><code>.rate(10)</code> → 10 tx/block → 2 tx/second average</li>
</ul>
<h3 id="duration-wall-clock-time"><a class="header" href="#duration-wall-clock-time">Duration: Wall-Clock Time</a></h3>
<p><code>.with_run_duration(Duration::from_secs(60))</code> means the scenario runs for <strong>60 seconds of real time</strong>, not 60 blocks.</p>
<p><strong>How many blocks will be produced?</strong>
Depends on consensus timing (slot time, active slot coefficient). Typical: 1-2 seconds per block.</p>
<p><strong>Rule of thumb:</strong></p>
<ul>
<li>60 seconds → ~30-60 blocks</li>
<li>120 seconds → ~60-120 blocks</li>
</ul>
<hr />
<h2 id="structuring-scenarios-in-a-test-suite"><a class="header" href="#structuring-scenarios-in-a-test-suite">Structuring Scenarios in a Test Suite</a></h2>
<h3 id="pattern-1-integration-test-module"><a class="header" href="#pattern-1-integration-test-module">Pattern 1: Integration Test Module</a></h3>
<pre><code class="language-rust ignore">// tests/integration_test.rs
use std::time::Duration;
use anyhow::Result;
use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
use testing_framework_runner_local::LocalDeployer;
use testing_framework_workflows::ScenarioBuilderExt;
#[tokio::test]
async fn test_consensus_liveness() -&gt; Result&lt;()&gt; {
let mut scenario = ScenarioBuilder::topology_with(|t| {
t.network_star().validators(3).executors(1)
})
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(30))
.build();
let deployer = LocalDeployer::default();
let runner = deployer.deploy(&amp;scenario).await?;
runner.run(&amp;mut scenario).await?;
Ok(())
}
#[tokio::test]
async fn test_transaction_inclusion() -&gt; Result&lt;()&gt; {
let mut scenario = ScenarioBuilder::topology_with(|t| {
t.network_star().validators(2).executors(1)
})
.wallets(10)
.transactions_with(|tx| tx.rate(5).users(5))
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(60))
.build();
let deployer = LocalDeployer::default();
let runner = deployer.deploy(&amp;scenario).await?;
runner.run(&amp;mut scenario).await?;
Ok(())
}</code></pre>
<h3 id="pattern-2-shared-scenario-builders"><a class="header" href="#pattern-2-shared-scenario-builders">Pattern 2: Shared Scenario Builders</a></h3>
<p>Extract common topology patterns:</p>
<pre><code class="language-rust ignore">// tests/helpers.rs
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn minimal_topology() -&gt; ScenarioBuilder {
ScenarioBuilder::topology_with(|t| {
t.network_star().validators(2).executors(1)
})
}
pub fn production_like_topology() -&gt; ScenarioBuilder {
ScenarioBuilder::topology_with(|t| {
t.network_star().validators(7).executors(3)
})
}
// tests/consensus_tests.rs
use std::time::Duration;
use helpers::*;
#[tokio::test]
async fn small_cluster_liveness() -&gt; anyhow::Result&lt;()&gt; {
let mut scenario = minimal_topology()
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(30))
.build();
// ... deploy and run
Ok(())
}
#[tokio::test]
async fn large_cluster_liveness() -&gt; anyhow::Result&lt;()&gt; {
let mut scenario = production_like_topology()
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(60))
.build();
// ... deploy and run
Ok(())
}</code></pre>
<h3 id="pattern-3-parameterized-scenarios"><a class="header" href="#pattern-3-parameterized-scenarios">Pattern 3: Parameterized Scenarios</a></h3>
<p>Test the same behavior across different scales:</p>
<pre><code class="language-rust ignore">use std::time::Duration;
use anyhow::Result;
use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
use testing_framework_runner_local::LocalDeployer;
use testing_framework_workflows::ScenarioBuilderExt;
async fn test_liveness_with_topology(validators: usize, executors: usize) -&gt; Result&lt;()&gt; {
let mut scenario = ScenarioBuilder::topology_with(|t| {
t.network_star()
.validators(validators)
.executors(executors)
})
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(60))
.build();
let deployer = LocalDeployer::default();
let runner = deployer.deploy(&amp;scenario).await?;
runner.run(&amp;mut scenario).await?;
Ok(())
}
#[tokio::test]
async fn liveness_small() -&gt; Result&lt;()&gt; {
test_liveness_with_topology(2, 1).await
}
#[tokio::test]
async fn liveness_medium() -&gt; Result&lt;()&gt; {
test_liveness_with_topology(5, 2).await
}
#[tokio::test]
async fn liveness_large() -&gt; Result&lt;()&gt; {
test_liveness_with_topology(10, 3).await
}</code></pre>
<hr />
<h2 id="what-belongs-where"><a class="header" href="#what-belongs-where">What Belongs Where?</a></h2>
<h3 id="topology"><a class="header" href="#topology">Topology</a></h3>
<p><strong>Do include:</strong></p>
<ul>
<li>Node counts (<code>.validators(3)</code>, <code>.executors(1)</code>)</li>
<li>Network shape (<code>.network_star()</code>)</li>
<li>Role split (validators vs. executors)</li>
</ul>
<p><strong>Don't include:</strong></p>
<ul>
<li>Traffic rates (workload concern)</li>
<li>Expected outcomes (expectation concern)</li>
<li>Runtime behavior (runner/duration concern)</li>
</ul>
<h3 id="workloads"><a class="header" href="#workloads">Workloads</a></h3>
<p><strong>Do include:</strong></p>
<ul>
<li>Transaction traffic (<code>.transactions_with(|tx| ...)</code>)</li>
<li>DA traffic (<code>.da_with(|da| ...)</code>)</li>
<li>Chaos injection (<code>.with_workload(RandomRestartWorkload::new(...))</code>)</li>
<li>Rates, users, timing</li>
</ul>
<p><strong>Don't include:</strong></p>
<ul>
<li>Node configuration (topology concern)</li>
<li>Success criteria (expectation concern)</li>
</ul>
<h3 id="expectations"><a class="header" href="#expectations">Expectations</a></h3>
<p><strong>Do include:</strong></p>
<ul>
<li>Health checks (<code>.expect_consensus_liveness()</code>)</li>
<li>Inclusion verification (built-in to workloads)</li>
<li>Custom assertions (<code>.with_expectation(MyExpectation::new())</code>)</li>
</ul>
<p><strong>Don't include:</strong></p>
<ul>
<li>Traffic generation (workload concern)</li>
<li>Cluster shape (topology concern)</li>
</ul>
<hr />
<h2 id="best-practices"><a class="header" href="#best-practices">Best Practices</a></h2>
<ol>
<li><strong>Keep scenarios focused</strong>: One scenario = one behavior under test</li>
<li><strong>Start small</strong>: 2-3 validators, 1 executor, 30-60 seconds</li>
<li><strong>Use descriptive names</strong>: <code>test_consensus_survives_validator_restart</code> not <code>test_1</code></li>
<li><strong>Extract common patterns</strong>: Shared topology builders, helper functions</li>
<li><strong>Document intent</strong>: Add comments explaining what you're testing and why</li>
<li><strong>Mind the units</strong>: <code>.rate(N)</code> is per-block, <code>.with_run_duration()</code> is wall-clock</li>
<li><strong>Set realistic durations</strong>: Allow enough time for multiple blocks + workload effects</li>
</ol>
<hr />
<h2 id="next-steps-1"><a class="header" href="#next-steps-1">Next Steps</a></h2>
<ul>
<li><strong><a href="workloads.html">Core Content: Workloads &amp; Expectations</a></strong> — Comprehensive reference for built-in workloads and expectations</li>
<li><strong><a href="examples.html">Examples</a></strong> — More scenario patterns (DA, chaos, advanced topologies)</li>
<li><strong><a href="running-scenarios.html">Running Scenarios</a></strong> — How execution works, artifacts produced, per-runner details</li>
<li><strong><a href="api-levels.html">API Levels</a></strong> — When to use builder DSL vs. direct instantiation</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="core-content-workloads--expectations"><a class="header" href="#core-content-workloads--expectations">Core Content: Workloads &amp; Expectations</a></h1>
<p>Workloads describe the activity a scenario generates; expectations describe the signals that must hold when that activity completes. This page is the <strong>canonical reference</strong> for all built-in workloads and expectations, including configuration knobs, defaults, prerequisites, and debugging guidance.</p>
<hr />
<h2 id="overview"><a class="header" href="#overview">Overview</a></h2>
<pre><code class="language-mermaid">flowchart TD
I[Inputs&lt;br/&gt;topology + wallets + rates] --&gt; Init[Workload init]
Init --&gt; Drive[Drive traffic]
Drive --&gt; Collect[Collect signals]
Collect --&gt; Eval[Expectations evaluate]
</code></pre>
<p><strong>Key concepts:</strong></p>
<ul>
<li><strong>Workloads</strong> run during the <strong>execution phase</strong> (generate traffic)</li>
<li><strong>Expectations</strong> run during the <strong>evaluation phase</strong> (check health signals)</li>
<li>Each workload can attach its own expectations automatically</li>
<li>Expectations can also be added explicitly</li>
</ul>
<hr />
<h2 id="built-in-workloads"><a class="header" href="#built-in-workloads">Built-in Workloads</a></h2>
<h3 id="1-transaction-workload"><a class="header" href="#1-transaction-workload">1. Transaction Workload</a></h3>
<p>Submits user-level transactions at a configurable rate to exercise transaction processing and inclusion paths.</p>
<p><strong>Import:</strong></p>
<pre><code class="language-rust ignore">use testing_framework_workflows::workloads::transaction::Workload;</code></pre>
<h4 id="configuration"><a class="header" href="#configuration">Configuration</a></h4>
<div class="table-wrapper"><table><thead><tr><th>Parameter</th><th>Type</th><th>Default</th><th>Description</th></tr></thead><tbody>
<tr><td><code>rate</code></td><td><code>u64</code></td><td><strong>Required</strong></td><td>Transactions per block (not per second!)</td></tr>
<tr><td><code>users</code></td><td><code>Option&lt;usize&gt;</code></td><td>All wallets</td><td>Number of distinct wallet accounts to use</td></tr>
</tbody></table>
</div>
<h4 id="dsl-usage"><a class="header" href="#dsl-usage">DSL Usage</a></h4>
<pre><code class="language-rust ignore">use testing_framework_workflows::ScenarioBuilderExt;
ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
.wallets(20) // Seed 20 wallet accounts
.transactions_with(|tx| {
tx.rate(10) // 10 transactions per block
.users(5) // Use only 5 of the 20 wallets
})
.with_run_duration(Duration::from_secs(60))
.build();</code></pre>
<h4 id="direct-instantiation"><a class="header" href="#direct-instantiation">Direct Instantiation</a></h4>
<pre><code class="language-rust ignore">use testing_framework_workflows::workloads::transaction;
let tx_workload = transaction::Workload::with_rate(10)
.expect("transaction rate must be non-zero");
ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
.wallets(20)
.with_workload(tx_workload)
.with_run_duration(Duration::from_secs(60))
.build();</code></pre>
<h4 id="prerequisites-1"><a class="header" href="#prerequisites-1">Prerequisites</a></h4>
<ol>
<li>
<p><strong>Wallet accounts must be seeded:</strong></p>
<pre><code class="language-rust ignore">.wallets(N) // Before .transactions_with()</code></pre>
<p>The workload will fail during <code>init()</code> if no wallets are configured.</p>
</li>
<li>
<p><strong>Proof generation must be fast:</strong></p>
<pre><code class="language-bash">export POL_PROOF_DEV_MODE=true
</code></pre>
<p>Without this, proof generation takes ~30-60 seconds per transaction, causing timeouts.</p>
</li>
<li>
<p><strong>Circuit artifacts must be available:</strong></p>
<ul>
<li>Automatically staged by <code>scripts/run/run-examples.sh</code></li>
<li>Or manually via <code>scripts/setup/setup-circuits-stack.sh</code> (recommended) / <code>scripts/setup/setup-nomos-circuits.sh</code></li>
</ul>
</li>
</ol>
<h4 id="attached-expectation"><a class="header" href="#attached-expectation">Attached Expectation</a></h4>
<p><strong>TxInclusionExpectation</strong> — Verifies that submitted transactions were included in blocks.</p>
<p><strong>What it checks:</strong></p>
<ul>
<li>At least <code>N</code> transactions were included on-chain (where N = rate × user count × expected block count)</li>
<li>Uses BlockFeed to count transactions across all observed blocks</li>
</ul>
<p><strong>Failure modes:</strong></p>
<ul>
<li>"Expected &gt;= X transactions, observed Y" (Y &lt; X)</li>
<li>Common causes: proof generation timeouts, node crashes, insufficient duration</li>
</ul>
<h4 id="what-failure-looks-like"><a class="header" href="#what-failure-looks-like">What Failure Looks Like</a></h4>
<pre><code class="language-text">Error: Expectation failed: TxInclusionExpectation
Expected: &gt;= 600 transactions (10 tx/block × 60 blocks)
Observed: 127 transactions
Possible causes:
- POL_PROOF_DEV_MODE not set (proof generation too slow)
- Duration too short (nodes still syncing)
- Node crashes (check logs for panics/OOM)
- Wallet accounts not seeded (check topology config)
</code></pre>
<p><strong>How to debug:</strong></p>
<ol>
<li>Check logs for proof generation timing:
<pre><code class="language-bash">grep "proof generation" $NOMOS_LOG_DIR/executor-0/*.log
</code></pre>
</li>
<li>Verify <code>POL_PROOF_DEV_MODE=true</code> was set</li>
<li>Increase duration: <code>.with_run_duration(Duration::from_secs(120))</code></li>
<li>Reduce rate: <code>.rate(5)</code> instead of <code>.rate(10)</code></li>
</ol>
<hr />
<h3 id="2-data-availability-da-workload"><a class="header" href="#2-data-availability-da-workload">2. Data Availability (DA) Workload</a></h3>
<p>Drives blob and channel activity to exercise data availability paths and storage.</p>
<p><strong>Import:</strong></p>
<pre><code class="language-rust ignore">use testing_framework_workflows::workloads::da::Workload;</code></pre>
<h4 id="configuration-1"><a class="header" href="#configuration-1">Configuration</a></h4>
<div class="table-wrapper"><table><thead><tr><th>Parameter</th><th>Type</th><th>Default</th><th>Description</th></tr></thead><tbody>
<tr><td><code>blob_rate_per_block</code></td><td><code>NonZeroU64</code></td><td><strong>Required</strong></td><td>Blobs to publish per block</td></tr>
<tr><td><code>channel_rate_per_block</code></td><td><code>NonZeroU64</code></td><td><strong>Required</strong></td><td>Channels to create per block</td></tr>
<tr><td><code>headroom_percent</code></td><td><code>u64</code></td><td><code>20</code></td><td>Extra capacity for channel planning (avoids saturation)</td></tr>
</tbody></table>
</div>
<h4 id="dsl-usage-1"><a class="header" href="#dsl-usage-1">DSL Usage</a></h4>
<pre><code class="language-rust ignore">use testing_framework_workflows::ScenarioBuilderExt;
ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
.da_with(|da| {
da.channel_rate(2) // 2 channels per block
.blob_rate(4) // 4 blobs per block
})
.with_run_duration(Duration::from_secs(120))
.build();</code></pre>
<h4 id="direct-instantiation-1"><a class="header" href="#direct-instantiation-1">Direct Instantiation</a></h4>
<pre><code class="language-rust ignore">use std::num::NonZeroU64;
use testing_framework_workflows::workloads::da;
let da_workload = da::Workload::with_rate(
NonZeroU64::new(4).unwrap(), // blob_rate_per_block
NonZeroU64::new(2).unwrap(), // channel_rate_per_block
20, // headroom_percent
);
ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
.with_workload(da_workload)
.with_run_duration(Duration::from_secs(120))
.build();</code></pre>
<h4 id="prerequisites-2"><a class="header" href="#prerequisites-2">Prerequisites</a></h4>
<ol>
<li>
<p><strong>Executors must be present:</strong></p>
<pre><code class="language-rust ignore">.executors(N) // At least 1 executor</code></pre>
<p>DA workload requires executor nodes to handle blob publishing.</p>
</li>
<li>
<p><strong>Sufficient duration:</strong>
Channel creation and blob publishing are slower than transaction submission. Allow 120+ seconds.</p>
</li>
<li>
<p><strong>Circuit artifacts:</strong>
Same as transaction workload (POL_PROOF_DEV_MODE, circuits staged).</p>
</li>
</ol>
<h4 id="attached-expectation-1"><a class="header" href="#attached-expectation-1">Attached Expectation</a></h4>
<p><strong>DaWorkloadExpectation</strong> — Verifies blobs and channels were created and published.</p>
<p><strong>What it checks:</strong></p>
<ul>
<li>At least <code>N</code> channels were created (where N = channel_rate × expected blocks)</li>
<li>At least <code>M</code> blobs were published (where M = blob_rate × expected blocks × headroom)</li>
<li>Uses BlockFeed and executor API to verify</li>
</ul>
<p><strong>Failure modes:</strong></p>
<ul>
<li>"Expected &gt;= X channels, observed Y" (Y &lt; X)</li>
<li>"Expected &gt;= X blobs, observed Y" (Y &lt; X)</li>
<li>Common causes: executor crashes, insufficient duration, DA saturation</li>
</ul>
<h4 id="what-failure-looks-like-1"><a class="header" href="#what-failure-looks-like-1">What Failure Looks Like</a></h4>
<pre><code class="language-text">Error: Expectation failed: DaWorkloadExpectation
Expected: &gt;= 60 channels (2 channels/block × 30 blocks)
Observed: 23 channels
Possible causes:
- Executors crashed or restarted (check executor logs)
- Duration too short (channels still being created)
- Blob publishing failed (check executor API errors)
- Network issues (check validator/executor connectivity)
</code></pre>
<p><strong>How to debug:</strong></p>
<ol>
<li>Check executor logs:
<pre><code class="language-bash">grep "channel\|blob" $NOMOS_LOG_DIR/executor-0/*.log
</code></pre>
</li>
<li>Verify executors stayed running:
<pre><code class="language-bash">grep "panic\|killed" $NOMOS_LOG_DIR/executor-*/*.log
</code></pre>
</li>
<li>Increase duration: <code>.with_run_duration(Duration::from_secs(180))</code></li>
<li>Reduce rates: <code>.channel_rate(1).blob_rate(2)</code></li>
</ol>
<hr />
<h3 id="3-chaos-workload-random-restart"><a class="header" href="#3-chaos-workload-random-restart">3. Chaos Workload (Random Restart)</a></h3>
<p>Triggers controlled node restarts to test resilience and recovery behaviors.</p>
<p><strong>Import:</strong></p>
<pre><code class="language-rust ignore">use testing_framework_workflows::workloads::chaos::RandomRestartWorkload;</code></pre>
<h4 id="configuration-2"><a class="header" href="#configuration-2">Configuration</a></h4>
<div class="table-wrapper"><table><thead><tr><th>Parameter</th><th>Type</th><th>Default</th><th>Description</th></tr></thead><tbody>
<tr><td><code>min_delay</code></td><td><code>Duration</code></td><td><strong>Required</strong></td><td>Minimum time between restart attempts</td></tr>
<tr><td><code>max_delay</code></td><td><code>Duration</code></td><td><strong>Required</strong></td><td>Maximum time between restart attempts</td></tr>
<tr><td><code>target_cooldown</code></td><td><code>Duration</code></td><td><strong>Required</strong></td><td>Minimum time before restarting same node again</td></tr>
<tr><td><code>include_validators</code></td><td><code>bool</code></td><td><strong>Required</strong></td><td>Whether to restart validators</td></tr>
<tr><td><code>include_executors</code></td><td><code>bool</code></td><td><strong>Required</strong></td><td>Whether to restart executors</td></tr>
</tbody></table>
</div>
<h4 id="usage"><a class="header" href="#usage">Usage</a></h4>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::{ScenarioBuilderExt, workloads::chaos::RandomRestartWorkload};
let scenario = ScenarioBuilder::topology_with(|t| {
t.network_star().validators(3).executors(2)
})
.enable_node_control() // REQUIRED for chaos
.with_workload(RandomRestartWorkload::new(
Duration::from_secs(45), // min_delay
Duration::from_secs(75), // max_delay
Duration::from_secs(120), // target_cooldown
true, // include_validators
true, // include_executors
))
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(180))
.build();</code></pre>
<h4 id="prerequisites-3"><a class="header" href="#prerequisites-3">Prerequisites</a></h4>
<ol>
<li>
<p><strong>Node control must be enabled:</strong></p>
<pre><code class="language-rust ignore">.enable_node_control()</code></pre>
<p>This adds <code>NodeControlCapability</code> to the scenario.</p>
</li>
<li>
<p><strong>Runner must support node control:</strong></p>
<ul>
<li><strong>Compose runner:</strong> Supported</li>
<li><strong>Local runner:</strong> Not supported</li>
<li><strong>K8s runner:</strong> Not yet implemented</li>
</ul>
</li>
<li>
<p><strong>Sufficient topology:</strong></p>
<ul>
<li>For validators: Need &gt;1 validator (workload skips if only 1)</li>
<li>For executors: Can restart all executors</li>
</ul>
</li>
<li>
<p><strong>Realistic timing:</strong></p>
<ul>
<li>Total duration should be 2-3× the max_delay + cooldown</li>
<li>Example: max_delay=75s, cooldown=120s → duration &gt;= 180s</li>
</ul>
</li>
</ol>
<h4 id="attached-expectation-2"><a class="header" href="#attached-expectation-2">Attached Expectation</a></h4>
<p>None. You must explicitly add expectations (typically <code>.expect_consensus_liveness()</code>).</p>
<p><strong>Why?</strong> Chaos workloads are about testing recovery under disruption. The appropriate expectation depends on what you're testing:</p>
<ul>
<li>Consensus survives restarts → <code>.expect_consensus_liveness()</code></li>
<li>Height converges after chaos → Custom expectation checking BlockFeed</li>
</ul>
<h4 id="what-failure-looks-like-2"><a class="header" href="#what-failure-looks-like-2">What Failure Looks Like</a></h4>
<pre><code class="language-text">Error: Workload failed: chaos_restart
Cause: NodeControlHandle not available
Possible causes:
- Forgot .enable_node_control() in scenario builder
- Using local runner (doesn't support node control)
- Using k8s runner (doesn't support node control)
</code></pre>
<p><strong>Or:</strong></p>
<pre><code class="language-text">Error: Expectation failed: ConsensusLiveness
Expected: &gt;= 20 blocks
Observed: 8 blocks
Possible causes:
- Restart frequency too high (nodes can't recover)
- Consensus timing too slow (increase duration)
- Too many validators restarted simultaneously
- Nodes crashed after restart (check logs)
</code></pre>
<p><strong>How to debug:</strong></p>
<ol>
<li>Check restart events in logs:
<pre><code class="language-bash">grep "restarting\|restart complete" $NOMOS_LOG_DIR/*/*.log
</code></pre>
</li>
<li>Verify node control is enabled:
<pre><code class="language-bash">grep "NodeControlHandle" $NOMOS_LOG_DIR/*/*.log
</code></pre>
</li>
<li>Increase cooldown: <code>Duration::from_secs(180)</code></li>
<li>Reduce restart scope: <code>include_validators = false</code> (test executors only)</li>
<li>Increase duration: <code>.with_run_duration(Duration::from_secs(300))</code></li>
</ol>
<hr />
<h2 id="built-in-expectations"><a class="header" href="#built-in-expectations">Built-in Expectations</a></h2>
<h3 id="1-consensus-liveness"><a class="header" href="#1-consensus-liveness">1. Consensus Liveness</a></h3>
<p>Verifies the system continues to produce blocks during the execution window.</p>
<p><strong>Import:</strong></p>
<pre><code class="language-rust ignore">use testing_framework_workflows::ScenarioBuilderExt;</code></pre>
<h4 id="dsl-usage-2"><a class="header" href="#dsl-usage-2">DSL Usage</a></h4>
<pre><code class="language-rust ignore">ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(60))
.build();</code></pre>
<h4 id="what-it-checks"><a class="header" href="#what-it-checks">What It Checks</a></h4>
<ul>
<li>At least <code>N</code> blocks were produced (where N = duration / expected_block_time)</li>
<li>Uses BlockFeed to count observed blocks</li>
<li>Compares against a minimum threshold (typically 50% of theoretical max)</li>
</ul>
<h4 id="failure-modes"><a class="header" href="#failure-modes">Failure Modes</a></h4>
<pre><code class="language-text">Error: Expectation failed: ConsensusLiveness
Expected: &gt;= 30 blocks
Observed: 3 blocks
Possible causes:
- Nodes crashed or never started (check logs)
- Consensus timing misconfigured (CONSENSUS_SLOT_TIME too high)
- Insufficient validators (need &gt;= 2 for BFT consensus)
- Duration too short (nodes still syncing)
</code></pre>
<h4 id="how-to-debug"><a class="header" href="#how-to-debug">How to Debug</a></h4>
<ol>
<li>Check if nodes started:
<pre><code class="language-bash">grep "node started\|listening on" $NOMOS_LOG_DIR/*/*.log
</code></pre>
</li>
<li>Check block production:
<pre><code class="language-bash">grep "block.*height" $NOMOS_LOG_DIR/validator-*/*.log
</code></pre>
</li>
<li>Check consensus participation:
<pre><code class="language-bash">grep "consensus.*slot\|proposal" $NOMOS_LOG_DIR/validator-*/*.log
</code></pre>
</li>
<li>Increase duration: <code>.with_run_duration(Duration::from_secs(120))</code></li>
<li>Check env vars: <code>echo $CONSENSUS_SLOT_TIME $CONSENSUS_ACTIVE_SLOT_COEFF</code></li>
</ol>
<hr />
<h3 id="2-workload-specific-expectations"><a class="header" href="#2-workload-specific-expectations">2. Workload-Specific Expectations</a></h3>
<p>Each workload automatically attaches its own expectation:</p>
<div class="table-wrapper"><table><thead><tr><th>Workload</th><th>Expectation</th><th>What It Checks</th></tr></thead><tbody>
<tr><td>Transaction</td><td><code>TxInclusionExpectation</code></td><td>Transactions were included in blocks</td></tr>
<tr><td>DA</td><td><code>DaWorkloadExpectation</code></td><td>Blobs and channels were created/published</td></tr>
<tr><td>Chaos</td><td>(None)</td><td>Add <code>.expect_consensus_liveness()</code> explicitly</td></tr>
</tbody></table>
</div>
<p>These expectations are added automatically when using the DSL (<code>.transactions_with()</code>, <code>.da_with()</code>).</p>
<hr />
<h2 id="configuration-quick-reference"><a class="header" href="#configuration-quick-reference">Configuration Quick Reference</a></h2>
<h3 id="transaction-workload"><a class="header" href="#transaction-workload">Transaction Workload</a></h3>
<pre><code class="language-rust ignore">.wallets(20)
.transactions_with(|tx| tx.rate(10).users(5))</code></pre>
<div class="table-wrapper"><table><thead><tr><th>What</th><th>Value</th><th>Unit</th></tr></thead><tbody>
<tr><td>Rate</td><td>10</td><td>tx/block</td></tr>
<tr><td>Users</td><td>5</td><td>wallet accounts</td></tr>
<tr><td>Wallets</td><td>20</td><td>total seeded</td></tr>
</tbody></table>
</div>
<h3 id="da-workload"><a class="header" href="#da-workload">DA Workload</a></h3>
<pre><code class="language-rust ignore">.da_with(|da| da.channel_rate(2).blob_rate(4))</code></pre>
<div class="table-wrapper"><table><thead><tr><th>What</th><th>Value</th><th>Unit</th></tr></thead><tbody>
<tr><td>Channel rate</td><td>2</td><td>channels/block</td></tr>
<tr><td>Blob rate</td><td>4</td><td>blobs/block</td></tr>
<tr><td>Headroom</td><td>20</td><td>percent</td></tr>
</tbody></table>
</div>
<h3 id="chaos-workload"><a class="header" href="#chaos-workload">Chaos Workload</a></h3>
<pre><code class="language-rust ignore">.enable_node_control()
.with_workload(RandomRestartWorkload::new(
Duration::from_secs(45), // min
Duration::from_secs(75), // max
Duration::from_secs(120), // cooldown
true, // validators
true, // executors
))</code></pre>
<hr />
<h2 id="common-patterns"><a class="header" href="#common-patterns">Common Patterns</a></h2>
<h3 id="pattern-1-multiple-workloads"><a class="header" href="#pattern-1-multiple-workloads">Pattern 1: Multiple Workloads</a></h3>
<pre><code class="language-rust ignore">ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
.wallets(20)
.transactions_with(|tx| tx.rate(5).users(10))
.da_with(|da| da.channel_rate(2).blob_rate(2))
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(120))
.build();</code></pre>
<p>All workloads run concurrently. Expectations for each workload run after the execution window ends.</p>
<h3 id="pattern-2-custom-expectation"><a class="header" href="#pattern-2-custom-expectation">Pattern 2: Custom Expectation</a></h3>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::Expectation;
struct MyCustomExpectation;
#[async_trait]
impl Expectation for MyCustomExpectation {
async fn evaluate(&amp;self, ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
// Access BlockFeed, metrics, topology, etc.
let block_count = ctx.block_feed()?.count();
if block_count &lt; 10 {
return Err("Not enough blocks".into());
}
Ok(())
}
}
ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
.with_expectation(MyCustomExpectation)
.with_run_duration(Duration::from_secs(60))
.build();</code></pre>
<hr />
<h2 id="debugging-checklist"><a class="header" href="#debugging-checklist">Debugging Checklist</a></h2>
<p>When a workload or expectation fails:</p>
<ol>
<li>Check logs: <code>$NOMOS_LOG_DIR/*/</code> or <code>docker compose logs</code> or <code>kubectl logs</code></li>
<li>Verify environment variables: <code>POL_PROOF_DEV_MODE</code>, <code>NOMOS_NODE_BIN</code>, etc.</li>
<li>Check prerequisites: wallets, executors, node control, circuits</li>
<li>Increase duration: Double the run duration and retry</li>
<li>Reduce rates: Half the traffic rates and retry</li>
<li>Check metrics: Prometheus queries for block height, tx count, DA stats</li>
<li>Reproduce locally: Use local runner for faster iteration</li>
</ol>
<hr />
<h2 id="see-also"><a class="header" href="#see-also">See Also</a></h2>
<ul>
<li><strong><a href="authoring-scenarios.html">Authoring Scenarios</a></strong> — Step-by-step tutorial for building scenarios</li>
<li><strong><a href="node-control.html">RunContext: BlockFeed &amp; Node Control</a></strong> — Learn how to use BlockFeed in expectations and access node control</li>
<li><strong><a href="examples.html">Examples</a></strong> — Concrete scenario patterns combining workloads and expectations</li>
<li><strong><a href="extending.html">Extending the Framework</a></strong> — Implement custom workloads and expectations</li>
<li><strong><a href="troubleshooting.html">Troubleshooting</a></strong> — Common failure scenarios and fixes</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="core-content-scenariobuilderext-patterns"><a class="header" href="#core-content-scenariobuilderext-patterns">Core Content: ScenarioBuilderExt Patterns</a></h1>
<blockquote>
<p><strong>When should I read this?</strong> After writing 2-3 scenarios. This page documents patterns that emerge from real usage—come back when you're refactoring or standardizing your test suite.</p>
</blockquote>
<p>Patterns that keep scenarios readable and reusable:</p>
<ul>
<li><strong>Topology-first</strong>: start by shaping the cluster (counts, layout) so later
steps inherit a clear foundation.</li>
<li><strong>Bundle defaults</strong>: use the DSL helpers to attach common expectations (like
liveness) whenever you add a matching workload, reducing forgotten checks.</li>
<li><strong>Intentional rates</strong>: express traffic in per-block terms to align with
protocol timing rather than wall-clock assumptions.</li>
<li><strong>Opt-in chaos</strong>: enable restart patterns only in scenarios meant to probe
resilience; keep functional smoke tests deterministic.</li>
<li><strong>Wallet clarity</strong>: seed only the number of actors you need; it keeps
transaction scenarios deterministic and interpretable.</li>
</ul>
<p>These patterns make scenario definitions self-explanatory while staying aligned
with the framework’s block-oriented timing model.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="best-practices-1"><a class="header" href="#best-practices-1">Best Practices</a></h1>
<p>This page collects proven patterns for authoring, running, and maintaining test scenarios that are reliable, maintainable, and actionable.</p>
<h2 id="scenario-design"><a class="header" href="#scenario-design">Scenario Design</a></h2>
<p><strong>State your intent</strong></p>
<ul>
<li>Document the goal of each scenario (throughput, DA validation, resilience) so expectation choices are obvious</li>
<li>Use descriptive variable names that explain topology purpose (e.g., <code>star_topology_3val_2exec</code> vs <code>topology</code>)</li>
<li>Add comments explaining why specific rates or durations were chosen</li>
</ul>
<p><strong>Keep runs meaningful</strong></p>
<ul>
<li>Choose durations that allow multiple blocks and make timing-based assertions trustworthy</li>
<li>Use <a href="faq.html#how-long-should-a-scenario-run">FAQ: Run Duration Calculator</a> to estimate minimum duration</li>
<li>Avoid runs shorter than 30 seconds unless testing startup behavior specifically</li>
</ul>
<p><strong>Separate concerns</strong></p>
<ul>
<li>Start with deterministic workloads for functional checks</li>
<li>Add chaos in dedicated resilience scenarios to avoid noisy failures</li>
<li>Don't mix high transaction load with aggressive chaos in the same test (hard to debug)</li>
</ul>
<p><strong>Start small, scale up</strong></p>
<ul>
<li>Begin with minimal topology (1-2 validators) to validate scenario logic</li>
<li>Gradually increase topology size and workload rates</li>
<li>Use Host runner for fast iteration, then validate on Compose before production</li>
</ul>
<h2 id="code-organization"><a class="header" href="#code-organization">Code Organization</a></h2>
<p><strong>Reuse patterns</strong></p>
<ul>
<li>Standardize on shared topology and workload presets so results are comparable across environments and teams</li>
<li>Extract common topology builders into helper functions</li>
<li>Create workspace-level constants for standard rates and durations</li>
</ul>
<p><strong>Example: Topology preset</strong></p>
<pre><code class="language-rust ignore">pub fn standard_da_topology() -&gt; GeneratedTopology {
TopologyBuilder::new()
.network_star()
.validators(3)
.executors(2)
.generate()
}</code></pre>
<p><strong>Example: Shared constants</strong></p>
<pre><code class="language-rust ignore">pub const STANDARD_TX_RATE: f64 = 10.0;
pub const STANDARD_DA_CHANNEL_RATE: f64 = 2.0;
pub const SHORT_RUN_DURATION: Duration = Duration::from_secs(60);
pub const LONG_RUN_DURATION: Duration = Duration::from_secs(300);</code></pre>
<h2 id="debugging--observability"><a class="header" href="#debugging--observability">Debugging &amp; Observability</a></h2>
<p><strong>Observe first, tune second</strong></p>
<ul>
<li>Rely on liveness and inclusion signals to interpret outcomes before tweaking rates or topology</li>
<li>Enable detailed logging (<code>RUST_LOG=debug</code>, <code>NOMOS_LOG_LEVEL=debug</code>) only after initial failure</li>
<li>Use <code>NOMOS_TESTS_KEEP_LOGS=1</code> to persist logs when debugging failures</li>
</ul>
<p><strong>Use BlockFeed effectively</strong></p>
<ul>
<li>Subscribe to BlockFeed in expectations for real-time block monitoring</li>
<li>Track block production rate to detect liveness issues early</li>
<li>Use block statistics (<code>block_feed.stats().total_transactions()</code>) to verify inclusion</li>
</ul>
<p><strong>Collect metrics</strong></p>
<ul>
<li>Set up Prometheus/Grafana via <code>scripts/setup/setup-observability.sh compose up</code> for visualizing node behavior</li>
<li>Use metrics to identify bottlenecks before adding more load</li>
<li>Monitor mempool size, block size, and consensus timing</li>
</ul>
<h2 id="environment--runner-selection"><a class="header" href="#environment--runner-selection">Environment &amp; Runner Selection</a></h2>
<p><strong>Environment fit</strong></p>
<ul>
<li>Pick runners that match the feedback loop you need:
<ul>
<li><strong>Host</strong>: Fast iteration during development, quick CI smoke tests</li>
<li><strong>Compose</strong>: Reproducible environments (recommended for CI), chaos testing</li>
<li><strong>K8s</strong>: Production-like fidelity, large topologies (10+ nodes)</li>
</ul>
</li>
</ul>
<p><strong>Runner-specific considerations</strong></p>
<div class="table-wrapper"><table><thead><tr><th>Runner</th><th>When to Use</th><th>When to Avoid</th></tr></thead><tbody>
<tr><td>Host</td><td>Development iteration, fast feedback</td><td>Chaos testing, container-specific issues</td></tr>
<tr><td>Compose</td><td>CI pipelines, chaos tests, reproducibility</td><td>Very large topologies (&gt;10 nodes)</td></tr>
<tr><td>K8s</td><td>Production-like testing, cluster behaviors</td><td>Local development, fast iteration</td></tr>
</tbody></table>
</div>
<p><strong>Minimal surprises</strong></p>
<ul>
<li>Seed only necessary wallets and keep configuration deltas explicit when moving between CI and developer machines</li>
<li>Use <code>versions.env</code> to pin node versions consistently across environments</li>
<li>Document non-default environment variables in scenario comments or README</li>
</ul>
<h2 id="cicd-integration"><a class="header" href="#cicd-integration">CI/CD Integration</a></h2>
<p><strong>Use matrix builds</strong></p>
<pre><code class="language-yaml">strategy:
matrix:
runner: [host, compose]
topology: [small, medium]
</code></pre>
<p><strong>Cache aggressively</strong></p>
<ul>
<li>Cache Rust build artifacts (<code>target/</code>)</li>
<li>Cache circuit parameters (<code>assets/stack/kzgrs_test_params/</code>)</li>
<li>Cache Docker layers (use BuildKit cache)</li>
</ul>
<p><strong>Collect logs on failure</strong></p>
<pre><code class="language-yaml">- name: Collect logs on failure
if: failure()
run: |
mkdir -p test-logs
find /tmp -name "nomos-*.log" -exec cp {} test-logs/ \;
- uses: actions/upload-artifact@v3
if: failure()
with:
name: test-logs-${{ matrix.runner }}
path: test-logs/
</code></pre>
<p><strong>Time limits</strong></p>
<ul>
<li>Set job timeout to prevent hung runs: <code>timeout-minutes: 30</code></li>
<li>Use shorter durations in CI (60s) vs local testing (300s)</li>
<li>Run expensive tests (k8s, large topologies) only on main branch or release tags</li>
</ul>
<p><strong>See also:</strong> <a href="ci-integration.html">CI Integration</a> for complete workflow examples</p>
<h2 id="anti-patterns-to-avoid"><a class="header" href="#anti-patterns-to-avoid">Anti-Patterns to Avoid</a></h2>
<p><strong>DON'T: Run without POL_PROOF_DEV_MODE</strong></p>
<pre><code class="language-bash"># BAD: Will hang/timeout on proof generation
cargo run -p runner-examples --bin local_runner
# GOOD: Fast mode for testing
POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
</code></pre>
<p><strong>DON'T: Use tiny durations</strong></p>
<pre><code class="language-rust ignore">// BAD: Not enough time for blocks to propagate
.with_run_duration(Duration::from_secs(5))
// GOOD: Allow multiple consensus rounds
.with_run_duration(Duration::from_secs(60))</code></pre>
<p><strong>DON'T: Ignore cleanup failures</strong></p>
<pre><code class="language-rust ignore">// BAD: Next run inherits leaked state
runner.run(&amp;mut scenario).await?;
// forgot to call cleanup or use CleanupGuard
// GOOD: Cleanup via guard (automatic on panic)
let _cleanup = CleanupGuard::new(runner.clone());
runner.run(&amp;mut scenario).await?;</code></pre>
<p><strong>DON'T: Mix concerns in one scenario</strong></p>
<pre><code class="language-rust ignore">// BAD: Hard to debug when it fails
.transactions_with(|tx| tx.rate(50).users(100)) // high load
.chaos_with(|c| c.restart().min_delay(...)) // AND chaos
.da_with(|da| da.channel_rate(10).blob_rate(20)) // AND DA stress
// GOOD: Separate tests for each concern
// Test 1: High transaction load only
// Test 2: Chaos resilience only
// Test 3: DA stress only</code></pre>
<p><strong>DON'T: Hardcode paths or ports</strong></p>
<pre><code class="language-rust ignore">// BAD: Breaks on different machines
let path = PathBuf::from("/home/user/circuits/kzgrs_test_params");
let port = 9000; // might conflict
// GOOD: Use env vars and dynamic allocation
let path = std::env::var("NOMOS_KZGRS_PARAMS_PATH")
.unwrap_or_else(|_| "assets/stack/kzgrs_test_params/kzgrs_test_params".to_string());
let port = get_available_tcp_port();</code></pre>
<p><strong>DON'T: Ignore resource limits</strong></p>
<pre><code class="language-bash"># BAD: Large topology without checking resources
scripts/run/run-examples.sh -v 20 -e 10 compose
# (might OOM or exhaust ulimits)
# GOOD: Scale gradually and monitor resources
scripts/run/run-examples.sh -v 3 -e 2 compose # start small
docker stats # monitor resource usage
# then increase if resources allow
</code></pre>
<h2 id="scenario-design-heuristics"><a class="header" href="#scenario-design-heuristics">Scenario Design Heuristics</a></h2>
<p><strong>Minimal viable topology</strong></p>
<ul>
<li>Consensus: 3 validators (minimum for Byzantine fault tolerance)</li>
<li>DA: 2+ executors (test dispersal and sampling)</li>
<li>Network: Star topology (simplest for debugging)</li>
</ul>
<p><strong>Workload rate selection</strong></p>
<ul>
<li>Start with 1-5 tx/s per user, then increase</li>
<li>DA: 1-2 channels, 1-3 blobs/channel initially</li>
<li>Chaos: 30s+ intervals between restarts (allow recovery)</li>
</ul>
<p><strong>Duration guidelines</strong></p>
<div class="table-wrapper"><table><thead><tr><th>Test Type</th><th>Minimum Duration</th><th>Typical Duration</th></tr></thead><tbody>
<tr><td>Smoke test</td><td>30s</td><td>60s</td></tr>
<tr><td>Integration test</td><td>60s</td><td>120s</td></tr>
<tr><td>Load test</td><td>120s</td><td>300s</td></tr>
<tr><td>Resilience test</td><td>120s</td><td>300s</td></tr>
<tr><td>Soak test</td><td>600s (10m)</td><td>3600s (1h)</td></tr>
</tbody></table>
</div>
<p><strong>Expectation selection</strong></p>
<div class="table-wrapper"><table><thead><tr><th>Test Goal</th><th>Expectations</th></tr></thead><tbody>
<tr><td>Basic functionality</td><td><code>expect_consensus_liveness()</code></td></tr>
<tr><td>Transaction handling</td><td><code>expect_consensus_liveness()</code> + custom inclusion check</td></tr>
<tr><td>DA correctness</td><td><code>expect_consensus_liveness()</code> + DA dispersal/sampling checks</td></tr>
<tr><td>Resilience</td><td><code>expect_consensus_liveness()</code> + recovery time measurement</td></tr>
</tbody></table>
</div>
<h2 id="testing-the-tests"><a class="header" href="#testing-the-tests">Testing the Tests</a></h2>
<p><strong>Validate scenarios before committing</strong></p>
<ol>
<li>Run on Host runner first (fast feedback)</li>
<li>Run on Compose runner (reproducibility check)</li>
<li>Check logs for warnings or errors</li>
<li>Verify cleanup (no leaked processes/containers)</li>
<li>Run 2-3 times to check for flakiness</li>
</ol>
<p><strong>Handling flaky tests</strong></p>
<ul>
<li>Increase run duration (timing-sensitive assertions need longer runs)</li>
<li>Reduce workload rates (might be saturating nodes)</li>
<li>Check resource limits (CPU/RAM/ulimits)</li>
<li>Add debugging output to identify race conditions</li>
<li>Consider if test is over-specified (too strict expectations)</li>
</ul>
<p><strong>See also:</strong></p>
<ul>
<li><a href="troubleshooting.html">Troubleshooting</a> for common failure patterns</li>
<li><a href="faq.html">FAQ</a> for design decisions and gotchas</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="usage-patterns"><a class="header" href="#usage-patterns">Usage Patterns</a></h1>
<ul>
<li><strong>Shape a topology, pick a runner</strong>: choose local for quick iteration, compose
for reproducible multi-node stacks with observability, or k8s for cluster-grade
validation.</li>
<li><strong>Compose workloads deliberately</strong>: pair transactions and data-availability
traffic for end-to-end coverage; add chaos only when assessing recovery and
resilience.</li>
<li><strong>Align expectations with goals</strong>: use liveness-style checks to confirm the
system keeps up with planned activity, and add workload-specific assertions for
inclusion or availability.</li>
<li><strong>Reuse plans across environments</strong>: keep the scenario constant while swapping
runners to compare behavior between developer machines and CI clusters.</li>
<li><strong>Iterate with clear signals</strong>: treat expectation outcomes as the primary
pass/fail indicator, and adjust topology or workloads based on what those
signals reveal.</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="examples"><a class="header" href="#examples">Examples</a></h1>
<p>Concrete scenario shapes that illustrate how to combine topologies, workloads,
and expectations.</p>
<p><strong>View Complete Source Code:</strong></p>
<ul>
<li><a href="https://github.com/logos-blockchain/logos-blockchain-testing/blob/master/examples/src/bin/local_runner.rs">local_runner.rs</a> — Host processes (local)</li>
<li><a href="https://github.com/logos-blockchain/logos-blockchain-testing/blob/master/examples/src/bin/compose_runner.rs">compose_runner.rs</a> — Docker Compose</li>
<li><a href="https://github.com/logos-blockchain/logos-blockchain-testing/blob/master/examples/src/bin/k8s_runner.rs">k8s_runner.rs</a> — Kubernetes</li>
</ul>
<p><strong>Runnable examples:</strong> The repo includes complete binaries in <code>examples/src/bin/</code>:</p>
<ul>
<li><code>local_runner.rs</code> — Host processes (local)</li>
<li><code>compose_runner.rs</code> — Docker Compose (requires image built)</li>
<li><code>k8s_runner.rs</code> — Kubernetes (requires cluster access and image loaded)</li>
</ul>
<p><strong>Recommended:</strong> Use <code>scripts/run/run-examples.sh -t &lt;duration&gt; -v &lt;validators&gt; -e &lt;executors&gt; &lt;mode&gt;</code> where mode is <code>host</code>, <code>compose</code>, or <code>k8s</code>.</p>
<p><strong>Alternative:</strong> Direct cargo run: <code>POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin &lt;name&gt;</code></p>
<p><strong>All runners require <code>POL_PROOF_DEV_MODE=true</code></strong> to avoid expensive proof generation.</p>
<p><strong>Code patterns</strong> below show how to build scenarios. Wrap these in <code>#[tokio::test]</code> functions for integration tests, or <code>#[tokio::main]</code> for binaries.</p>
<h2 id="simple-consensus-liveness"><a class="header" href="#simple-consensus-liveness">Simple consensus liveness</a></h2>
<p>Minimal test that validates basic block production:</p>
<pre><code class="language-rust ignore">use std::time::Duration;
use anyhow::Result;
use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
use testing_framework_runner_local::LocalDeployer;
use testing_framework_workflows::ScenarioBuilderExt;
pub async fn simple_consensus() -&gt; Result&lt;()&gt; {
let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(0))
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(30))
.build();
let deployer = LocalDeployer::default();
let runner = deployer.deploy(&amp;plan).await?;
let _handle = runner.run(&amp;mut plan).await?;
Ok(())
}</code></pre>
<p><strong>When to use</strong>: smoke tests for consensus on minimal hardware.</p>
<h2 id="transaction-workload-1"><a class="header" href="#transaction-workload-1">Transaction workload</a></h2>
<p>Test consensus under transaction load:</p>
<pre><code class="language-rust ignore">use std::time::Duration;
use anyhow::Result;
use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
use testing_framework_runner_local::LocalDeployer;
use testing_framework_workflows::ScenarioBuilderExt;
pub async fn transaction_workload() -&gt; Result&lt;()&gt; {
let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(2).executors(0))
.wallets(20)
.transactions_with(|txs| txs.rate(5).users(10))
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(60))
.build();
let deployer = LocalDeployer::default();
let runner = deployer.deploy(&amp;plan).await?;
let _handle = runner.run(&amp;mut plan).await?;
Ok(())
}</code></pre>
<p><strong>When to use</strong>: validate transaction submission and inclusion.</p>
<h2 id="da--transaction-workload"><a class="header" href="#da--transaction-workload">DA + transaction workload</a></h2>
<p>Combined test stressing both transaction and DA layers:</p>
<pre><code class="language-rust ignore">use std::time::Duration;
use anyhow::Result;
use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
use testing_framework_runner_local::LocalDeployer;
use testing_framework_workflows::ScenarioBuilderExt;
pub async fn da_and_transactions() -&gt; Result&lt;()&gt; {
let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
.wallets(30)
.transactions_with(|txs| txs.rate(5).users(15))
.da_with(|da| da.channel_rate(2).blob_rate(2))
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(90))
.build();
let deployer = LocalDeployer::default();
let runner = deployer.deploy(&amp;plan).await?;
let _handle = runner.run(&amp;mut plan).await?;
Ok(())
}</code></pre>
<p><strong>When to use</strong>: end-to-end coverage of transaction and DA layers.</p>
<h2 id="chaos-resilience"><a class="header" href="#chaos-resilience">Chaos resilience</a></h2>
<p>Test system resilience under node restarts:</p>
<pre><code class="language-rust ignore">use std::time::Duration;
use anyhow::Result;
use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
use testing_framework_runner_compose::ComposeDeployer;
use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt};
pub async fn chaos_resilience() -&gt; Result&lt;()&gt; {
let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(4).executors(2))
.enable_node_control()
.wallets(20)
.transactions_with(|txs| txs.rate(3).users(10))
.chaos_with(|c| {
c.restart()
.min_delay(Duration::from_secs(20))
.max_delay(Duration::from_secs(40))
.target_cooldown(Duration::from_secs(30))
.apply()
})
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(120))
.build();
let deployer = ComposeDeployer::default();
let runner = deployer.deploy(&amp;plan).await?;
let _handle = runner.run(&amp;mut plan).await?;
Ok(())
}</code></pre>
<p><strong>When to use</strong>: resilience validation and operational readiness drills.</p>
<p><strong>Note</strong>: Chaos tests require <code>ComposeDeployer</code> or another runner with node control support.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="advanced-examples"><a class="header" href="#advanced-examples">Advanced Examples</a></h1>
<blockquote>
<p><strong>When should I read this?</strong> Skim now to see what's possible, revisit later when you need load testing, chaos scenarios, or custom extensions. Start with <a href="examples.html">basic examples</a> first.</p>
</blockquote>
<p>Realistic advanced scenarios demonstrating framework capabilities for production testing.</p>
<p><strong>Adapt from Complete Source:</strong></p>
<ul>
<li><a href="https://github.com/logos-blockchain/logos-blockchain-testing/blob/master/examples/src/bin/compose_runner.rs">compose_runner.rs</a> — Compose examples with workloads</li>
<li><a href="https://github.com/logos-blockchain/logos-blockchain-testing/blob/master/examples/src/bin/k8s_runner.rs">k8s_runner.rs</a> — K8s production patterns</li>
<li><a href="https://github.com/logos-blockchain/logos-blockchain-testing/blob/master/testing-framework/workflows/src/workloads/chaos.rs">Chaos testing patterns</a> — Node control implementation</li>
</ul>
<h2 id="summary-1"><a class="header" href="#summary-1">Summary</a></h2>
<div class="table-wrapper"><table><thead><tr><th>Example</th><th>Topology</th><th>Workloads</th><th>Deployer</th><th>Key Feature</th></tr></thead><tbody>
<tr><td>Load Progression</td><td>3 validators + 2 executors</td><td>Increasing tx rate</td><td>Compose</td><td>Dynamic load testing</td></tr>
<tr><td>Sustained Load</td><td>4 validators + 2 executors</td><td>High tx + DA rate</td><td>Compose</td><td>Stress testing</td></tr>
<tr><td>Aggressive Chaos</td><td>4 validators + 2 executors</td><td>Frequent restarts + traffic</td><td>Compose</td><td>Resilience validation</td></tr>
</tbody></table>
</div>
<h2 id="load-progression-test"><a class="header" href="#load-progression-test">Load Progression Test</a></h2>
<p>Test consensus under progressively increasing transaction load:</p>
<pre><code class="language-rust ignore">use std::time::Duration;
use anyhow::Result;
use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
use testing_framework_runner_compose::ComposeDeployer;
use testing_framework_workflows::ScenarioBuilderExt;
pub async fn load_progression_test() -&gt; Result&lt;()&gt; {
for rate in [5, 10, 20, 30] {
println!("Testing with rate: {}", rate);
let mut plan =
ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
.wallets(50)
.transactions_with(|txs| txs.rate(rate).users(20))
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(60))
.build();
let deployer = ComposeDeployer::default();
let runner = deployer.deploy(&amp;plan).await?;
let _handle = runner.run(&amp;mut plan).await?;
}
Ok(())
}</code></pre>
<p><strong>When to use:</strong> Finding the maximum sustainable transaction rate for a given topology.</p>
<h2 id="sustained-load-test"><a class="header" href="#sustained-load-test">Sustained Load Test</a></h2>
<p>Run high transaction and DA load for extended duration:</p>
<pre><code class="language-rust ignore">use std::time::Duration;
use anyhow::Result;
use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
use testing_framework_runner_compose::ComposeDeployer;
use testing_framework_workflows::ScenarioBuilderExt;
pub async fn sustained_load_test() -&gt; Result&lt;()&gt; {
let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(4).executors(2))
.wallets(100)
.transactions_with(|txs| txs.rate(15).users(50))
.da_with(|da| da.channel_rate(2).blob_rate(3))
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(300))
.build();
let deployer = ComposeDeployer::default();
let runner = deployer.deploy(&amp;plan).await?;
let _handle = runner.run(&amp;mut plan).await?;
Ok(())
}</code></pre>
<p><strong>When to use:</strong> Validating stability under continuous high load over extended periods.</p>
<h2 id="aggressive-chaos-test"><a class="header" href="#aggressive-chaos-test">Aggressive Chaos Test</a></h2>
<p>Frequent node restarts with active traffic:</p>
<pre><code class="language-rust ignore">use std::time::Duration;
use anyhow::Result;
use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
use testing_framework_runner_compose::ComposeDeployer;
use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt};
pub async fn aggressive_chaos_test() -&gt; Result&lt;()&gt; {
let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(4).executors(2))
.enable_node_control()
.wallets(50)
.transactions_with(|txs| txs.rate(10).users(20))
.chaos_with(|c| {
c.restart()
.min_delay(Duration::from_secs(10))
.max_delay(Duration::from_secs(20))
.target_cooldown(Duration::from_secs(15))
.apply()
})
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(180))
.build();
let deployer = ComposeDeployer::default();
let runner = deployer.deploy(&amp;plan).await?;
let _handle = runner.run(&amp;mut plan).await?;
Ok(())
}</code></pre>
<p><strong>When to use:</strong> Validating recovery and liveness under aggressive failure conditions.</p>
<p><strong>Note:</strong> Requires <code>ComposeDeployer</code> for node control support.</p>
<h2 id="extension-ideas"><a class="header" href="#extension-ideas">Extension Ideas</a></h2>
<p>These scenarios require custom implementations but demonstrate framework extensibility:</p>
<h3 id="mempool--transaction-handling"><a class="header" href="#mempool--transaction-handling">Mempool &amp; Transaction Handling</a></h3>
<h4 id="transaction-propagation--inclusion-test"><a class="header" href="#transaction-propagation--inclusion-test">Transaction Propagation &amp; Inclusion Test</a></h4>
<p><strong>Concept:</strong> Submit the same batch of independent transactions to different nodes in randomized order/offsets, then verify all transactions are included and final state matches across nodes.</p>
<p><strong>Requirements:</strong></p>
<ul>
<li><strong>Custom workload:</strong> Generates a fixed batch of transactions and submits the same set to different nodes via <code>ctx.node_clients()</code>, with randomized submission order and timing offsets per node</li>
<li><strong>Custom expectation:</strong> Verifies all transactions appear in blocks (order may vary), final state matches across all nodes (compare balances or state roots), and no transactions are dropped</li>
</ul>
<p><strong>Why useful:</strong> Exercises mempool propagation, proposer fairness, and transaction inclusion guarantees under realistic race conditions. Tests that the protocol maintains consistency regardless of which node receives transactions first.</p>
<p><strong>Implementation notes:</strong> Requires both a custom <code>Workload</code> implementation (to submit same transactions to multiple nodes with jitter) and a custom <code>Expectation</code> implementation (to verify inclusion and state consistency).</p>
<h4 id="cross-validator-mempool-divergence--convergence"><a class="header" href="#cross-validator-mempool-divergence--convergence">Cross-Validator Mempool Divergence &amp; Convergence</a></h4>
<p><strong>Concept:</strong> Drive different transaction subsets into different validators (or differing arrival orders) to create temporary mempool divergence, then verify mempools/blocks converge to contain the union (no permanent divergence).</p>
<p><strong>Requirements:</strong></p>
<ul>
<li><strong>Custom workload:</strong> Targets specific nodes via <code>ctx.node_clients()</code> with disjoint or jittered transaction batches</li>
<li><strong>Custom expectation:</strong> After a convergence window, verifies that all transactions appear in blocks (order may vary) or that mempool contents converge across nodes</li>
<li>Run normal workloads during convergence period</li>
</ul>
<p><strong>Expectations:</strong></p>
<ul>
<li>Temporary mempool divergence is acceptable (different nodes see different transactions initially)</li>
<li>After convergence window, all transactions appear in blocks or mempools converge</li>
<li>No transactions are permanently dropped despite initial divergence</li>
<li>Mempool gossip/reconciliation mechanisms work correctly</li>
</ul>
<p><strong>Why useful:</strong> Exercises mempool gossip and reconciliation under uneven input or latency. Ensures no node "drops" transactions seen elsewhere, validating that mempool synchronization mechanisms correctly propagate transactions across the network even when they arrive at different nodes in different orders.</p>
<p><strong>Implementation notes:</strong> Requires both a custom <code>Workload</code> implementation (to inject disjoint/jittered batches per node) and a custom <code>Expectation</code> implementation (to verify mempool convergence or block inclusion). Uses existing <code>ctx.node_clients()</code> capability—no new infrastructure needed.</p>
<h4 id="adaptive-mempool-pressure-test"><a class="header" href="#adaptive-mempool-pressure-test">Adaptive Mempool Pressure Test</a></h4>
<p><strong>Concept:</strong> Ramp transaction load over time to observe mempool growth, fee prioritization/eviction, and block saturation behavior, detecting performance regressions and ensuring backpressure/eviction work under increasing load.</p>
<p><strong>Requirements:</strong></p>
<ul>
<li><strong>Custom workload:</strong> Steadily increases transaction rate over time (optional: use fee tiers)</li>
<li><strong>Custom expectation:</strong> Monitors mempool size, evictions, and throughput (blocks/txs per slot), flagging runaway growth or stalls</li>
<li>Run for extended duration to observe pressure buildup</li>
</ul>
<p><strong>Expectations:</strong></p>
<ul>
<li>Mempool size grows predictably with load (not runaway growth)</li>
<li>Fee prioritization/eviction mechanisms activate under pressure</li>
<li>Block saturation behavior is acceptable (blocks fill appropriately)</li>
<li>Throughput (blocks/txs per slot) remains stable or degrades gracefully</li>
<li>No stalls or unbounded mempool growth</li>
</ul>
<p><strong>Why useful:</strong> Detects performance regressions in mempool management. Ensures backpressure and eviction mechanisms work correctly under increasing load, preventing memory exhaustion or unbounded growth. Validates that fee prioritization correctly selects high-value transactions when mempool is full.</p>
<p><strong>Implementation notes:</strong> Can be built with current workload model (ramping rate). Requires custom <code>Expectation</code> implementation that reads mempool metrics (via node HTTP APIs or Prometheus) and monitors throughput to judge behavior. No new infrastructure needed—uses existing observability capabilities.</p>
<h4 id="invalid-transaction-fuzzing"><a class="header" href="#invalid-transaction-fuzzing">Invalid Transaction Fuzzing</a></h4>
<p><strong>Concept:</strong> Submit malformed transactions and verify they're rejected properly.</p>
<p><strong>Implementation approach:</strong></p>
<ul>
<li>Custom workload that generates invalid transactions (bad signatures, insufficient funds, malformed structure)</li>
<li>Expectation verifies mempool rejects them and they never appear in blocks</li>
<li>Test mempool resilience and filtering</li>
</ul>
<p><strong>Why useful:</strong> Ensures mempool doesn't crash or include invalid transactions under fuzzing.</p>
<h3 id="network--gossip"><a class="header" href="#network--gossip">Network &amp; Gossip</a></h3>
<h4 id="gossip-latency-gradient-scenario"><a class="header" href="#gossip-latency-gradient-scenario">Gossip Latency Gradient Scenario</a></h4>
<p><strong>Concept:</strong> Test consensus robustness under skewed gossip delays by partitioning nodes into latency tiers (tier A ≈10ms, tier B ≈100ms, tier C ≈300ms) and observing propagation lag, fork rate, and eventual convergence.</p>
<p><strong>Requirements:</strong></p>
<ul>
<li>Partition nodes into three groups (tiers)</li>
<li>Apply per-group network delay via chaos: <code>netem</code>/<code>iptables</code> in compose; NetworkPolicy + <code>netem</code> sidecar in k8s</li>
<li>Run standard workload (transactions/block production)</li>
<li>Optional: Remove delays at end to check recovery</li>
</ul>
<p><strong>Expectations:</strong></p>
<ul>
<li><strong>Propagation:</strong> Messages reach all tiers within acceptable bounds</li>
<li><strong>Safety:</strong> No divergent finalized heads; fork rate stays within tolerance</li>
<li><strong>Liveness:</strong> Chain keeps advancing; convergence after delays relaxed (if healed)</li>
</ul>
<p><strong>Why useful:</strong> Real networks have heterogeneous latency. This stress-tests proposer selection and fork resolution when some peers are "far" (high latency), validating that consensus remains safe and live under realistic network conditions.</p>
<p><strong>Current blocker:</strong> Runner support for per-group delay injection (network delay via <code>netem</code>/<code>iptables</code>) is not present today. Would require new chaos plumbing in compose/k8s deployers to inject network delays per node group.</p>
<h4 id="byzantine-gossip-flooding-libp2p-peer"><a class="header" href="#byzantine-gossip-flooding-libp2p-peer">Byzantine Gossip Flooding (libp2p Peer)</a></h4>
<p><strong>Concept:</strong> Spin up a custom workload/sidecar that runs a libp2p host, joins the cluster's gossip mesh, and publishes a high rate of syntactically valid but useless/stale messages to selected topics, testing gossip backpressure, scoring, and queue handling under a "malicious" peer.</p>
<p><strong>Requirements:</strong></p>
<ul>
<li>Custom workload/sidecar that implements a libp2p host</li>
<li>Join the cluster's gossip mesh as a peer</li>
<li>Publish high-rate syntactically valid but useless/stale messages to selected gossip topics</li>
<li>Run alongside normal workloads (transactions/block production)</li>
</ul>
<p><strong>Expectations:</strong></p>
<ul>
<li>Gossip backpressure mechanisms prevent message flooding from overwhelming nodes</li>
<li>Peer scoring correctly identifies and penalizes the malicious peer</li>
<li>Queue handling remains stable under flood conditions</li>
<li>Normal consensus operation continues despite malicious peer</li>
</ul>
<p><strong>Why useful:</strong> Tests Byzantine behavior (malicious peer) which is critical for consensus protocol robustness. More realistic than RPC spam since it uses the actual gossip protocol. Validates that gossip backpressure, peer scoring, and queue management correctly handle adversarial peers without disrupting consensus.</p>
<p><strong>Current blocker:</strong> Requires adding gossip-capable helper (libp2p integration) to the framework. Would need a custom workload/sidecar implementation that can join the gossip mesh and inject messages. The rest of the scenario can use existing runners/workloads.</p>
<h4 id="network-partition-recovery"><a class="header" href="#network-partition-recovery">Network Partition Recovery</a></h4>
<p><strong>Concept:</strong> Test consensus recovery after network partitions.</p>
<p><strong>Requirements:</strong></p>
<ul>
<li>Needs <code>block_peer()</code> / <code>unblock_peer()</code> methods in <code>NodeControlHandle</code></li>
<li>Partition subsets of validators, wait, then restore connectivity</li>
<li>Verify chain convergence after partition heals</li>
</ul>
<p><strong>Why useful:</strong> Tests the most realistic failure mode in distributed systems.</p>
<p><strong>Current blocker:</strong> Node control doesn't yet support network-level actions (only process restarts).</p>
<h3 id="time--timing"><a class="header" href="#time--timing">Time &amp; Timing</a></h3>
<h4 id="time-shifted-blocks-clock-skew-test"><a class="header" href="#time-shifted-blocks-clock-skew-test">Time-Shifted Blocks (Clock Skew Test)</a></h4>
<p><strong>Concept:</strong> Test consensus and timestamp handling when nodes run with skewed clocks (e.g., +1s, −1s, +200ms jitter) to surface timestamp validation issues, reorg sensitivity, and clock drift handling.</p>
<p><strong>Requirements:</strong></p>
<ul>
<li>Assign per-node time offsets (e.g., +1s, −1s, +200ms jitter)</li>
<li>Run normal workload (transactions/block production)</li>
<li>Observe whether blocks are accepted/propagated and the chain stays consistent</li>
</ul>
<p><strong>Expectations:</strong></p>
<ul>
<li>Blocks with skewed timestamps are handled correctly (accepted or rejected per protocol rules)</li>
<li>Chain remains consistent across nodes despite clock differences</li>
<li>No unexpected reorgs or chain splits due to timestamp validation issues</li>
</ul>
<p><strong>Why useful:</strong> Clock skew is a common real-world issue in distributed systems. This validates that consensus correctly handles timestamp validation and maintains safety/liveness when nodes have different clock offsets, preventing timestamp-based attacks or failures.</p>
<p><strong>Current blocker:</strong> Runner ability to skew per-node clocks (e.g., privileged containers with <code>libfaketime</code>/<code>chrony</code> or time-offset netns) is not available today. Would require a new chaos/time-skew hook in deployers to inject clock offsets per node.</p>
<h4 id="block-timing-consistency"><a class="header" href="#block-timing-consistency">Block Timing Consistency</a></h4>
<p><strong>Concept:</strong> Verify block production intervals stay within expected bounds.</p>
<p><strong>Implementation approach:</strong></p>
<ul>
<li>Custom expectation that consumes <code>BlockFeed</code></li>
<li>Collect block timestamps during run</li>
<li>Assert intervals are within <code>(slot_duration * active_slot_coeff) ± tolerance</code></li>
</ul>
<p><strong>Why useful:</strong> Validates consensus timing under various loads.</p>
<h3 id="topology--membership"><a class="header" href="#topology--membership">Topology &amp; Membership</a></h3>
<h4 id="dynamic-topology-churn-scenario"><a class="header" href="#dynamic-topology-churn-scenario">Dynamic Topology (Churn) Scenario</a></h4>
<p><strong>Concept:</strong> Nodes join and leave mid-run (new identities/addresses added; some nodes permanently removed) to exercise peer discovery, bootstrapping, reputation, and load balancing under churn.</p>
<p><strong>Requirements:</strong></p>
<ul>
<li>Runner must be able to spin up new nodes with fresh keys/addresses at runtime</li>
<li>Update peer lists and bootstraps dynamically as nodes join/leave</li>
<li>Optionally tear down nodes permanently (not just restart)</li>
<li>Run normal workloads (transactions/block production) during churn</li>
</ul>
<p><strong>Expectations:</strong></p>
<ul>
<li>New nodes successfully discover and join the network</li>
<li>Peer discovery mechanisms correctly handle dynamic topology changes</li>
<li>Reputation systems adapt to new/removed peers</li>
<li>Load balancing adjusts to changing node set</li>
<li>Consensus remains safe and live despite topology churn</li>
</ul>
<p><strong>Why useful:</strong> Real networks experience churn (nodes joining/leaving). Unlike restarts (which preserve topology), churn changes the actual topology size and peer set, testing how the protocol handles dynamic membership. This exercises peer discovery, bootstrapping, reputation systems, and load balancing under realistic conditions.</p>
<p><strong>Current blocker:</strong> Runner support for dynamic node addition/removal at runtime is not available today. Chaos today only restarts existing nodes; churn would require the ability to spin up new nodes with fresh identities/addresses, update peer lists/bootstraps dynamically, and permanently remove nodes. Would need new topology management capabilities in deployers.</p>
<h3 id="api--external-interfaces"><a class="header" href="#api--external-interfaces">API &amp; External Interfaces</a></h3>
<h4 id="api-dosstress-test"><a class="header" href="#api-dosstress-test">API DoS/Stress Test</a></h4>
<p><strong>Concept:</strong> Adversarial workload floods node HTTP/WS APIs with high QPS and malformed/bursty requests; expectation checks nodes remain responsive or rate-limit without harming consensus.</p>
<p><strong>Requirements:</strong></p>
<ul>
<li><strong>Custom workload:</strong> Targets node HTTP/WS API endpoints with mixed valid/invalid requests at high rate</li>
<li><strong>Custom expectation:</strong> Monitors error rates, latency, and confirms block production/liveness unaffected</li>
<li>Run alongside normal workloads (transactions/block production)</li>
</ul>
<p><strong>Expectations:</strong></p>
<ul>
<li>Nodes remain responsive or correctly rate-limit under API flood</li>
<li>Error rates/latency are acceptable (rate limiting works)</li>
<li>Block production/liveness unaffected by API abuse</li>
<li>Consensus continues normally despite API stress</li>
</ul>
<p><strong>Why useful:</strong> Validates API hardening under abuse and ensures control/telemetry endpoints don't destabilize the node. Tests that API abuse is properly isolated from consensus operations, preventing DoS attacks on API endpoints from affecting blockchain functionality.</p>
<p><strong>Implementation notes:</strong> Requires custom <code>Workload</code> implementation that directs high-QPS traffic to node APIs (via <code>ctx.node_clients()</code> or direct HTTP clients) and custom <code>Expectation</code> implementation that monitors API responsiveness metrics and consensus liveness. Uses existing node API access—no new infrastructure needed.</p>
<h3 id="state--correctness"><a class="header" href="#state--correctness">State &amp; Correctness</a></h3>
<h4 id="wallet-balance-verification"><a class="header" href="#wallet-balance-verification">Wallet Balance Verification</a></h4>
<p><strong>Concept:</strong> Track wallet balances and verify state consistency.</p>
<p><strong>Description:</strong> After transaction workload completes, query all wallet balances via node API and verify total supply is conserved. Requires tracking initial state, submitted transactions, and final balances. Validates that the ledger maintains correctness under load (no funds lost or created). This is a <strong>state assertion</strong> expectation that checks correctness, not just liveness.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="cucumberbdd-interface"><a class="header" href="#cucumberbdd-interface">Cucumber/BDD Interface</a></h1>
<p>The Logos testing repo includes a small Cucumber (Gherkin) harness for “smoke” scenarios. It is useful when you want readable acceptance-style checks, but it intentionally exposes a limited surface area compared to Rust scenarios.</p>
<hr />
<h2 id="what-exists-today"><a class="header" href="#what-exists-today">What Exists Today</a></h2>
<ul>
<li>Step definitions live in <code>testing-framework/cucumber</code>.</li>
<li>The runnable entrypoints are binaries in <code>examples</code> (crate <code>runner-examples</code>):
<ul>
<li><code>cucumber_host</code> (local/host deployer)</li>
<li><code>cucumber_compose</code> (compose deployer)</li>
</ul>
</li>
<li>Feature files live in <code>examples/cucumber/features/</code>.</li>
<li>Supported deployers: <code>local</code> and <code>compose</code> (no k8s runner integration in Cucumber yet).</li>
</ul>
<hr />
<h2 id="example-feature-matches-current-steps"><a class="header" href="#example-feature-matches-current-steps">Example Feature (Matches Current Steps)</a></h2>
<p>This is the shape used by the repo’s smoke features:</p>
<pre><code class="language-gherkin">Feature: Testing Framework - Local Runner
Scenario: Run a local smoke scenario (tx + DA + liveness)
Given deployer is "local"
And topology has 1 validators and 1 executors
And run duration is 60 seconds
And wallets total funds is 1000000000 split across 50 users
And transactions rate is 1 per block
And data availability channel rate is 1 per block and blob rate is 1 per block
And expect consensus liveness
When run scenario
Then scenario should succeed
</code></pre>
<hr />
<h2 id="running-the-smoke-features"><a class="header" href="#running-the-smoke-features">Running The Smoke Features</a></h2>
<p>Local runner smoke:</p>
<pre><code class="language-bash">POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin cucumber_host
</code></pre>
<p>Compose runner smoke:</p>
<pre><code class="language-bash">POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin cucumber_compose
</code></pre>
<hr />
<h2 id="available-steps-current"><a class="header" href="#available-steps-current">Available Steps (Current)</a></h2>
<p>Topology / runner selection:</p>
<ul>
<li><code>Given deployer is "local"|"compose"</code></li>
<li><code>Given topology has &lt;validators&gt; validators and &lt;executors&gt; executors</code></li>
</ul>
<p>Run configuration:</p>
<ul>
<li><code>Given run duration is &lt;seconds&gt; seconds</code></li>
<li><code>Given wallets total funds is &lt;funds&gt; split across &lt;users&gt; users</code></li>
</ul>
<p>Workloads:</p>
<ul>
<li><code>Given transactions rate is &lt;rate&gt; per block</code></li>
<li><code>Given transactions rate is &lt;rate&gt; per block using &lt;users&gt; users</code></li>
<li><code>Given data availability channel rate is &lt;channel_rate&gt; per block and blob rate is &lt;blob_rate&gt; per block</code></li>
</ul>
<p>Expectations:</p>
<ul>
<li><code>Given expect consensus liveness</code></li>
<li><code>Given consensus liveness lag allowance is &lt;blocks&gt;</code></li>
</ul>
<p>Execution + assertion:</p>
<ul>
<li><code>When run scenario</code></li>
<li><code>Then scenario should succeed</code></li>
</ul>
<hr />
<h2 id="notes"><a class="header" href="#notes">Notes</a></h2>
<ul>
<li>The Cucumber harness builds scenarios using the same core + workflow builder APIs as the Rust examples, so the same prerequisites apply (notably <code>POL_PROOF_DEV_MODE=true</code> for practical runs).</li>
<li>If you need more flexibility (custom workloads/expectations, richer checks, node control/chaos), write Rust scenarios instead: see <a href="examples.html">Examples</a> and <a href="extending.html">Extending the Framework</a>.</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="running-scenarios"><a class="header" href="#running-scenarios">Running Scenarios</a></h1>
<p>This page focuses on how scenarios are executed (deploy → run → evaluate → cleanup), what artifacts you get back, and how that differs across runners.</p>
<p>For “just run something that works” commands, see <a href="running-examples.html">Running Examples</a>.</p>
<hr />
<h2 id="execution-flow-high-level"><a class="header" href="#execution-flow-high-level">Execution Flow (High Level)</a></h2>
<p>When you run a built scenario via a deployer, the run follows the same shape:</p>
<pre><code class="language-mermaid">flowchart TD
Build[Scenario built] --&gt; Deploy[Deploy]
Deploy --&gt; Capture[Capture]
Capture --&gt; Execute[Execute]
Execute --&gt; Evaluate[Evaluate]
Evaluate --&gt; Cleanup[Cleanup]
</code></pre>
<ul>
<li><strong>Deploy</strong>: provision infrastructure and start nodes (processes/containers/pods)</li>
<li><strong>Capture</strong>: establish clients/observability and capture initial state</li>
<li><strong>Execute</strong>: run workloads for the configured wall-clock duration</li>
<li><strong>Evaluate</strong>: run expectations (after the execution window ends)</li>
<li><strong>Cleanup</strong>: stop resources and finalize artifacts</li>
</ul>
<hr />
<h2 id="the-core-api"><a class="header" href="#the-core-api">The Core API</a></h2>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::{Deployer as _, ScenarioBuilder};
use testing_framework_runner_local::LocalDeployer;
use testing_framework_workflows::ScenarioBuilderExt;
async fn run_once() -&gt; anyhow::Result&lt;()&gt; {
let mut scenario = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
.wallets(20)
.transactions_with(|tx| tx.rate(1).users(5))
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(60))
.build()?;
let runner = LocalDeployer::default().deploy(&amp;scenario).await?;
runner.run(&amp;mut scenario).await?;
Ok(())
}</code></pre>
<p>Notes:</p>
<ul>
<li><code>with_run_duration(...)</code> is wall-clock time, not “number of blocks”.</li>
<li><code>.transactions_with(...)</code> rates are per-block.</li>
<li>Most users should run scenarios via <code>scripts/run/run-examples.sh</code> unless they are embedding the framework in their own test crate.</li>
</ul>
<hr />
<h2 id="runner-differences"><a class="header" href="#runner-differences">Runner Differences</a></h2>
<h3 id="local-host-runner"><a class="header" href="#local-host-runner">Local (Host) Runner</a></h3>
<ul>
<li><strong>Best for</strong>: fast iteration and debugging</li>
<li><strong>Logs/state</strong>: stored under a temporary run directory unless you set <code>NOMOS_TESTS_KEEP_LOGS=1</code> and/or <code>NOMOS_LOG_DIR=...</code></li>
<li><strong>Limitations</strong>: no node-control capability (chaos workflows that require node control won’t work here)</li>
</ul>
<p>Run the built-in local examples:</p>
<pre><code class="language-bash">POL_PROOF_DEV_MODE=true \
scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
</code></pre>
<h3 id="compose-runner"><a class="header" href="#compose-runner">Compose Runner</a></h3>
<ul>
<li><strong>Best for</strong>: reproducible multi-node environments and node control</li>
<li><strong>Logs</strong>: primarily via <code>docker compose logs</code> (and any node-level log configuration you apply)</li>
<li><strong>Debugging</strong>: set <code>COMPOSE_RUNNER_PRESERVE=1</code> to keep the environment up after a run</li>
</ul>
<p>Run the built-in compose examples:</p>
<pre><code class="language-bash">POL_PROOF_DEV_MODE=true \
scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
</code></pre>
<h3 id="k8s-runner"><a class="header" href="#k8s-runner">K8s Runner</a></h3>
<ul>
<li><strong>Best for</strong>: production-like behavior, cluster scheduling/networking</li>
<li><strong>Logs</strong>: <code>kubectl logs ...</code></li>
<li><strong>Debugging</strong>: set <code>K8S_RUNNER_PRESERVE=1</code> and <code>K8S_RUNNER_NAMESPACE=...</code> to keep resources around</li>
</ul>
<p>Run the built-in k8s examples:</p>
<pre><code class="language-bash">POL_PROOF_DEV_MODE=true \
scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
</code></pre>
<hr />
<h2 id="artifacts--where-to-look"><a class="header" href="#artifacts--where-to-look">Artifacts &amp; Where to Look</a></h2>
<ul>
<li><strong>Node logs</strong>: configure via <code>NOMOS_LOG_DIR</code>, <code>NOMOS_LOG_LEVEL</code>, <code>NOMOS_LOG_FILTER</code> (see <a href="logging-observability.html">Logging &amp; Observability</a>)</li>
<li><strong>Runner logs</strong>: controlled by <code>RUST_LOG</code> (runner process only)</li>
<li><strong>Keep run directories</strong>: set <code>NOMOS_TESTS_KEEP_LOGS=1</code></li>
<li><strong>Compose environment preservation</strong>: set <code>COMPOSE_RUNNER_PRESERVE=1</code></li>
<li><strong>K8s environment preservation</strong>: set <code>K8S_RUNNER_PRESERVE=1</code></li>
</ul>
<hr />
<h2 id="see-also-1"><a class="header" href="#see-also-1">See Also</a></h2>
<ul>
<li><a href="scenario-lifecycle.html">Scenario Lifecycle</a></li>
<li><a href="running-examples.html">Running Examples</a></li>
<li><a href="troubleshooting.html">Troubleshooting Scenarios</a></li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="runners"><a class="header" href="#runners">Runners</a></h1>
<p>Runners turn a scenario plan into a live environment while keeping the plan
unchanged. Choose based on feedback speed, reproducibility, and fidelity. For
environment and operational considerations, see <a href="operations-overview.html">Operations Overview</a>.</p>
<p><strong>Important:</strong> All runners require <code>POL_PROOF_DEV_MODE=true</code> to avoid expensive Groth16 proof generation that causes timeouts.</p>
<h2 id="host-runner-local-processes"><a class="header" href="#host-runner-local-processes">Host runner (local processes)</a></h2>
<ul>
<li>Launches node processes directly on the host (via <code>LocalDeployer</code>).</li>
<li>Binary: <code>local_runner.rs</code>, script mode: <code>host</code></li>
<li>Fastest feedback loop and minimal orchestration overhead.</li>
<li>Best for development-time iteration and debugging.</li>
<li><strong>Can run in CI</strong> for fast smoke tests.</li>
<li><strong>Node control:</strong> Not supported (chaos workloads not available)</li>
</ul>
<p><strong>Run with:</strong> <code>scripts/run/run-examples.sh -t 60 -v 1 -e 1 host</code></p>
<h2 id="docker-compose-runner"><a class="header" href="#docker-compose-runner">Docker Compose runner</a></h2>
<ul>
<li>Starts nodes in containers to provide a reproducible multi-node stack on a
single machine (via <code>ComposeDeployer</code>).</li>
<li>Binary: <code>compose_runner.rs</code>, script mode: <code>compose</code></li>
<li>Discovers service ports and wires observability for convenient inspection.</li>
<li>Good balance between fidelity and ease of setup.</li>
<li><strong>Recommended for CI pipelines</strong> (isolated environment, reproducible).</li>
<li><strong>Node control:</strong> Supported (can restart nodes for chaos testing)</li>
</ul>
<p><strong>Run with:</strong> <code>scripts/run/run-examples.sh -t 60 -v 1 -e 1 compose</code></p>
<h2 id="kubernetes-runner"><a class="header" href="#kubernetes-runner">Kubernetes runner</a></h2>
<ul>
<li>Deploys nodes onto a cluster for higher-fidelity, longer-running scenarios (via <code>K8sDeployer</code>).</li>
<li>Binary: <code>k8s_runner.rs</code>, script mode: <code>k8s</code></li>
<li>Suits CI with cluster access or shared test environments where cluster behavior
and scheduling matter.</li>
<li><strong>Node control:</strong> Not supported yet (chaos workloads not available)</li>
</ul>
<p><strong>Run with:</strong> <code>scripts/run/run-examples.sh -t 60 -v 1 -e 1 k8s</code></p>
<h3 id="common-expectations"><a class="header" href="#common-expectations">Common expectations</a></h3>
<ul>
<li>All runners require at least one validator and, for transaction scenarios,
access to seeded wallets.</li>
<li>Readiness probes gate workload start so traffic begins only after nodes are
reachable.</li>
<li>Environment flags can relax timeouts or increase tracing when diagnostics are
needed.</li>
</ul>
<h2 id="runner-comparison"><a class="header" href="#runner-comparison">Runner Comparison</a></h2>
<pre><code class="language-mermaid">flowchart TB
subgraph Host["Host Runner (Local)"]
H1["Speed: Fast"]
H2["Isolation: Shared host"]
H3["Setup: Minimal"]
H4["Chaos: Not supported"]
H5["CI: Quick smoke tests"]
end
subgraph Compose["Compose Runner (Docker)"]
C1["Speed: Medium"]
C2["Isolation: Containerized"]
C3["Setup: Image build required"]
C4["Chaos: Supported"]
C5["CI: Recommended"]
end
subgraph K8s["K8s Runner (Cluster)"]
K1["Speed: Slower"]
K2["Isolation: Pod-level"]
K3["Setup: Cluster + image"]
K4["Chaos: Not yet supported"]
K5["CI: Large-scale tests"]
end
Decision{Choose Based On}
Decision --&gt;|Fast iteration| Host
Decision --&gt;|Reproducibility| Compose
Decision --&gt;|Production-like| K8s
style Host fill:#e1f5ff
style Compose fill:#e1ffe1
style K8s fill:#ffe1f5
</code></pre>
<h2 id="detailed-feature-matrix"><a class="header" href="#detailed-feature-matrix">Detailed Feature Matrix</a></h2>
<div class="table-wrapper"><table><thead><tr><th>Feature</th><th>Host</th><th>Compose</th><th>K8s</th></tr></thead><tbody>
<tr><td><strong>Speed</strong></td><td>Fastest</td><td>Medium</td><td>Slowest</td></tr>
<tr><td><strong>Setup Time</strong></td><td>&lt; 1 min</td><td>2-5 min</td><td>5-10 min</td></tr>
<tr><td><strong>Isolation</strong></td><td>Process-level</td><td>Container</td><td>Pod + namespace</td></tr>
<tr><td><strong>Node Control</strong></td><td>No</td><td>Yes</td><td>Not yet</td></tr>
<tr><td><strong>Observability</strong></td><td>Basic</td><td>External stack</td><td>Cluster-wide</td></tr>
<tr><td><strong>CI Integration</strong></td><td>Smoke tests</td><td>Recommended</td><td>Heavy tests</td></tr>
<tr><td><strong>Resource Usage</strong></td><td>Low</td><td>Medium</td><td>High</td></tr>
<tr><td><strong>Reproducibility</strong></td><td>Environment-dependent</td><td>High</td><td>Highest</td></tr>
<tr><td><strong>Network Fidelity</strong></td><td>Localhost only</td><td>Virtual network</td><td>Real cluster</td></tr>
<tr><td><strong>Parallel Runs</strong></td><td>Port conflicts</td><td>Isolated</td><td>Namespace isolation</td></tr>
</tbody></table>
</div>
<h2 id="decision-guide"><a class="header" href="#decision-guide">Decision Guide</a></h2>
<pre><code class="language-mermaid">flowchart TD
Start[Need to run tests?] --&gt; Q1{Local development?}
Q1 --&gt;|Yes| Q2{Testing chaos?}
Q1 --&gt;|No| Q5{Have cluster access?}
Q2 --&gt;|Yes| UseCompose[Use Compose]
Q2 --&gt;|No| Q3{Need isolation?}
Q3 --&gt;|Yes| UseCompose
Q3 --&gt;|No| UseHost[Use Host]
Q5 --&gt;|Yes| Q6{Large topology?}
Q5 --&gt;|No| Q7{CI pipeline?}
Q6 --&gt;|Yes| UseK8s[Use K8s]
Q6 --&gt;|No| UseCompose
Q7 --&gt;|Yes| Q8{Docker available?}
Q7 --&gt;|No| UseHost
Q8 --&gt;|Yes| UseCompose
Q8 --&gt;|No| UseHost
style UseHost fill:#e1f5ff
style UseCompose fill:#e1ffe1
style UseK8s fill:#ffe1f5
</code></pre>
<h3 id="quick-recommendations"><a class="header" href="#quick-recommendations">Quick Recommendations</a></h3>
<p><strong>Use Host Runner when:</strong></p>
<ul>
<li>Iterating rapidly during development</li>
<li>Running quick smoke tests</li>
<li>Testing on a laptop with limited resources</li>
<li>Don't need chaos testing</li>
</ul>
<p><strong>Use Compose Runner when:</strong></p>
<ul>
<li>Need reproducible test environments</li>
<li>Testing chaos scenarios (node restarts)</li>
<li>Running in CI pipelines</li>
<li>Want containerized isolation</li>
</ul>
<p><strong>Use K8s Runner when:</strong></p>
<ul>
<li>Testing large-scale topologies (10+ nodes)</li>
<li>Need production-like environment</li>
<li>Have cluster access in CI</li>
<li>Testing cluster-specific behaviors</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="runcontext-blockfeed--node-control"><a class="header" href="#runcontext-blockfeed--node-control">RunContext: BlockFeed &amp; Node Control</a></h1>
<p>The deployer supplies a <code>RunContext</code> that workloads and expectations share. It
provides:</p>
<ul>
<li>Topology descriptors (<code>GeneratedTopology</code>)</li>
<li>Client handles (<code>NodeClients</code> / <code>ClusterClient</code>) for HTTP/RPC calls</li>
<li>Metrics (<code>RunMetrics</code>, <code>Metrics</code>) and block feed</li>
<li>Optional <code>NodeControlHandle</code> for managing nodes</li>
</ul>
<h2 id="blockfeed-observing-block-production"><a class="header" href="#blockfeed-observing-block-production">BlockFeed: Observing Block Production</a></h2>
<p>The <code>BlockFeed</code> is a broadcast stream of block observations that allows workloads and expectations to monitor blockchain progress in real-time. It polls a validator node continuously and broadcasts new blocks to all subscribers.</p>
<h3 id="what-blockfeed-provides"><a class="header" href="#what-blockfeed-provides">What BlockFeed Provides</a></h3>
<p><strong>Real-time block stream:</strong></p>
<ul>
<li>Subscribe to receive <code>BlockRecord</code> notifications as blocks are produced</li>
<li>Each record includes the block header (<code>HeaderId</code>) and full block payload</li>
<li>Backed by a background task that polls node storage every second</li>
</ul>
<p><strong>Block statistics:</strong></p>
<ul>
<li>Track total transactions across all observed blocks</li>
<li>Access via <code>block_feed.stats().total_transactions()</code></li>
</ul>
<p><strong>Broadcast semantics:</strong></p>
<ul>
<li>Multiple subscribers can receive the same blocks independently</li>
<li>Late subscribers start receiving from current block (no history replay)</li>
<li>Lagged subscribers skip missed blocks automatically</li>
</ul>
<h3 id="accessing-blockfeed"><a class="header" href="#accessing-blockfeed">Accessing BlockFeed</a></h3>
<p>BlockFeed is available through <code>RunContext</code>:</p>
<pre><code class="language-rust ignore">let block_feed = ctx.block_feed();</code></pre>
<h3 id="usage-in-expectations"><a class="header" href="#usage-in-expectations">Usage in Expectations</a></h3>
<p>Expectations typically use BlockFeed to verify block production and inclusion of transactions/data.</p>
<p><strong>Example: Counting blocks during a run</strong></p>
<pre><code class="language-rust ignore">use std::sync::{
Arc,
atomic::{AtomicU64, Ordering},
};
use async_trait::async_trait;
use testing_framework_core::scenario::{DynError, Expectation, RunContext};
struct MinimumBlocksExpectation {
min_blocks: u64,
captured_blocks: Option&lt;Arc&lt;AtomicU64&gt;&gt;,
}
#[async_trait]
impl Expectation for MinimumBlocksExpectation {
fn name(&amp;self) -&gt; &amp;'static str {
"minimum_blocks"
}
async fn start_capture(&amp;mut self, ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
let block_count = Arc::new(AtomicU64::new(0));
let block_count_task = Arc::clone(&amp;block_count);
// Subscribe to block feed
let mut receiver = ctx.block_feed().subscribe();
// Spawn a task to count blocks
tokio::spawn(async move {
loop {
match receiver.recv().await {
Ok(_record) =&gt; {
block_count_task.fetch_add(1, Ordering::Relaxed);
}
Err(tokio::sync::broadcast::error::RecvError::Lagged(skipped)) =&gt; {
tracing::debug!(skipped, "receiver lagged, skipping blocks");
}
Err(tokio::sync::broadcast::error::RecvError::Closed) =&gt; {
tracing::debug!("block feed closed");
break;
}
}
}
});
self.captured_blocks = Some(block_count);
Ok(())
}
async fn evaluate(&amp;mut self, ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
let blocks = self.captured_blocks
.as_ref()
.expect("start_capture must be called first")
.load(Ordering::Relaxed);
if blocks &lt; self.min_blocks {
return Err(format!(
"expected at least {} blocks, observed {}",
self.min_blocks, blocks
).into());
}
tracing::info!(blocks, min = self.min_blocks, "minimum blocks expectation passed");
Ok(())
}
}</code></pre>
<p><strong>Example: Inspecting block contents</strong></p>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::{DynError, RunContext};
async fn start_capture(ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
let mut receiver = ctx.block_feed().subscribe();
tokio::spawn(async move {
loop {
match receiver.recv().await {
Ok(record) =&gt; {
// Access block header
let header_id = &amp;record.header;
// Access full block
let tx_count = record.block.transactions().len();
tracing::debug!(
?header_id,
tx_count,
"observed block"
);
// Process transactions, DA blobs, etc.
}
Err(tokio::sync::broadcast::error::RecvError::Closed) =&gt; break,
Err(_) =&gt; continue,
}
}
});
Ok(())
}</code></pre>
<h3 id="usage-in-workloads"><a class="header" href="#usage-in-workloads">Usage in Workloads</a></h3>
<p>Workloads can use BlockFeed to coordinate timing or wait for specific conditions before proceeding.</p>
<p><strong>Example: Wait for N blocks before starting</strong></p>
<pre><code class="language-rust ignore">use async_trait::async_trait;
use testing_framework_core::scenario::{DynError, RunContext, Workload};
struct DelayedWorkload {
wait_blocks: usize,
}
#[async_trait]
impl Workload for DelayedWorkload {
fn name(&amp;self) -&gt; &amp;str {
"delayed_workload"
}
async fn start(&amp;self, ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
tracing::info!(wait_blocks = self.wait_blocks, "waiting for blocks before starting");
// Subscribe to block feed
let mut receiver = ctx.block_feed().subscribe();
let mut count = 0;
// Wait for N blocks
while count &lt; self.wait_blocks {
match receiver.recv().await {
Ok(_) =&gt; count += 1,
Err(tokio::sync::broadcast::error::RecvError::Lagged(_)) =&gt; continue,
Err(tokio::sync::broadcast::error::RecvError::Closed) =&gt; {
return Err("block feed closed before reaching target".into());
}
}
}
tracing::info!("warmup complete, starting actual workload");
// Now do the actual work
// ...
Ok(())
}
}</code></pre>
<p><strong>Example: Rate limiting based on block production</strong></p>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::{DynError, RunContext};
async fn generate_request() -&gt; Option&lt;()&gt; {
None
}
async fn start(ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
let clients = ctx.node_clients().validator_clients();
let mut receiver = ctx.block_feed().subscribe();
let mut pending_requests: Vec&lt;()&gt; = Vec::new();
loop {
tokio::select! {
// Issue a batch on each new block.
Ok(_record) = receiver.recv() =&gt; {
if !pending_requests.is_empty() {
tracing::debug!(count = pending_requests.len(), "issuing requests on new block");
for _req in pending_requests.drain(..) {
let _info = clients[0].consensus_info().await?;
}
}
}
// Generate work continuously.
Some(req) = generate_request() =&gt; {
pending_requests.push(req);
}
}
}
}</code></pre>
<h3 id="blockfeed-vs-direct-polling"><a class="header" href="#blockfeed-vs-direct-polling">BlockFeed vs Direct Polling</a></h3>
<p><strong>Use BlockFeed when:</strong></p>
<ul>
<li>You need to react to blocks as they're produced</li>
<li>Multiple components need to observe the same blocks</li>
<li>You want automatic retry/reconnect logic</li>
<li>You're tracking statistics across many blocks</li>
</ul>
<p><strong>Use direct polling when:</strong></p>
<ul>
<li>You need to query specific historical blocks</li>
<li>You're checking final state after workloads complete</li>
<li>You need transaction receipts or other indexed data</li>
<li>You're implementing a one-time health check</li>
</ul>
<p>Example direct polling in expectations:</p>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::{DynError, RunContext};
async fn evaluate(ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
let client = &amp;ctx.node_clients().validator_clients()[0];
// Poll current height once
let info = client.consensus_info().await?;
tracing::info!(height = info.height, "final block height");
// This is simpler than BlockFeed for one-time checks
Ok(())
}</code></pre>
<h3 id="block-statistics"><a class="header" href="#block-statistics">Block Statistics</a></h3>
<p>Access aggregated statistics without subscribing to the feed:</p>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::{DynError, RunContext};
async fn evaluate(ctx: &amp;RunContext, expected_min: u64) -&gt; Result&lt;(), DynError&gt; {
let stats = ctx.block_feed().stats();
let total_txs = stats.total_transactions();
tracing::info!(total_txs, "transactions observed across all blocks");
if total_txs &lt; expected_min {
return Err(format!(
"expected at least {} transactions, observed {}",
expected_min, total_txs
).into());
}
Ok(())
}</code></pre>
<h3 id="important-notes"><a class="header" href="#important-notes">Important Notes</a></h3>
<p><strong>Subscription timing:</strong></p>
<ul>
<li>Subscribe in <code>start_capture()</code> for expectations</li>
<li>Subscribe in <code>start()</code> for workloads</li>
<li>Late subscribers miss historical blocks (no replay)</li>
</ul>
<p><strong>Lagged receivers:</strong></p>
<ul>
<li>If your subscriber is too slow, it may lag behind</li>
<li>Handle <code>RecvError::Lagged(skipped)</code> gracefully</li>
<li>Consider increasing processing speed or reducing block rate</li>
</ul>
<p><strong>Feed lifetime:</strong></p>
<ul>
<li>BlockFeed runs for the entire scenario duration</li>
<li>Automatically cleaned up when the run completes</li>
<li>Closed channels signal graceful shutdown</li>
</ul>
<p><strong>Performance:</strong></p>
<ul>
<li>BlockFeed polls nodes every 1 second</li>
<li>Broadcasts to all subscribers with minimal overhead</li>
<li>Suitable for scenarios with hundreds of blocks</li>
</ul>
<h3 id="real-world-examples"><a class="header" href="#real-world-examples">Real-World Examples</a></h3>
<p>The framework's built-in expectations use BlockFeed extensively:</p>
<ul>
<li><strong><code>ConsensusLiveness</code></strong>: Doesn't directly subscribe but uses block feed stats to verify progress</li>
<li><strong><code>DataAvailabilityExpectation</code></strong>: Subscribes to inspect DA blobs in each block and track inscription/dispersal</li>
<li><strong><code>TransactionInclusion</code></strong>: Subscribes to find specific transactions in blocks</li>
</ul>
<p>See <a href="examples.html">Examples</a> and <a href="workloads.html">Workloads &amp; Expectations</a> for more patterns.</p>
<hr />
<h2 id="current-chaos-capabilities-and-limitations"><a class="header" href="#current-chaos-capabilities-and-limitations">Current Chaos Capabilities and Limitations</a></h2>
<p>The framework currently supports <strong>process-level chaos</strong> (node restarts) for
resilience testing:</p>
<p><strong>Supported:</strong></p>
<ul>
<li>Restart validators (<code>restart_validator</code>)</li>
<li>Restart executors (<code>restart_executor</code>)</li>
<li>Random restart workload via <code>.chaos().restart()</code></li>
</ul>
<p><strong>Not Yet Supported:</strong></p>
<ul>
<li>Network partitions (blocking peers, packet loss)</li>
<li>Resource constraints (CPU throttling, memory limits)</li>
<li>Byzantine behavior injection (invalid blocks, bad signatures)</li>
<li>Selective peer blocking/unblocking</li>
</ul>
<p>For network partition testing, see <a href="examples-advanced.html#extension-ideas">Extension Ideas</a>
which describes the proposed <code>block_peer</code>/<code>unblock_peer</code> API (not yet implemented).</p>
<h2 id="accessing-node-control-in-workloadsexpectations"><a class="header" href="#accessing-node-control-in-workloadsexpectations">Accessing node control in workloads/expectations</a></h2>
<p>Check for control support and use it conditionally:</p>
<pre><code class="language-rust ignore">use async_trait::async_trait;
use testing_framework_core::scenario::{DynError, RunContext, Workload};
struct RestartWorkload;
#[async_trait]
impl Workload for RestartWorkload {
fn name(&amp;self) -&gt; &amp;str {
"restart_workload"
}
async fn start(&amp;self, ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
if let Some(control) = ctx.node_control() {
// Restart the first validator (index 0) if supported.
control.restart_validator(0).await?;
}
Ok(())
}
}</code></pre>
<p>When chaos workloads need control, require <code>enable_node_control()</code> in the
scenario builder and deploy with a runner that supports it.</p>
<h2 id="current-api-surface"><a class="header" href="#current-api-surface">Current API surface</a></h2>
<p>The <code>NodeControlHandle</code> trait currently provides:</p>
<pre><code class="language-rust ignore">use async_trait::async_trait;
use testing_framework_core::scenario::DynError;
#[async_trait]
pub trait NodeControlHandle: Send + Sync {
async fn restart_validator(&amp;self, index: usize) -&gt; Result&lt;(), DynError&gt;;
async fn restart_executor(&amp;self, index: usize) -&gt; Result&lt;(), DynError&gt;;
}</code></pre>
<p>Future extensions may include peer blocking/unblocking or other control
operations. For now, focus on restart-based chaos patterns as shown in the
chaos workload examples.</p>
<h2 id="considerations"><a class="header" href="#considerations">Considerations</a></h2>
<ul>
<li>Always guard control usage: not all runners expose <code>NodeControlHandle</code>.</li>
<li>Treat control as best-effort: failures should surface as test failures, but
workloads should degrade gracefully when control is absent.</li>
<li>Combine control actions with expectations (e.g., restart then assert height
convergence) to keep scenarios meaningful.</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="chaos-workloads"><a class="header" href="#chaos-workloads">Chaos Workloads</a></h1>
<blockquote>
<p><strong>When should I read this?</strong> You don't need chaos testing to be productive with the framework. Focus on basic scenarios first—chaos is for resilience validation and operational readiness drills once your core tests are stable.</p>
</blockquote>
<p>Chaos in the framework uses node control to introduce failures and validate
recovery. The built-in restart workload lives in
<code>testing_framework_workflows::workloads::chaos::RandomRestartWorkload</code>.</p>
<h2 id="how-it-works-1"><a class="header" href="#how-it-works-1">How it works</a></h2>
<ul>
<li>Requires <code>NodeControlCapability</code> (<code>enable_node_control()</code> in the scenario
builder) and a runner that provides a <code>NodeControlHandle</code>.</li>
<li>Randomly selects nodes (validators, executors) to restart based on your
include/exclude flags.</li>
<li>Respects min/max delay between restarts and a target cooldown to avoid
flapping the same node too frequently.</li>
<li>Runs alongside other workloads; expectations should account for the added
disruption.</li>
<li>Support varies by runner: node control is not provided by the local runner
and is not yet implemented for the k8s runner. Use a runner that advertises
<code>NodeControlHandle</code> support (e.g., compose) for chaos workloads.</li>
</ul>
<h2 id="usage-1"><a class="header" href="#usage-1">Usage</a></h2>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::{ScenarioBuilderExt, workloads::chaos::RandomRestartWorkload};
pub fn random_restart_plan() -&gt; testing_framework_core::scenario::Scenario&lt;
testing_framework_core::scenario::NodeControlCapability,
&gt; {
ScenarioBuilder::topology_with(|t| t.network_star().validators(2).executors(1))
.enable_node_control()
.with_workload(RandomRestartWorkload::new(
Duration::from_secs(45), // min delay
Duration::from_secs(75), // max delay
Duration::from_secs(120), // target cooldown
true, // include validators
true, // include executors
))
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(150))
.build()
}</code></pre>
<h2 id="expectations-to-pair"><a class="header" href="#expectations-to-pair">Expectations to pair</a></h2>
<ul>
<li><strong>Consensus liveness</strong>: ensure blocks keep progressing despite restarts.</li>
<li><strong>Height convergence</strong>: optionally check all nodes converge after the chaos
window.</li>
<li>Any workload-specific inclusion checks if you’re also driving tx/DA traffic.</li>
</ul>
<h2 id="best-practices-2"><a class="header" href="#best-practices-2">Best practices</a></h2>
<ul>
<li>Keep delays/cooldowns realistic; avoid back-to-back restarts that would never
happen in production.</li>
<li>Limit chaos scope: toggle validators vs executors based on what you want to
test.</li>
<li>Combine with observability: monitor metrics/logs to explain failures.</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="topology--chaos-patterns"><a class="header" href="#topology--chaos-patterns">Topology &amp; Chaos Patterns</a></h1>
<p>This page focuses on cluster manipulation: node control, chaos patterns, and
what the tooling supports today.</p>
<h2 id="node-control-availability"><a class="header" href="#node-control-availability">Node control availability</a></h2>
<ul>
<li><strong>Supported</strong>: restart/peer control via <code>NodeControlHandle</code> (compose runner).</li>
<li><strong>Not supported</strong>: local runner does not expose node control; k8s runner does
not support it yet.</li>
</ul>
<h2 id="chaos-patterns-to-consider"><a class="header" href="#chaos-patterns-to-consider">Chaos patterns to consider</a></h2>
<ul>
<li><strong>Restarts</strong>: random restarts with minimum delay/cooldown to test recovery.</li>
<li><strong>Partitions</strong>: block/unblock peers to simulate partial isolation, then assert
height convergence after healing.</li>
<li><strong>Validator churn</strong>: stop one validator and start another (new key) mid-run to
test membership changes; expect convergence.</li>
<li><strong>Load SLOs</strong>: push tx/DA rates and assert inclusion/availability budgets
instead of only liveness.</li>
<li><strong>API probes</strong>: poll HTTP/RPC endpoints during chaos to ensure external
contracts stay healthy (shape + latency).</li>
</ul>
<h2 id="expectations-to-pair-1"><a class="header" href="#expectations-to-pair-1">Expectations to pair</a></h2>
<ul>
<li><strong>Liveness/height convergence</strong> after chaos windows.</li>
<li><strong>SLO checks</strong>: inclusion latency, DA responsiveness, API latency/shape.</li>
<li><strong>Recovery checks</strong>: ensure nodes that were isolated or restarted catch up to
cluster height within a timeout.</li>
</ul>
<h2 id="guidance"><a class="header" href="#guidance">Guidance</a></h2>
<ul>
<li>Keep chaos realistic: avoid flapping or patterns you wouldn't operate in prod.</li>
<li>Scope chaos: choose validators vs executors intentionally; don't restart all
nodes at once unless you're testing full outages.</li>
<li>Combine chaos with observability: capture block feed/metrics and API health so
failures are diagnosable.</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="part-iii--developer-reference"><a class="header" href="#part-iii--developer-reference">Part III — Developer Reference</a></h1>
<p>Deep dives for contributors who extend the framework, evolve its abstractions,
or maintain the crate set.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="scenario-model-developer-level"><a class="header" href="#scenario-model-developer-level">Scenario Model (Developer Level)</a></h1>
<p>The scenario model defines clear, composable responsibilities:</p>
<ul>
<li><strong>Topology</strong>: a declarative description of the cluster—how many nodes, their
roles, and the broad network and data-availability characteristics. It
represents the intended shape of the system under test.</li>
<li><strong>Scenario</strong>: a plan combining topology, workloads, expectations, and a run
window. Building a scenario validates prerequisites (like seeded wallets) and
ensures the run lasts long enough to observe meaningful block progression.</li>
<li><strong>Workloads</strong>: asynchronous tasks that generate traffic or conditions. They
use shared context to interact with the deployed cluster and may bundle
default expectations.</li>
<li><strong>Expectations</strong>: post-run assertions. They can capture baselines before
workloads start and evaluate success once activity stops.</li>
<li><strong>Runtime</strong>: coordinates workloads and expectations for the configured
duration, enforces cooldowns when control actions occur, and ensures cleanup
so runs do not leak resources.</li>
</ul>
<p>Developers extending the model should keep these boundaries strict: topology
describes, scenarios assemble, deployers provision, runners orchestrate,
workloads drive, and expectations judge outcomes. For guidance on adding new
capabilities, see <a href="extending.html">Extending the Framework</a>.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="api-levels-builder-dsl-vs-direct-instantiation"><a class="header" href="#api-levels-builder-dsl-vs-direct-instantiation">API Levels: Builder DSL vs. Direct Instantiation</a></h1>
<p>The framework supports two styles for constructing scenarios:</p>
<ol>
<li><strong>High-level Builder DSL</strong> (recommended): fluent helper methods (e.g. <code>.transactions_with(...)</code>)</li>
<li><strong>Low-level direct instantiation</strong>: construct workload/expectation types explicitly, then attach them</li>
</ol>
<p>Both styles produce the same runtime behavior because they ultimately call the same core builder APIs.</p>
<h2 id="high-level-builder-dsl-recommended"><a class="header" href="#high-level-builder-dsl-recommended">High-Level Builder DSL (Recommended)</a></h2>
<p>The DSL is implemented as extension traits (primarily <code>testing_framework_workflows::ScenarioBuilderExt</code>) on the core scenario builder.</p>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
let plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
.wallets(5)
.transactions_with(|txs| txs.rate(5).users(3))
.da_with(|da| da.channel_rate(1).blob_rate(1).headroom_percent(20))
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(60))
.build();</code></pre>
<p><strong>When to use:</strong></p>
<ul>
<li>Most test code (smoke, regression, CI)</li>
<li>When you want sensible defaults and minimal boilerplate</li>
</ul>
<h2 id="low-level-direct-instantiation"><a class="header" href="#low-level-direct-instantiation">Low-Level Direct Instantiation</a></h2>
<p>Direct instantiation gives you explicit control over the concrete types you attach:</p>
<pre><code class="language-rust ignore">use std::{
num::{NonZeroU64, NonZeroUsize},
time::Duration,
};
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::{
expectations::ConsensusLiveness,
workloads::{da, transaction},
};
let tx_workload = transaction::Workload::with_rate(5)
.expect("transaction rate must be non-zero")
.with_user_limit(NonZeroUsize::new(3));
let da_workload = da::Workload::with_rate(
NonZeroU64::new(1).unwrap(), // blob rate per block
NonZeroU64::new(1).unwrap(), // channel rate per block
da::Workload::default_headroom_percent(),
);
let plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
.wallets(5)
.with_workload(tx_workload)
.with_workload(da_workload)
.with_expectation(ConsensusLiveness::default())
.with_run_duration(Duration::from_secs(60))
.build();</code></pre>
<p><strong>When to use:</strong></p>
<ul>
<li>Custom workload/expectation implementations</li>
<li>Reusing preconfigured workload instances across multiple scenarios</li>
<li>Debugging / exploring the underlying workload types</li>
</ul>
<h2 id="method-correspondence"><a class="header" href="#method-correspondence">Method Correspondence</a></h2>
<div class="table-wrapper"><table><thead><tr><th>High-Level DSL</th><th>Low-Level Direct</th></tr></thead><tbody>
<tr><td><code>.transactions_with(|txs| txs.rate(5).users(3))</code></td><td><code>.with_workload(transaction::Workload::with_rate(5).expect(...).with_user_limit(...))</code></td></tr>
<tr><td><code>.da_with(|da| da.blob_rate(1).channel_rate(1))</code></td><td><code>.with_workload(da::Workload::with_rate(...))</code></td></tr>
<tr><td><code>.expect_consensus_liveness()</code></td><td><code>.with_expectation(ConsensusLiveness::default())</code></td></tr>
</tbody></table>
</div>
<h2 id="bundled-expectations-important"><a class="header" href="#bundled-expectations-important">Bundled Expectations (Important)</a></h2>
<p>Workloads can bundle expectations by implementing <code>Workload::expectations()</code>.</p>
<p>These bundled expectations are attached automatically whenever you call <code>.with_workload(...)</code> (including when you use the DSL), because the core builder expands workload expectations during attachment.</p>
<h2 id="mixing-both-styles"><a class="header" href="#mixing-both-styles">Mixing Both Styles</a></h2>
<p>Mixing is common: use the DSL for built-ins, and direct instantiation for custom pieces.</p>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::{ScenarioBuilderExt, workloads::transaction};
let tx_workload = transaction::Workload::with_rate(5)
.expect("transaction rate must be non-zero");
let plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
.wallets(5)
.with_workload(tx_workload) // direct instantiation
.expect_consensus_liveness() // DSL
.with_run_duration(Duration::from_secs(60))
.build();</code></pre>
<h2 id="implementation-detail-how-the-dsl-works"><a class="header" href="#implementation-detail-how-the-dsl-works">Implementation Detail (How the DSL Works)</a></h2>
<p>The DSL methods are thin wrappers. For example:</p>
<p><code>builder.transactions_with(|txs| txs.rate(5).users(3))</code></p>
<p>is roughly equivalent to:</p>
<p><code>builder.transactions().rate(5).users(3).apply()</code></p>
<h2 id="troubleshooting"><a class="header" href="#troubleshooting">Troubleshooting</a></h2>
<p><strong>DSL method not found</strong></p>
<ul>
<li>Ensure the extension traits are in scope, e.g. <code>use testing_framework_workflows::ScenarioBuilderExt;</code></li>
<li>Cross-check method names in <a href="dsl-cheat-sheet.html">Builder API Quick Reference</a></li>
</ul>
<h2 id="see-also-2"><a class="header" href="#see-also-2">See Also</a></h2>
<ul>
<li><a href="dsl-cheat-sheet.html">Builder API Quick Reference</a></li>
<li><a href="custom-workload-example.html">Example: New Workload &amp; Expectation (Rust)</a></li>
<li><a href="extending.html">Extending the Framework</a></li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="extending-the-framework-1"><a class="header" href="#extending-the-framework-1">Extending the Framework</a></h1>
<p>This guide shows how to extend the framework with custom workloads, expectations, runners, and topology helpers. Each section includes the trait outline and a minimal code example.</p>
<h2 id="adding-a-workload"><a class="header" href="#adding-a-workload">Adding a Workload</a></h2>
<p><strong>Steps:</strong></p>
<ol>
<li>Implement <code>testing_framework_core::scenario::Workload</code></li>
<li>Provide a name and any bundled expectations</li>
<li>Use <code>init</code> to derive inputs from topology/metrics; fail fast if prerequisites missing</li>
<li>Use <code>start</code> to drive async traffic using <code>RunContext</code> clients</li>
<li>Expose from <code>testing-framework/workflows</code> and optionally add a DSL helper</li>
</ol>
<p><strong>Trait outline:</strong></p>
<pre><code class="language-rust ignore">use async_trait::async_trait;
use testing_framework_core::scenario::{
DynError, Expectation, RunContext, RunMetrics, Workload,
};
use testing_framework_core::topology::generation::GeneratedTopology;
struct MyExpectation;
#[async_trait]
impl Expectation for MyExpectation {
fn name(&amp;self) -&gt; &amp;str {
"my_expectation"
}
async fn evaluate(&amp;mut self, _ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
Ok(())
}
}
pub struct MyWorkload {
// Configuration fields
target_rate: u64,
}
impl MyWorkload {
pub fn new(target_rate: u64) -&gt; Self {
Self { target_rate }
}
}
#[async_trait]
impl Workload for MyWorkload {
fn name(&amp;self) -&gt; &amp;str {
"my_workload"
}
fn expectations(&amp;self) -&gt; Vec&lt;Box&lt;dyn Expectation&gt;&gt; {
// Return bundled expectations that should run with this workload
vec![Box::new(MyExpectation)]
}
fn init(
&amp;mut self,
topology: &amp;GeneratedTopology,
_run_metrics: &amp;RunMetrics,
) -&gt; Result&lt;(), DynError&gt; {
// Validate prerequisites (e.g., enough nodes, wallet data present)
if topology.validators().is_empty() {
return Err("no validators available".into());
}
Ok(())
}
async fn start(&amp;self, ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
// Drive async activity: submit transactions, query nodes, etc.
let clients = ctx.node_clients().validator_clients();
for client in clients {
let info = client.consensus_info().await?;
tracing::info!(height = info.height, "workload queried node");
}
Ok(())
}
}</code></pre>
<p><strong>Key points:</strong></p>
<ul>
<li><code>name()</code> identifies the workload in logs</li>
<li><code>expectations()</code> bundles default checks (can be empty)</li>
<li><code>init()</code> validates topology before run starts</li>
<li><code>start()</code> executes concurrently with other workloads; it should complete before run duration expires</li>
</ul>
<p>See <a href="custom-workload-example.html">Example: New Workload &amp; Expectation</a> for a complete, runnable example.</p>
<h2 id="adding-an-expectation"><a class="header" href="#adding-an-expectation">Adding an Expectation</a></h2>
<p><strong>Steps:</strong></p>
<ol>
<li>Implement <code>testing_framework_core::scenario::Expectation</code></li>
<li>Use <code>start_capture</code> to snapshot baseline metrics (optional)</li>
<li>Use <code>evaluate</code> to assert outcomes after workloads finish</li>
<li>Return descriptive errors; the runner aggregates them</li>
<li>Export from <code>testing-framework/workflows</code> if reusable</li>
</ol>
<p><strong>Trait outline:</strong></p>
<pre><code class="language-rust ignore">use async_trait::async_trait;
use testing_framework_core::scenario::{DynError, Expectation, RunContext};
pub struct MyExpectation {
expected_value: u64,
captured_baseline: Option&lt;u64&gt;,
}
impl MyExpectation {
pub fn new(expected_value: u64) -&gt; Self {
Self {
expected_value,
captured_baseline: None,
}
}
}
#[async_trait]
impl Expectation for MyExpectation {
fn name(&amp;self) -&gt; &amp;str {
"my_expectation"
}
async fn start_capture(&amp;mut self, ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
// Optional: capture baseline state before workloads start
let client = ctx.node_clients().validator_clients().first()
.ok_or("no validators")?;
let info = client.consensus_info().await?;
self.captured_baseline = Some(info.height);
tracing::info!(baseline = self.captured_baseline, "captured baseline");
Ok(())
}
async fn evaluate(&amp;mut self, ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
// Assert the expected condition holds after workloads finish
let client = ctx.node_clients().validator_clients().first()
.ok_or("no validators")?;
let info = client.consensus_info().await?;
let final_height = info.height;
let baseline = self.captured_baseline.unwrap_or(0);
let delta = final_height.saturating_sub(baseline);
if delta &lt; self.expected_value {
return Err(format!(
"expected at least {} blocks, got {}",
self.expected_value, delta
).into());
}
tracing::info!(delta, "expectation passed");
Ok(())
}
}</code></pre>
<p><strong>Key points:</strong></p>
<ul>
<li><code>name()</code> identifies the expectation in logs</li>
<li><code>start_capture()</code> runs before workloads start (optional)</li>
<li><code>evaluate()</code> runs after workloads finish; return descriptive errors</li>
<li>Expectations run sequentially; keep them fast</li>
</ul>
<h2 id="adding-a-runner-deployer"><a class="header" href="#adding-a-runner-deployer">Adding a Runner (Deployer)</a></h2>
<p><strong>Steps:</strong></p>
<ol>
<li>Implement <code>testing_framework_core::scenario::Deployer&lt;Caps&gt;</code> for your capability type</li>
<li>Deploy infrastructure and return a <code>Runner</code></li>
<li>Construct <code>NodeClients</code> and spawn a <code>BlockFeed</code></li>
<li>Build a <code>RunContext</code> and provide a <code>CleanupGuard</code> for teardown</li>
</ol>
<p><strong>Trait outline:</strong></p>
<pre><code class="language-rust ignore">use async_trait::async_trait;
use testing_framework_core::scenario::{
CleanupGuard, Deployer, DynError, Metrics, NodeClients, RunContext, Runner, Scenario,
spawn_block_feed,
};
use testing_framework_core::topology::deployment::Topology;
pub struct MyDeployer {
// Configuration: cluster connection details, etc.
}
impl MyDeployer {
pub fn new() -&gt; Self {
Self {}
}
}
#[async_trait]
impl Deployer&lt;()&gt; for MyDeployer {
type Error = DynError;
async fn deploy(&amp;self, scenario: &amp;Scenario&lt;()&gt;) -&gt; Result&lt;Runner, Self::Error&gt; {
// 1. Launch nodes using scenario.topology()
// 2. Wait for readiness (e.g., consensus info endpoint responds)
// 3. Build NodeClients for validators/executors
// 4. Spawn a block feed for expectations (optional but recommended)
// 5. Create NodeControlHandle if you support restarts (optional)
// 6. Return a Runner wrapping RunContext + CleanupGuard
tracing::info!("deploying scenario with MyDeployer");
let topology: Option&lt;Topology&gt; = None; // Some(topology) if you spawned one
let node_clients = NodeClients::default(); // Or NodeClients::from_topology(...)
let client = node_clients
.any_client()
.ok_or("no api clients available")?
.clone();
let (block_feed, block_feed_guard) = spawn_block_feed(client).await?;
let telemetry = Metrics::empty(); // or Metrics::from_prometheus(...)
let node_control = None; // or Some(Arc&lt;dyn NodeControlHandle&gt;)
let context = RunContext::new(
scenario.topology().clone(),
topology,
node_clients,
scenario.duration(),
telemetry,
block_feed,
node_control,
);
// If you also have other resources to clean up (containers/pods/etc),
// wrap them in your own CleanupGuard implementation and call
// CleanupGuard::cleanup(Box::new(block_feed_guard)) inside it.
Ok(Runner::new(context, Some(Box::new(block_feed_guard))))
}
}</code></pre>
<p><strong>Key points:</strong></p>
<ul>
<li><code>deploy()</code> must return a fully prepared <code>Runner</code></li>
<li>Block until nodes are ready before returning (avoid false negatives)</li>
<li>Use a <code>CleanupGuard</code> to tear down resources on failure (and on <code>RunHandle</code> drop)</li>
<li>If you want chaos workloads, also provide a <code>NodeControlHandle</code> via <code>RunContext</code></li>
</ul>
<h2 id="adding-topology-helpers"><a class="header" href="#adding-topology-helpers">Adding Topology Helpers</a></h2>
<p><strong>Steps:</strong></p>
<ol>
<li>Extend <code>testing_framework_core::topology::config::TopologyBuilder</code> with new layouts</li>
<li>Keep defaults safe: ensure at least one participant, clamp dispersal factors</li>
<li>Consider adding configuration presets for specialized parameters</li>
</ol>
<p><strong>Example:</strong></p>
<pre><code class="language-rust ignore">use testing_framework_core::topology::{
config::TopologyBuilder,
configs::network::Libp2pNetworkLayout,
};
pub trait TopologyBuilderExt {
fn network_full(self) -&gt; Self;
}
impl TopologyBuilderExt for TopologyBuilder {
fn network_full(self) -&gt; Self {
self.with_network_layout(Libp2pNetworkLayout::Full)
}
}</code></pre>
<p><strong>Key points:</strong></p>
<ul>
<li>Maintain method chaining (return <code>&amp;mut Self</code>)</li>
<li>Validate inputs: clamp factors, enforce minimums</li>
<li>Document assumptions (e.g., "requires at least 4 nodes")</li>
</ul>
<h2 id="adding-a-dsl-helper"><a class="header" href="#adding-a-dsl-helper">Adding a DSL Helper</a></h2>
<p>To expose your custom workload through the high-level DSL, add a trait extension:</p>
<pre><code class="language-rust ignore">use async_trait::async_trait;
use testing_framework_core::scenario::{DynError, RunContext, ScenarioBuilder, Workload};
#[derive(Default)]
pub struct MyWorkloadBuilder {
target_rate: u64,
some_option: bool,
}
impl MyWorkloadBuilder {
pub const fn target_rate(mut self, target_rate: u64) -&gt; Self {
self.target_rate = target_rate;
self
}
pub const fn some_option(mut self, some_option: bool) -&gt; Self {
self.some_option = some_option;
self
}
pub const fn build(self) -&gt; MyWorkload {
MyWorkload {
target_rate: self.target_rate,
some_option: self.some_option,
}
}
}
pub struct MyWorkload {
target_rate: u64,
some_option: bool,
}
#[async_trait]
impl Workload for MyWorkload {
fn name(&amp;self) -&gt; &amp;str {
"my_workload"
}
async fn start(&amp;self, _ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
Ok(())
}
}
pub trait MyWorkloadDsl {
fn my_workload_with(
self,
f: impl FnOnce(MyWorkloadBuilder) -&gt; MyWorkloadBuilder,
) -&gt; Self;
}
impl MyWorkloadDsl for ScenarioBuilder {
fn my_workload_with(
self,
f: impl FnOnce(MyWorkloadBuilder) -&gt; MyWorkloadBuilder,
) -&gt; Self {
let builder = f(MyWorkloadBuilder::default());
self.with_workload(builder.build())
}
}</code></pre>
<p>Users can then call:</p>
<pre><code class="language-rust ignore">ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(1))
.my_workload_with(|w| {
w.target_rate(10)
.some_option(true)
})
.build()</code></pre>
<h2 id="see-also-3"><a class="header" href="#see-also-3">See Also</a></h2>
<ul>
<li><a href="api-levels.html">API Levels: Builder DSL vs. Direct</a> - Understanding the two API levels</li>
<li><a href="custom-workload-example.html">Custom Workload Example</a> - Complete runnable example</li>
<li><a href="internal-crate-reference.html">Internal Crate Reference</a> - Where to add new code</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="example-new-workload--expectation-rust"><a class="header" href="#example-new-workload--expectation-rust">Example: New Workload &amp; Expectation (Rust)</a></h1>
<p>A minimal, end-to-end illustration of adding a custom workload and matching
expectation. This shows the shape of the traits and where to plug into the
framework; expand the logic to fit your real test.</p>
<h2 id="workload-simple-reachability-probe"><a class="header" href="#workload-simple-reachability-probe">Workload: simple reachability probe</a></h2>
<p>Key ideas:</p>
<ul>
<li><strong>name</strong>: identifies the workload in logs.</li>
<li><strong>expectations</strong>: workloads can bundle defaults so callers don’t forget checks.</li>
<li><strong>init</strong>: derive inputs from the generated topology (e.g., pick a target node).</li>
<li><strong>start</strong>: drive async activity using the shared <code>RunContext</code>.</li>
</ul>
<pre><code class="language-rust ignore">use async_trait::async_trait;
use testing_framework_core::{
scenario::{DynError, Expectation, RunContext, RunMetrics, Workload},
topology::generation::GeneratedTopology,
};
pub struct ReachabilityWorkload {
target_idx: usize,
}
impl ReachabilityWorkload {
pub fn new(target_idx: usize) -&gt; Self {
Self { target_idx }
}
}
#[async_trait]
impl Workload for ReachabilityWorkload {
fn name(&amp;self) -&gt; &amp;str {
"reachability_workload"
}
fn expectations(&amp;self) -&gt; Vec&lt;Box&lt;dyn Expectation&gt;&gt; {
vec![Box::new(
crate::custom_workload_example_expectation::ReachabilityExpectation::new(
self.target_idx,
),
)]
}
fn init(
&amp;mut self,
topology: &amp;GeneratedTopology,
_run_metrics: &amp;RunMetrics,
) -&gt; Result&lt;(), DynError&gt; {
if topology.validators().get(self.target_idx).is_none() {
return Err(Box::new(std::io::Error::new(
std::io::ErrorKind::Other,
"no validator at requested index",
)));
}
Ok(())
}
async fn start(&amp;self, ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
let client = ctx
.node_clients()
.validator_clients()
.get(self.target_idx)
.ok_or_else(|| {
Box::new(std::io::Error::new(
std::io::ErrorKind::Other,
"missing target client",
)) as DynError
})?;
// Lightweight API call to prove reachability.
client
.consensus_info()
.await
.map(|_| ())
.map_err(|e| e.into())
}
}</code></pre>
<h2 id="expectation-confirm-the-target-stayed-reachable"><a class="header" href="#expectation-confirm-the-target-stayed-reachable">Expectation: confirm the target stayed reachable</a></h2>
<p>Key ideas:</p>
<ul>
<li><strong>start_capture</strong>: snapshot baseline if needed (not used here).</li>
<li><strong>evaluate</strong>: assert the condition after workloads finish.</li>
</ul>
<pre><code class="language-rust ignore">use async_trait::async_trait;
use testing_framework_core::scenario::{DynError, Expectation, RunContext};
pub struct ReachabilityExpectation {
target_idx: usize,
}
impl ReachabilityExpectation {
pub fn new(target_idx: usize) -&gt; Self {
Self { target_idx }
}
}
#[async_trait]
impl Expectation for ReachabilityExpectation {
fn name(&amp;self) -&gt; &amp;str {
"target_reachable"
}
async fn evaluate(&amp;mut self, ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
let client = ctx
.node_clients()
.validator_clients()
.get(self.target_idx)
.ok_or_else(|| {
Box::new(std::io::Error::new(
std::io::ErrorKind::Other,
"missing target client",
)) as DynError
})?;
client
.consensus_info()
.await
.map(|_| ())
.map_err(|e| e.into())
}
}</code></pre>
<h2 id="how-to-wire-it"><a class="header" href="#how-to-wire-it">How to wire it</a></h2>
<ul>
<li>Build your scenario as usual and call <code>.with_workload(ReachabilityWorkload::new(0))</code>.</li>
<li>The bundled expectation is attached automatically; you can add more with
<code>.with_expectation(...)</code> if needed.</li>
<li>Keep the logic minimal and fast for smoke tests; grow it into richer probes
for deeper scenarios.</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="internal-crate-reference"><a class="header" href="#internal-crate-reference">Internal Crate Reference</a></h1>
<p>High-level roles of the crates that make up the framework:</p>
<ul>
<li>
<p><strong>Configs</strong> (<code>testing-framework/configs/</code>): Prepares reusable configuration primitives for nodes, networking, tracing, data availability, and wallets, shared by all scenarios and runners. Includes topology generation and circuit asset resolution.</p>
</li>
<li>
<p><strong>Core scenario orchestration</strong> (<code>testing-framework/core/</code>): Houses the topology and scenario model, runtime coordination, node clients, and readiness/health probes. Defines <code>Deployer</code> and <code>Runner</code> traits, <code>ScenarioBuilder</code>, and <code>RunContext</code>.</p>
</li>
<li>
<p><strong>Workflows</strong> (<code>testing-framework/workflows/</code>): Packages workloads (transaction, DA, chaos) and expectations (consensus liveness) into reusable building blocks. Offers fluent DSL extensions (<code>ScenarioBuilderExt</code>, <code>ChaosBuilderExt</code>).</p>
</li>
<li>
<p><strong>Runners</strong> (<code>testing-framework/runners/{local,compose,k8s}/</code>): Implements deployment backends (local host, Docker Compose, Kubernetes) that all consume the same scenario plan. Each provides a <code>Deployer</code> implementation (<code>LocalDeployer</code>, <code>ComposeDeployer</code>, <code>K8sDeployer</code>).</p>
</li>
<li>
<p><strong>Runner Examples</strong> (crate name: <code>runner-examples</code>, path: <code>examples/</code>): Runnable binaries demonstrating framework usage and serving as living documentation. These are the <strong>primary entry point</strong> for running scenarios (<code>examples/src/bin/local_runner.rs</code>, <code>examples/src/bin/compose_runner.rs</code>, <code>examples/src/bin/k8s_runner.rs</code>).</p>
</li>
</ul>
<h2 id="where-to-add-new-capabilities"><a class="header" href="#where-to-add-new-capabilities">Where to Add New Capabilities</a></h2>
<div class="table-wrapper"><table><thead><tr><th>What You're Adding</th><th>Where It Goes</th><th>Examples</th></tr></thead><tbody>
<tr><td><strong>Node config parameter</strong></td><td><code>testing-framework/configs/src/topology/configs/</code></td><td>Slot duration, log levels, DA params</td></tr>
<tr><td><strong>Topology feature</strong></td><td><code>testing-framework/core/src/topology/</code></td><td>New network layouts, node roles</td></tr>
<tr><td><strong>Scenario capability</strong></td><td><code>testing-framework/core/src/scenario/</code></td><td>New capabilities, context methods</td></tr>
<tr><td><strong>Workload</strong></td><td><code>testing-framework/workflows/src/workloads/</code></td><td>New traffic generators</td></tr>
<tr><td><strong>Expectation</strong></td><td><code>testing-framework/workflows/src/expectations/</code></td><td>New success criteria</td></tr>
<tr><td><strong>Builder API</strong></td><td><code>testing-framework/workflows/src/builder/</code></td><td>DSL extensions, fluent methods</td></tr>
<tr><td><strong>Deployer</strong></td><td><code>testing-framework/runners/</code></td><td>New deployment backends</td></tr>
<tr><td><strong>Example scenario</strong></td><td><code>examples/src/bin/</code></td><td>Demonstration binaries</td></tr>
</tbody></table>
</div>
<h2 id="extension-workflow"><a class="header" href="#extension-workflow">Extension Workflow</a></h2>
<h3 id="adding-a-new-workload"><a class="header" href="#adding-a-new-workload">Adding a New Workload</a></h3>
<ol>
<li><strong>Define the workload</strong> in <code>testing-framework/workflows/src/workloads/your_workload.rs</code>:</li>
</ol>
<pre><code class="language-rust ignore">use async_trait::async_trait;
use testing_framework_core::scenario::{DynError, RunContext, Workload};
pub struct YourWorkload;
#[async_trait]
impl Workload for YourWorkload {
fn name(&amp;self) -&gt; &amp;'static str {
"your_workload"
}
async fn start(&amp;self, _ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
// implementation
Ok(())
}
}</code></pre>
<ol start="2">
<li><strong>Add builder extension</strong> in <code>testing-framework/workflows/src/builder/mod.rs</code>:</li>
</ol>
<pre><code class="language-rust ignore">pub struct YourWorkloadBuilder;
impl YourWorkloadBuilder {
pub fn some_config(self) -&gt; Self {
self
}
}
pub trait ScenarioBuilderExt: Sized {
fn your_workload(self) -&gt; YourWorkloadBuilder;
}</code></pre>
<ol start="3">
<li><strong>Use in examples</strong> in <code>examples/src/bin/your_scenario.rs</code>:</li>
</ol>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::ScenarioBuilder;
pub struct YourWorkloadBuilder;
impl YourWorkloadBuilder {
pub fn some_config(self) -&gt; Self {
self
}
}
pub trait YourWorkloadDslExt: Sized {
fn your_workload_with&lt;F&gt;(self, configurator: F) -&gt; Self
where
F: FnOnce(YourWorkloadBuilder) -&gt; YourWorkloadBuilder;
}
impl&lt;Caps&gt; YourWorkloadDslExt for testing_framework_core::scenario::Builder&lt;Caps&gt; {
fn your_workload_with&lt;F&gt;(self, configurator: F) -&gt; Self
where
F: FnOnce(YourWorkloadBuilder) -&gt; YourWorkloadBuilder,
{
let _ = configurator(YourWorkloadBuilder);
self
}
}
pub fn use_in_examples() {
let _plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(0))
.your_workload_with(|w| w.some_config())
.build();
}</code></pre>
<h3 id="adding-a-new-expectation"><a class="header" href="#adding-a-new-expectation">Adding a New Expectation</a></h3>
<ol>
<li><strong>Define the expectation</strong> in <code>testing-framework/workflows/src/expectations/your_expectation.rs</code>:</li>
</ol>
<pre><code class="language-rust ignore">use async_trait::async_trait;
use testing_framework_core::scenario::{DynError, Expectation, RunContext};
pub struct YourExpectation;
#[async_trait]
impl Expectation for YourExpectation {
fn name(&amp;self) -&gt; &amp;'static str {
"your_expectation"
}
async fn evaluate(&amp;mut self, _ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
// implementation
Ok(())
}
}</code></pre>
<ol start="2">
<li><strong>Add builder extension</strong> in <code>testing-framework/workflows/src/builder/mod.rs</code>:</li>
</ol>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::ScenarioBuilder;
pub trait YourExpectationDslExt: Sized {
fn expect_your_condition(self) -&gt; Self;
}
impl&lt;Caps&gt; YourExpectationDslExt for testing_framework_core::scenario::Builder&lt;Caps&gt; {
fn expect_your_condition(self) -&gt; Self {
self
}
}
pub fn use_in_examples() {
let _plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(0))
.expect_your_condition()
.build();
}</code></pre>
<h3 id="adding-a-new-deployer"><a class="header" href="#adding-a-new-deployer">Adding a New Deployer</a></h3>
<ol>
<li><strong>Implement <code>Deployer</code> trait</strong> in <code>testing-framework/runners/your_runner/src/deployer.rs</code>:</li>
</ol>
<pre><code class="language-rust ignore">use async_trait::async_trait;
use testing_framework_core::scenario::{Deployer, Runner, Scenario};
#[derive(Debug)]
pub struct YourError;
pub struct YourDeployer;
#[async_trait]
impl Deployer for YourDeployer {
type Error = YourError;
async fn deploy(&amp;self, _scenario: &amp;Scenario&lt;()&gt;) -&gt; Result&lt;Runner, Self::Error&gt; {
// Provision infrastructure
// Wait for readiness
// Return Runner
todo!()
}
}</code></pre>
<ol start="2">
<li>
<p><strong>Provide cleanup</strong> and handle node control if supported.</p>
</li>
<li>
<p><strong>Add example</strong> in <code>examples/src/bin/your_runner.rs</code>.</p>
</li>
</ol>
<p>For detailed examples, see <a href="extending.html">Extending the Framework</a> and <a href="custom-workload-example.html">Custom Workload Example</a>.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="part-iv--operations--deployment"><a class="header" href="#part-iv--operations--deployment">Part IV — Operations &amp; Deployment</a></h1>
<p>This section covers operational aspects of running the testing framework: prerequisites, deployment configuration, continuous integration, and observability.</p>
<h2 id="what-youll-learn"><a class="header" href="#what-youll-learn">What You'll Learn</a></h2>
<ul>
<li><strong>Prerequisites &amp; Setup</strong>: Required files, binaries, circuit assets, and environment configuration</li>
<li><strong>Running Examples</strong>: How to execute scenarios across host, compose, and k8s runners</li>
<li><strong>CI Integration</strong>: Automating tests in continuous integration pipelines with caching and matrix testing</li>
<li><strong>Environment Variables</strong>: Complete reference of all configuration variables</li>
<li><strong>Logging &amp; Observability</strong>: Log collection strategies, metrics integration, and debugging techniques</li>
</ul>
<h2 id="who-this-section-is-for"><a class="header" href="#who-this-section-is-for">Who This Section Is For</a></h2>
<ul>
<li><strong>Operators</strong> setting up the framework for the first time</li>
<li><strong>DevOps Engineers</strong> integrating tests into CI/CD pipelines</li>
<li><strong>Developers</strong> debugging test failures or performance issues</li>
<li><strong>Platform Engineers</strong> deploying across different environments (local, Docker, Kubernetes)</li>
</ul>
<h2 id="navigation"><a class="header" href="#navigation">Navigation</a></h2>
<p>This section is organized for progressive depth:</p>
<ol>
<li>Start with <a href="operations-overview.html">Operations Overview</a> for the big picture</li>
<li>Follow <a href="prerequisites.html">Prerequisites &amp; Setup</a> to prepare your environment</li>
<li>Use <a href="running-examples.html">Running Examples</a> to execute your first scenarios</li>
<li>Integrate with <a href="ci-integration.html">CI Integration</a> for automated testing</li>
<li>Reference <a href="environment-variables.html">Environment Variables</a> for complete configuration options</li>
<li>Debug with <a href="logging-observability.html">Logging &amp; Observability</a> when issues arise</li>
</ol>
<h2 id="key-principles"><a class="header" href="#key-principles">Key Principles</a></h2>
<p><strong>Operational Hygiene:</strong> Assets present, prerequisites satisfied, observability reachable</p>
<p><strong>Environment Fit:</strong> Choose the right deployment target based on isolation, reproducibility, and resource needs</p>
<p><strong>Clear Signals:</strong> Verify runners report node readiness before starting workloads</p>
<p><strong>Failure Triage:</strong> Map failures to specific causes—missing prerequisites, platform issues, or unmet expectations</p>
<hr />
<p>Ready to get started? Begin with <a href="operations-overview.html">Operations Overview</a> →</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="operations--deployment-overview"><a class="header" href="#operations--deployment-overview">Operations &amp; Deployment Overview</a></h1>
<p>Operational readiness focuses on prerequisites, environment fit, and clear signals that ensure your test scenarios run reliably across different deployment targets.</p>
<h2 id="core-principles"><a class="header" href="#core-principles">Core Principles</a></h2>
<ul>
<li><strong>Prerequisites First</strong>: Ensure all required files, binaries, and assets are in place before attempting to run scenarios</li>
<li><strong>Environment Fit</strong>: Choose the right deployment target (host, compose, k8s) based on your isolation, reproducibility, and resource needs</li>
<li><strong>Clear Signals</strong>: Verify runners report node readiness before starting workloads to avoid false negatives</li>
<li><strong>Failure Triage</strong>: Map failures to specific causes—missing prerequisites, platform issues, or unmet expectations</li>
</ul>
<h2 id="key-operational-concerns"><a class="header" href="#key-operational-concerns">Key Operational Concerns</a></h2>
<p><strong>Prerequisites:</strong></p>
<ul>
<li><code>versions.env</code> file at repository root (required by helper scripts)</li>
<li>Node binaries (<code>nomos-node</code>, <code>nomos-executor</code>) available or built on demand</li>
<li>Platform requirements met (Docker for compose, cluster access for k8s)</li>
<li>Circuit assets for DA workloads</li>
</ul>
<p><strong>Artifacts:</strong></p>
<ul>
<li>KZG parameters (circuit assets) for Data Availability scenarios</li>
<li>Docker images for compose/k8s deployments</li>
<li>Binary bundles for reproducible builds</li>
</ul>
<p><strong>Environment Configuration:</strong></p>
<ul>
<li><code>POL_PROOF_DEV_MODE=true</code> is <strong>REQUIRED for all runners</strong> to avoid expensive proof generation</li>
<li>Logging configured via <code>NOMOS_LOG_*</code> variables</li>
<li>Observability endpoints (Prometheus, Grafana) optional but useful</li>
</ul>
<p><strong>Readiness &amp; Health:</strong></p>
<ul>
<li>Runners verify node readiness before starting workloads</li>
<li>Health checks prevent premature workload execution</li>
<li>Consensus liveness expectations validate basic operation</li>
</ul>
<h2 id="runner-agnostic-design"><a class="header" href="#runner-agnostic-design">Runner-Agnostic Design</a></h2>
<p>The framework is intentionally <strong>runner-agnostic</strong>: the same scenario plan runs across all deployment targets. Understanding which operational concerns apply to each runner helps you choose the right fit.</p>
<div class="table-wrapper"><table><thead><tr><th>Concern</th><th>Host</th><th>Compose</th><th>Kubernetes</th></tr></thead><tbody>
<tr><td><strong>Topology</strong></td><td>Full support</td><td>Full support</td><td>Full support</td></tr>
<tr><td><strong>Workloads</strong></td><td>All workloads</td><td>All workloads</td><td>All workloads</td></tr>
<tr><td><strong>Expectations</strong></td><td>All expectations</td><td>All expectations</td><td>All expectations</td></tr>
<tr><td><strong>Chaos / Node Control</strong></td><td>Not supported</td><td>Supported</td><td>Not yet</td></tr>
<tr><td><strong>Metrics / Observability</strong></td><td>Manual setup</td><td>External stack</td><td>Cluster-wide</td></tr>
<tr><td><strong>Log Collection</strong></td><td>Temp files</td><td>Container logs</td><td>Pod logs</td></tr>
<tr><td><strong>Isolation</strong></td><td>Process-level</td><td>Container</td><td>Pod + namespace</td></tr>
<tr><td><strong>Setup Time</strong></td><td>&lt; 1 min</td><td>2-5 min</td><td>5-10 min</td></tr>
<tr><td><strong>CI Recommended?</strong></td><td>Smoke tests</td><td>Primary</td><td>Large-scale only</td></tr>
</tbody></table>
</div>
<p><strong>Key insight:</strong> Operational concerns (prerequisites, environment variables) are largely <strong>consistent</strong> across runners, while deployment-specific concerns (isolation, chaos support) vary by backend.</p>
<h2 id="operational-workflow"><a class="header" href="#operational-workflow">Operational Workflow</a></h2>
<pre><code class="language-mermaid">flowchart LR
Setup[Prerequisites &amp; Setup] --&gt; Run[Run Scenarios]
Run --&gt; Monitor[Monitor &amp; Observe]
Monitor --&gt; Debug{Success?}
Debug --&gt;|No| Triage[Failure Triage]
Triage --&gt; Setup
Debug --&gt;|Yes| Done[Complete]
</code></pre>
<ol>
<li><strong>Setup</strong>: Verify prerequisites, configure environment, prepare assets</li>
<li><strong>Run</strong>: Execute scenarios using appropriate runner (host/compose/k8s)</li>
<li><strong>Monitor</strong>: Collect logs, metrics, and observability signals</li>
<li><strong>Triage</strong>: When failures occur, map to root causes and fix prerequisites</li>
</ol>
<h2 id="documentation-structure-1"><a class="header" href="#documentation-structure-1">Documentation Structure</a></h2>
<p>This Operations &amp; Deployment section covers:</p>
<ul>
<li><a href="prerequisites.html">Prerequisites &amp; Setup</a> — Required files, binaries, and environment setup</li>
<li><a href="running-examples.html">Running Examples</a> — How to run scenarios across different runners</li>
<li><a href="ci-integration.html">CI Integration</a> — Automating tests in continuous integration pipelines</li>
<li><a href="environment-variables.html">Environment Variables</a> — Complete reference of configuration variables</li>
<li><a href="logging-observability.html">Logging &amp; Observability</a> — Log collection, metrics, and debugging</li>
</ul>
<p><strong>Philosophy:</strong> Treat operational hygiene—assets present, prerequisites satisfied, observability reachable—as the first step to reliable scenario outcomes.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="prerequisites--setup"><a class="header" href="#prerequisites--setup">Prerequisites &amp; Setup</a></h1>
<p>This page covers everything you need before running your first scenario.</p>
<h2 id="required-files"><a class="header" href="#required-files">Required Files</a></h2>
<h3 id="versionsenv-required"><a class="header" href="#versionsenv-required"><code>versions.env</code> (Required)</a></h3>
<p>All helper scripts require a <code>versions.env</code> file at the repository root:</p>
<pre><code class="language-bash">VERSION=v0.3.1
NOMOS_NODE_REV=abc123def456789
NOMOS_BUNDLE_VERSION=v1
</code></pre>
<p><strong>What it defines:</strong></p>
<ul>
<li><code>VERSION</code> — Circuit release tag for KZG parameters</li>
<li><code>NOMOS_NODE_REV</code> — Git revision of nomos-node to build/fetch</li>
<li><code>NOMOS_BUNDLE_VERSION</code> — Bundle schema version</li>
</ul>
<p><strong>Where it's used:</strong></p>
<ul>
<li><code>scripts/run/run-examples.sh</code></li>
<li><code>scripts/build/build-bundle.sh</code></li>
<li><code>scripts/setup/setup-nomos-circuits.sh</code></li>
<li>CI workflows</li>
</ul>
<p><strong>Error if missing:</strong></p>
<pre><code class="language-text">ERROR: versions.env not found at repository root
This file is required and should define:
VERSION=&lt;circuit release tag&gt;
NOMOS_NODE_REV=&lt;nomos-node git revision&gt;
NOMOS_BUNDLE_VERSION=&lt;bundle schema version&gt;
</code></pre>
<p><strong>Fix:</strong> Ensure you're in the repository root. The file should already exist in the checked-out repo.</p>
<h2 id="node-binaries"><a class="header" href="#node-binaries">Node Binaries</a></h2>
<p>Scenarios need compiled <code>nomos-node</code> and <code>nomos-executor</code> binaries.</p>
<h3 id="option-1-use-helper-scripts-recommended"><a class="header" href="#option-1-use-helper-scripts-recommended">Option 1: Use Helper Scripts (Recommended)</a></h3>
<pre><code class="language-bash">scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
</code></pre>
<p>This automatically:</p>
<ul>
<li>Clones/updates nomos-node checkout</li>
<li>Builds required binaries</li>
<li>Sets <code>NOMOS_NODE_BIN</code> / <code>NOMOS_EXECUTOR_BIN</code></li>
</ul>
<h3 id="option-2-manual-build"><a class="header" href="#option-2-manual-build">Option 2: Manual Build</a></h3>
<p>If you have a sibling <code>nomos-node</code> checkout:</p>
<pre><code class="language-bash">cd ../nomos-node
cargo build --release --bin nomos-node --bin nomos-executor
# Set environment variables
export NOMOS_NODE_BIN=$PWD/target/release/nomos-node
export NOMOS_EXECUTOR_BIN=$PWD/target/release/nomos-executor
# Return to testing framework
cd ../nomos-testing
</code></pre>
<h3 id="option-3-prebuilt-bundles-ci"><a class="header" href="#option-3-prebuilt-bundles-ci">Option 3: Prebuilt Bundles (CI)</a></h3>
<p>CI workflows use prebuilt artifacts:</p>
<pre><code class="language-yaml">- name: Download nomos binaries
uses: actions/download-artifact@v3
with:
name: nomos-binaries-linux
path: .tmp/
- name: Extract bundle
run: |
tar -xzf .tmp/nomos-binaries-linux-*.tar.gz -C .tmp/
export NOMOS_NODE_BIN=$PWD/.tmp/nomos-node
export NOMOS_EXECUTOR_BIN=$PWD/.tmp/nomos-executor
</code></pre>
<h2 id="circuit-assets-kzg-parameters"><a class="header" href="#circuit-assets-kzg-parameters">Circuit Assets (KZG Parameters)</a></h2>
<p>Data Availability (DA) workloads require KZG cryptographic parameters.</p>
<h3 id="asset-location"><a class="header" href="#asset-location">Asset Location</a></h3>
<p><strong>Default path:</strong> <code>testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params</code></p>
<p>Note: The directory <code>kzgrs_test_params/</code> contains a file named <code>kzgrs_test_params</code>. This is the proving key file (~120MB).</p>
<p><strong>Container path (compose/k8s):</strong> <code>/kzgrs_test_params/kzgrs_test_params</code></p>
<h3 id="getting-assets"><a class="header" href="#getting-assets">Getting Assets</a></h3>
<p><strong>Option 1: Use helper script</strong> (recommended):</p>
<pre><code class="language-bash"># Fetch circuits
scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits
# Copy to default location
mkdir -p testing-framework/assets/stack/kzgrs_test_params
cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/
# Verify (should be ~120MB)
ls -lh testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params
</code></pre>
<p><strong>Option 2: Let <code>run-examples.sh</code> handle it</strong>:</p>
<pre><code class="language-bash">scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
</code></pre>
<p>This automatically fetches and places assets.</p>
<h3 id="override-path"><a class="header" href="#override-path">Override Path</a></h3>
<p>Set <code>NOMOS_KZGRS_PARAMS_PATH</code> to use a custom location:</p>
<pre><code class="language-bash">NOMOS_KZGRS_PARAMS_PATH=/custom/path/to/kzgrs_test_params \
cargo run -p runner-examples --bin local_runner
</code></pre>
<h3 id="when-are-assets-needed"><a class="header" href="#when-are-assets-needed">When Are Assets Needed?</a></h3>
<div class="table-wrapper"><table><thead><tr><th>Runner</th><th>When Required</th></tr></thead><tbody>
<tr><td><strong>Host (local)</strong></td><td>Always (for DA workloads)</td></tr>
<tr><td><strong>Compose</strong></td><td>During image build (baked into image)</td></tr>
<tr><td><strong>K8s</strong></td><td>During image build + mounted via hostPath</td></tr>
</tbody></table>
</div>
<p><strong>Error without assets:</strong></p>
<pre><code class="language-text">Error: Custom { kind: NotFound, error: "Circuit file not found at: testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params" }
</code></pre>
<h2 id="platform-requirements"><a class="header" href="#platform-requirements">Platform Requirements</a></h2>
<h3 id="host-runner-local-processes-1"><a class="header" href="#host-runner-local-processes-1">Host Runner (Local Processes)</a></h3>
<p><strong>Requires:</strong></p>
<ul>
<li>Rust nightly toolchain</li>
<li>Node binaries built</li>
<li>KZG circuit assets (for DA workloads)</li>
<li>Available ports (18080+, 3100+, etc.)</li>
</ul>
<p><strong>No Docker required.</strong></p>
<p><strong>Best for:</strong></p>
<ul>
<li>Quick iteration</li>
<li>Development</li>
<li>Smoke tests</li>
</ul>
<h3 id="compose-runner-docker-compose"><a class="header" href="#compose-runner-docker-compose">Compose Runner (Docker Compose)</a></h3>
<p><strong>Requires:</strong></p>
<ul>
<li>Docker daemon running</li>
<li>Docker image built: <code>logos-blockchain-testing:local</code></li>
<li>KZG assets baked into image</li>
<li>Docker Desktop (macOS) or Docker Engine (Linux)</li>
</ul>
<p><strong>Platform notes (macOS / Apple silicon):</strong></p>
<ul>
<li>Prefer <code>NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64</code> for native performance</li>
<li>Use <code>linux/amd64</code> only if targeting amd64 environments (slower via emulation)</li>
</ul>
<p><strong>Best for:</strong></p>
<ul>
<li>Reproducible environments</li>
<li>CI testing</li>
<li>Chaos workloads (node control support)</li>
</ul>
<h3 id="k8s-runner-kubernetes"><a class="header" href="#k8s-runner-kubernetes">K8s Runner (Kubernetes)</a></h3>
<p><strong>Requires:</strong></p>
<ul>
<li>Kubernetes cluster (Docker Desktop K8s, minikube, kind, or remote)</li>
<li><code>kubectl</code> configured</li>
<li>Docker image built and loaded/pushed</li>
<li>KZG assets baked into image + mounted via hostPath</li>
</ul>
<p><strong>Local cluster setup:</strong></p>
<pre><code class="language-bash"># Docker Desktop: Enable Kubernetes in settings
# OR: Use kind
kind create cluster
kind load docker-image logos-blockchain-testing:local
# OR: Use minikube
minikube start
minikube image load logos-blockchain-testing:local
</code></pre>
<p><strong>Remote cluster:</strong> Push image to registry and set <code>NOMOS_TESTNET_IMAGE</code>.</p>
<p><strong>Best for:</strong></p>
<ul>
<li>Production-like testing</li>
<li>Resource isolation</li>
<li>Large topologies</li>
</ul>
<h2 id="critical-environment-variable"><a class="header" href="#critical-environment-variable">Critical Environment Variable</a></h2>
<p><strong><code>POL_PROOF_DEV_MODE=true</code> is REQUIRED for ALL runners!</strong></p>
<p>Without this, proof generation uses expensive Groth16 proving, causing:</p>
<ul>
<li>Tests "hang" for minutes</li>
<li>CPU spikes to 100%</li>
<li>Timeouts and failures</li>
</ul>
<p><strong>Always set:</strong></p>
<pre><code class="language-bash">POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
POL_PROOF_DEV_MODE=true scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
# etc.
</code></pre>
<p><strong>Or add to your shell profile:</strong></p>
<pre><code class="language-bash"># ~/.bashrc or ~/.zshrc
export POL_PROOF_DEV_MODE=true
</code></pre>
<h2 id="quick-setup-check"><a class="header" href="#quick-setup-check">Quick Setup Check</a></h2>
<p>Run this checklist before your first scenario:</p>
<pre><code class="language-bash"># 1. Verify versions.env exists
cat versions.env
# 2. Check circuit assets (for DA workloads)
ls -lh testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params
# 3. Verify POL_PROOF_DEV_MODE is set
echo $POL_PROOF_DEV_MODE # Should print: true
# 4. For compose/k8s: verify Docker is running
docker ps
# 5. For compose/k8s: verify image exists
docker images | grep logos-blockchain-testing
# 6. For host runner: verify node binaries (if not using scripts)
$NOMOS_NODE_BIN --version
$NOMOS_EXECUTOR_BIN --version
</code></pre>
<h2 id="recommended-use-helper-scripts"><a class="header" href="#recommended-use-helper-scripts">Recommended: Use Helper Scripts</a></h2>
<p>The easiest path is to let the helper scripts handle everything:</p>
<pre><code class="language-bash"># Host runner
scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
# Compose runner
scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
# K8s runner
scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
</code></pre>
<p>These scripts:</p>
<ul>
<li>Verify <code>versions.env</code> exists</li>
<li>Clone/build nomos-node if needed</li>
<li>Fetch circuit assets if missing</li>
<li>Build Docker images (compose/k8s)</li>
<li>Load images into cluster (k8s)</li>
<li>Run the scenario with proper environment</li>
</ul>
<p><strong>Next Steps:</strong></p>
<ul>
<li><a href="running-examples.html">Running Examples</a> — Learn how to run scenarios</li>
<li><a href="environment-variables.html">Environment Variables</a> — Full variable reference</li>
<li><a href="troubleshooting.html">Troubleshooting</a> — Common issues and fixes</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="running-examples"><a class="header" href="#running-examples">Running Examples</a></h1>
<p>The framework provides three runner modes: <strong>host</strong> (local processes), <strong>compose</strong> (Docker Compose), and <strong>k8s</strong> (Kubernetes).</p>
<h2 id="quick-start-recommended"><a class="header" href="#quick-start-recommended">Quick Start (Recommended)</a></h2>
<p>Use <code>scripts/run/run-examples.sh</code> for all modes—it handles all setup automatically:</p>
<pre><code class="language-bash"># Host mode (local processes)
scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
# Compose mode (Docker Compose)
scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
# K8s mode (Kubernetes)
scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
</code></pre>
<p><strong>Parameters:</strong></p>
<ul>
<li><code>-t 60</code> — Run duration in seconds</li>
<li><code>-v 3</code> — Number of validators</li>
<li><code>-e 1</code> — Number of executors</li>
<li><code>host|compose|k8s</code> — Deployment mode</li>
</ul>
<p>This script handles:</p>
<ul>
<li>Circuit asset setup</li>
<li>Binary building/bundling</li>
<li>Image building (compose/k8s)</li>
<li>Image loading into cluster (k8s)</li>
<li>Execution with proper environment</li>
</ul>
<p><strong>Note:</strong> For <code>k8s</code> runs against non-local clusters (e.g. EKS), the cluster pulls images from a registry. In that case, build + push your image separately (see <code>scripts/build/build_test_image.sh</code>) and set <code>NOMOS_TESTNET_IMAGE</code> to the pushed reference.</p>
<h2 id="quick-smoke-matrix"><a class="header" href="#quick-smoke-matrix">Quick Smoke Matrix</a></h2>
<p>For a small "does everything still run?" matrix across all runners:</p>
<pre><code class="language-bash">scripts/run/run-test-matrix.sh -t 120 -v 1 -e 1
</code></pre>
<p>This runs host, compose, and k8s modes with various image-build configurations. Useful after making runner/image/script changes. Forwards <code>--metrics-*</code> options through to <code>scripts/run/run-examples.sh</code>.</p>
<p><strong>Common options:</strong></p>
<ul>
<li><code>--modes host,compose,k8s</code> — Restrict which modes run</li>
<li><code>--no-clean</code> — Skip <code>scripts/ops/clean.sh</code> step</li>
<li><code>--no-bundles</code> — Skip <code>scripts/build/build-bundle.sh</code> (reuses existing <code>.tmp</code> tarballs)</li>
<li><code>--no-image-build</code> — Skip the “rebuild image” variants in the matrix (compose/k8s)</li>
<li><code>--allow-nonzero-progress</code> — Soft-pass expectation failures if logs show non-zero progress (local iteration only)</li>
<li><code>--force-k8s-image-build</code> — Allow the k8s image-build variant even on non-docker-desktop clusters</li>
</ul>
<p><strong>Environment overrides:</strong></p>
<ul>
<li><code>VERSION=v0.3.1</code> — Circuit version</li>
<li><code>NOMOS_NODE_REV=&lt;commit&gt;</code> — nomos-node git revision</li>
<li><code>NOMOS_BINARIES_TAR=path/to/bundle.tar.gz</code> — Use prebuilt bundle</li>
<li><code>NOMOS_SKIP_IMAGE_BUILD=1</code> — Skip image rebuild inside <code>run-examples.sh</code> (compose/k8s)</li>
<li><code>NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64|linux/amd64</code> — Docker platform for bundle builds (macOS/Windows)</li>
<li><code>COMPOSE_CIRCUITS_PLATFORM=linux-aarch64|linux-x86_64</code> — Circuits platform for image builds</li>
<li><code>SLOW_TEST_ENV=true</code> — Doubles built-in readiness timeouts (useful in CI / constrained laptops)</li>
<li><code>TESTNET_PRINT_ENDPOINTS=1</code> — Print <code>TESTNET_ENDPOINTS</code> / <code>TESTNET_PPROF</code> lines during deploy</li>
</ul>
<h2 id="dev-workflow-updating-nomos-node-revision"><a class="header" href="#dev-workflow-updating-nomos-node-revision">Dev Workflow: Updating nomos-node Revision</a></h2>
<p>The repo pins a <code>nomos-node</code> revision in <code>versions.env</code> for reproducible builds. To update it or point to a local checkout:</p>
<pre><code class="language-bash"># Pin to a new git revision (updates versions.env + Cargo.toml git revs)
scripts/ops/update-nomos-rev.sh --rev &lt;git_sha&gt;
# Use a local nomos-node checkout instead (for development)
scripts/ops/update-nomos-rev.sh --path /path/to/nomos-node
# If Cargo.toml was marked skip-worktree, clear it
scripts/ops/update-nomos-rev.sh --unskip-worktree
</code></pre>
<p><strong>Notes:</strong></p>
<ul>
<li>Don't commit absolute <code>NOMOS_NODE_PATH</code> values; prefer <code>--rev</code> for shared history/CI</li>
<li>After changing rev/path, expect <code>Cargo.lock</code> to update on the next <code>cargo build</code>/<code>cargo test</code></li>
</ul>
<h2 id="cleanup-helper"><a class="header" href="#cleanup-helper">Cleanup Helper</a></h2>
<p>If you hit Docker build failures, I/O errors, or disk space issues:</p>
<pre><code class="language-bash">scripts/ops/clean.sh
</code></pre>
<p>For extra Docker cache cleanup:</p>
<pre><code class="language-bash">scripts/ops/clean.sh --docker
</code></pre>
<hr />
<h2 id="host-runner-direct-cargo-run"><a class="header" href="#host-runner-direct-cargo-run">Host Runner (Direct Cargo Run)</a></h2>
<p>For manual control, run the <code>local_runner</code> binary directly:</p>
<pre><code class="language-bash">POL_PROOF_DEV_MODE=true \
NOMOS_NODE_BIN=/path/to/nomos-node \
NOMOS_EXECUTOR_BIN=/path/to/nomos-executor \
cargo run -p runner-examples --bin local_runner
</code></pre>
<h3 id="host-runner-environment-variables"><a class="header" href="#host-runner-environment-variables">Host Runner Environment Variables</a></h3>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>NOMOS_DEMO_VALIDATORS</code></td><td>1</td><td>Number of validators (legacy: <code>LOCAL_DEMO_VALIDATORS</code>)</td></tr>
<tr><td><code>NOMOS_DEMO_EXECUTORS</code></td><td>1</td><td>Number of executors (legacy: <code>LOCAL_DEMO_EXECUTORS</code>)</td></tr>
<tr><td><code>NOMOS_DEMO_RUN_SECS</code></td><td>60</td><td>Run duration in seconds (legacy: <code>LOCAL_DEMO_RUN_SECS</code>)</td></tr>
<tr><td><code>NOMOS_NODE_BIN</code></td><td>—</td><td>Path to nomos-node binary (required)</td></tr>
<tr><td><code>NOMOS_EXECUTOR_BIN</code></td><td>—</td><td>Path to nomos-executor binary (required)</td></tr>
<tr><td><code>NOMOS_LOG_DIR</code></td><td>None</td><td>Directory for per-node log files</td></tr>
<tr><td><code>NOMOS_TESTS_KEEP_LOGS</code></td><td>0</td><td>Keep per-run temporary directories (useful for debugging/CI)</td></tr>
<tr><td><code>NOMOS_TESTS_TRACING</code></td><td>false</td><td>Enable debug tracing preset</td></tr>
<tr><td><code>NOMOS_LOG_LEVEL</code></td><td>info</td><td>Global log level: error, warn, info, debug, trace</td></tr>
<tr><td><code>NOMOS_LOG_FILTER</code></td><td>None</td><td>Fine-grained module filtering (e.g., <code>cryptarchia=trace,nomos_da_sampling=debug</code>)</td></tr>
<tr><td><code>POL_PROOF_DEV_MODE</code></td><td>—</td><td><strong>REQUIRED</strong>: Set to <code>true</code> for all runners</td></tr>
</tbody></table>
</div>
<p><strong>Note:</strong> Requires circuit assets and host binaries. Use <code>scripts/run/run-examples.sh host</code> to handle setup automatically.</p>
<hr />
<h2 id="compose-runner-direct-cargo-run"><a class="header" href="#compose-runner-direct-cargo-run">Compose Runner (Direct Cargo Run)</a></h2>
<p>For manual control, run the <code>compose_runner</code> binary directly. Compose requires a Docker image with embedded assets.</p>
<h3 id="option-1-prebuilt-bundle-recommended"><a class="header" href="#option-1-prebuilt-bundle-recommended">Option 1: Prebuilt Bundle (Recommended)</a></h3>
<pre><code class="language-bash"># 1. Build a Linux bundle (includes binaries + circuits)
scripts/build/build-bundle.sh --platform linux
# Creates .tmp/nomos-binaries-linux-v0.3.1.tar.gz
# 2. Build image (embeds bundle assets)
export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
scripts/build/build_test_image.sh
# 3. Run
NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin compose_runner
</code></pre>
<h3 id="option-2-manual-circuitimage-setup"><a class="header" href="#option-2-manual-circuitimage-setup">Option 2: Manual Circuit/Image Setup</a></h3>
<pre><code class="language-bash"># Fetch and copy circuits
scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits
cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/
# Build image
scripts/build/build_test_image.sh
# Run
NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin compose_runner
</code></pre>
<h3 id="platform-note-macos--apple-silicon"><a class="header" href="#platform-note-macos--apple-silicon">Platform Note (macOS / Apple Silicon)</a></h3>
<ul>
<li>Docker Desktop runs a <code>linux/arm64</code> engine by default</li>
<li>For native performance: <code>NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64</code> (recommended for local testing)</li>
<li>For amd64 targets: <code>NOMOS_BUNDLE_DOCKER_PLATFORM=linux/amd64</code> (slower via emulation)</li>
</ul>
<h3 id="compose-runner-environment-variables"><a class="header" href="#compose-runner-environment-variables">Compose Runner Environment Variables</a></h3>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>NOMOS_TESTNET_IMAGE</code></td><td>—</td><td>Image tag (required, must match built image)</td></tr>
<tr><td><code>POL_PROOF_DEV_MODE</code></td><td>—</td><td><strong>REQUIRED</strong>: Set to <code>true</code> for all runners</td></tr>
<tr><td><code>NOMOS_DEMO_VALIDATORS</code></td><td>1</td><td>Number of validators</td></tr>
<tr><td><code>NOMOS_DEMO_EXECUTORS</code></td><td>1</td><td>Number of executors</td></tr>
<tr><td><code>NOMOS_DEMO_RUN_SECS</code></td><td>60</td><td>Run duration in seconds</td></tr>
<tr><td><code>COMPOSE_NODE_PAIRS</code></td><td>—</td><td>Alternative topology format: "validators×executors" (e.g., <code>3x2</code>)</td></tr>
<tr><td><code>NOMOS_METRICS_QUERY_URL</code></td><td>None</td><td>Prometheus-compatible base URL for runner to query</td></tr>
<tr><td><code>NOMOS_METRICS_OTLP_INGEST_URL</code></td><td>None</td><td>Full OTLP HTTP ingest URL for node metrics export</td></tr>
<tr><td><code>NOMOS_GRAFANA_URL</code></td><td>None</td><td>Grafana base URL for printing/logging</td></tr>
<tr><td><code>COMPOSE_RUNNER_HOST</code></td><td>127.0.0.1</td><td>Host address for port mappings</td></tr>
<tr><td><code>COMPOSE_RUNNER_PRESERVE</code></td><td>0</td><td>Keep containers running after test</td></tr>
<tr><td><code>NOMOS_LOG_LEVEL</code></td><td>info</td><td>Node log level (stdout/stderr)</td></tr>
<tr><td><code>NOMOS_LOG_FILTER</code></td><td>None</td><td>Fine-grained module filtering</td></tr>
</tbody></table>
</div>
<p><strong>Config file option:</strong> <code>testing-framework/assets/stack/cfgsync.yaml</code> (<code>tracing_settings.logger</code>) — Switch node logs between stdout/stderr and file output</p>
<h3 id="compose-specific-features"><a class="header" href="#compose-specific-features">Compose-Specific Features</a></h3>
<ul>
<li><strong>Node control support</strong>: Only runner that supports chaos testing (<code>.enable_node_control()</code> + chaos workloads)</li>
<li><strong>External observability</strong>: Set <code>NOMOS_METRICS_*</code> / <code>NOMOS_GRAFANA_URL</code> to enable telemetry links and querying
<ul>
<li>Quickstart: <code>scripts/setup/setup-observability.sh compose up</code> then <code>scripts/setup/setup-observability.sh compose env</code></li>
</ul>
</li>
</ul>
<p><strong>Important:</strong></p>
<ul>
<li>Containers expect KZG parameters at <code>/kzgrs_test_params/kzgrs_test_params</code> (note the repeated filename)</li>
<li>Use <code>scripts/run/run-examples.sh compose</code> to handle all setup automatically</li>
</ul>
<hr />
<h2 id="k8s-runner-direct-cargo-run"><a class="header" href="#k8s-runner-direct-cargo-run">K8s Runner (Direct Cargo Run)</a></h2>
<p>For manual control, run the <code>k8s_runner</code> binary directly. K8s requires the same image setup as Compose.</p>
<h3 id="prerequisites-4"><a class="header" href="#prerequisites-4">Prerequisites</a></h3>
<ol>
<li><strong>Kubernetes cluster</strong> with <code>kubectl</code> configured</li>
<li><strong>Test image built</strong> (same as Compose, preferably with prebuilt bundle)</li>
<li><strong>Image available in cluster</strong> (loaded or pushed to registry)</li>
</ol>
<h3 id="build-and-load-image"><a class="header" href="#build-and-load-image">Build and Load Image</a></h3>
<pre><code class="language-bash"># 1. Build image with bundle (recommended)
scripts/build/build-bundle.sh --platform linux
export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
scripts/build/build_test_image.sh
# 2. Load into cluster (choose one)
export NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local
# For kind:
kind load docker-image logos-blockchain-testing:local
# For minikube:
minikube image load logos-blockchain-testing:local
# For remote cluster (push to registry):
docker tag logos-blockchain-testing:local your-registry/logos-blockchain-testing:latest
docker push your-registry/logos-blockchain-testing:latest
export NOMOS_TESTNET_IMAGE=your-registry/logos-blockchain-testing:latest
</code></pre>
<h3 id="run-the-example"><a class="header" href="#run-the-example">Run the Example</a></h3>
<pre><code class="language-bash">export NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local
export POL_PROOF_DEV_MODE=true
cargo run -p runner-examples --bin k8s_runner
</code></pre>
<h3 id="k8s-runner-environment-variables"><a class="header" href="#k8s-runner-environment-variables">K8s Runner Environment Variables</a></h3>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>NOMOS_TESTNET_IMAGE</code></td><td>—</td><td>Image tag (required)</td></tr>
<tr><td><code>POL_PROOF_DEV_MODE</code></td><td>—</td><td><strong>REQUIRED</strong>: Set to <code>true</code> for all runners</td></tr>
<tr><td><code>NOMOS_DEMO_VALIDATORS</code></td><td>1</td><td>Number of validators</td></tr>
<tr><td><code>NOMOS_DEMO_EXECUTORS</code></td><td>1</td><td>Number of executors</td></tr>
<tr><td><code>NOMOS_DEMO_RUN_SECS</code></td><td>60</td><td>Run duration in seconds</td></tr>
<tr><td><code>NOMOS_METRICS_QUERY_URL</code></td><td>None</td><td>Prometheus-compatible base URL for runner to query (PromQL)</td></tr>
<tr><td><code>NOMOS_METRICS_OTLP_INGEST_URL</code></td><td>None</td><td>Full OTLP HTTP ingest URL for node metrics export</td></tr>
<tr><td><code>NOMOS_GRAFANA_URL</code></td><td>None</td><td>Grafana base URL for printing/logging</td></tr>
<tr><td><code>K8S_RUNNER_NAMESPACE</code></td><td>Random</td><td>Kubernetes namespace (pin for debugging)</td></tr>
<tr><td><code>K8S_RUNNER_RELEASE</code></td><td>Random</td><td>Helm release name (pin for debugging)</td></tr>
<tr><td><code>K8S_RUNNER_NODE_HOST</code></td><td>—</td><td>NodePort host resolution for non-local clusters</td></tr>
<tr><td><code>K8S_RUNNER_DEBUG</code></td><td>0</td><td>Log Helm stdout/stderr for install commands</td></tr>
<tr><td><code>K8S_RUNNER_PRESERVE</code></td><td>0</td><td>Keep namespace/release after run (for debugging)</td></tr>
</tbody></table>
</div>
<h3 id="k8s--observability-optional"><a class="header" href="#k8s--observability-optional">K8s + Observability (Optional)</a></h3>
<pre><code class="language-bash">export NOMOS_METRICS_QUERY_URL=http://your-prometheus:9090
# Prometheus OTLP receiver example:
export NOMOS_METRICS_OTLP_INGEST_URL=http://your-prometheus:9090/api/v1/otlp/v1/metrics
# Optional: print Grafana link in TESTNET_ENDPOINTS
export NOMOS_GRAFANA_URL=http://your-grafana:3000
cargo run -p runner-examples --bin k8s_runner
</code></pre>
<p><strong>Notes:</strong></p>
<ul>
<li><code>NOMOS_METRICS_QUERY_URL</code> must be reachable from the runner process (often via <code>kubectl port-forward</code>)</li>
<li><code>NOMOS_METRICS_OTLP_INGEST_URL</code> must be reachable from nodes (pods/containers) and is backend-specific
<ul>
<li>Quickstart installer: <code>scripts/setup/setup-observability.sh k8s install</code> then <code>scripts/setup/setup-observability.sh k8s env</code></li>
<li>Optional dashboards: <code>scripts/setup/setup-observability.sh k8s dashboards</code></li>
</ul>
</li>
</ul>
<h3 id="via-scriptsrunrun-examplessh-recommended"><a class="header" href="#via-scriptsrunrun-examplessh-recommended">Via <code>scripts/run/run-examples.sh</code> (Recommended)</a></h3>
<pre><code class="language-bash">scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s \
--metrics-query-url http://your-prometheus:9090 \
--metrics-otlp-ingest-url http://your-prometheus:9090/api/v1/otlp/v1/metrics
</code></pre>
<h3 id="in-code-optional"><a class="header" href="#in-code-optional">In Code (Optional)</a></h3>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ObservabilityBuilderExt as _;
let plan = ScenarioBuilder::with_node_counts(1, 1)
.with_metrics_query_url_str("http://your-prometheus:9090")
.with_metrics_otlp_ingest_url_str("http://your-prometheus:9090/api/v1/otlp/v1/metrics")
.build();</code></pre>
<h3 id="important-k8s-notes"><a class="header" href="#important-k8s-notes">Important K8s Notes</a></h3>
<ul>
<li>K8s runner mounts <code>testing-framework/assets/stack/kzgrs_test_params</code> as a hostPath volume</li>
<li>File path inside pods: <code>/kzgrs_test_params/kzgrs_test_params</code></li>
<li><strong>No node control support yet</strong>: Chaos workloads (<code>.enable_node_control()</code>) will fail</li>
<li>Optimized for local clusters (Docker Desktop K8s / minikube / kind)
<ul>
<li>Remote clusters require additional setup (registry push, PV/CSI for assets, etc.)</li>
</ul>
</li>
<li>Use <code>scripts/run/run-examples.sh k8s</code> to handle all setup automatically</li>
</ul>
<h2 id="next-steps-2"><a class="header" href="#next-steps-2">Next Steps</a></h2>
<ul>
<li><a href="ci-integration.html">CI Integration</a> — Automate tests in continuous integration</li>
<li><a href="environment-variables.html">Environment Variables</a> — Full variable reference</li>
<li><a href="logging-observability.html">Logging &amp; Observability</a> — Log collection and metrics</li>
<li><a href="troubleshooting.html">Troubleshooting</a> — Common issues and fixes</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="ci-integration"><a class="header" href="#ci-integration">CI Integration</a></h1>
<p>Both <strong>LocalDeployer</strong> and <strong>ComposeDeployer</strong> work well in CI environments. Choose based on your tradeoffs.</p>
<h2 id="runner-comparison-for-ci"><a class="header" href="#runner-comparison-for-ci">Runner Comparison for CI</a></h2>
<p><strong>LocalDeployer (Host Runner):</strong></p>
<ul>
<li>Faster startup (no Docker overhead)</li>
<li>Good for quick smoke tests</li>
<li><strong>Trade-off:</strong> Less isolation (processes share host resources)</li>
</ul>
<p><strong>ComposeDeployer (Recommended for CI):</strong></p>
<ul>
<li>Better isolation (containerized)</li>
<li>Reproducible environment</li>
<li>Can integrate with external Prometheus/Grafana (optional)</li>
<li><strong>Trade-offs:</strong> Slower startup (Docker image build), requires Docker daemon</li>
</ul>
<p><strong>K8sDeployer:</strong></p>
<ul>
<li>Production-like environment</li>
<li>Full resource isolation</li>
<li><strong>Trade-offs:</strong> Slowest (cluster setup + image loading), requires cluster access</li>
<li>Best for nightly/weekly runs or production validation</li>
</ul>
<p><strong>Existing Examples:</strong></p>
<p>See <code>.github/workflows/lint.yml</code> (jobs: <code>host_smoke</code>, <code>compose_smoke</code>) for CI examples running the demo scenarios in this repository.</p>
<h2 id="complete-ci-workflow-example"><a class="header" href="#complete-ci-workflow-example">Complete CI Workflow Example</a></h2>
<p>Here's a comprehensive GitHub Actions workflow demonstrating host and compose runners with caching, matrix testing, and log collection:</p>
<pre><code class="language-yaml">name: Testing Framework CI
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
env:
POL_PROOF_DEV_MODE: true
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
jobs:
# Quick smoke test with host runner (no Docker)
host_smoke:
name: Host Runner Smoke Test
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up Rust toolchain
uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: nightly
override: true
- name: Cache Rust dependencies
uses: actions/cache@v3
with:
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
target/
key: ${{ runner.os }}-cargo-host-${{ hashFiles('**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-host-
- name: Cache nomos-node build
uses: actions/cache@v3
with:
path: |
../nomos-node/target/release/nomos-node
../nomos-node/target/release/nomos-executor
key: ${{ runner.os }}-nomos-${{ hashFiles('../nomos-node/**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-nomos-
- name: Run host smoke test
run: |
# Use run-examples.sh which handles setup automatically
scripts/run/run-examples.sh -t 120 -v 3 -e 1 host
- name: Upload logs on failure
if: failure()
uses: actions/upload-artifact@v3
with:
name: host-runner-logs
path: |
.tmp/
*.log
retention-days: 7
# Compose runner matrix (with Docker)
compose_matrix:
name: Compose Runner (${{ matrix.topology }})
runs-on: ubuntu-latest
timeout-minutes: 25
strategy:
fail-fast: false
matrix:
topology:
- "3v1e"
- "5v1e"
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up Rust toolchain
uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: nightly
override: true
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Cache Rust dependencies
uses: actions/cache@v3
with:
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
target/
key: ${{ runner.os }}-cargo-compose-${{ hashFiles('**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-compose-
- name: Cache Docker layers
uses: actions/cache@v3
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-${{ hashFiles('Dockerfile', 'scripts/build/build_test_image.sh') }}
restore-keys: |
${{ runner.os }}-buildx-
- name: Run compose test
env:
TOPOLOGY: ${{ matrix.topology }}
run: |
# Build and run with the specified topology
scripts/run/run-examples.sh -t 120 -v ${TOPOLOGY:0:1} -e ${TOPOLOGY:2:1} compose
- name: Collect Docker logs on failure
if: failure()
run: |
mkdir -p logs
for container in $(docker ps -a --filter "name=nomos-compose-" -q); do
docker logs $container &gt; logs/$(docker inspect --format='{{.Name}}' $container).log 2&gt;&amp;1
done
- name: Upload logs and artifacts
if: failure()
uses: actions/upload-artifact@v3
with:
name: compose-${{ matrix.topology }}-logs
path: |
logs/
.tmp/
retention-days: 7
- name: Clean up Docker resources
if: always()
run: |
docker compose down -v 2&gt;/dev/null || true
docker ps -a --filter "name=nomos-compose-" -q | xargs -r docker rm -f
# Cucumber/BDD integration tests (if enabled)
cucumber_tests:
name: Cucumber BDD Tests
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up Rust toolchain
uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: nightly
override: true
- name: Cache dependencies
uses: actions/cache@v3
with:
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
target/
key: ${{ runner.os }}-cargo-cucumber-${{ hashFiles('**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-cucumber-
- name: Run Cucumber tests
run: |
# Build prerequisites
scripts/build/build-bundle.sh --platform linux
export NOMOS_BINARIES_TAR=$(ls -t .tmp/nomos-binaries-linux-*.tar.gz | head -1)
# Run Cucumber tests (host runner)
cargo test -p runner-examples --bin cucumber_host
- name: Upload test report
if: always()
uses: actions/upload-artifact@v3
with:
name: cucumber-report
path: |
target/cucumber-reports/
retention-days: 14
# Summary job (requires all tests to pass)
ci_success:
name: CI Success
needs: [host_smoke, compose_matrix, cucumber_tests]
runs-on: ubuntu-latest
if: always()
steps:
- name: Check all jobs
run: |
if [[ "${{ needs.host_smoke.result }}" != "success" ]] || \
[[ "${{ needs.compose_matrix.result }}" != "success" ]] || \
[[ "${{ needs.cucumber_tests.result }}" != "success" ]]; then
echo "One or more CI jobs failed"
exit 1
fi
echo "All CI jobs passed!"
</code></pre>
<h2 id="workflow-features"><a class="header" href="#workflow-features">Workflow Features</a></h2>
<ol>
<li><strong>Matrix Testing:</strong> Runs compose tests with different topologies (<code>3v1e</code>, <code>5v1e</code>)</li>
<li><strong>Caching:</strong> Caches Rust dependencies, Docker layers, and nomos-node builds for faster runs</li>
<li><strong>Log Collection:</strong> Automatically uploads logs and artifacts when tests fail</li>
<li><strong>Timeout Protection:</strong> Reasonable timeouts prevent jobs from hanging indefinitely</li>
<li><strong>Cucumber Integration:</strong> Shows how to integrate BDD tests into CI</li>
<li><strong>Clean Teardown:</strong> Ensures Docker resources are cleaned up even on failure</li>
</ol>
<h2 id="customization-points"><a class="header" href="#customization-points">Customization Points</a></h2>
<p><strong>Topology Matrix:</strong></p>
<p>Add more topologies for comprehensive testing:</p>
<pre><code class="language-yaml">matrix:
topology:
- "3v1e"
- "5v1e"
- "10v2e" # Larger scale
</code></pre>
<p><strong>Timeout Adjustments:</strong></p>
<p>Increase <code>timeout-minutes</code> for longer-running scenarios or slower environments:</p>
<pre><code class="language-yaml">timeout-minutes: 30 # Instead of 15
</code></pre>
<p><strong>Artifact Retention:</strong></p>
<p>Change <code>retention-days</code> based on your storage needs:</p>
<pre><code class="language-yaml">retention-days: 14 # Keep logs for 2 weeks
</code></pre>
<p><strong>Conditional Execution:</strong></p>
<p>Run expensive tests only on merge to main:</p>
<pre><code class="language-yaml">if: github.event_name == 'push' &amp;&amp; github.ref == 'refs/heads/main'
</code></pre>
<h2 id="best-practices-3"><a class="header" href="#best-practices-3">Best Practices</a></h2>
<h3 id="required-set-pol_proof_dev_mode"><a class="header" href="#required-set-pol_proof_dev_mode">Required: Set POL_PROOF_DEV_MODE</a></h3>
<p><strong>Always set <code>POL_PROOF_DEV_MODE=true</code> globally</strong> in your workflow env:</p>
<pre><code class="language-yaml">env:
POL_PROOF_DEV_MODE: true # REQUIRED!
</code></pre>
<p>Without this, tests will hang due to expensive proof generation.</p>
<h3 id="use-helper-scripts"><a class="header" href="#use-helper-scripts">Use Helper Scripts</a></h3>
<p>Prefer <code>scripts/run/run-examples.sh</code> which handles all setup automatically:</p>
<pre><code class="language-bash">scripts/run/run-examples.sh -t 120 -v 3 -e 1 host
</code></pre>
<p>This is more reliable than manual <code>cargo run</code> commands.</p>
<h3 id="cache-aggressively"><a class="header" href="#cache-aggressively">Cache Aggressively</a></h3>
<p>Cache Rust dependencies, nomos-node builds, and Docker layers to speed up CI:</p>
<pre><code class="language-yaml">- name: Cache Rust dependencies
uses: actions/cache@v3
with:
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
target/
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
</code></pre>
<h3 id="collect-logs-on-failure"><a class="header" href="#collect-logs-on-failure">Collect Logs on Failure</a></h3>
<p>Always upload logs when tests fail for easier debugging:</p>
<pre><code class="language-yaml">- name: Upload logs on failure
if: failure()
uses: actions/upload-artifact@v3
with:
name: test-logs
path: |
.tmp/
*.log
retention-days: 7
</code></pre>
<h3 id="split-workflows-for-faster-iteration"><a class="header" href="#split-workflows-for-faster-iteration">Split Workflows for Faster Iteration</a></h3>
<p>For large projects, split host/compose/k8s into separate workflow files:</p>
<ul>
<li><code>.github/workflows/test-host.yml</code> — Fast smoke tests</li>
<li><code>.github/workflows/test-compose.yml</code> — Reproducible integration tests</li>
<li><code>.github/workflows/test-k8s.yml</code> — Production-like validation (nightly)</li>
</ul>
<h3 id="run-k8s-tests-less-frequently"><a class="header" href="#run-k8s-tests-less-frequently">Run K8s Tests Less Frequently</a></h3>
<p>K8s tests are slower. Consider running them only on main branch or scheduled:</p>
<pre><code class="language-yaml">on:
push:
branches: [main]
schedule:
- cron: '0 2 * * *' # Daily at 2 AM
</code></pre>
<h2 id="platform-specific-notes"><a class="header" href="#platform-specific-notes">Platform-Specific Notes</a></h2>
<h3 id="ubuntu-runners"><a class="header" href="#ubuntu-runners">Ubuntu Runners</a></h3>
<ul>
<li>Docker pre-installed and running</li>
<li>Best for compose/k8s runners</li>
<li>Most common choice</li>
</ul>
<h3 id="macos-runners"><a class="header" href="#macos-runners">macOS Runners</a></h3>
<ul>
<li>Docker Desktop not installed by default</li>
<li>Slower and more expensive</li>
<li>Use only if testing macOS-specific issues</li>
</ul>
<h3 id="self-hosted-runners"><a class="header" href="#self-hosted-runners">Self-Hosted Runners</a></h3>
<ul>
<li>Cache Docker images locally for faster builds</li>
<li>Set resource limits (<code>SLOW_TEST_ENV=true</code> if needed)</li>
<li>Ensure cleanup scripts run (<code>docker system prune</code>)</li>
</ul>
<h2 id="debugging-ci-failures"><a class="header" href="#debugging-ci-failures">Debugging CI Failures</a></h2>
<h3 id="enable-debug-logging"><a class="header" href="#enable-debug-logging">Enable Debug Logging</a></h3>
<p>Add debug environment variables temporarily:</p>
<pre><code class="language-yaml">env:
RUST_LOG: debug
NOMOS_LOG_LEVEL: debug
</code></pre>
<h3 id="preserve-containers-compose"><a class="header" href="#preserve-containers-compose">Preserve Containers (Compose)</a></h3>
<p>Set <code>COMPOSE_RUNNER_PRESERVE=1</code> to keep containers running for inspection:</p>
<pre><code class="language-yaml">- name: Run compose test (preserve on failure)
env:
COMPOSE_RUNNER_PRESERVE: 1
run: scripts/run/run-examples.sh -t 120 -v 3 -e 1 compose
</code></pre>
<h3 id="access-artifacts"><a class="header" href="#access-artifacts">Access Artifacts</a></h3>
<p>Download uploaded artifacts from the GitHub Actions UI to inspect logs locally.</p>
<h2 id="next-steps-3"><a class="header" href="#next-steps-3">Next Steps</a></h2>
<ul>
<li><a href="running-examples.html">Running Examples</a> — Manual execution for local development</li>
<li><a href="environment-variables.html">Environment Variables</a> — Full variable reference</li>
<li><a href="troubleshooting.html">Troubleshooting</a> — Common CI-specific issues</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="environment-variables-reference"><a class="header" href="#environment-variables-reference">Environment Variables Reference</a></h1>
<p>Complete reference of environment variables used by the testing framework, organized by category.</p>
<h2 id="critical-variables"><a class="header" href="#critical-variables">Critical Variables</a></h2>
<p>These MUST be set for successful test runs:</p>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Required</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>POL_PROOF_DEV_MODE</code></td><td><strong>YES</strong></td><td>—</td><td><strong>REQUIRED for all runners</strong>. Set to <code>true</code> to use fast dev-mode proving instead of expensive Groth16. Without this, tests will hang/timeout.</td></tr>
</tbody></table>
</div>
<p><strong>Example:</strong></p>
<pre><code class="language-bash">export POL_PROOF_DEV_MODE=true
</code></pre>
<p>Or add to your shell profile (<code>~/.bashrc</code>, <code>~/.zshrc</code>):</p>
<pre><code class="language-bash"># Required for nomos-testing framework
export POL_PROOF_DEV_MODE=true
</code></pre>
<hr />
<h2 id="runner-selection--topology"><a class="header" href="#runner-selection--topology">Runner Selection &amp; Topology</a></h2>
<p>Control which runner to use and the test topology:</p>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>NOMOS_DEMO_VALIDATORS</code></td><td>1</td><td>Number of validators (all runners)</td></tr>
<tr><td><code>NOMOS_DEMO_EXECUTORS</code></td><td>1</td><td>Number of executors (all runners)</td></tr>
<tr><td><code>NOMOS_DEMO_RUN_SECS</code></td><td>60</td><td>Run duration in seconds (all runners)</td></tr>
<tr><td><code>LOCAL_DEMO_VALIDATORS</code></td><td>—</td><td>Legacy: Number of validators (host runner only)</td></tr>
<tr><td><code>LOCAL_DEMO_EXECUTORS</code></td><td>—</td><td>Legacy: Number of executors (host runner only)</td></tr>
<tr><td><code>LOCAL_DEMO_RUN_SECS</code></td><td>—</td><td>Legacy: Run duration (host runner only)</td></tr>
<tr><td><code>COMPOSE_NODE_PAIRS</code></td><td>—</td><td>Compose-specific topology format: "validators×executors" (e.g., <code>3x2</code>)</td></tr>
</tbody></table>
</div>
<p><strong>Example:</strong></p>
<pre><code class="language-bash"># Run with 5 validators, 2 executors, for 120 seconds
NOMOS_DEMO_VALIDATORS=5 \
NOMOS_DEMO_EXECUTORS=2 \
NOMOS_DEMO_RUN_SECS=120 \
scripts/run/run-examples.sh -t 120 -v 5 -e 2 host
</code></pre>
<hr />
<h2 id="node-binaries-host-runner"><a class="header" href="#node-binaries-host-runner">Node Binaries (Host Runner)</a></h2>
<p>Required for host runner when not using helper scripts:</p>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Required</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>NOMOS_NODE_BIN</code></td><td>Yes (host)</td><td>—</td><td>Path to <code>nomos-node</code> binary</td></tr>
<tr><td><code>NOMOS_EXECUTOR_BIN</code></td><td>Yes (host)</td><td>—</td><td>Path to <code>nomos-executor</code> binary</td></tr>
<tr><td><code>NOMOS_NODE_PATH</code></td><td>No</td><td>—</td><td>Path to nomos-node git checkout (dev workflow)</td></tr>
</tbody></table>
</div>
<p><strong>Example:</strong></p>
<pre><code class="language-bash">export NOMOS_NODE_BIN=/path/to/nomos-node/target/release/nomos-node
export NOMOS_EXECUTOR_BIN=/path/to/nomos-node/target/release/nomos-executor
</code></pre>
<hr />
<h2 id="docker-images-compose--k8s"><a class="header" href="#docker-images-compose--k8s">Docker Images (Compose / K8s)</a></h2>
<p>Required for compose and k8s runners:</p>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Required</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>NOMOS_TESTNET_IMAGE</code></td><td>Yes (compose/k8s)</td><td><code>logos-blockchain-testing:local</code></td><td>Docker image tag for node containers</td></tr>
<tr><td><code>NOMOS_TESTNET_IMAGE_PULL_POLICY</code></td><td>No</td><td><code>IfNotPresent</code> (local) / <code>Always</code> (ECR)</td><td>K8s <code>imagePullPolicy</code> used by the runner</td></tr>
<tr><td><code>NOMOS_BINARIES_TAR</code></td><td>No</td><td>—</td><td>Path to prebuilt bundle (<code>.tar.gz</code>) for image build</td></tr>
<tr><td><code>NOMOS_SKIP_IMAGE_BUILD</code></td><td>No</td><td>0</td><td>Skip image rebuild (compose/k8s); assumes image already exists</td></tr>
<tr><td><code>NOMOS_FORCE_IMAGE_BUILD</code></td><td>No</td><td>0</td><td>Force rebuilding the image even when the script would normally skip it (e.g. non-local k8s)</td></tr>
</tbody></table>
</div>
<p><strong>Example:</strong></p>
<pre><code class="language-bash"># Using prebuilt bundle
export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
export NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local
scripts/build/build_test_image.sh
# Using pre-existing image (skip build)
export NOMOS_SKIP_IMAGE_BUILD=1
scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
</code></pre>
<hr />
<h2 id="circuit-assets-kzg-parameters-1"><a class="header" href="#circuit-assets-kzg-parameters-1">Circuit Assets (KZG Parameters)</a></h2>
<p>Circuit asset configuration for DA workloads:</p>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>NOMOS_KZGRS_PARAMS_PATH</code></td><td><code>testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params</code></td><td>Path to KZG proving key file</td></tr>
<tr><td><code>NOMOS_KZG_DIR_REL</code></td><td><code>testing-framework/assets/stack/kzgrs_test_params</code></td><td>Directory containing KZG assets (relative to workspace root)</td></tr>
<tr><td><code>NOMOS_KZG_FILE</code></td><td><code>kzgrs_test_params</code></td><td>Filename of the proving key within <code>NOMOS_KZG_DIR_REL</code></td></tr>
<tr><td><code>NOMOS_KZG_CONTAINER_PATH</code></td><td><code>/kzgrs_test_params/kzgrs_test_params</code></td><td>File path where the node expects KZG params inside containers</td></tr>
<tr><td><code>NOMOS_KZG_MODE</code></td><td>Runner-specific</td><td>K8s only: <code>hostPath</code> (mount from host) or <code>inImage</code> (embed into image)</td></tr>
<tr><td><code>NOMOS_KZG_IN_IMAGE_PARAMS_PATH</code></td><td><code>/opt/nomos/kzg-params/kzgrs_test_params</code></td><td>K8s <code>inImage</code> mode: where the proving key is stored inside the image</td></tr>
<tr><td><code>VERSION</code></td><td>From <code>versions.env</code></td><td>Circuit release tag (used by helper scripts)</td></tr>
<tr><td><code>NOMOS_CIRCUITS</code></td><td>—</td><td>Directory containing fetched circuit bundles (set by <code>scripts/setup/setup-circuits-stack.sh</code>)</td></tr>
<tr><td><code>NOMOS_CIRCUITS_VERSION</code></td><td>—</td><td>Legacy alias for <code>VERSION</code> (supported by some build scripts)</td></tr>
<tr><td><code>NOMOS_CIRCUITS_PLATFORM</code></td><td>Auto-detected</td><td>Override circuits platform (e.g. <code>linux-x86_64</code>, <code>macos-aarch64</code>)</td></tr>
<tr><td><code>NOMOS_CIRCUITS_HOST_DIR_REL</code></td><td><code>.tmp/nomos-circuits-host</code></td><td>Output dir for host circuits bundle (relative to repo root)</td></tr>
<tr><td><code>NOMOS_CIRCUITS_LINUX_DIR_REL</code></td><td><code>.tmp/nomos-circuits-linux</code></td><td>Output dir for linux circuits bundle (relative to repo root)</td></tr>
<tr><td><code>NOMOS_CIRCUITS_NONINTERACTIVE</code></td><td>0</td><td>Set to <code>1</code> to overwrite outputs without prompting in setup scripts</td></tr>
<tr><td><code>NOMOS_CIRCUITS_REBUILD_RAPIDSNARK</code></td><td>0</td><td>Set to <code>1</code> to force rebuilding rapidsnark (host bundle only)</td></tr>
</tbody></table>
</div>
<p><strong>Example:</strong></p>
<pre><code class="language-bash"># Use custom circuit assets
NOMOS_KZGRS_PARAMS_PATH=/custom/path/to/kzgrs_test_params \
cargo run -p runner-examples --bin local_runner
</code></pre>
<hr />
<h2 id="node-logging"><a class="header" href="#node-logging">Node Logging</a></h2>
<p>Control node log output (not framework runner logs):</p>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>NOMOS_LOG_LEVEL</code></td><td><code>info</code></td><td>Global log level: <code>error</code>, <code>warn</code>, <code>info</code>, <code>debug</code>, <code>trace</code></td></tr>
<tr><td><code>NOMOS_LOG_FILTER</code></td><td>—</td><td>Fine-grained module filtering (e.g., <code>cryptarchia=trace,nomos_da_sampling=debug</code>)</td></tr>
<tr><td><code>NOMOS_LOG_DIR</code></td><td>—</td><td>Host runner: directory for per-node log files (persistent). Compose/k8s: use <code>cfgsync.yaml</code> for file logging.</td></tr>
<tr><td><code>NOMOS_TESTS_KEEP_LOGS</code></td><td>0</td><td>Keep per-run temporary directories (useful for debugging/CI artifacts)</td></tr>
<tr><td><code>NOMOS_TESTS_TRACING</code></td><td>false</td><td>Enable debug tracing preset (combine with <code>NOMOS_LOG_DIR</code> unless external tracing backends configured)</td></tr>
</tbody></table>
</div>
<p><strong>Important:</strong> Node logging ignores <code>RUST_LOG</code>; use <code>NOMOS_LOG_LEVEL</code> and <code>NOMOS_LOG_FILTER</code> for node logs.</p>
<p><strong>Example:</strong></p>
<pre><code class="language-bash"># Debug logging to files
NOMOS_LOG_DIR=/tmp/test-logs \
NOMOS_LOG_LEVEL=debug \
NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug" \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin local_runner
# Inspect logs
ls /tmp/test-logs/
# nomos-node-0.2024-12-18T14-30-00.log
# nomos-node-1.2024-12-18T14-30-00.log
</code></pre>
<p><strong>Common filter targets:</strong></p>
<div class="table-wrapper"><table><thead><tr><th>Target Prefix</th><th>Subsystem</th></tr></thead><tbody>
<tr><td><code>cryptarchia</code></td><td>Consensus (Cryptarchia)</td></tr>
<tr><td><code>nomos_da_sampling</code></td><td>DA sampling service</td></tr>
<tr><td><code>nomos_da_dispersal</code></td><td>DA dispersal service</td></tr>
<tr><td><code>nomos_da_verifier</code></td><td>DA verification</td></tr>
<tr><td><code>nomos_blend</code></td><td>Mix network/privacy layer</td></tr>
<tr><td><code>chain_service</code></td><td>Chain service (node APIs/state)</td></tr>
<tr><td><code>chain_network</code></td><td>P2P networking</td></tr>
<tr><td><code>chain_leader</code></td><td>Leader election</td></tr>
</tbody></table>
</div>
<hr />
<h2 id="observability--metrics"><a class="header" href="#observability--metrics">Observability &amp; Metrics</a></h2>
<p>Optional observability integration:</p>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>NOMOS_METRICS_QUERY_URL</code></td><td>—</td><td>Prometheus-compatible base URL for runner to query (e.g., <code>http://localhost:9090</code>)</td></tr>
<tr><td><code>NOMOS_METRICS_OTLP_INGEST_URL</code></td><td>—</td><td>Full OTLP HTTP ingest URL for node metrics export (e.g., <code>http://localhost:9090/api/v1/otlp/v1/metrics</code>)</td></tr>
<tr><td><code>NOMOS_GRAFANA_URL</code></td><td>—</td><td>Grafana base URL for printing/logging (e.g., <code>http://localhost:3000</code>)</td></tr>
<tr><td><code>NOMOS_OTLP_ENDPOINT</code></td><td>—</td><td>OTLP trace endpoint (optional)</td></tr>
<tr><td><code>NOMOS_OTLP_METRICS_ENDPOINT</code></td><td>—</td><td>OTLP metrics endpoint (optional)</td></tr>
</tbody></table>
</div>
<p><strong>Example:</strong></p>
<pre><code class="language-bash"># Enable Prometheus querying
export NOMOS_METRICS_QUERY_URL=http://localhost:9090
export NOMOS_METRICS_OTLP_INGEST_URL=http://localhost:9090/api/v1/otlp/v1/metrics
export NOMOS_GRAFANA_URL=http://localhost:3000
scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
</code></pre>
<hr />
<h2 id="compose-runner-specific"><a class="header" href="#compose-runner-specific">Compose Runner Specific</a></h2>
<p>Variables specific to Docker Compose deployment:</p>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>COMPOSE_RUNNER_HOST</code></td><td><code>127.0.0.1</code></td><td>Host address for port mappings</td></tr>
<tr><td><code>COMPOSE_RUNNER_PRESERVE</code></td><td>0</td><td>Keep containers running after test (for debugging)</td></tr>
<tr><td><code>COMPOSE_RUNNER_HTTP_TIMEOUT_SECS</code></td><td>—</td><td>Override HTTP readiness timeout (seconds)</td></tr>
<tr><td><code>COMPOSE_RUNNER_HOST_GATEWAY</code></td><td><code>host.docker.internal:host-gateway</code></td><td>Controls <code>extra_hosts</code> entry injected into compose (set to <code>disable</code> to omit)</td></tr>
<tr><td><code>TESTNET_RUNNER_PRESERVE</code></td><td>—</td><td>Alias for <code>COMPOSE_RUNNER_PRESERVE</code></td></tr>
</tbody></table>
</div>
<p><strong>Example:</strong></p>
<pre><code class="language-bash"># Keep containers after test for debugging
COMPOSE_RUNNER_PRESERVE=1 \
scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
# Containers remain running
docker ps --filter "name=nomos-compose-"
docker logs &lt;container-id&gt;
</code></pre>
<hr />
<h2 id="k8s-runner-specific"><a class="header" href="#k8s-runner-specific">K8s Runner Specific</a></h2>
<p>Variables specific to Kubernetes deployment:</p>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>K8S_RUNNER_NAMESPACE</code></td><td>Random UUID</td><td>Kubernetes namespace (pin for debugging)</td></tr>
<tr><td><code>K8S_RUNNER_RELEASE</code></td><td>Random UUID</td><td>Helm release name (pin for debugging)</td></tr>
<tr><td><code>K8S_RUNNER_NODE_HOST</code></td><td>—</td><td>NodePort host resolution for non-local clusters</td></tr>
<tr><td><code>K8S_RUNNER_DEBUG</code></td><td>0</td><td>Log Helm stdout/stderr for install commands</td></tr>
<tr><td><code>K8S_RUNNER_PRESERVE</code></td><td>0</td><td>Keep namespace/release after run (for debugging)</td></tr>
<tr><td><code>K8S_RUNNER_DEPLOYMENT_TIMEOUT_SECS</code></td><td>—</td><td>Override deployment readiness timeout</td></tr>
<tr><td><code>K8S_RUNNER_HTTP_TIMEOUT_SECS</code></td><td>—</td><td>Override HTTP readiness timeout (port-forwards)</td></tr>
<tr><td><code>K8S_RUNNER_HTTP_PROBE_TIMEOUT_SECS</code></td><td>—</td><td>Override HTTP readiness timeout (NodePort probes)</td></tr>
<tr><td><code>K8S_RUNNER_PROMETHEUS_HTTP_TIMEOUT_SECS</code></td><td>—</td><td>Override Prometheus readiness timeout</td></tr>
<tr><td><code>K8S_RUNNER_PROMETHEUS_HTTP_PROBE_TIMEOUT_SECS</code></td><td>—</td><td>Override Prometheus NodePort probe timeout</td></tr>
</tbody></table>
</div>
<p><strong>Example:</strong></p>
<pre><code class="language-bash"># Pin namespace for debugging
K8S_RUNNER_NAMESPACE=nomos-test-debug \
K8S_RUNNER_PRESERVE=1 \
K8S_RUNNER_DEBUG=1 \
scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
# Inspect resources
kubectl get pods -n nomos-test-debug
kubectl logs -n nomos-test-debug -l nomos/logical-role=validator
</code></pre>
<hr />
<h2 id="platform--build-configuration"><a class="header" href="#platform--build-configuration">Platform &amp; Build Configuration</a></h2>
<p>Platform-specific build configuration:</p>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>NOMOS_BUNDLE_DOCKER_PLATFORM</code></td><td>Host arch</td><td>Docker platform for bundle builds: <code>linux/arm64</code> or <code>linux/amd64</code> (macOS/Windows hosts)</td></tr>
<tr><td><code>NOMOS_BIN_PLATFORM</code></td><td>—</td><td>Legacy alias for <code>NOMOS_BUNDLE_DOCKER_PLATFORM</code></td></tr>
<tr><td><code>COMPOSE_CIRCUITS_PLATFORM</code></td><td>Host arch</td><td>Circuits platform for image builds: <code>linux-aarch64</code> or <code>linux-x86_64</code></td></tr>
<tr><td><code>NOMOS_EXTRA_FEATURES</code></td><td>—</td><td>Extra cargo features to enable when building bundles (used by <code>scripts/build/build-bundle.sh</code>)</td></tr>
</tbody></table>
</div>
<p><strong>macOS / Apple Silicon:</strong></p>
<pre><code class="language-bash"># Native performance (recommended for local testing)
export NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64
# Or target amd64 (slower via emulation)
export NOMOS_BUNDLE_DOCKER_PLATFORM=linux/amd64
</code></pre>
<hr />
<h2 id="timeouts--performance"><a class="header" href="#timeouts--performance">Timeouts &amp; Performance</a></h2>
<p>Timeout and performance tuning:</p>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>SLOW_TEST_ENV</code></td><td>false</td><td>Doubles built-in readiness timeouts (useful in CI / constrained laptops)</td></tr>
<tr><td><code>TESTNET_PRINT_ENDPOINTS</code></td><td>0</td><td>Print <code>TESTNET_ENDPOINTS</code> / <code>TESTNET_PPROF</code> lines during deploy (set automatically by <code>scripts/run/run-examples.sh</code>)</td></tr>
<tr><td><code>NOMOS_DISPERSAL_TIMEOUT_SECS</code></td><td>20</td><td>DA dispersal timeout (seconds)</td></tr>
<tr><td><code>NOMOS_RETRY_COOLDOWN_SECS</code></td><td>3</td><td>Cooldown between retries (seconds)</td></tr>
<tr><td><code>NOMOS_GRACE_PERIOD_SECS</code></td><td>1200</td><td>Grace period before enforcing strict time-based expectations (seconds)</td></tr>
<tr><td><code>NOMOS_PRUNE_DURATION_SECS</code></td><td>30</td><td>Prune step duration (seconds)</td></tr>
<tr><td><code>NOMOS_PRUNE_INTERVAL_SECS</code></td><td>5</td><td>Interval between prune cycles (seconds)</td></tr>
<tr><td><code>NOMOS_SHARE_DURATION_SECS</code></td><td>5</td><td>Share duration (seconds)</td></tr>
<tr><td><code>NOMOS_COMMITMENTS_WAIT_SECS</code></td><td>1</td><td>Commitments wait duration (seconds)</td></tr>
<tr><td><code>NOMOS_SDP_TRIGGER_DELAY_SECS</code></td><td>5</td><td>SDP trigger delay (seconds)</td></tr>
</tbody></table>
</div>
<p><strong>Example:</strong></p>
<pre><code class="language-bash"># Increase timeouts for slow environments
SLOW_TEST_ENV=true \
scripts/run/run-examples.sh -t 120 -v 5 -e 2 compose
</code></pre>
<hr />
<h2 id="node-configuration-advanced"><a class="header" href="#node-configuration-advanced">Node Configuration (Advanced)</a></h2>
<p>Node-level configuration passed through to nomos-node/nomos-executor:</p>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>CONSENSUS_SLOT_TIME</code></td><td>—</td><td>Consensus slot time (seconds)</td></tr>
<tr><td><code>CONSENSUS_ACTIVE_SLOT_COEFF</code></td><td>—</td><td>Active slot coefficient (0.0-1.0)</td></tr>
<tr><td><code>NOMOS_USE_AUTONAT</code></td><td>Unset</td><td>If set, use AutoNAT instead of a static loopback address for libp2p NAT settings</td></tr>
<tr><td><code>NOMOS_CFGSYNC_PORT</code></td><td>4400</td><td>Port used for cfgsync service inside the stack</td></tr>
<tr><td><code>NOMOS_TIME_BACKEND</code></td><td><code>monotonic</code></td><td>Select time backend (used by compose/k8s stack scripts and deployers)</td></tr>
</tbody></table>
</div>
<p><strong>Example:</strong></p>
<pre><code class="language-bash"># Faster block production
CONSENSUS_SLOT_TIME=5 \
CONSENSUS_ACTIVE_SLOT_COEFF=0.9 \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin local_runner
</code></pre>
<hr />
<h2 id="framework-runner-logging-not-node-logs"><a class="header" href="#framework-runner-logging-not-node-logs">Framework Runner Logging (Not Node Logs)</a></h2>
<p>Control framework runner process logs (uses <code>RUST_LOG</code>, not <code>NOMOS_*</code>):</p>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>RUST_LOG</code></td><td>—</td><td>Framework runner log level (e.g., <code>debug</code>, <code>info</code>)</td></tr>
<tr><td><code>RUST_BACKTRACE</code></td><td>—</td><td>Enable Rust backtraces on panic (<code>1</code> or <code>full</code>)</td></tr>
<tr><td><code>CARGO_TERM_COLOR</code></td><td>—</td><td>Cargo output color (<code>always</code>, <code>never</code>, <code>auto</code>)</td></tr>
</tbody></table>
</div>
<p><strong>Example:</strong></p>
<pre><code class="language-bash"># Debug framework runner (not nodes)
RUST_LOG=debug \
RUST_BACKTRACE=1 \
cargo run -p runner-examples --bin local_runner
</code></pre>
<hr />
<h2 id="helper-script-variables"><a class="header" href="#helper-script-variables">Helper Script Variables</a></h2>
<p>Variables used by helper scripts (<code>scripts/run/run-examples.sh</code>, etc.):</p>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>NOMOS_NODE_REV</code></td><td>From <code>versions.env</code></td><td>nomos-node git revision to build/fetch</td></tr>
<tr><td><code>NOMOS_BUNDLE_VERSION</code></td><td>From <code>versions.env</code></td><td>Bundle schema version</td></tr>
<tr><td><code>NOMOS_IMAGE_SELECTION</code></td><td>—</td><td>Internal: image selection mode set by <code>run-examples.sh</code> (<code>local</code>/<code>ecr</code>/<code>auto</code>)</td></tr>
<tr><td><code>NOMOS_NODE_APPLY_PATCHES</code></td><td>1</td><td>Set to <code>0</code> to disable applying local patches when building bundles</td></tr>
<tr><td><code>NOMOS_NODE_PATCH_DIR</code></td><td><code>patches/nomos-node</code></td><td>Patch directory applied to nomos-node checkout during bundle builds</td></tr>
<tr><td><code>NOMOS_NODE_PATCH_LEVEL</code></td><td>—</td><td>Patch application level (<code>all</code> or an integer) for bundle builds</td></tr>
</tbody></table>
</div>
<hr />
<h2 id="quick-reference-examples"><a class="header" href="#quick-reference-examples">Quick Reference Examples</a></h2>
<h3 id="minimal-host-run"><a class="header" href="#minimal-host-run">Minimal Host Run</a></h3>
<pre><code class="language-bash">POL_PROOF_DEV_MODE=true \
scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
</code></pre>
<h3 id="debug-logging-host"><a class="header" href="#debug-logging-host">Debug Logging (Host)</a></h3>
<pre><code class="language-bash">POL_PROOF_DEV_MODE=true \
NOMOS_LOG_DIR=/tmp/logs \
NOMOS_LOG_LEVEL=debug \
NOMOS_LOG_FILTER="cryptarchia=trace" \
scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
</code></pre>
<h3 id="compose-with-observability"><a class="header" href="#compose-with-observability">Compose with Observability</a></h3>
<pre><code class="language-bash">POL_PROOF_DEV_MODE=true \
NOMOS_METRICS_QUERY_URL=http://localhost:9090 \
NOMOS_GRAFANA_URL=http://localhost:3000 \
scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
</code></pre>
<h3 id="k8s-with-debug"><a class="header" href="#k8s-with-debug">K8s with Debug</a></h3>
<pre><code class="language-bash">POL_PROOF_DEV_MODE=true \
K8S_RUNNER_NAMESPACE=nomos-debug \
K8S_RUNNER_DEBUG=1 \
K8S_RUNNER_PRESERVE=1 \
scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
</code></pre>
<h3 id="ci-environment"><a class="header" href="#ci-environment">CI Environment</a></h3>
<pre><code class="language-yaml">env:
POL_PROOF_DEV_MODE: true
RUST_BACKTRACE: 1
NOMOS_TESTS_KEEP_LOGS: 1
</code></pre>
<hr />
<h2 id="see-also-4"><a class="header" href="#see-also-4">See Also</a></h2>
<ul>
<li><a href="prerequisites.html">Prerequisites &amp; Setup</a> — Required files and setup</li>
<li><a href="running-examples.html">Running Examples</a> — How to run scenarios</li>
<li><a href="logging-observability.html">Logging &amp; Observability</a> — Log collection details</li>
<li><a href="ci-integration.html">CI Integration</a> — CI-specific variables</li>
<li><a href="troubleshooting.html">Troubleshooting</a> — Common issues with variables</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="logging--observability"><a class="header" href="#logging--observability">Logging &amp; Observability</a></h1>
<p>Comprehensive guide to log collection, metrics, and debugging across all runners.</p>
<h2 id="node-logging-vs-framework-logging"><a class="header" href="#node-logging-vs-framework-logging">Node Logging vs Framework Logging</a></h2>
<p><strong>Critical distinction:</strong> Node logs and framework logs use different configuration mechanisms.</p>
<div class="table-wrapper"><table><thead><tr><th>Component</th><th>Controlled By</th><th>Purpose</th></tr></thead><tbody>
<tr><td><strong>Framework binaries</strong> (<code>cargo run -p runner-examples --bin local_runner</code>)</td><td><code>RUST_LOG</code></td><td>Runner orchestration, deployment logs</td></tr>
<tr><td><strong>Node processes</strong> (validators, executors spawned by runner)</td><td><code>NOMOS_LOG_LEVEL</code>, <code>NOMOS_LOG_FILTER</code> (+ <code>NOMOS_LOG_DIR</code> on host runner)</td><td>Consensus, DA, mempool, network logs</td></tr>
</tbody></table>
</div>
<p><strong>Common mistake:</strong> Setting <code>RUST_LOG=debug</code> only increases verbosity of the runner binary itself. Node logs remain at their default level unless you also set <code>NOMOS_LOG_LEVEL=debug</code>.</p>
<p><strong>Example:</strong></p>
<pre><code class="language-bash"># This only makes the RUNNER verbose, not the nodes:
RUST_LOG=debug cargo run -p runner-examples --bin local_runner
# This makes the NODES verbose:
NOMOS_LOG_LEVEL=debug cargo run -p runner-examples --bin local_runner
# Both verbose (typically not needed):
RUST_LOG=debug NOMOS_LOG_LEVEL=debug cargo run -p runner-examples --bin local_runner
</code></pre>
<h2 id="logging-environment-variables"><a class="header" href="#logging-environment-variables">Logging Environment Variables</a></h2>
<p>See <a href="environment-variables.html">Environment Variables Reference</a> for complete details. Quick summary:</p>
<div class="table-wrapper"><table><thead><tr><th>Variable</th><th>Default</th><th>Effect</th></tr></thead><tbody>
<tr><td><code>NOMOS_LOG_DIR</code></td><td>None (console only)</td><td>Host runner: directory for per-node log files. Compose/k8s: use <code>cfgsync.yaml</code></td></tr>
<tr><td><code>NOMOS_LOG_LEVEL</code></td><td><code>info</code></td><td>Global log level: <code>error</code>, <code>warn</code>, <code>info</code>, <code>debug</code>, <code>trace</code></td></tr>
<tr><td><code>NOMOS_LOG_FILTER</code></td><td>None</td><td>Fine-grained target filtering (e.g., <code>cryptarchia=trace,nomos_da_sampling=debug</code>)</td></tr>
<tr><td><code>NOMOS_TESTS_TRACING</code></td><td>false</td><td>Enable debug tracing preset</td></tr>
<tr><td><code>NOMOS_OTLP_ENDPOINT</code></td><td>None</td><td>OTLP trace endpoint (optional)</td></tr>
<tr><td><code>NOMOS_OTLP_METRICS_ENDPOINT</code></td><td>None</td><td>OTLP metrics endpoint (optional)</td></tr>
</tbody></table>
</div>
<p><strong>Example:</strong> Full debug logging to files:</p>
<pre><code class="language-bash">NOMOS_TESTS_TRACING=true \
NOMOS_LOG_DIR=/tmp/test-logs \
NOMOS_LOG_LEVEL=debug \
NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug,nomos_da_dispersal=debug,nomos_da_verifier=debug" \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin local_runner
</code></pre>
<h2 id="per-node-log-files"><a class="header" href="#per-node-log-files">Per-Node Log Files</a></h2>
<p>When <code>NOMOS_LOG_DIR</code> is set, each node writes logs to separate files:</p>
<p><strong>File naming pattern:</strong></p>
<ul>
<li><strong>Validators</strong>: Prefix <code>nomos-node-0</code>, <code>nomos-node-1</code>, etc. (may include timestamp suffix)</li>
<li><strong>Executors</strong>: Prefix <code>nomos-executor-0</code>, <code>nomos-executor-1</code>, etc. (may include timestamp suffix)</li>
</ul>
<p><strong>Example filenames:</strong></p>
<ul>
<li><code>nomos-node-0.2024-12-18T14-30-00.log</code></li>
<li><code>nomos-node-1.2024-12-18T14-30-00.log</code></li>
<li><code>nomos-executor-0.2024-12-18T14-30-00.log</code></li>
</ul>
<p><strong>Local runner note:</strong> The local runner uses per-run temporary directories under the current working directory and removes them after the run unless <code>NOMOS_TESTS_KEEP_LOGS=1</code>. Use <code>NOMOS_LOG_DIR=/path/to/logs</code> to write per-node log files to a stable location.</p>
<h2 id="filter-target-names"><a class="header" href="#filter-target-names">Filter Target Names</a></h2>
<p>Common target prefixes for <code>NOMOS_LOG_FILTER</code>:</p>
<div class="table-wrapper"><table><thead><tr><th>Target Prefix</th><th>Subsystem</th></tr></thead><tbody>
<tr><td><code>cryptarchia</code></td><td>Consensus (Cryptarchia)</td></tr>
<tr><td><code>nomos_da_sampling</code></td><td>DA sampling service</td></tr>
<tr><td><code>nomos_da_dispersal</code></td><td>DA dispersal service</td></tr>
<tr><td><code>nomos_da_verifier</code></td><td>DA verification</td></tr>
<tr><td><code>nomos_blend</code></td><td>Mix network/privacy layer</td></tr>
<tr><td><code>chain_service</code></td><td>Chain service (node APIs/state)</td></tr>
<tr><td><code>chain_network</code></td><td>P2P networking</td></tr>
<tr><td><code>chain_leader</code></td><td>Leader election</td></tr>
</tbody></table>
</div>
<p><strong>Example filter:</strong></p>
<pre><code class="language-bash">NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug,chain_service=info,chain_network=info"
</code></pre>
<hr />
<h2 id="accessing-logs-by-runner"><a class="header" href="#accessing-logs-by-runner">Accessing Logs by Runner</a></h2>
<h3 id="local-runner-host-processes"><a class="header" href="#local-runner-host-processes">Local Runner (Host Processes)</a></h3>
<p><strong>Default (temporary directories, auto-cleanup):</strong></p>
<pre><code class="language-bash">POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
# Logs written to temporary directories in working directory
# Automatically cleaned up after test completes
</code></pre>
<p><strong>Persistent file output:</strong></p>
<pre><code class="language-bash">NOMOS_LOG_DIR=/tmp/local-logs \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin local_runner
# After test completes:
ls /tmp/local-logs/
# Files with prefix: nomos-node-0*, nomos-node-1*, nomos-executor-0*
# May include timestamps in filename
</code></pre>
<p><strong>Tip:</strong> Use <code>NOMOS_LOG_DIR</code> for persistent per-node log files, and <code>NOMOS_TESTS_KEEP_LOGS=1</code> if you want to keep the per-run temporary directories (configs/state) for post-mortem inspection.</p>
<h3 id="compose-runner-docker-containers"><a class="header" href="#compose-runner-docker-containers">Compose Runner (Docker Containers)</a></h3>
<p><strong>Via Docker logs (default, recommended):</strong></p>
<pre><code class="language-bash"># List containers (note the UUID prefix in names)
docker ps --filter "name=nomos-compose-"
# Stream logs from specific container
docker logs -f &lt;container-id-or-name&gt;
# Or use name pattern matching:
docker logs -f $(docker ps --filter "name=nomos-compose-.*-validator-0" -q | head -1)
# Show last 100 lines
docker logs --tail 100 &lt;container-id&gt;
</code></pre>
<p><strong>Via file collection (advanced):</strong></p>
<p>To write per-node log files inside containers, set <code>tracing_settings.logger: !File</code> in <code>testing-framework/assets/stack/cfgsync.yaml</code> (and ensure the directory is writable). To access them, you must either:</p>
<ol>
<li><strong>Copy files out after the run:</strong></li>
</ol>
<pre><code class="language-bash"># Ensure cfgsync.yaml is configured to log to /logs
NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin compose_runner
# After test, copy files from containers:
docker ps --filter "name=nomos-compose-"
docker cp &lt;container-id&gt;:/logs/node* /tmp/
</code></pre>
<ol start="2">
<li><strong>Mount a host volume</strong> (requires modifying compose template):</li>
</ol>
<pre><code class="language-yaml">volumes:
- /tmp/host-logs:/logs # Add to docker-compose.yml.tera
</code></pre>
<p><strong>Recommendation:</strong> Use <code>docker logs</code> by default. File collection inside containers is complex and rarely needed.</p>
<p><strong>Keep containers for debugging:</strong></p>
<pre><code class="language-bash">COMPOSE_RUNNER_PRESERVE=1 \
NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
cargo run -p runner-examples --bin compose_runner
# Containers remain running after test—inspect with docker logs or docker exec
</code></pre>
<p><strong>Compose debugging variables:</strong></p>
<ul>
<li><code>COMPOSE_RUNNER_HOST=127.0.0.1</code> — host used for readiness probes</li>
<li><code>COMPOSE_RUNNER_HOST_GATEWAY=host.docker.internal:host-gateway</code> — controls <code>extra_hosts</code> entry (set to <code>disable</code> to omit)</li>
<li><code>TESTNET_RUNNER_PRESERVE=1</code> — alias for <code>COMPOSE_RUNNER_PRESERVE=1</code></li>
<li><code>COMPOSE_RUNNER_HTTP_TIMEOUT_SECS=&lt;secs&gt;</code> — override HTTP readiness timeout</li>
</ul>
<p><strong>Note:</strong> Container names follow pattern <code>nomos-compose-{uuid}-validator-{index}-1</code> where <code>{uuid}</code> changes per run.</p>
<h3 id="k8s-runner-kubernetes-pods"><a class="header" href="#k8s-runner-kubernetes-pods">K8s Runner (Kubernetes Pods)</a></h3>
<p><strong>Via kubectl logs (use label selectors):</strong></p>
<pre><code class="language-bash"># List pods
kubectl get pods
# Stream logs using label selectors (recommended)
# Helm chart labels:
# - nomos/logical-role=validator|executor
# - nomos/validator-index / nomos/executor-index
kubectl logs -l nomos/logical-role=validator -f
kubectl logs -l nomos/logical-role=executor -f
# Stream logs from specific pod
kubectl logs -f nomos-validator-0
# Previous logs from crashed pods
kubectl logs --previous -l nomos/logical-role=validator
</code></pre>
<p><strong>Download logs for offline analysis:</strong></p>
<pre><code class="language-bash"># Using label selectors
kubectl logs -l nomos/logical-role=validator --tail=1000 &gt; all-validators.log
kubectl logs -l nomos/logical-role=executor --tail=1000 &gt; all-executors.log
# Specific pods
kubectl logs nomos-validator-0 &gt; validator-0.log
kubectl logs nomos-executor-1 &gt; executor-1.log
</code></pre>
<p><strong>K8s debugging variables:</strong></p>
<ul>
<li><code>K8S_RUNNER_DEBUG=1</code> — logs Helm stdout/stderr for install commands</li>
<li><code>K8S_RUNNER_PRESERVE=1</code> — keep namespace/release after run</li>
<li><code>K8S_RUNNER_NODE_HOST=&lt;ip|hostname&gt;</code> — override NodePort host resolution</li>
<li><code>K8S_RUNNER_NAMESPACE=&lt;name&gt;</code> / <code>K8S_RUNNER_RELEASE=&lt;name&gt;</code> — pin namespace/release (useful for debugging)</li>
</ul>
<p><strong>Specify namespace (if not using default):</strong></p>
<pre><code class="language-bash">kubectl logs -n my-namespace -l nomos/logical-role=validator -f
</code></pre>
<p><strong>Note:</strong> K8s runner is optimized for local clusters (Docker Desktop K8s, minikube, kind). Remote clusters require additional setup.</p>
<hr />
<h2 id="otlp-and-telemetry"><a class="header" href="#otlp-and-telemetry">OTLP and Telemetry</a></h2>
<p><strong>OTLP exporters are optional.</strong> If you see errors about unreachable OTLP endpoints, it's safe to ignore them unless you're actively collecting traces/metrics.</p>
<p><strong>To enable OTLP:</strong></p>
<pre><code class="language-bash">NOMOS_OTLP_ENDPOINT=http://localhost:4317 \
NOMOS_OTLP_METRICS_ENDPOINT=http://localhost:4318 \
cargo run -p runner-examples --bin local_runner
</code></pre>
<p><strong>To silence OTLP errors:</strong> Simply leave these variables unset (the default).</p>
<hr />
<h2 id="observability-prometheus-and-node-apis"><a class="header" href="#observability-prometheus-and-node-apis">Observability: Prometheus and Node APIs</a></h2>
<p>Runners expose metrics and node HTTP endpoints for expectation code and debugging.</p>
<h3 id="prometheus-compatible-metrics-querying-optional"><a class="header" href="#prometheus-compatible-metrics-querying-optional">Prometheus-Compatible Metrics Querying (Optional)</a></h3>
<ul>
<li>Runners do <strong>not</strong> provision Prometheus automatically</li>
<li>For a ready-to-run stack, use <code>scripts/setup/setup-observability.sh</code>:
<ul>
<li>Compose: <code>scripts/setup/setup-observability.sh compose up</code> then <code>scripts/setup/setup-observability.sh compose env</code></li>
<li>K8s: <code>scripts/setup/setup-observability.sh k8s install</code> then <code>scripts/setup/setup-observability.sh k8s env</code></li>
</ul>
</li>
<li>Provide <code>NOMOS_METRICS_QUERY_URL</code> (PromQL base URL) to enable <code>ctx.telemetry()</code> queries</li>
<li>Access from expectations when configured: <code>ctx.telemetry().prometheus().map(|p| p.base_url())</code></li>
</ul>
<p><strong>Example:</strong></p>
<pre><code class="language-bash"># Start observability stack (Compose)
scripts/setup/setup-observability.sh compose up
# Get environment variables
eval $(scripts/setup/setup-observability.sh compose env)
# Run scenario with metrics
POL_PROOF_DEV_MODE=true \
scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
</code></pre>
<h3 id="grafana-optional"><a class="header" href="#grafana-optional">Grafana (Optional)</a></h3>
<ul>
<li>Runners do <strong>not</strong> provision Grafana automatically (but <code>scripts/setup/setup-observability.sh</code> can)</li>
<li>If you set <code>NOMOS_GRAFANA_URL</code>, the deployer prints it in <code>TESTNET_ENDPOINTS</code></li>
<li>Dashboards live in <code>testing-framework/assets/stack/monitoring/grafana/dashboards/</code> (the bundled stack auto-provisions them)</li>
</ul>
<p><strong>Example:</strong></p>
<pre><code class="language-bash"># Bring up the bundled Prometheus+Grafana stack (optional)
scripts/setup/setup-observability.sh compose up
eval $(scripts/setup/setup-observability.sh compose env)
export NOMOS_GRAFANA_URL=http://localhost:3000
POL_PROOF_DEV_MODE=true scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
</code></pre>
<p><strong>Default bundled Grafana login:</strong> <code>admin</code> / <code>admin</code> (see <code>scripts/observability/compose/docker-compose.yml</code>).</p>
<h3 id="node-apis"><a class="header" href="#node-apis">Node APIs</a></h3>
<ul>
<li>Access from expectations: <code>ctx.node_clients().validator_clients().get(0)</code></li>
<li>Endpoints: consensus info, network info, DA membership, etc.</li>
<li>See <code>testing-framework/core/src/nodes/api_client.rs</code> for available methods</li>
</ul>
<p><strong>Example usage in expectations:</strong></p>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::{DynError, RunContext};
async fn evaluate(ctx: &amp;RunContext) -&gt; Result&lt;(), DynError&gt; {
let client = &amp;ctx.node_clients().validator_clients()[0];
let info = client.consensus_info().await?;
tracing::info!(height = info.height, "consensus info from validator 0");
Ok(())
}</code></pre>
<hr />
<h2 id="observability-flow"><a class="header" href="#observability-flow">Observability Flow</a></h2>
<pre><code class="language-mermaid">flowchart TD
Expose[Runner exposes endpoints/ports] --&gt; Collect[Runtime collects block/health signals]
Collect --&gt; Consume[Expectations consume signals&lt;br/&gt;decide pass/fail]
Consume --&gt; Inspect[Operators inspect logs/metrics&lt;br/&gt;when failures arise]
</code></pre>
<hr />
<h2 id="quick-reference"><a class="header" href="#quick-reference">Quick Reference</a></h2>
<h3 id="debug-logging-host-1"><a class="header" href="#debug-logging-host-1">Debug Logging (Host)</a></h3>
<pre><code class="language-bash">NOMOS_LOG_DIR=/tmp/logs \
NOMOS_LOG_LEVEL=debug \
NOMOS_LOG_FILTER="cryptarchia=trace" \
POL_PROOF_DEV_MODE=true \
scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
</code></pre>
<h3 id="compose-with-observability-1"><a class="header" href="#compose-with-observability-1">Compose with Observability</a></h3>
<pre><code class="language-bash"># Start observability stack
scripts/setup/setup-observability.sh compose up
eval $(scripts/setup/setup-observability.sh compose env)
# Run with metrics
POL_PROOF_DEV_MODE=true \
scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
# Access Grafana at http://localhost:3000
</code></pre>
<h3 id="k8s-with-debug-1"><a class="header" href="#k8s-with-debug-1">K8s with Debug</a></h3>
<pre><code class="language-bash">K8S_RUNNER_NAMESPACE=nomos-debug \
K8S_RUNNER_DEBUG=1 \
K8S_RUNNER_PRESERVE=1 \
POL_PROOF_DEV_MODE=true \
scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
# Inspect logs
kubectl logs -n nomos-debug -l nomos/logical-role=validator
</code></pre>
<hr />
<h2 id="see-also-5"><a class="header" href="#see-also-5">See Also</a></h2>
<ul>
<li><a href="environment-variables.html">Environment Variables</a> — Complete variable reference</li>
<li><a href="troubleshooting.html">Troubleshooting</a> — Log-related debugging (see "Where to Find Logs")</li>
<li><a href="running-examples.html">Running Examples</a> — Runner-specific logging details</li>
<li><a href="prerequisites.html">Prerequisites &amp; Setup</a> — Setup before running</li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="part-v--appendix"><a class="header" href="#part-v--appendix">Part V — Appendix</a></h1>
<p>Quick reference materials, troubleshooting guides, and supplementary information.</p>
<h2 id="contents"><a class="header" href="#contents">Contents</a></h2>
<ul>
<li><strong>Builder API Quick Reference</strong>: Cheat sheet for DSL methods</li>
<li><strong>Troubleshooting Scenarios</strong>: Common issues and their solutions, including "What Failure Looks Like" with realistic examples</li>
<li><strong>FAQ</strong>: Frequently asked questions</li>
<li><strong>Glossary</strong>: Terminology reference</li>
</ul>
<h2 id="when-to-use-this-section"><a class="header" href="#when-to-use-this-section">When to Use This Section</a></h2>
<ul>
<li><strong>Quick lookups</strong>: Find DSL method signatures without reading full guides</li>
<li><strong>Debugging failures</strong>: Match symptoms to known issues and fixes</li>
<li><strong>Clarifying concepts</strong>: Look up unfamiliar terms in the glossary</li>
<li><strong>Common questions</strong>: Check FAQ before asking for help</li>
</ul>
<p>This section complements the main documentation with practical reference materials that you'll return to frequently during development and operations.</p>
<hr />
<p>Jump to:</p>
<ul>
<li><a href="dsl-cheat-sheet.html">Builder API Quick Reference</a></li>
<li><a href="troubleshooting.html">Troubleshooting Scenarios</a></li>
<li><a href="faq.html">FAQ</a></li>
<li><a href="glossary.html">Glossary</a></li>
</ul>
<div style="break-before: page; page-break-before: always;"></div><h1 id="builder-api-quick-reference"><a class="header" href="#builder-api-quick-reference">Builder API Quick Reference</a></h1>
<p>Quick reference for the scenario builder DSL. All methods are chainable.</p>
<h2 id="imports"><a class="header" href="#imports">Imports</a></h2>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
use testing_framework_runner_compose::ComposeDeployer;
use testing_framework_runner_k8s::K8sDeployer;
use testing_framework_runner_local::LocalDeployer;
use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt};</code></pre>
<h2 id="topology-1"><a class="header" href="#topology-1">Topology</a></h2>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::{Builder, ScenarioBuilder};
pub fn topology() -&gt; Builder&lt;()&gt; {
ScenarioBuilder::topology_with(|t| {
t.network_star() // Star topology (all connect to seed node)
.validators(3) // Number of validator nodes
.executors(2) // Number of executor nodes
})
}</code></pre>
<h2 id="wallets"><a class="header" href="#wallets">Wallets</a></h2>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn wallets_plan() -&gt; testing_framework_core::scenario::Scenario&lt;()&gt; {
ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0))
.wallets(50) // Seed 50 funded wallet accounts
.build()
}</code></pre>
<h2 id="transaction-workload-2"><a class="header" href="#transaction-workload-2">Transaction Workload</a></h2>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn transactions_plan() -&gt; testing_framework_core::scenario::Scenario&lt;()&gt; {
ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0))
.wallets(50)
.transactions_with(|txs| {
txs.rate(5) // 5 transactions per block
.users(20) // Use 20 of the seeded wallets
}) // Finish transaction workload config
.build()
}</code></pre>
<h2 id="da-workload-1"><a class="header" href="#da-workload-1">DA Workload</a></h2>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn da_plan() -&gt; testing_framework_core::scenario::Scenario&lt;()&gt; {
ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(1))
.wallets(50)
.da_with(|da| {
da.channel_rate(1) // number of DA channels to run
.blob_rate(2) // target 2 blobs per block (headroom applied)
.headroom_percent(20) // optional headroom when sizing channels
}) // Finish DA workload config
.build()
}</code></pre>
<h2 id="chaos-workload-requires-enable_node_control"><a class="header" href="#chaos-workload-requires-enable_node_control">Chaos Workload (Requires <code>enable_node_control()</code>)</a></h2>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::{NodeControlCapability, ScenarioBuilder};
use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt};
pub fn chaos_plan() -&gt; testing_framework_core::scenario::Scenario&lt;NodeControlCapability&gt; {
ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
.enable_node_control() // Enable node control capability
.chaos_with(|c| {
c.restart() // Random restart chaos
.min_delay(Duration::from_secs(30)) // Min time between restarts
.max_delay(Duration::from_secs(60)) // Max time between restarts
.target_cooldown(Duration::from_secs(45)) // Cooldown after restart
.apply() // Required for chaos configuration
})
.build()
}</code></pre>
<h2 id="expectations-1"><a class="header" href="#expectations-1">Expectations</a></h2>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn expectations_plan() -&gt; testing_framework_core::scenario::Scenario&lt;()&gt; {
ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0))
.expect_consensus_liveness() // Assert blocks are produced continuously
.build()
}</code></pre>
<h2 id="run-duration"><a class="header" href="#run-duration">Run Duration</a></h2>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn run_duration_plan() -&gt; testing_framework_core::scenario::Scenario&lt;()&gt; {
ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0))
.with_run_duration(Duration::from_secs(120)) // Run for 120 seconds
.build()
}</code></pre>
<h2 id="build"><a class="header" href="#build">Build</a></h2>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn build_plan() -&gt; testing_framework_core::scenario::Scenario&lt;()&gt; {
ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0)).build() // Construct the final Scenario
}</code></pre>
<h2 id="deployers-1"><a class="header" href="#deployers-1">Deployers</a></h2>
<pre><code class="language-rust ignore">use testing_framework_runner_compose::ComposeDeployer;
use testing_framework_runner_k8s::K8sDeployer;
use testing_framework_runner_local::LocalDeployer;
pub fn deployers() {
// Local processes
let _deployer = LocalDeployer::default();
// Docker Compose
let _deployer = ComposeDeployer::default();
// Kubernetes
let _deployer = K8sDeployer::default();
}</code></pre>
<h2 id="execution"><a class="header" href="#execution">Execution</a></h2>
<pre><code class="language-rust ignore">use anyhow::Result;
use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
use testing_framework_runner_local::LocalDeployer;
use testing_framework_workflows::ScenarioBuilderExt;
pub async fn execution() -&gt; Result&lt;()&gt; {
let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0))
.expect_consensus_liveness()
.build();
let deployer = LocalDeployer::default();
let runner = deployer.deploy(&amp;plan).await?;
let _handle = runner.run(&amp;mut plan).await?;
Ok(())
}</code></pre>
<h2 id="complete-example"><a class="header" href="#complete-example">Complete Example</a></h2>
<pre><code class="language-rust ignore">use std::time::Duration;
use anyhow::Result;
use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
use testing_framework_runner_local::LocalDeployer;
use testing_framework_workflows::ScenarioBuilderExt;
pub async fn run_test() -&gt; Result&lt;()&gt; {
let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
.wallets(50)
.transactions_with(|txs| {
txs.rate(5) // 5 transactions per block
.users(20)
})
.da_with(|da| {
da.channel_rate(1) // number of DA channels
.blob_rate(2) // target 2 blobs per block
.headroom_percent(20) // optional channel headroom
})
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(90))
.build();
let deployer = LocalDeployer::default();
let runner = deployer.deploy(&amp;plan).await?;
let _handle = runner.run(&amp;mut plan).await?;
Ok(())
}</code></pre>
<div style="break-before: page; page-break-before: always;"></div><h1 id="troubleshooting-scenarios"><a class="header" href="#troubleshooting-scenarios">Troubleshooting Scenarios</a></h1>
<p><strong>Prerequisites for All Runners:</strong></p>
<ul>
<li><strong><code>versions.env</code> file</strong> at repository root (required by helper scripts)</li>
<li><strong><code>POL_PROOF_DEV_MODE=true</code></strong> MUST be set for all runners (host, compose, k8s) to avoid expensive Groth16 proof generation that causes timeouts</li>
<li><strong>KZG circuit assets</strong> must be present at <code>testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params</code> (note the repeated filename) for DA workloads</li>
</ul>
<p><strong>Platform/Environment Notes:</strong></p>
<ul>
<li><strong>macOS + Docker Desktop (Apple silicon):</strong> prefer <code>NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64</code> for local compose/k8s runs to avoid slow/fragile amd64 emulation builds.</li>
<li><strong>Disk space:</strong> bundle/image builds are storage-heavy. If you see I/O errors or Docker build failures, check free space and prune old artifacts (<code>.tmp/</code>, <code>target/</code>, and Docker build cache) before retrying.</li>
<li><strong>K8s runner scope:</strong> the default Helm chart mounts KZG params via <code>hostPath</code> and uses a local image tag (<code>logos-blockchain-testing:local</code>). This is intended for local clusters (Docker Desktop / minikube / kind), not remote managed clusters without additional setup.
<ul>
<li>Quick cleanup: <code>scripts/ops/clean.sh</code> (and <code>scripts/ops/clean.sh --docker</code> if needed).</li>
<li>Destructive cleanup (last resort): <code>scripts/ops/clean.sh --docker-system --dangerous</code> (add <code>--volumes</code> if you also want to prune Docker volumes).</li>
</ul>
</li>
</ul>
<p><strong>Recommended:</strong> Use <code>scripts/run/run-examples.sh</code> which handles all setup automatically.</p>
<h2 id="quick-symptom-guide"><a class="header" href="#quick-symptom-guide">Quick Symptom Guide</a></h2>
<p>Common symptoms and likely causes:</p>
<ul>
<li><strong>No or slow block progression</strong>: missing <code>POL_PROOF_DEV_MODE=true</code>, missing KZG circuit assets (<code>/kzgrs_test_params/kzgrs_test_params</code> file) for DA workloads, too-short run window, port conflicts, or resource exhaustion—set required env vars, verify assets exist, extend duration, check node logs for startup errors.</li>
<li><strong>Transactions not included</strong>: unfunded or misconfigured wallets (check <code>.wallets(N)</code> vs <code>.users(M)</code>), transaction rate exceeding block capacity, or rates exceeding block production speed—reduce rate, increase wallet count, verify wallet setup in logs.</li>
<li><strong>Chaos stalls the run</strong>: chaos (node control) only works with ComposeDeployer; host runner (LocalDeployer) and K8sDeployer don't support it (won't "stall", just can't execute chaos workloads). With compose, aggressive restart cadence can prevent consensus recovery—widen restart intervals.</li>
<li><strong>Observability gaps</strong>: metrics or logs unreachable because ports clash or services are not exposed—adjust observability ports and confirm runner wiring.</li>
<li><strong>Flaky behavior across runs</strong>: mixing chaos with functional smoke tests or inconsistent topology between environments—separate deterministic and chaos scenarios and standardize topology presets.</li>
</ul>
<h2 id="what-failure-looks-like-3"><a class="header" href="#what-failure-looks-like-3">What Failure Looks Like</a></h2>
<p>This section shows what you'll actually see when common issues occur. Each example includes realistic console output and the fix.</p>
<h3 id="1-missing-pol_proof_dev_modetrue-most-common"><a class="header" href="#1-missing-pol_proof_dev_modetrue-most-common">1. Missing <code>POL_PROOF_DEV_MODE=true</code> (Most Common!)</a></h3>
<p><strong>Symptoms:</strong></p>
<ul>
<li>Test "hangs" with no visible progress</li>
<li>CPU usage spikes to 100%</li>
<li>Eventually hits timeout after several minutes</li>
<li>Nodes appear to start but blocks aren't produced</li>
</ul>
<p><strong>What you'll see:</strong></p>
<pre><code class="language-text">$ cargo run -p runner-examples --bin local_runner
Finished dev [unoptimized + debuginfo] target(s) in 0.48s
Running `target/debug/local_runner`
[INFO runner_examples::local_runner] Starting local runner scenario
[INFO testing_framework_runner_local] Launching 3 validators
[INFO testing_framework_runner_local] Waiting for node readiness...
(hangs here for 5+ minutes, CPU at 100%)
thread 'main' panicked at 'readiness timeout expired'
</code></pre>
<p><strong>Root Cause:</strong> Groth16 proof generation is extremely slow without dev mode. The system tries to compute real cryptographic proofs, which can take minutes per block.</p>
<p><strong>Fix:</strong></p>
<pre><code class="language-bash">POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
</code></pre>
<p><strong>Prevention:</strong> Set this in your shell profile or <code>.env</code> file so you never forget it.</p>
<hr />
<h3 id="2-missing-versionsenv-file"><a class="header" href="#2-missing-versionsenv-file">2. Missing <code>versions.env</code> File</a></h3>
<p><strong>Symptoms:</strong></p>
<ul>
<li>Helper scripts fail immediately</li>
<li>Error about missing file at repo root</li>
<li>Scripts can't determine which circuit/node versions to use</li>
</ul>
<p><strong>What you'll see:</strong></p>
<pre><code class="language-text">$ scripts/run/run-examples.sh -t 60 -v 1 -e 1 host
ERROR: versions.env not found at repository root
This file is required and should define:
VERSION=&lt;circuit release tag&gt;
NOMOS_NODE_REV=&lt;nomos-node git revision&gt;
NOMOS_BUNDLE_VERSION=&lt;bundle schema version&gt;
</code></pre>
<p><strong>Root Cause:</strong> Helper scripts need <code>versions.env</code> to know which versions to build/fetch.</p>
<p><strong>Fix:</strong> Ensure you're in the repository root directory. The <code>versions.env</code> file should already exist—verify it's present:</p>
<pre><code class="language-bash">cat versions.env
# Should show:
# VERSION=v0.3.1
# NOMOS_NODE_REV=abc123def456
# NOMOS_BUNDLE_VERSION=v1
</code></pre>
<hr />
<h3 id="3-missing-kzg-circuit-assets-da-workloads"><a class="header" href="#3-missing-kzg-circuit-assets-da-workloads">3. Missing KZG Circuit Assets (DA Workloads)</a></h3>
<p><strong>Symptoms:</strong></p>
<ul>
<li>DA workload tests fail</li>
<li>Error messages about missing circuit files</li>
<li>Nodes crash during DA operations</li>
</ul>
<p><strong>What you'll see:</strong></p>
<pre><code class="language-text">$ POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
[INFO testing_framework_runner_local] Starting DA workload
[ERROR nomos_da_dispersal] Failed to load KZG parameters
Error: Custom { kind: NotFound, error: "Circuit file not found at: testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params" }
thread 'main' panicked at 'workload init failed'
</code></pre>
<p><strong>Root Cause:</strong> DA (Data Availability) workloads require KZG cryptographic parameters. The file must exist at: <code>testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params</code> (note the repeated filename).</p>
<p><strong>Fix (recommended):</strong></p>
<pre><code class="language-bash"># Use run-examples.sh which handles setup automatically
scripts/run/run-examples.sh -t 60 -v 1 -e 1 host
</code></pre>
<p><strong>Fix (manual):</strong></p>
<pre><code class="language-bash"># Fetch circuits
scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits
# Copy to expected location
mkdir -p testing-framework/assets/stack/kzgrs_test_params
cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/
# Verify (should be ~120MB)
ls -lh testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params
</code></pre>
<hr />
<h3 id="4-node-binaries-not-found"><a class="header" href="#4-node-binaries-not-found">4. Node Binaries Not Found</a></h3>
<p><strong>Symptoms:</strong></p>
<ul>
<li>Error about missing <code>nomos-node</code> or <code>nomos-executor</code> binary</li>
<li>"file not found" or "no such file or directory"</li>
<li>Environment variables <code>NOMOS_NODE_BIN</code> / <code>NOMOS_EXECUTOR_BIN</code> not set</li>
</ul>
<p><strong>What you'll see:</strong></p>
<pre><code class="language-text">$ POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
[INFO testing_framework_runner_local] Spawning validator 0
Error: Os { code: 2, kind: NotFound, message: "No such file or directory" }
thread 'main' panicked at 'failed to spawn nomos-node process'
</code></pre>
<p><strong>Root Cause:</strong> The local runner needs compiled <code>nomos-node</code> and <code>nomos-executor</code> binaries, but doesn't know where they are.</p>
<p><strong>Fix (recommended):</strong></p>
<pre><code class="language-bash"># Use run-examples.sh which builds binaries automatically
scripts/run/run-examples.sh -t 60 -v 1 -e 1 host
</code></pre>
<p><strong>Fix (manual - set paths explicitly):</strong></p>
<pre><code class="language-bash"># Build binaries first
cd ../nomos-node # or wherever your nomos-node checkout is
cargo build --release --bin nomos-node --bin nomos-executor
# Set environment variables
export NOMOS_NODE_BIN=$PWD/target/release/nomos-node
export NOMOS_EXECUTOR_BIN=$PWD/target/release/nomos-executor
# Return to testing framework
cd ../nomos-testing
POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
</code></pre>
<hr />
<h3 id="5-docker-daemon-not-running-compose"><a class="header" href="#5-docker-daemon-not-running-compose">5. Docker Daemon Not Running (Compose)</a></h3>
<p><strong>Symptoms:</strong></p>
<ul>
<li>Compose tests fail immediately</li>
<li>"Cannot connect to Docker daemon"</li>
<li>Docker commands don't work</li>
</ul>
<p><strong>What you'll see:</strong></p>
<pre><code class="language-text">$ scripts/run/run-examples.sh -t 60 -v 1 -e 1 compose
[INFO runner_examples::compose_runner] Starting compose deployment
Error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
thread 'main' panicked at 'compose deployment failed'
</code></pre>
<p><strong>Root Cause:</strong> Docker Desktop isn't running, or your user doesn't have permission to access Docker.</p>
<p><strong>Fix:</strong></p>
<pre><code class="language-bash"># macOS: Start Docker Desktop application
open -a Docker
# Linux: Start Docker daemon
sudo systemctl start docker
# Verify Docker is working
docker ps
# If permission denied, add your user to docker group (Linux)
sudo usermod -aG docker $USER
# Then log out and log back in
</code></pre>
<hr />
<h3 id="6-image-not-found-composek8s"><a class="header" href="#6-image-not-found-composek8s">6. Image Not Found (Compose/K8s)</a></h3>
<p><strong>Symptoms:</strong></p>
<ul>
<li>Compose/K8s tests fail during deployment</li>
<li>"Image not found: logos-blockchain-testing:local"</li>
<li>Containers fail to start</li>
</ul>
<p><strong>What you'll see:</strong></p>
<pre><code class="language-text">$ POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin compose_runner
[INFO testing_framework_runner_compose] Starting compose deployment
Error: Failed to pull image 'logos-blockchain-testing:local': No such image
thread 'main' panicked at 'compose deployment failed'
</code></pre>
<p><strong>Root Cause:</strong> The Docker image hasn't been built yet, or was pruned.</p>
<p><strong>Fix (recommended):</strong></p>
<pre><code class="language-bash"># Use run-examples.sh which builds the image automatically
scripts/run/run-examples.sh -t 60 -v 1 -e 1 compose
</code></pre>
<p><strong>Fix (manual):</strong></p>
<pre><code class="language-bash"># 1. Build Linux bundle
scripts/build/build-bundle.sh --platform linux
# 2. Set bundle path
export NOMOS_BINARIES_TAR=$(ls -t .tmp/nomos-binaries-linux-*.tar.gz | head -1)
# 3. Build Docker image
scripts/build/build_test_image.sh
# 4. Verify image exists
docker images | grep logos-blockchain-testing
# 5. For kind/minikube: load image into cluster
kind load docker-image logos-blockchain-testing:local
# OR: minikube image load logos-blockchain-testing:local
</code></pre>
<hr />
<h3 id="7-port-conflicts"><a class="header" href="#7-port-conflicts">7. Port Conflicts</a></h3>
<p><strong>Symptoms:</strong></p>
<ul>
<li>"Address already in use" errors</li>
<li>Tests fail during node startup</li>
<li>Observability stack (Prometheus/Grafana) won't start</li>
</ul>
<p><strong>What you'll see:</strong></p>
<pre><code class="language-text">$ POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
[INFO testing_framework_runner_local] Launching validator 0 on port 18080
Error: Os { code: 48, kind: AddrInUse, message: "Address already in use" }
thread 'main' panicked at 'failed to bind port 18080'
</code></pre>
<p><strong>Root Cause:</strong> Previous test didn't clean up properly, or another service is using the port.</p>
<p><strong>Fix:</strong></p>
<pre><code class="language-bash"># Find processes using the port
lsof -i :18080 # macOS/Linux
netstat -ano | findstr :18080 # Windows
# Kill orphaned nomos processes
pkill nomos-node
pkill nomos-executor
# For compose: ensure containers are stopped
docker compose down
docker ps -a --filter "name=nomos-compose-" -q | xargs docker rm -f
# Check if port is now free
lsof -i :18080 # Should return nothing
</code></pre>
<p><strong>For Observability Stack Port Conflicts:</strong></p>
<pre><code class="language-bash"># Edit ports in observability compose file
vim scripts/observability/compose/docker-compose.yml
# Change conflicting port mappings:
# ports:
# - "9090:9090" # Prometheus - change to "19090:9090" if needed
# - "3000:3000" # Grafana - change to "13000:3000" if needed
</code></pre>
<hr />
<h3 id="8-wallet-seeding-failed-insufficient-funds"><a class="header" href="#8-wallet-seeding-failed-insufficient-funds">8. Wallet Seeding Failed (Insufficient Funds)</a></h3>
<p><strong>Symptoms:</strong></p>
<ul>
<li>Transaction workload reports wallet issues</li>
<li>"Insufficient funds" errors</li>
<li>Transactions aren't being submitted</li>
</ul>
<p><strong>What you'll see:</strong></p>
<pre><code class="language-text">$ POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
[INFO testing_framework_workflows] Starting transaction workload with 10 users
[ERROR testing_framework_workflows] Wallet seeding failed: requested 10 users but only 3 wallets available
thread 'main' panicked at 'workload init failed: insufficient wallets'
</code></pre>
<p><strong>Root Cause:</strong> Topology configured fewer wallets than the workload needs. Transaction workload has <code>.users(M)</code> but topology only has <code>.wallets(N)</code> where N &lt; M.</p>
<p><strong>Fix:</strong></p>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
let scenario = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
.wallets(20) // ← Increase wallet count
.transactions_with(|tx| {
tx.users(10) // ← Must be ≤ wallets(20)
.rate(5)
})
.build();</code></pre>
<hr />
<h3 id="9-resource-exhaustion-oom--cpu"><a class="header" href="#9-resource-exhaustion-oom--cpu">9. Resource Exhaustion (OOM / CPU)</a></h3>
<p><strong>Symptoms:</strong></p>
<ul>
<li>Nodes crash randomly</li>
<li>"OOM Killed" messages</li>
<li>Test becomes flaky under load</li>
<li>Docker containers restart repeatedly</li>
</ul>
<p><strong>What you'll see:</strong></p>
<pre><code class="language-text">$ docker ps --filter "name=nomos-compose-"
CONTAINER ID STATUS
abc123def456 Restarting (137) 30 seconds ago # 137 = OOM killed
$ docker logs abc123def456
[INFO nomos_node] Starting validator
[INFO consensus] Processing block
Killed # ← OOM killer terminated the process
</code></pre>
<p><strong>Root Cause:</strong> Too many nodes, too much workload traffic, or insufficient Docker resources.</p>
<p><strong>Fix:</strong></p>
<pre><code class="language-bash"># 1. Reduce topology size
# In your scenario:
# .topology(Topology::preset_3v1e()) # Instead of preset_10v2e()
# 2. Reduce workload rates
# .workload(TransactionWorkload::new().rate(5.0)) # Instead of rate(100.0)
# 3. Increase Docker resources (Docker Desktop)
# Settings → Resources → Memory: 8GB minimum (12GB+ recommended for large topologies)
# Settings → Resources → CPUs: 4+ cores recommended
# 4. Increase file descriptor limits (Linux/macOS)
ulimit -n 4096
# 5. Close other heavy applications (browsers, IDEs, etc.)
</code></pre>
<hr />
<h3 id="10-logs-disappear-after-run"><a class="header" href="#10-logs-disappear-after-run">10. Logs Disappear After Run</a></h3>
<p><strong>Symptoms:</strong></p>
<ul>
<li>Test completes but no logs on disk</li>
<li>Can't debug failures because logs are gone</li>
<li>Temporary directories cleaned up automatically</li>
</ul>
<p><strong>What you'll see:</strong></p>
<pre><code class="language-text">$ POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
[INFO runner_examples] Test complete, cleaning up
[INFO testing_framework_runner_local] Removing temporary directories
$ ls .tmp/
# Empty or missing
</code></pre>
<p><strong>Root Cause:</strong> Framework cleans up temporary directories by default to avoid disk bloat.</p>
<p><strong>Fix:</strong></p>
<pre><code class="language-bash"># Persist logs to a specific directory
NOMOS_LOG_DIR=/tmp/test-logs \
NOMOS_TESTS_KEEP_LOGS=1 \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin local_runner
# Logs persist after run
ls /tmp/test-logs/
# nomos-node-0.2024-12-18T14-30-00.log
# nomos-node-1.2024-12-18T14-30-00.log
# ...
</code></pre>
<hr />
<h3 id="11-consensus-timing-too-tight--run-duration-too-short"><a class="header" href="#11-consensus-timing-too-tight--run-duration-too-short">11. Consensus Timing Too Tight / Run Duration Too Short</a></h3>
<p><strong>Symptoms:</strong></p>
<ul>
<li>"Consensus liveness expectation failed"</li>
<li>Only 1-2 blocks produced (or zero)</li>
<li>Nodes appear healthy but not making progress</li>
</ul>
<p><strong>What you'll see:</strong></p>
<pre><code class="language-text">$ POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
[INFO testing_framework_core] Starting workloads
[INFO testing_framework_core] Run window: 10 seconds
[INFO testing_framework_core] Evaluating expectations
[ERROR testing_framework_core] Consensus liveness expectation failed: expected min 5 blocks, got 1
thread 'main' panicked at 'expectations failed'
</code></pre>
<p><strong>Root Cause:</strong> Run duration too short for consensus parameters. If <code>CONSENSUS_SLOT_TIME=20s</code> but run duration is only <code>10s</code>, you can't produce many blocks.</p>
<p><strong>Fix:</strong></p>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
// Increase run duration to allow more blocks.
let scenario = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
.expect_consensus_liveness()
.with_run_duration(Duration::from_secs(120)) // ← Give more time
.build();</code></pre>
<p><strong>Or adjust consensus timing (if you control node config):</strong></p>
<pre><code class="language-bash"># Faster block production (shorter slot time)
CONSENSUS_SLOT_TIME=5 \
CONSENSUS_ACTIVE_SLOT_COEFF=0.9 \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin local_runner
</code></pre>
<hr />
<h2 id="summary-quick-checklist-for-failed-runs"><a class="header" href="#summary-quick-checklist-for-failed-runs">Summary: Quick Checklist for Failed Runs</a></h2>
<p>When a test fails, check these in order:</p>
<ol>
<li><strong><code>POL_PROOF_DEV_MODE=true</code> is set</strong> (REQUIRED for all runners)</li>
<li><strong><code>versions.env</code> exists at repo root</strong></li>
<li><strong>KZG circuit assets present</strong> (for DA workloads): <code>testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params</code></li>
<li><strong>Node binaries available</strong> (<code>NOMOS_NODE_BIN</code> / <code>NOMOS_EXECUTOR_BIN</code> set, or using <code>run-examples.sh</code>)</li>
<li><strong>Docker daemon running</strong> (for compose/k8s)</li>
<li><strong>Docker image built</strong> (<code>logos-blockchain-testing:local</code> exists for compose/k8s)</li>
<li><strong>No port conflicts</strong> (<code>lsof -i :18080</code>, kill orphaned processes)</li>
<li><strong>Sufficient wallets</strong> (<code>.wallets(N)</code> ≥ <code>.users(M)</code>)</li>
<li><strong>Enough resources</strong> (Docker memory 8GB+, ulimit -n 4096)</li>
<li><strong>Run duration appropriate</strong> (long enough for consensus timing)</li>
<li><strong>Logs persisted</strong> (<code>NOMOS_LOG_DIR</code> + <code>NOMOS_TESTS_KEEP_LOGS=1</code> if needed)</li>
</ol>
<p><strong>Still stuck?</strong> Check node logs (see <a href="troubleshooting.html#where-to-find-logs">Where to Find Logs</a>) for the actual error.</p>
<h2 id="where-to-find-logs"><a class="header" href="#where-to-find-logs">Where to Find Logs</a></h2>
<h3 id="log-location-quick-reference"><a class="header" href="#log-location-quick-reference">Log Location Quick Reference</a></h3>
<div class="table-wrapper"><table><thead><tr><th>Runner</th><th>Default Output</th><th>With <code>NOMOS_LOG_DIR</code> + Flags</th><th>Access Command</th></tr></thead><tbody>
<tr><td><strong>Host</strong> (local)</td><td>Per-run temporary directories under the current working directory (removed unless <code>NOMOS_TESTS_KEEP_LOGS=1</code>)</td><td>Per-node files with prefix <code>nomos-node-{index}</code> (set <code>NOMOS_LOG_DIR</code>)</td><td><code>cat $NOMOS_LOG_DIR/nomos-node-0*</code></td></tr>
<tr><td><strong>Compose</strong></td><td>Docker container stdout/stderr</td><td>Set <code>tracing_settings.logger: !File</code> in <code>testing-framework/assets/stack/cfgsync.yaml</code> (and mount a writable directory)</td><td><code>docker ps</code> then <code>docker logs &lt;container-id&gt;</code></td></tr>
<tr><td><strong>K8s</strong></td><td>Pod stdout/stderr</td><td>Set <code>tracing_settings.logger: !File</code> in <code>testing-framework/assets/stack/cfgsync.yaml</code> (and mount a writable directory)</td><td><code>kubectl logs -l nomos/logical-role=validator</code></td></tr>
</tbody></table>
</div>
<p><strong>Important Notes:</strong></p>
<ul>
<li><strong>Host runner</strong> (local processes): Per-run temporary directories are created under the current working directory and removed after the run unless <code>NOMOS_TESTS_KEEP_LOGS=1</code>. To write per-node log files to a stable location, set <code>NOMOS_LOG_DIR=/path/to/logs</code>.</li>
<li><strong>Compose/K8s</strong>: Node log destination is controlled by <code>testing-framework/assets/stack/cfgsync.yaml</code> (<code>tracing_settings.logger</code>). By default, rely on <code>docker logs</code> or <code>kubectl logs</code>.</li>
<li><strong>File naming</strong>: Log files use prefix <code>nomos-node-{index}*</code> or <code>nomos-executor-{index}*</code> with timestamps, e.g., <code>nomos-node-0.2024-12-01T10-30-45.log</code> (NOT just <code>.log</code> suffix).</li>
<li><strong>Container names</strong>: Compose containers include project UUID, e.g., <code>nomos-compose-&lt;uuid&gt;-validator-0-1</code> where <code>&lt;uuid&gt;</code> is randomly generated per run</li>
</ul>
<h3 id="accessing-node-logs-by-runner"><a class="header" href="#accessing-node-logs-by-runner">Accessing Node Logs by Runner</a></h3>
<h4 id="local-runner"><a class="header" href="#local-runner">Local Runner</a></h4>
<p><strong>Console output (default):</strong></p>
<pre><code class="language-bash">POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner 2&gt;&amp;1 | tee test.log
</code></pre>
<p><strong>Persistent file output:</strong></p>
<pre><code class="language-bash">NOMOS_LOG_DIR=/tmp/debug-logs \
NOMOS_LOG_LEVEL=debug \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin local_runner
# Inspect logs (note: filenames include timestamps):
ls /tmp/debug-logs/
# Example: nomos-node-0.2024-12-01T10-30-45.log
tail -f /tmp/debug-logs/nomos-node-0* # Use wildcard to match timestamp
</code></pre>
<h4 id="compose-runner-1"><a class="header" href="#compose-runner-1">Compose Runner</a></h4>
<p><strong>Stream live logs:</strong></p>
<pre><code class="language-bash"># List running containers (note the UUID prefix in names)
docker ps --filter "name=nomos-compose-"
# Find your container ID or name from the list, then:
docker logs -f &lt;container-id&gt;
# Or filter by name pattern:
docker logs -f $(docker ps --filter "name=nomos-compose-.*-validator-0" -q | head -1)
# Show last 100 lines
docker logs --tail 100 &lt;container-id&gt;
</code></pre>
<p><strong>Keep containers for post-mortem debugging:</strong></p>
<pre><code class="language-bash">COMPOSE_RUNNER_PRESERVE=1 \
NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin compose_runner
# OR: Use run-examples.sh (handles setup automatically)
COMPOSE_RUNNER_PRESERVE=1 scripts/run/run-examples.sh -t 60 -v 1 -e 1 compose
# After test failure, containers remain running:
docker ps --filter "name=nomos-compose-"
docker exec -it &lt;container-id&gt; /bin/sh
docker logs &lt;container-id&gt; &gt; debug.log
</code></pre>
<p><strong>Note:</strong> Container names follow the pattern <code>nomos-compose-{uuid}-validator-{index}-1</code> or <code>nomos-compose-{uuid}-executor-{index}-1</code>, where <code>{uuid}</code> is randomly generated per run.</p>
<h4 id="k8s-runner-1"><a class="header" href="#k8s-runner-1">K8s Runner</a></h4>
<p><strong>Important:</strong> Always verify your namespace and use label selectors instead of assuming pod names.</p>
<p><strong>Stream pod logs (use label selectors):</strong></p>
<pre><code class="language-bash"># Check your namespace first
kubectl config view --minify | grep namespace
# All validator pods (add -n &lt;namespace&gt; if not using default)
kubectl logs -l nomos/logical-role=validator -f
# All executor pods
kubectl logs -l nomos/logical-role=executor -f
# Specific pod by name (find exact name first)
kubectl get pods -l nomos/logical-role=validator # Find the exact pod name
kubectl logs -f &lt;actual-pod-name&gt; # Then use it
# With explicit namespace
kubectl logs -n my-namespace -l nomos/logical-role=validator -f
</code></pre>
<p><strong>Download logs from crashed pods:</strong></p>
<pre><code class="language-bash"># Previous logs from crashed pod
kubectl get pods -l nomos/logical-role=validator # Find crashed pod name first
kubectl logs --previous &lt;actual-pod-name&gt; &gt; crashed-validator.log
# Or use label selector for all crashed validators
for pod in $(kubectl get pods -l nomos/logical-role=validator -o name); do
kubectl logs --previous $pod &gt; $(basename $pod)-previous.log 2&gt;&amp;1
done
</code></pre>
<p><strong>Access logs from all pods:</strong></p>
<pre><code class="language-bash"># All pods in current namespace
for pod in $(kubectl get pods -o name); do
echo "=== $pod ==="
kubectl logs $pod
done &gt; all-logs.txt
# Or use label selectors (recommended)
kubectl logs -l nomos/logical-role=validator --tail=500 &gt; validators.log
kubectl logs -l nomos/logical-role=executor --tail=500 &gt; executors.log
# With explicit namespace
kubectl logs -n my-namespace -l nomos/logical-role=validator --tail=500 &gt; validators.log
</code></pre>
<h2 id="debugging-workflow"><a class="header" href="#debugging-workflow">Debugging Workflow</a></h2>
<p>When a test fails, follow this sequence:</p>
<h3 id="1-check-framework-output"><a class="header" href="#1-check-framework-output">1. Check Framework Output</a></h3>
<p>Start with the test harness output—did expectations fail? Was there a deployment error?</p>
<p><strong>Look for:</strong></p>
<ul>
<li>Expectation failure messages</li>
<li>Timeout errors</li>
<li>Deployment/readiness failures</li>
</ul>
<h3 id="2-verify-node-readiness"><a class="header" href="#2-verify-node-readiness">2. Verify Node Readiness</a></h3>
<p>Ensure all nodes started successfully and became ready before workloads began.</p>
<p><strong>Commands:</strong></p>
<pre><code class="language-bash"># Local: check process list
ps aux | grep nomos
# Compose: check container status (note UUID in names)
docker ps -a --filter "name=nomos-compose-"
# K8s: check pod status (use label selectors, add -n &lt;namespace&gt; if needed)
kubectl get pods -l nomos/logical-role=validator
kubectl get pods -l nomos/logical-role=executor
kubectl describe pod &lt;actual-pod-name&gt; # Get name from above first
</code></pre>
<h3 id="3-inspect-node-logs"><a class="header" href="#3-inspect-node-logs">3. Inspect Node Logs</a></h3>
<p>Focus on the first node that exhibited problems or the node with the highest index (often the last to start).</p>
<p><strong>Common error patterns:</strong></p>
<ul>
<li>"ERROR: versions.env missing" → missing required <code>versions.env</code> file at repository root</li>
<li>"Failed to bind address" → port conflict</li>
<li>"Connection refused" → peer not ready or network issue</li>
<li>"Proof verification failed" or "Proof generation timeout" → missing <code>POL_PROOF_DEV_MODE=true</code> (REQUIRED for all runners)</li>
<li>"Failed to load KZG parameters" or "Circuit file not found" → missing KZG circuit assets at <code>testing-framework/assets/stack/kzgrs_test_params/</code></li>
<li>"Insufficient funds" → wallet seeding issue (increase <code>.wallets(N)</code> or reduce <code>.users(M)</code>)</li>
</ul>
<h3 id="4-check-log-levels"><a class="header" href="#4-check-log-levels">4. Check Log Levels</a></h3>
<p>If logs are too sparse, increase verbosity:</p>
<pre><code class="language-bash">NOMOS_LOG_LEVEL=debug \
NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug" \
cargo run -p runner-examples --bin local_runner
</code></pre>
<p>If metric updates are polluting your logs (fields like <code>counter.*</code> / <code>gauge.*</code>), move those events to a dedicated <code>tracing</code> target (e.g. <code>target: "nomos_metrics"</code>) and set <code>NOMOS_LOG_FILTER="nomos_metrics=off,..."</code> so they don’t get formatted into log output.</p>
<h3 id="5-verify-observability-endpoints"><a class="header" href="#5-verify-observability-endpoints">5. Verify Observability Endpoints</a></h3>
<p>If expectations report observability issues:</p>
<p><strong>Prometheus (Compose):</strong></p>
<pre><code class="language-bash">curl http://localhost:9090/-/healthy
</code></pre>
<p><strong>Node HTTP APIs:</strong></p>
<pre><code class="language-bash">curl http://localhost:18080/consensus/info # Adjust port per node
</code></pre>
<h3 id="6-compare-with-known-good-scenario"><a class="header" href="#6-compare-with-known-good-scenario">6. Compare with Known-Good Scenario</a></h3>
<p>Run a minimal baseline test (e.g., 2 validators, consensus liveness only). If it passes, the issue is in your workload or topology configuration.</p>
<h2 id="common-error-messages"><a class="header" href="#common-error-messages">Common Error Messages</a></h2>
<h3 id="consensus-liveness-expectation-failed"><a class="header" href="#consensus-liveness-expectation-failed">"Consensus liveness expectation failed"</a></h3>
<ul>
<li><strong>Cause</strong>: Not enough blocks produced during the run window, missing
<code>POL_PROOF_DEV_MODE=true</code> (causes slow proof generation), or missing KZG
assets for DA workloads.</li>
<li><strong>Fix</strong>:
<ol>
<li>Verify <code>POL_PROOF_DEV_MODE=true</code> is set (REQUIRED for all runners).</li>
<li>Verify KZG assets exist at
<code>testing-framework/assets/stack/kzgrs_test_params/</code> (for DA workloads).</li>
<li>Extend <code>with_run_duration()</code> to allow more blocks.</li>
<li>Check node logs for proof generation or DA errors.</li>
<li>Reduce transaction/DA rate if nodes are overwhelmed.</li>
</ol>
</li>
</ul>
<h3 id="wallet-seeding-failed"><a class="header" href="#wallet-seeding-failed">"Wallet seeding failed"</a></h3>
<ul>
<li><strong>Cause</strong>: Topology doesn't have enough funded wallets for the workload.</li>
<li><strong>Fix</strong>: Increase <code>.wallets(N)</code> count or reduce <code>.users(M)</code> in the transaction
workload (ensure N ≥ M).</li>
</ul>
<h3 id="node-control-not-available"><a class="header" href="#node-control-not-available">"Node control not available"</a></h3>
<ul>
<li><strong>Cause</strong>: Runner doesn't support node control (only ComposeDeployer does), or
<code>enable_node_control()</code> wasn't called.</li>
<li><strong>Fix</strong>:
<ol>
<li>Use ComposeDeployer for chaos tests (LocalDeployer and K8sDeployer don't
support node control).</li>
<li>Ensure <code>.enable_node_control()</code> is called in the scenario before <code>.chaos()</code>.</li>
</ol>
</li>
</ul>
<h3 id="readiness-timeout"><a class="header" href="#readiness-timeout">"Readiness timeout"</a></h3>
<ul>
<li><strong>Cause</strong>: Nodes didn't become responsive within expected time (often due to
missing prerequisites).</li>
<li><strong>Fix</strong>:
<ol>
<li><strong>Verify <code>POL_PROOF_DEV_MODE=true</code> is set</strong> (REQUIRED for all runners—without
it, proof generation is too slow).</li>
<li>Check node logs for startup errors (port conflicts, missing assets).</li>
<li>Verify network connectivity between nodes.</li>
<li>For DA workloads, ensure KZG circuit assets are present.</li>
</ol>
</li>
</ul>
<h3 id="error-versionsenv-missing"><a class="header" href="#error-versionsenv-missing">"ERROR: versions.env missing"</a></h3>
<ul>
<li><strong>Cause</strong>: Helper scripts (<code>run-examples.sh</code>, <code>build-bundle.sh</code>, <code>setup-circuits-stack.sh</code>) require <code>versions.env</code> file at repository root.</li>
<li><strong>Fix</strong>: Ensure you're running from the repository root directory. The <code>versions.env</code> file should already exist and contains:</li>
</ul>
<pre><code class="language-text"> VERSION=&lt;circuit release tag&gt;
NOMOS_NODE_REV=&lt;nomos-node git revision&gt;
NOMOS_BUNDLE_VERSION=&lt;bundle schema version&gt;
</code></pre>
<p>Use the checked-in <code>versions.env</code> at the repository root as the source of truth.</p>
<h3 id="port-already-in-use"><a class="header" href="#port-already-in-use">"Port already in use"</a></h3>
<ul>
<li><strong>Cause</strong>: Previous test didn't clean up, or another process holds the port.</li>
<li><strong>Fix</strong>: Kill orphaned processes (<code>pkill nomos-node</code>), wait for Docker cleanup
(<code>docker compose down</code>), or restart Docker.</li>
</ul>
<h3 id="image-not-found-logos-blockchain-testinglocal"><a class="header" href="#image-not-found-logos-blockchain-testinglocal">"Image not found: logos-blockchain-testing:local"</a></h3>
<ul>
<li><strong>Cause</strong>: Docker image not built for Compose/K8s runners, or KZG assets not
baked into the image.</li>
<li><strong>Fix (recommended)</strong>: Use run-examples.sh which handles everything:
<pre><code class="language-bash">scripts/run/run-examples.sh -t 60 -v 1 -e 1 compose
</code></pre>
</li>
<li><strong>Fix (manual)</strong>:
<ol>
<li>Build bundle: <code>scripts/build/build-bundle.sh --platform linux</code></li>
<li>Set bundle path: <code>export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz</code></li>
<li>Build image: <code>scripts/build/build_test_image.sh</code></li>
<li><strong>kind/minikube:</strong> load the image into the cluster nodes (e.g. <code>kind load docker-image logos-blockchain-testing:local</code>, or <code>minikube image load ...</code>), or push to a registry and set <code>NOMOS_TESTNET_IMAGE</code> accordingly.</li>
</ol>
</li>
</ul>
<h3 id="failed-to-load-kzg-parameters-or-circuit-file-not-found"><a class="header" href="#failed-to-load-kzg-parameters-or-circuit-file-not-found">"Failed to load KZG parameters" or "Circuit file not found"</a></h3>
<ul>
<li><strong>Cause</strong>: DA workload requires KZG circuit assets. The file <code>testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params</code> (note repeated filename) must exist. Inside containers, it's at <code>/kzgrs_test_params/kzgrs_test_params</code>.</li>
<li><strong>Fix (recommended)</strong>: Use run-examples.sh which handles setup:
<pre><code class="language-bash">scripts/run/run-examples.sh -t 60 -v 1 -e 1 &lt;mode&gt;
</code></pre>
</li>
<li><strong>Fix (manual)</strong>:
<ol>
<li>Fetch assets: <code>scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits</code></li>
<li>Copy to expected path: <code>cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/</code></li>
<li>Verify file exists: <code>ls -lh testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params</code></li>
<li>For Compose/K8s: rebuild image with assets baked in</li>
</ol>
</li>
</ul>
<p>For detailed logging configuration and observability setup, see <a href="logging-observability.html">Logging &amp; Observability</a>.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="faq"><a class="header" href="#faq">FAQ</a></h1>
<p><strong>Why block-oriented timing?</strong><br />
Slots advance at a fixed rate (NTP-synchronized, 2s by default), so reasoning
about blocks and consensus intervals keeps assertions aligned with protocol
behavior rather than arbitrary wall-clock durations.</p>
<p><strong>Can I reuse the same scenario across runners?</strong><br />
Yes. The plan stays the same; swap runners (local, compose, k8s) to target
different environments.</p>
<p><strong>When should I enable chaos workloads?</strong><br />
Only when testing resilience or operational recovery; keep functional smoke
tests deterministic.</p>
<p><strong>How long should runs be?</strong><br />
The framework enforces a minimum of <strong>2× slot duration</strong> (4 seconds with default 2s slots), but practical recommendations:</p>
<ul>
<li><strong>Smoke tests</strong>: 30s minimum (~14 blocks with default 2s slots, 0.9 coefficient)</li>
<li><strong>Transaction workloads</strong>: 60s+ (~27 blocks) to observe inclusion patterns</li>
<li><strong>DA workloads</strong>: 90s+ (~40 blocks) to account for dispersal and sampling</li>
<li><strong>Chaos tests</strong>: 120s+ (~54 blocks) to allow recovery after restarts</li>
</ul>
<p>Very short runs (&lt; 30s) risk false confidence—one or two lucky blocks don't prove liveness.</p>
<p><strong>Do I always need seeded wallets?</strong><br />
Only for transaction scenarios. Data-availability or pure chaos scenarios may
not require them, but liveness checks still need validators producing blocks.</p>
<p><strong>What if expectations fail but workloads “look fine”?</strong><br />
Trust expectations first—they capture the intended success criteria. Use the
observability signals and runner logs to pinpoint why the system missed the
target.</p>
<div style="break-before: page; page-break-before: always;"></div><h1 id="glossary"><a class="header" href="#glossary">Glossary</a></h1>
<ul>
<li><strong>Validator</strong>: node role responsible for participating in consensus and block
production.</li>
<li><strong>Executor</strong>: a validator node with the DA dispersal service enabled. Executors
can submit transactions and disperse blob data to the DA network, in addition
to performing all validator functions.</li>
<li><strong>DA (Data Availability)</strong>: subsystem ensuring blobs or channel data are
published and retrievable for validation.</li>
<li><strong>Deployer</strong>: component that provisions infrastructure (spawns processes,
creates containers, or launches pods), waits for readiness, and returns a
Runner. Examples: LocalDeployer, ComposeDeployer, K8sDeployer.</li>
<li><strong>Runner</strong>: component returned by deployers that orchestrates scenario
execution—starts workloads, observes signals, evaluates expectations, and
triggers cleanup.</li>
<li><strong>Workload</strong>: traffic or behavior generator that exercises the system during a
scenario run.</li>
<li><strong>Expectation</strong>: post-run assertion that judges whether the system met the
intended success criteria.</li>
<li><strong>Topology</strong>: declarative description of the cluster shape, roles, and
high-level parameters for a scenario.</li>
<li><strong>Scenario</strong>: immutable plan combining topology, workloads, expectations, and
run duration.</li>
<li><strong>Blockfeed</strong>: stream of block observations used for liveness or inclusion
signals during a run.</li>
<li><strong>Control capability</strong>: the ability for a runner to start, stop, or restart
nodes, used by chaos workloads.</li>
<li><strong>Slot duration</strong>: time interval between consensus rounds in Cryptarchia. Blocks
are produced at multiples of the slot duration based on lottery outcomes.</li>
<li><strong>Block cadence</strong>: observed rate of block production in a live network, measured
in blocks per second or seconds per block.</li>
<li><strong>Cooldown</strong>: waiting period after a chaos action (e.g., node restart) before
triggering the next action, allowing the system to stabilize.</li>
<li><strong>Run window</strong>: total duration a scenario executes, specified via
<code>with_run_duration()</code>. Framework auto-extends to at least 2× slot duration.</li>
<li><strong>Readiness probe</strong>: health check performed by runners to ensure nodes are
reachable and responsive before starting workloads. Prevents false negatives
from premature traffic.</li>
<li><strong>Liveness</strong>: property that the system continues making progress (producing
blocks) under specified conditions. Contrasts with safety/correctness which
verifies that state transitions are accurate.</li>
<li><strong>State assertion</strong>: expectation that verifies specific values in the system
state (e.g., wallet balances, UTXO sets) rather than just progress signals.
Also called "correctness expectations."</li>
<li><strong>Mantle transaction</strong>: transaction type in Logos that can contain UTXO transfers
(LedgerTx) and operations (Op), including channel data (ChannelBlob).</li>
<li><strong>Channel</strong>: logical grouping for DA blobs; each blob belongs to a channel and
references a parent blob in the same channel, creating a chain of related data.</li>
<li><strong>POL_PROOF_DEV_MODE</strong>: environment variable that disables expensive Groth16 zero-knowledge
proof generation for leader election. <strong>Required for all runners</strong> (local, compose, k8s)
for practical testing—without it, proof generation causes timeouts. Should never be
used in production environments.</li>
</ul>
<hr />
<h2 id="external-resources"><a class="header" href="#external-resources">External Resources</a></h2>
<ul>
<li><strong><a href="https://nomos-tech.notion.site/project">Logos Project Documentation</a></strong> — Protocol specifications, node internals, and architecture details</li>
</ul>
</main>
<nav class="nav-wrapper" aria-label="Page navigation">
<!-- Mobile navigation buttons -->
<div style="clear: both"></div>
</nav>
</div>
</div>
<nav class="nav-wide-wrapper" aria-label="Page navigation">
</nav>
</div>
<script>
window.playground_copyable = true;
</script>
<script src="elasticlunr.min.js"></script>
<script src="mark.min.js"></script>
<script src="searcher.js"></script>
<script src="clipboard.min.js"></script>
<script src="highlight.js"></script>
<script src="book.js"></script>
<!-- Custom JS scripts -->
<script src="theme/mermaid-init.js"></script>
<script>
window.addEventListener('load', function() {
window.setTimeout(window.print, 100);
});
</script>
</div>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink