logos-blockchain-testing/docs/testing-philosophy.html
2025-12-20 09:51:51 +01:00

382 lines
24 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE HTML>
<html lang="en" class="light" dir="ltr">
<head>
<!-- Book generated using mdBook -->
<meta charset="UTF-8">
<title>Testing Philosophy - Logos Blockchain Testing Framework Book</title>
<!-- Custom HTML head -->
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#ffffff">
<link rel="icon" href="favicon.svg">
<link rel="shortcut icon" href="favicon.png">
<link rel="stylesheet" href="css/variables.css">
<link rel="stylesheet" href="css/general.css">
<link rel="stylesheet" href="css/chrome.css">
<link rel="stylesheet" href="css/print.css" media="print">
<!-- Fonts -->
<link rel="stylesheet" href="FontAwesome/css/font-awesome.css">
<link rel="stylesheet" href="fonts/fonts.css">
<!-- Highlight.js Stylesheets -->
<link rel="stylesheet" href="highlight.css">
<link rel="stylesheet" href="tomorrow-night.css">
<link rel="stylesheet" href="ayu-highlight.css">
<!-- Custom theme stylesheets -->
</head>
<body class="sidebar-visible no-js">
<div id="body-container">
<!-- Provide site root to javascript -->
<script>
var path_to_root = "";
var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "navy" : "light";
</script>
<!-- Work around some values being stored in localStorage wrapped in quotes -->
<script>
try {
var theme = localStorage.getItem('mdbook-theme');
var sidebar = localStorage.getItem('mdbook-sidebar');
if (theme.startsWith('"') && theme.endsWith('"')) {
localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));
}
if (sidebar.startsWith('"') && sidebar.endsWith('"')) {
localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));
}
} catch (e) { }
</script>
<!-- Set the theme before any content is loaded, prevents flash -->
<script>
var theme;
try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }
if (theme === null || theme === undefined) { theme = default_theme; }
var html = document.querySelector('html');
html.classList.remove('light')
html.classList.add(theme);
var body = document.querySelector('body');
body.classList.remove('no-js')
body.classList.add('js');
</script>
<input type="checkbox" id="sidebar-toggle-anchor" class="hidden">
<!-- Hide / unhide sidebar before it is displayed -->
<script>
var body = document.querySelector('body');
var sidebar = null;
var sidebar_toggle = document.getElementById("sidebar-toggle-anchor");
if (document.body.clientWidth >= 1080) {
try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }
sidebar = sidebar || 'visible';
} else {
sidebar = 'hidden';
}
sidebar_toggle.checked = sidebar === 'visible';
body.classList.remove('sidebar-visible');
body.classList.add("sidebar-" + sidebar);
</script>
<nav id="sidebar" class="sidebar" aria-label="Table of contents">
<div class="sidebar-scrollbox">
<ol class="chapter"><li class="chapter-item expanded "><a href="project-context-primer.html"><strong aria-hidden="true">1.</strong> Project Context Primer</a></li><li class="chapter-item expanded "><a href="what-you-will-learn.html"><strong aria-hidden="true">2.</strong> What You Will Learn</a></li><li class="chapter-item expanded "><a href="quickstart.html"><strong aria-hidden="true">3.</strong> Quickstart</a></li><li class="chapter-item expanded "><a href="part-i.html"><strong aria-hidden="true">4.</strong> Part I — Foundations</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="introduction.html"><strong aria-hidden="true">4.1.</strong> Introduction</a></li><li class="chapter-item expanded "><a href="architecture-overview.html"><strong aria-hidden="true">4.2.</strong> Architecture Overview</a></li><li class="chapter-item expanded "><a href="testing-philosophy.html" class="active"><strong aria-hidden="true">4.3.</strong> Testing Philosophy</a></li><li class="chapter-item expanded "><a href="scenario-lifecycle.html"><strong aria-hidden="true">4.4.</strong> Scenario Lifecycle</a></li><li class="chapter-item expanded "><a href="design-rationale.html"><strong aria-hidden="true">4.5.</strong> Design Rationale</a></li></ol></li><li class="chapter-item expanded "><a href="part-ii.html"><strong aria-hidden="true">5.</strong> Part II — User Guide</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="workspace-layout.html"><strong aria-hidden="true">5.1.</strong> Workspace Layout</a></li><li class="chapter-item expanded "><a href="annotated-tree.html"><strong aria-hidden="true">5.2.</strong> Annotated Tree</a></li><li class="chapter-item expanded "><a href="authoring-scenarios.html"><strong aria-hidden="true">5.3.</strong> Authoring Scenarios</a></li><li class="chapter-item expanded "><a href="workloads.html"><strong aria-hidden="true">5.4.</strong> Core Content: Workloads & Expectations</a></li><li class="chapter-item expanded "><a href="scenario-builder-ext-patterns.html"><strong aria-hidden="true">5.5.</strong> Core Content: ScenarioBuilderExt Patterns</a></li><li class="chapter-item expanded "><a href="best-practices.html"><strong aria-hidden="true">5.6.</strong> Best Practices</a></li><li class="chapter-item expanded "><a href="usage-patterns.html"><strong aria-hidden="true">5.7.</strong> Usage Patterns</a></li><li class="chapter-item expanded "><a href="examples.html"><strong aria-hidden="true">5.8.</strong> Examples</a></li><li class="chapter-item expanded "><a href="examples-advanced.html"><strong aria-hidden="true">5.9.</strong> Advanced & Artificial Examples</a></li><li class="chapter-item expanded "><a href="cucumber-bdd.html"><strong aria-hidden="true">5.10.</strong> Cucumber/BDD Interface</a></li><li class="chapter-item expanded "><a href="running-scenarios.html"><strong aria-hidden="true">5.11.</strong> Running Scenarios</a></li><li class="chapter-item expanded "><a href="runners.html"><strong aria-hidden="true">5.12.</strong> Runners</a></li><li class="chapter-item expanded "><a href="node-control.html"><strong aria-hidden="true">5.13.</strong> RunContext: BlockFeed & Node Control</a></li><li class="chapter-item expanded "><a href="chaos.html"><strong aria-hidden="true">5.14.</strong> Chaos Workloads</a></li><li class="chapter-item expanded "><a href="topology-chaos.html"><strong aria-hidden="true">5.15.</strong> Topology & Chaos Patterns</a></li></ol></li><li class="chapter-item expanded "><a href="part-iii.html"><strong aria-hidden="true">6.</strong> Part III — Developer Reference</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="scenario-model.html"><strong aria-hidden="true">6.1.</strong> Scenario Model (Developer Level)</a></li><li class="chapter-item expanded "><a href="api-levels.html"><strong aria-hidden="true">6.2.</strong> API Levels: Builder DSL vs. Direct</a></li><li class="chapter-item expanded "><a href="extending.html"><strong aria-hidden="true">6.3.</strong> Extending the Framework</a></li><li class="chapter-item expanded "><a href="custom-workload-example.html"><strong aria-hidden="true">6.4.</strong> Example: New Workload & Expectation (Rust)</a></li><li class="chapter-item expanded "><a href="internal-crate-reference.html"><strong aria-hidden="true">6.5.</strong> Internal Crate Reference</a></li></ol></li><li class="chapter-item expanded "><a href="part-iv.html"><strong aria-hidden="true">7.</strong> Part IV — Operations & Deployment</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="operations-overview.html"><strong aria-hidden="true">7.1.</strong> Overview</a></li><li class="chapter-item expanded "><a href="prerequisites.html"><strong aria-hidden="true">7.2.</strong> Prerequisites & Setup</a></li><li class="chapter-item expanded "><a href="running-examples.html"><strong aria-hidden="true">7.3.</strong> Running Examples</a></li><li class="chapter-item expanded "><a href="ci-integration.html"><strong aria-hidden="true">7.4.</strong> CI Integration</a></li><li class="chapter-item expanded "><a href="environment-variables.html"><strong aria-hidden="true">7.5.</strong> Environment Variables</a></li><li class="chapter-item expanded "><a href="logging-observability.html"><strong aria-hidden="true">7.6.</strong> Logging & Observability</a></li></ol></li><li class="chapter-item expanded "><a href="part-v.html"><strong aria-hidden="true">8.</strong> Part V — Appendix</a></li><li><ol class="section"><li class="chapter-item expanded "><a href="dsl-cheat-sheet.html"><strong aria-hidden="true">8.1.</strong> Builder API Quick Reference</a></li><li class="chapter-item expanded "><a href="troubleshooting.html"><strong aria-hidden="true">8.2.</strong> Troubleshooting Scenarios</a></li><li class="chapter-item expanded "><a href="faq.html"><strong aria-hidden="true">8.3.</strong> FAQ</a></li><li class="chapter-item expanded "><a href="glossary.html"><strong aria-hidden="true">8.4.</strong> Glossary</a></li></ol></li></ol>
</div>
<div id="sidebar-resize-handle" class="sidebar-resize-handle">
<div class="sidebar-resize-indicator"></div>
</div>
</nav>
<!-- Track and set sidebar scroll position -->
<script>
var sidebarScrollbox = document.querySelector('#sidebar .sidebar-scrollbox');
sidebarScrollbox.addEventListener('click', function(e) {
if (e.target.tagName === 'A') {
sessionStorage.setItem('sidebar-scroll', sidebarScrollbox.scrollTop);
}
}, { passive: true });
var sidebarScrollTop = sessionStorage.getItem('sidebar-scroll');
sessionStorage.removeItem('sidebar-scroll');
if (sidebarScrollTop) {
// preserve sidebar scroll position when navigating via links within sidebar
sidebarScrollbox.scrollTop = sidebarScrollTop;
} else {
// scroll sidebar to current active section when navigating via "next/previous chapter" buttons
var activeSection = document.querySelector('#sidebar .active');
if (activeSection) {
activeSection.scrollIntoView({ block: 'center' });
}
}
</script>
<div id="page-wrapper" class="page-wrapper">
<div class="page">
<div id="menu-bar-hover-placeholder"></div>
<div id="menu-bar" class="menu-bar sticky">
<div class="left-buttons">
<label id="sidebar-toggle" class="icon-button" for="sidebar-toggle-anchor" title="Toggle Table of Contents" aria-label="Toggle Table of Contents" aria-controls="sidebar">
<i class="fa fa-bars"></i>
</label>
<button id="theme-toggle" class="icon-button" type="button" title="Change theme" aria-label="Change theme" aria-haspopup="true" aria-expanded="false" aria-controls="theme-list">
<i class="fa fa-paint-brush"></i>
</button>
<ul id="theme-list" class="theme-popup" aria-label="Themes" role="menu">
<li role="none"><button role="menuitem" class="theme" id="light">Light</button></li>
<li role="none"><button role="menuitem" class="theme" id="rust">Rust</button></li>
<li role="none"><button role="menuitem" class="theme" id="coal">Coal</button></li>
<li role="none"><button role="menuitem" class="theme" id="navy">Navy</button></li>
<li role="none"><button role="menuitem" class="theme" id="ayu">Ayu</button></li>
</ul>
<button id="search-toggle" class="icon-button" type="button" title="Search. (Shortkey: s)" aria-label="Toggle Searchbar" aria-expanded="false" aria-keyshortcuts="S" aria-controls="searchbar">
<i class="fa fa-search"></i>
</button>
</div>
<h1 class="menu-title">Logos Blockchain Testing Framework Book</h1>
<div class="right-buttons">
<a href="print.html" title="Print this book" aria-label="Print this book">
<i id="print-button" class="fa fa-print"></i>
</a>
</div>
</div>
<div id="search-wrapper" class="hidden">
<form id="searchbar-outer" class="searchbar-outer">
<input type="search" id="searchbar" name="searchbar" placeholder="Search this book ..." aria-controls="searchresults-outer" aria-describedby="searchresults-header">
</form>
<div id="searchresults-outer" class="searchresults-outer hidden">
<div id="searchresults-header" class="searchresults-header"></div>
<ul id="searchresults">
</ul>
</div>
</div>
<!-- Apply ARIA attributes after the sidebar and the sidebar toggle button are added to the DOM -->
<script>
document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');
document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');
Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {
link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);
});
</script>
<div id="content" class="content">
<main>
<h1 id="testing-philosophy"><a class="header" href="#testing-philosophy">Testing Philosophy</a></h1>
<p>This framework embodies specific principles that shape how you author and run
scenarios. Understanding these principles helps you write effective tests and
interpret results correctly.</p>
<h2 id="declarative-over-imperative"><a class="header" href="#declarative-over-imperative">Declarative over Imperative</a></h2>
<p>Describe <strong>what</strong> you want to test, not <strong>how</strong> to orchestrate it:</p>
<pre><code class="language-rust ignore">use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn declarative_over_imperative() {
// Good: declarative
let _plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(2).executors(1))
.transactions_with(|txs| {
txs.rate(5) // 5 transactions per block
})
.expect_consensus_liveness()
.build();
// Bad: imperative (framework doesn't work this way)
// spawn_validator(); spawn_executor();
// loop { submit_tx(); check_block(); }
}</code></pre>
<p><strong>Why it matters:</strong> The framework handles deployment, readiness, and cleanup.
You focus on test intent, not infrastructure orchestration.</p>
<h2 id="protocol-time-not-wall-time"><a class="header" href="#protocol-time-not-wall-time">Protocol Time, Not Wall Time</a></h2>
<p>Reason in <strong>blocks</strong> and <strong>consensus intervals</strong>, not wall-clock seconds.</p>
<p><strong>Consensus defaults:</strong></p>
<ul>
<li>Slot duration: 2 seconds (NTP-synchronized, configurable via <code>CONSENSUS_SLOT_TIME</code>)</li>
<li>Active slot coefficient: 0.9 (90% block probability per slot, configurable via <code>CONSENSUS_ACTIVE_SLOT_COEFF</code>)</li>
<li>Expected rate: ~27 blocks per minute</li>
</ul>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn protocol_time_not_wall_time() {
// Good: protocol-oriented thinking
let _plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(2).executors(1))
.transactions_with(|txs| {
txs.rate(5) // 5 transactions per block
})
.with_run_duration(Duration::from_secs(60)) // Let framework calculate expected blocks
.expect_consensus_liveness() // "Did we produce the expected blocks?"
.build();
// Bad: wall-clock assumptions
// "I expect exactly 30 blocks in 60 seconds"
// This breaks on slow CI where slot timing might drift
}</code></pre>
<p><strong>Why it matters:</strong> Slot timing is fixed (2s by default, NTP-synchronized), so the
expected number of blocks is predictable: ~27 blocks in 60s with the default
0.9 active slot coefficient. The framework calculates expected blocks from slot
duration and run window, making assertions protocol-based rather than tied to
specific wall-clock expectations. Assert on "blocks produced relative to slots"
not "blocks produced in exact wall-clock seconds".</p>
<h2 id="determinism-first-chaos-when-needed"><a class="header" href="#determinism-first-chaos-when-needed">Determinism First, Chaos When Needed</a></h2>
<p><strong>Default scenarios are repeatable:</strong></p>
<ul>
<li>Fixed topology</li>
<li>Predictable traffic rates</li>
<li>Deterministic checks</li>
</ul>
<p><strong>Chaos is opt-in:</strong></p>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt};
pub fn determinism_first() {
// Separate: functional test (deterministic)
let _plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(2).executors(1))
.transactions_with(|txs| {
txs.rate(5) // 5 transactions per block
})
.expect_consensus_liveness()
.build();
// Separate: chaos test (introduces randomness)
let _chaos_plan =
ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
.enable_node_control()
.chaos_with(|c| {
c.restart()
.min_delay(Duration::from_secs(30))
.max_delay(Duration::from_secs(60))
.target_cooldown(Duration::from_secs(45))
.apply()
})
.transactions_with(|txs| {
txs.rate(5) // 5 transactions per block
})
.expect_consensus_liveness()
.build();
}</code></pre>
<p><strong>Why it matters:</strong> Mixing determinism with chaos creates noisy, hard-to-debug
failures. Separate concerns make failures actionable.</p>
<h2 id="observable-health-signals"><a class="header" href="#observable-health-signals">Observable Health Signals</a></h2>
<p>Prefer <strong>user-facing signals</strong> over internal state:</p>
<p><strong>Good checks:</strong></p>
<ul>
<li>Blocks progressing at expected rate (liveness)</li>
<li>Transactions included within N blocks (inclusion)</li>
<li>DA blobs retrievable (availability)</li>
</ul>
<p><strong>Avoid internal checks:</strong></p>
<ul>
<li>Memory pool size</li>
<li>Internal service state</li>
<li>Cache hit rates</li>
</ul>
<p><strong>Why it matters:</strong> User-facing signals reflect actual system health.
Internal state can be "healthy" while the system is broken from a user
perspective.</p>
<h2 id="minimum-run-windows"><a class="header" href="#minimum-run-windows">Minimum Run Windows</a></h2>
<p>Always run long enough for <strong>meaningful block production</strong>:</p>
<pre><code class="language-rust ignore">use std::time::Duration;
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ScenarioBuilderExt;
pub fn minimum_run_windows() {
// Bad: too short (~2 blocks with default 2s slots, 0.9 coeff)
let _too_short = ScenarioBuilder::with_node_counts(1, 0)
.with_run_duration(Duration::from_secs(5))
.expect_consensus_liveness()
.build();
// Good: enough blocks for assertions (~27 blocks with default 2s slots, 0.9
// coeff)
let _good = ScenarioBuilder::with_node_counts(1, 0)
.with_run_duration(Duration::from_secs(60))
.expect_consensus_liveness()
.build();
}</code></pre>
<p><strong>Note:</strong> Block counts assume default consensus parameters:</p>
<ul>
<li>Slot duration: 2 seconds (configurable via <code>CONSENSUS_SLOT_TIME</code>)</li>
<li>Active slot coefficient: 0.9 (90% block probability per slot, configurable via <code>CONSENSUS_ACTIVE_SLOT_COEFF</code>)</li>
<li>Formula: <code>blocks ≈ (duration / slot_duration) × active_slot_coeff</code></li>
</ul>
<p>If upstream changes these parameters, adjust your duration expectations accordingly.</p>
<p>The framework enforces minimum durations (at least 2× slot duration), but be explicit. Very short runs risk false confidence—one lucky block doesn't prove liveness.</p>
<h2 id="summary"><a class="header" href="#summary">Summary</a></h2>
<p>These principles keep scenarios:</p>
<ul>
<li><strong>Portable</strong> across environments (protocol time, declarative)</li>
<li><strong>Debuggable</strong> (determinism, separation of concerns)</li>
<li><strong>Meaningful</strong> (observable signals, sufficient duration)</li>
</ul>
<p>When authoring scenarios, ask: "Does this test the protocol behavior or
my local environment quirks?"</p>
</main>
<nav class="nav-wrapper" aria-label="Page navigation">
<!-- Mobile navigation buttons -->
<a rel="prev" href="architecture-overview.html" class="mobile-nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left">
<i class="fa fa-angle-left"></i>
</a>
<a rel="next prefetch" href="scenario-lifecycle.html" class="mobile-nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right">
<i class="fa fa-angle-right"></i>
</a>
<div style="clear: both"></div>
</nav>
</div>
</div>
<nav class="nav-wide-wrapper" aria-label="Page navigation">
<a rel="prev" href="architecture-overview.html" class="nav-chapters previous" title="Previous chapter" aria-label="Previous chapter" aria-keyshortcuts="Left">
<i class="fa fa-angle-left"></i>
</a>
<a rel="next prefetch" href="scenario-lifecycle.html" class="nav-chapters next" title="Next chapter" aria-label="Next chapter" aria-keyshortcuts="Right">
<i class="fa fa-angle-right"></i>
</a>
</nav>
</div>
<script>
window.playground_copyable = true;
</script>
<script src="elasticlunr.min.js"></script>
<script src="mark.min.js"></script>
<script src="searcher.js"></script>
<script src="clipboard.min.js"></script>
<script src="highlight.js"></script>
<script src="book.js"></script>
<!-- Custom JS scripts -->
<script src="theme/mermaid-init.js"></script>
</div>
</body>
</html>