Document not found (404)
+This URL is invalid, sorry. Please use the navigation bar or search to continue.
+ +This URL is invalid, sorry. Please use the navigation bar or search to continue.
+ +Directory structure with key paths annotated:
+logos-blockchain-testing/
+├─ testing-framework/ # Core library crates
+│ ├─ configs/ # Node config builders, topology generation, tracing/logging config
+│ ├─ core/ # Scenario model (ScenarioBuilder), runtime (Runner, Deployer), topology, node spawning
+│ ├─ workflows/ # Workloads (transactions, DA, chaos), expectations (liveness), builder DSL extensions
+│ ├─ runners/ # Deployment backends
+│ │ ├─ local/ # LocalDeployer (spawns local processes)
+│ │ ├─ compose/ # ComposeDeployer (Docker Compose + Prometheus)
+│ │ └─ k8s/ # K8sDeployer (Kubernetes Helm)
+│ └─ assets/ # Docker/K8s stack assets
+│ └─ stack/
+│ ├─ kzgrs_test_params/ # KZG circuit parameters directory
+│ │ └─ kzgrs_test_params # Actual proving key file (note repeated name)
+│ ├─ monitoring/ # Prometheus config
+│ ├─ scripts/ # Container entrypoints
+│ └─ cfgsync.yaml # Config sync server template
+│
+├─ examples/ # PRIMARY ENTRY POINT: runnable binaries
+│ └─ src/bin/
+│ ├─ local_runner.rs # Host processes demo (LocalDeployer)
+│ ├─ compose_runner.rs # Docker Compose demo (ComposeDeployer)
+│ └─ k8s_runner.rs # Kubernetes demo (K8sDeployer)
+│
+├─ scripts/ # Helper utilities
+│ ├─ run-examples.sh # Convenience script (handles setup + runs examples)
+│ ├─ build-bundle.sh # Build prebuilt binaries+circuits bundle
+│ ├─ setup-circuits-stack.sh # Fetch KZG parameters (Linux + host)
+│ └─ setup-nomos-circuits.sh # Legacy circuit fetcher
+│
+└─ book/ # This documentation (mdBook)
+testing-framework/Core library crates providing the testing API.
+| Crate | Purpose | Key Exports |
|---|---|---|
configs | Node configuration builders | Topology generation, tracing config |
core | Scenario model & runtime | ScenarioBuilder, Deployer, Runner |
workflows | Workloads & expectations | ScenarioBuilderExt, ChaosBuilderExt |
runners/local | Local process deployer | LocalDeployer |
runners/compose | Docker Compose deployer | ComposeDeployer |
runners/k8s | Kubernetes deployer | K8sDeployer |
testing-framework/assets/stack/Docker/K8s deployment assets:
+kzgrs_test_params/kzgrs_test_params: Circuit parameters file (note repeated name; override via NOMOS_KZGRS_PARAMS_PATH)monitoring/: Prometheus configscripts/: Container entrypointsscripts/Convenience utilities:
+run-examples.sh: All-in-one script for host/compose/k8s modes (recommended)build-bundle.sh: Create prebuilt binaries+circuits bundle for compose/k8sbuild_test_image.sh: Build the compose/k8s Docker image (bakes in assets)setup-circuits-stack.sh: Fetch KZG parameters for both Linux and hostcfgsync.yaml: Configuration sync server templateexamples/ (Start Here!)Runnable binaries demonstrating framework usage:
+local_runner.rs — Local processescompose_runner.rs — Docker Compose (requires NOMOS_TESTNET_IMAGE built)k8s_runner.rs — Kubernetes (requires cluster + image)Run with: POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin <name>
All runners require POL_PROOF_DEV_MODE=true to avoid expensive proof generation.
scripts/Helper utilities:
+setup-nomos-circuits.sh: Fetch KZG parameters from releasesCompose runner includes:
+http://localhost:9090 (metrics scraping)ctx.telemetry().prometheus().map(|p| p.base_url())Logging controlled by:
+NOMOS_LOG_DIR — Write per-node log filesNOMOS_LOG_LEVEL — Global log level (error/warn/info/debug/trace)NOMOS_LOG_FILTER — Target-specific filtering (e.g., cryptarchia=trace,nomos_da_sampling=debug)NOMOS_TESTS_TRACING — Enable file logging for local runnerSee Logging & Observability for details.
+| To Do This | Go Here |
|---|---|
| Run an example | examples/src/bin/ → cargo run -p runner-examples --bin <name> |
| Write a custom scenario | testing-framework/core/ → Implement using ScenarioBuilder |
| Add a new workload | testing-framework/workflows/src/workloads/ → Implement Workload trait |
| Add a new expectation | testing-framework/workflows/src/expectations/ → Implement Expectation trait |
| Modify node configs | testing-framework/configs/src/topology/configs/ |
| Extend builder DSL | testing-framework/workflows/src/builder/ → Add trait methods |
| Add a new deployer | testing-framework/runners/ → Implement Deployer trait |
For detailed guidance, see Internal Crate Reference.
+ +This repository publishes the Rust API documentation (generated via cargo doc) to GitHub Pages under rustdoc/.
cargo doc --workspace --no-deps and open target/doc/index.html.The framework supports two styles for constructing scenarios:
+.transactions_with(...))Both styles produce the same runtime behavior because they ultimately call the same core builder APIs.
+The DSL is implemented as extension traits (primarily testing_framework_workflows::ScenarioBuilderExt) on the core scenario builder.
use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+let plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .wallets(5)
+ .transactions_with(|txs| txs.rate(5).users(3))
+ .da_with(|da| da.channel_rate(1).blob_rate(1).headroom_percent(20))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();When to use:
+Direct instantiation gives you explicit control over the concrete types you attach:
+use std::{
+ num::{NonZeroU64, NonZeroUsize},
+ time::Duration,
+};
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::{
+ expectations::ConsensusLiveness,
+ workloads::{da, transaction},
+};
+
+let tx_workload = transaction::Workload::with_rate(5)
+ .expect("transaction rate must be non-zero")
+ .with_user_limit(NonZeroUsize::new(3));
+
+let da_workload = da::Workload::with_rate(
+ NonZeroU64::new(1).unwrap(), // blob rate per block
+ NonZeroU64::new(1).unwrap(), // channel rate per block
+ da::Workload::default_headroom_percent(),
+);
+
+let plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .wallets(5)
+ .with_workload(tx_workload)
+ .with_workload(da_workload)
+ .with_expectation(ConsensusLiveness::default())
+ .with_run_duration(Duration::from_secs(60))
+ .build();When to use:
+| High-Level DSL | Low-Level Direct |
|---|---|
.transactions_with(|txs| txs.rate(5).users(3)) | .with_workload(transaction::Workload::with_rate(5).expect(...).with_user_limit(...)) |
.da_with(|da| da.blob_rate(1).channel_rate(1)) | .with_workload(da::Workload::with_rate(...)) |
.expect_consensus_liveness() | .with_expectation(ConsensusLiveness::default()) |
Workloads can bundle expectations by implementing Workload::expectations().
These bundled expectations are attached automatically whenever you call .with_workload(...) (including when you use the DSL), because the core builder expands workload expectations during attachment.
Mixing is common: use the DSL for built-ins, and direct instantiation for custom pieces.
+use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::{ScenarioBuilderExt, workloads::transaction};
+
+let tx_workload = transaction::Workload::with_rate(5)
+ .expect("transaction rate must be non-zero");
+
+let plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .wallets(5)
+ .with_workload(tx_workload) // direct instantiation
+ .expect_consensus_liveness() // DSL
+ .with_run_duration(Duration::from_secs(60))
+ .build();The DSL methods are thin wrappers. For example:
+builder.transactions_with(|txs| txs.rate(5).users(3))
is roughly equivalent to:
+builder.transactions().rate(5).users(3).apply()
DSL method not found
+use testing_framework_workflows::ScenarioBuilderExt;The framework follows a clear flow: Topology → Scenario → Deployer → Runner → Workloads → Expectations.
+flowchart LR
+ A(Topology<br/>shape cluster) --> B(Scenario<br/>plan)
+ B --> C(Deployer<br/>provision & readiness)
+ C --> D(Runner<br/>orchestrate execution)
+ D --> E(Workloads<br/>drive traffic)
+ E --> F(Expectations<br/>verify outcomes)
+flowchart TB
+ subgraph Examples["Runner Examples"]
+ LocalBin[local_runner.rs]
+ ComposeBin[compose_runner.rs]
+ K8sBin[k8s_runner.rs]
+ CucumberBin[cucumber_*.rs]
+ end
+
+ subgraph Workflows["Workflows (Batteries Included)"]
+ DSL[ScenarioBuilderExt<br/>Fluent API]
+ TxWorkload[Transaction Workload]
+ DAWorkload[DA Workload]
+ ChaosWorkload[Chaos Workload]
+ Expectations[Built-in Expectations]
+ end
+
+ subgraph Core["Core Framework"]
+ ScenarioModel[Scenario Model]
+ Traits[Deployer + Runner Traits]
+ BlockFeed[BlockFeed]
+ NodeClients[Node Clients]
+ Topology[Topology Generation]
+ end
+
+ subgraph Deployers["Runner Implementations"]
+ LocalDeployer[LocalDeployer]
+ ComposeDeployer[ComposeDeployer]
+ K8sDeployer[K8sDeployer]
+ end
+
+ subgraph Support["Supporting Crates"]
+ Configs[Configs & Topology]
+ Nodes[Node API Clients]
+ Cucumber[Cucumber Extensions]
+ end
+
+ Examples --> Workflows
+ Examples --> Deployers
+ Workflows --> Core
+ Deployers --> Core
+ Deployers --> Support
+ Core --> Support
+ Workflows --> Support
+
+ style Examples fill:#e1f5ff
+ style Workflows fill:#e1ffe1
+ style Core fill:#fff4e1
+ style Deployers fill:#ffe1f5
+ style Support fill:#f0f0f0
+Runner Examples (Entry Points)
+Workflows (High-Level API)
+ScenarioBuilderExt trait provides fluent DSLCore Framework (Foundation)
+Scenario model and lifecycle orchestrationDeployer and Runner traits (extension points)BlockFeed for real-time block observationRunContext providing node clients and metricsRunner Implementations
+LocalDeployer - spawns processes on hostComposeDeployer - orchestrates Docker ComposeK8sDeployer - deploys to Kubernetes clusterDeployer traitSupporting Crates
+configs - Topology configuration and generationnodes - HTTP/RPC client for node APIscucumber - BDD/Gherkin integrationflowchart LR
+ Custom[Your Code] -.implements.-> Workload[Workload Trait]
+ Custom -.implements.-> Expectation[Expectation Trait]
+ Custom -.implements.-> Deployer[Deployer Trait]
+
+ Workload --> Core[Core Framework]
+ Expectation --> Core
+ Deployer --> Core
+
+ style Custom fill:#ffe1f5
+ style Core fill:#fff4e1
+Extend by implementing:
+Workload - Custom traffic generation patternsExpectation - Custom success criteriaDeployer - Support for new deployment targetsSee Extending the Framework for details.
+Each layer has a narrow responsibility so that cluster shape, deployment choice, +traffic generation, and health checks can evolve independently while fitting +together predictably.
+The framework is consumed via runnable example binaries in examples/src/bin/:
local_runner.rs — Spawns nodes as host processescompose_runner.rs — Deploys via Docker Compose (requires NOMOS_TESTNET_IMAGE built)k8s_runner.rs — Deploys via Kubernetes Helm (requires cluster + image)Recommended: Use the convenience script:
+scripts/run/run-examples.sh -t <duration> -v <validators> -e <executors> <mode>
+# mode: host, compose, or k8s
+This handles circuit setup, binary building/bundling, image building, and execution.
+Alternative: Direct cargo run (requires manual setup):
+POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin <name>
+Important: All runners require POL_PROOF_DEV_MODE=true to avoid expensive Groth16 proof generation that causes timeouts.
These binaries use the framework API (ScenarioBuilder) to construct and execute scenarios.
Scenarios are defined using a fluent builder pattern:
+use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn scenario_plan() -> testing_framework_core::scenario::Scenario<()> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .wallets(50)
+ .transactions_with(|txs| txs.rate(5).users(20))
+ .da_with(|da| da.channel_rate(1).blob_rate(2))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(90))
+ .build()
+}Key API Points:
+.topology_with(|t| { t.validators(N).executors(M) }) closure pattern_with closures (transactions_with, da_with, chaos_with).enable_node_control() and a compatible runnerThree deployer implementations:
+| Deployer | Backend | Prerequisites | Node Control |
|---|---|---|---|
LocalDeployer | Host processes | Binaries (built on demand or via bundle) | No |
ComposeDeployer | Docker Compose | Image with embedded assets/binaries | Yes |
K8sDeployer | Kubernetes Helm | Cluster + image loaded | Not yet |
Compose-specific features:
+NOMOS_METRICS_QUERY_URL / NOMOS_METRICS_OTLP_INGEST_URL / NOMOS_GRAFANA_URL as needed)NOMOS_OTLP_ENDPOINT, NOMOS_OTLP_METRICS_ENDPOINT)Built via scripts/build/build_test_image.sh:
testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_paramsrun_nomos_node.sh, run_nomos_executor.shNOMOS_TESTNET_IMAGE (default: logos-blockchain-testing:local)scripts/build/build-bundle.sh --platform linux and set NOMOS_BINARIES_TAR before building imageKZG parameters required for DA workloads:
+testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params (note repeated filename—directory contains file kzgrs_test_params)/kzgrs_test_params/kzgrs_test_params (for compose/k8s)NOMOS_KZGRS_PARAMS_PATH=/custom/path/to/file (must point to file)scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/circuits or use scripts/run/run-examples.shTemplates and configs in testing-framework/runners/compose/assets/:
docker-compose.yml.tera — Stack template (validators, executors)testing-framework/assets/stack/cfgsync.yamltesting-framework/assets/stack/monitoring/Two separate logging pipelines:
+| Component | Configuration | Output |
|---|---|---|
| Runner binaries | RUST_LOG | Framework orchestration logs |
| Node processes | NOMOS_LOG_LEVEL, NOMOS_LOG_FILTER (+ NOMOS_LOG_DIR on host runner) | Consensus, DA, mempool logs |
Node logging:
+NOMOS_TESTS_TRACING=true + NOMOS_LOG_DIR for persistent files.docker logs). To write per-node files, set tracing_settings.logger: !File in testing-framework/assets/stack/cfgsync.yaml (and mount a writable directory).kubectl logs). To write per-node files, set tracing_settings.logger: !File in testing-framework/assets/stack/cfgsync.yaml (and mount a writable directory).File naming: Per-node files use prefix nomos-node-{index} or nomos-executor-{index} (may include timestamps).
Prometheus-compatible metrics querying (optional):
+NOMOS_METRICS_QUERY_URL.ctx.telemetry().prometheus().map(|p| p.base_url())Grafana dashboards (optional):
+testing-framework/assets/stack/monitoring/grafana/dashboards/ and can be imported into your Grafana.NOMOS_GRAFANA_URL, the deployer prints it in TESTNET_ENDPOINTS.Node APIs:
+ctx.node_clients().validator_clients().get(0)OTLP (optional):
+NOMOS_OTLP_ENDPOINT=http://localhost:4317NOMOS_OTLP_METRICS_ENDPOINT=http://localhost:4318For detailed logging configuration, see Logging & Observability.
+ +Creating a scenario is a declarative exercise. This page walks you through the core authoring loop with concrete examples, explains the units and timing model, and shows how to structure scenarios in Rust test suites.
+Every scenario follows the same pattern:
+flowchart LR
+ A[1. Topology] --> B[2. Workloads]
+ B --> C[3. Expectations]
+ C --> D[4. Duration]
+ D --> E[5. Deploy & Run]
+Let's build a minimal consensus liveness test step-by-step.
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+let scenario = ScenarioBuilder::topology_with(|t| {
+ t.network_star() // Star network (one gateway + nodes)
+ .validators(3) // 3 validator nodes
+ .executors(1) // 1 executor node
+})What goes in topology?
+network_star() is currently the only built-in layout)What does NOT go in topology?
+.wallets(20) // Seed funded wallet accounts for transaction workloads
+.transactions_with(|tx| {
+ tx.rate(10) // 10 transactions per block
+ .users(5) // distributed across 5 wallets
+})What goes in workloads?
+Units explained:
+.rate(10) = 10 transactions per block (not per second!).users(5) = use 5 distinct wallet accounts.expect_consensus_liveness()What goes in expectations?
+When do expectations run? +After the duration window ends, during the evaluation phase of the scenario lifecycle.
+use std::time::Duration;
+
+.with_run_duration(Duration::from_secs(60))How long is enough?
+What happens during this window?
+.build();
+
+// Choose a runner
+use testing_framework_core::scenario::Deployer;
+use testing_framework_runner_local::LocalDeployer;
+
+let deployer = LocalDeployer::default();
+let runner = deployer.deploy(&scenario).await?;
+let _result = runner.run(&mut scenario).await?;Putting it all together:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+#[tokio::test]
+async fn hello_consensus_liveness() -> Result<()> {
+ let mut scenario = ScenarioBuilder::topology_with(|t| {
+ t.network_star()
+ .validators(3)
+ .executors(1)
+ })
+ .wallets(20)
+ .transactions_with(|tx| tx.rate(10).users(5))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&scenario).await?;
+ runner.run(&mut scenario).await?;
+
+ Ok(())
+}Run it:
+POL_PROOF_DEV_MODE=true cargo test hello_consensus_liveness
+Wrong mental model: .rate(10) = 10 tx/second
Correct mental model: .rate(10) = 10 tx/block
Why? The blockchain produces blocks at variable rates depending on consensus timing. The framework submits the configured rate per block to ensure predictable load regardless of block time.
+Example:
+.rate(10) → 10 tx/block → 5 tx/second average.rate(10) → 10 tx/block → 2 tx/second average.with_run_duration(Duration::from_secs(60)) means the scenario runs for 60 seconds of real time, not 60 blocks.
How many blocks will be produced? +Depends on consensus timing (slot time, active slot coefficient). Typical: 1-2 seconds per block.
+Rule of thumb:
+// tests/integration_test.rs
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+#[tokio::test]
+async fn test_consensus_liveness() -> Result<()> {
+ let mut scenario = ScenarioBuilder::topology_with(|t| {
+ t.network_star().validators(3).executors(1)
+ })
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(30))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&scenario).await?;
+ runner.run(&mut scenario).await?;
+ Ok(())
+}
+
+#[tokio::test]
+async fn test_transaction_inclusion() -> Result<()> {
+ let mut scenario = ScenarioBuilder::topology_with(|t| {
+ t.network_star().validators(2).executors(1)
+ })
+ .wallets(10)
+ .transactions_with(|tx| tx.rate(5).users(5))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&scenario).await?;
+ runner.run(&mut scenario).await?;
+ Ok(())
+}Extract common topology patterns:
+// tests/helpers.rs
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn minimal_topology() -> ScenarioBuilder {
+ ScenarioBuilder::topology_with(|t| {
+ t.network_star().validators(2).executors(1)
+ })
+}
+
+pub fn production_like_topology() -> ScenarioBuilder {
+ ScenarioBuilder::topology_with(|t| {
+ t.network_star().validators(7).executors(3)
+ })
+}
+
+// tests/consensus_tests.rs
+use std::time::Duration;
+
+use helpers::*;
+
+#[tokio::test]
+async fn small_cluster_liveness() -> anyhow::Result<()> {
+ let mut scenario = minimal_topology()
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(30))
+ .build();
+ // ... deploy and run
+ Ok(())
+}
+
+#[tokio::test]
+async fn large_cluster_liveness() -> anyhow::Result<()> {
+ let mut scenario = production_like_topology()
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+ // ... deploy and run
+ Ok(())
+}Test the same behavior across different scales:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+async fn test_liveness_with_topology(validators: usize, executors: usize) -> Result<()> {
+ let mut scenario = ScenarioBuilder::topology_with(|t| {
+ t.network_star()
+ .validators(validators)
+ .executors(executors)
+ })
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&scenario).await?;
+ runner.run(&mut scenario).await?;
+ Ok(())
+}
+
+#[tokio::test]
+async fn liveness_small() -> Result<()> {
+ test_liveness_with_topology(2, 1).await
+}
+
+#[tokio::test]
+async fn liveness_medium() -> Result<()> {
+ test_liveness_with_topology(5, 2).await
+}
+
+#[tokio::test]
+async fn liveness_large() -> Result<()> {
+ test_liveness_with_topology(10, 3).await
+}Do include:
+.validators(3), .executors(1)).network_star())Don't include:
+Do include:
+.transactions_with(|tx| ...)).da_with(|da| ...)).with_workload(RandomRestartWorkload::new(...)))Don't include:
+Do include:
+.expect_consensus_liveness()).with_expectation(MyExpectation::new()))Don't include:
+test_consensus_survives_validator_restart not test_1.rate(N) is per-block, .with_run_duration() is wall-clockThis page collects proven patterns for authoring, running, and maintaining test scenarios that are reliable, maintainable, and actionable.
+State your intent
+star_topology_3val_2exec vs topology)Keep runs meaningful
+Separate concerns
+Start small, scale up
+Reuse patterns
+Example: Topology preset
+pub fn standard_da_topology() -> GeneratedTopology {
+ TopologyBuilder::new()
+ .network_star()
+ .validators(3)
+ .executors(2)
+ .generate()
+}Example: Shared constants
+pub const STANDARD_TX_RATE: f64 = 10.0;
+pub const STANDARD_DA_CHANNEL_RATE: f64 = 2.0;
+pub const SHORT_RUN_DURATION: Duration = Duration::from_secs(60);
+pub const LONG_RUN_DURATION: Duration = Duration::from_secs(300);Observe first, tune second
+RUST_LOG=debug, NOMOS_LOG_LEVEL=debug) only after initial failureNOMOS_TESTS_KEEP_LOGS=1 to persist logs when debugging failuresUse BlockFeed effectively
+block_feed.stats().total_transactions()) to verify inclusionCollect metrics
+scripts/setup/setup-observability.sh compose up for visualizing node behaviorEnvironment fit
+Runner-specific considerations
+| Runner | When to Use | When to Avoid |
|---|---|---|
| Host | Development iteration, fast feedback | Chaos testing, container-specific issues |
| Compose | CI pipelines, chaos tests, reproducibility | Very large topologies (>10 nodes) |
| K8s | Production-like testing, cluster behaviors | Local development, fast iteration |
Minimal surprises
+versions.env to pin node versions consistently across environmentsUse matrix builds
+strategy:
+ matrix:
+ runner: [host, compose]
+ topology: [small, medium]
+Cache aggressively
+target/)assets/stack/kzgrs_test_params/)Collect logs on failure
+- name: Collect logs on failure
+ if: failure()
+ run: |
+ mkdir -p test-logs
+ find /tmp -name "nomos-*.log" -exec cp {} test-logs/ \;
+- uses: actions/upload-artifact@v3
+ if: failure()
+ with:
+ name: test-logs-${{ matrix.runner }}
+ path: test-logs/
+Time limits
+timeout-minutes: 30See also: CI Integration for complete workflow examples
+DON'T: Run without POL_PROOF_DEV_MODE
+# BAD: Will hang/timeout on proof generation
+cargo run -p runner-examples --bin local_runner
+
+# GOOD: Fast mode for testing
+POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+DON'T: Use tiny durations
+// BAD: Not enough time for blocks to propagate
+.with_run_duration(Duration::from_secs(5))
+
+// GOOD: Allow multiple consensus rounds
+.with_run_duration(Duration::from_secs(60))DON'T: Ignore cleanup failures
+// BAD: Next run inherits leaked state
+runner.run(&mut scenario).await?;
+// forgot to call cleanup or use CleanupGuard
+
+// GOOD: Cleanup via guard (automatic on panic)
+let _cleanup = CleanupGuard::new(runner.clone());
+runner.run(&mut scenario).await?;DON'T: Mix concerns in one scenario
+// BAD: Hard to debug when it fails
+.transactions_with(|tx| tx.rate(50).users(100)) // high load
+.chaos_with(|c| c.restart().min_delay(...)) // AND chaos
+.da_with(|da| da.channel_rate(10).blob_rate(20)) // AND DA stress
+
+// GOOD: Separate tests for each concern
+// Test 1: High transaction load only
+// Test 2: Chaos resilience only
+// Test 3: DA stress onlyDON'T: Hardcode paths or ports
+// BAD: Breaks on different machines
+let path = PathBuf::from("/home/user/circuits/kzgrs_test_params");
+let port = 9000; // might conflict
+
+// GOOD: Use env vars and dynamic allocation
+let path = std::env::var("NOMOS_KZGRS_PARAMS_PATH")
+ .unwrap_or_else(|_| "assets/stack/kzgrs_test_params/kzgrs_test_params".to_string());
+let port = get_available_tcp_port();DON'T: Ignore resource limits
+# BAD: Large topology without checking resources
+scripts/run/run-examples.sh -v 20 -e 10 compose
+# (might OOM or exhaust ulimits)
+
+# GOOD: Scale gradually and monitor resources
+scripts/run/run-examples.sh -v 3 -e 2 compose # start small
+docker stats # monitor resource usage
+# then increase if resources allow
+Minimal viable topology
+Workload rate selection
+Duration guidelines
+| Test Type | Minimum Duration | Typical Duration |
|---|---|---|
| Smoke test | 30s | 60s |
| Integration test | 60s | 120s |
| Load test | 120s | 300s |
| Resilience test | 120s | 300s |
| Soak test | 600s (10m) | 3600s (1h) |
Expectation selection
+| Test Goal | Expectations |
|---|---|
| Basic functionality | expect_consensus_liveness() |
| Transaction handling | expect_consensus_liveness() + custom inclusion check |
| DA correctness | expect_consensus_liveness() + DA dispersal/sampling checks |
| Resilience | expect_consensus_liveness() + recovery time measurement |
Validate scenarios before committing
+Handling flaky tests
+See also:
+++When should I read this? You don't need chaos testing to be productive with the framework. Focus on basic scenarios first—chaos is for resilience validation and operational readiness drills once your core tests are stable.
+
Chaos in the framework uses node control to introduce failures and validate
+recovery. The built-in restart workload lives in
+testing_framework_workflows::workloads::chaos::RandomRestartWorkload.
NodeControlCapability (enable_node_control() in the scenario
+builder) and a runner that provides a NodeControlHandle.NodeControlHandle support (e.g., compose) for chaos workloads.use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::{ScenarioBuilderExt, workloads::chaos::RandomRestartWorkload};
+
+pub fn random_restart_plan() -> testing_framework_core::scenario::Scenario<
+ testing_framework_core::scenario::NodeControlCapability,
+> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(2).executors(1))
+ .enable_node_control()
+ .with_workload(RandomRestartWorkload::new(
+ Duration::from_secs(45), // min delay
+ Duration::from_secs(75), // max delay
+ Duration::from_secs(120), // target cooldown
+ true, // include validators
+ true, // include executors
+ ))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(150))
+ .build()
+}Both LocalDeployer and ComposeDeployer work well in CI environments. Choose based on your tradeoffs.
+LocalDeployer (Host Runner):
+ComposeDeployer (Recommended for CI):
+K8sDeployer:
+Existing Examples:
+See .github/workflows/lint.yml (jobs: host_smoke, compose_smoke) for CI examples running the demo scenarios in this repository.
Here's a comprehensive GitHub Actions workflow demonstrating host and compose runners with caching, matrix testing, and log collection:
+name: Testing Framework CI
+
+on:
+ push:
+ branches: [main, develop]
+ pull_request:
+ branches: [main]
+
+env:
+ POL_PROOF_DEV_MODE: true
+ CARGO_TERM_COLOR: always
+ RUST_BACKTRACE: 1
+
+jobs:
+ # Quick smoke test with host runner (no Docker)
+ host_smoke:
+ name: Host Runner Smoke Test
+ runs-on: ubuntu-latest
+ timeout-minutes: 15
+
+ steps:
+ - name: Checkout repository
+ uses: actions/checkout@v4
+
+ - name: Set up Rust toolchain
+ uses: actions-rs/toolchain@v1
+ with:
+ profile: minimal
+ toolchain: nightly
+ override: true
+
+ - name: Cache Rust dependencies
+ uses: actions/cache@v3
+ with:
+ path: |
+ ~/.cargo/bin/
+ ~/.cargo/registry/index/
+ ~/.cargo/registry/cache/
+ ~/.cargo/git/db/
+ target/
+ key: ${{ runner.os }}-cargo-host-${{ hashFiles('**/Cargo.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-cargo-host-
+
+ - name: Cache nomos-node build
+ uses: actions/cache@v3
+ with:
+ path: |
+ ../nomos-node/target/release/nomos-node
+ ../nomos-node/target/release/nomos-executor
+ key: ${{ runner.os }}-nomos-${{ hashFiles('../nomos-node/**/Cargo.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-nomos-
+
+ - name: Run host smoke test
+ run: |
+ # Use run-examples.sh which handles setup automatically
+ scripts/run/run-examples.sh -t 120 -v 3 -e 1 host
+
+ - name: Upload logs on failure
+ if: failure()
+ uses: actions/upload-artifact@v3
+ with:
+ name: host-runner-logs
+ path: |
+ .tmp/
+ *.log
+ retention-days: 7
+
+ # Compose runner matrix (with Docker)
+ compose_matrix:
+ name: Compose Runner (${{ matrix.topology }})
+ runs-on: ubuntu-latest
+ timeout-minutes: 25
+
+ strategy:
+ fail-fast: false
+ matrix:
+ topology:
+ - "3v1e"
+ - "5v1e"
+
+ steps:
+ - name: Checkout repository
+ uses: actions/checkout@v4
+
+ - name: Set up Rust toolchain
+ uses: actions-rs/toolchain@v1
+ with:
+ profile: minimal
+ toolchain: nightly
+ override: true
+
+ - name: Set up Docker Buildx
+ uses: docker/setup-buildx-action@v2
+
+ - name: Cache Rust dependencies
+ uses: actions/cache@v3
+ with:
+ path: |
+ ~/.cargo/bin/
+ ~/.cargo/registry/index/
+ ~/.cargo/registry/cache/
+ ~/.cargo/git/db/
+ target/
+ key: ${{ runner.os }}-cargo-compose-${{ hashFiles('**/Cargo.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-cargo-compose-
+
+ - name: Cache Docker layers
+ uses: actions/cache@v3
+ with:
+ path: /tmp/.buildx-cache
+ key: ${{ runner.os }}-buildx-${{ hashFiles('Dockerfile', 'scripts/build/build_test_image.sh') }}
+ restore-keys: |
+ ${{ runner.os }}-buildx-
+
+ - name: Run compose test
+ env:
+ TOPOLOGY: ${{ matrix.topology }}
+ run: |
+ # Build and run with the specified topology
+ scripts/run/run-examples.sh -t 120 -v ${TOPOLOGY:0:1} -e ${TOPOLOGY:2:1} compose
+
+ - name: Collect Docker logs on failure
+ if: failure()
+ run: |
+ mkdir -p logs
+ for container in $(docker ps -a --filter "name=nomos-compose-" -q); do
+ docker logs $container > logs/$(docker inspect --format='{{.Name}}' $container).log 2>&1
+ done
+
+ - name: Upload logs and artifacts
+ if: failure()
+ uses: actions/upload-artifact@v3
+ with:
+ name: compose-${{ matrix.topology }}-logs
+ path: |
+ logs/
+ .tmp/
+ retention-days: 7
+
+ - name: Clean up Docker resources
+ if: always()
+ run: |
+ docker compose down -v 2>/dev/null || true
+ docker ps -a --filter "name=nomos-compose-" -q | xargs -r docker rm -f
+
+ # Cucumber/BDD integration tests (if enabled)
+ cucumber_tests:
+ name: Cucumber BDD Tests
+ runs-on: ubuntu-latest
+ timeout-minutes: 20
+
+ steps:
+ - name: Checkout repository
+ uses: actions/checkout@v4
+
+ - name: Set up Rust toolchain
+ uses: actions-rs/toolchain@v1
+ with:
+ profile: minimal
+ toolchain: nightly
+ override: true
+
+ - name: Cache dependencies
+ uses: actions/cache@v3
+ with:
+ path: |
+ ~/.cargo/bin/
+ ~/.cargo/registry/index/
+ ~/.cargo/registry/cache/
+ ~/.cargo/git/db/
+ target/
+ key: ${{ runner.os }}-cargo-cucumber-${{ hashFiles('**/Cargo.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-cargo-cucumber-
+
+ - name: Run Cucumber tests
+ run: |
+ # Build prerequisites
+ scripts/build/build-bundle.sh --platform linux
+ export NOMOS_BINARIES_TAR=$(ls -t .tmp/nomos-binaries-linux-*.tar.gz | head -1)
+
+ # Run Cucumber tests (host runner)
+ cargo test -p runner-examples --bin cucumber_host
+
+ - name: Upload test report
+ if: always()
+ uses: actions/upload-artifact@v3
+ with:
+ name: cucumber-report
+ path: |
+ target/cucumber-reports/
+ retention-days: 14
+
+ # Summary job (requires all tests to pass)
+ ci_success:
+ name: CI Success
+ needs: [host_smoke, compose_matrix, cucumber_tests]
+ runs-on: ubuntu-latest
+ if: always()
+
+ steps:
+ - name: Check all jobs
+ run: |
+ if [[ "${{ needs.host_smoke.result }}" != "success" ]] || \
+ [[ "${{ needs.compose_matrix.result }}" != "success" ]] || \
+ [[ "${{ needs.cucumber_tests.result }}" != "success" ]]; then
+ echo "One or more CI jobs failed"
+ exit 1
+ fi
+ echo "All CI jobs passed!"
+3v1e, 5v1e)Topology Matrix:
+Add more topologies for comprehensive testing:
+matrix:
+ topology:
+ - "3v1e"
+ - "5v1e"
+ - "10v2e" # Larger scale
+Timeout Adjustments:
+Increase timeout-minutes for longer-running scenarios or slower environments:
timeout-minutes: 30 # Instead of 15
+Artifact Retention:
+Change retention-days based on your storage needs:
retention-days: 14 # Keep logs for 2 weeks
+Conditional Execution:
+Run expensive tests only on merge to main:
+if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+Always set POL_PROOF_DEV_MODE=true globally in your workflow env:
env:
+ POL_PROOF_DEV_MODE: true # REQUIRED!
+Without this, tests will hang due to expensive proof generation.
+Prefer scripts/run/run-examples.sh which handles all setup automatically:
scripts/run/run-examples.sh -t 120 -v 3 -e 1 host
+This is more reliable than manual cargo run commands.
Cache Rust dependencies, nomos-node builds, and Docker layers to speed up CI:
+- name: Cache Rust dependencies
+ uses: actions/cache@v3
+ with:
+ path: |
+ ~/.cargo/bin/
+ ~/.cargo/registry/index/
+ ~/.cargo/registry/cache/
+ ~/.cargo/git/db/
+ target/
+ key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
+Always upload logs when tests fail for easier debugging:
+- name: Upload logs on failure
+ if: failure()
+ uses: actions/upload-artifact@v3
+ with:
+ name: test-logs
+ path: |
+ .tmp/
+ *.log
+ retention-days: 7
+For large projects, split host/compose/k8s into separate workflow files:
+.github/workflows/test-host.yml — Fast smoke tests.github/workflows/test-compose.yml — Reproducible integration tests.github/workflows/test-k8s.yml — Production-like validation (nightly)K8s tests are slower. Consider running them only on main branch or scheduled:
+on:
+ push:
+ branches: [main]
+ schedule:
+ - cron: '0 2 * * *' # Daily at 2 AM
+SLOW_TEST_ENV=true if needed)docker system prune)Add debug environment variables temporarily:
+env:
+ RUST_LOG: debug
+ NOMOS_LOG_LEVEL: debug
+Set COMPOSE_RUNNER_PRESERVE=1 to keep containers running for inspection:
- name: Run compose test (preserve on failure)
+ env:
+ COMPOSE_RUNNER_PRESERVE: 1
+ run: scripts/run/run-examples.sh -t 120 -v 3 -e 1 compose
+Download uploaded artifacts from the GitHub Actions UI to inspect logs locally.
+The Logos testing repo includes a small Cucumber (Gherkin) harness for “smoke” scenarios. It is useful when you want readable acceptance-style checks, but it intentionally exposes a limited surface area compared to Rust scenarios.
+testing-framework/cucumber.examples (crate runner-examples):
+cucumber_host (local/host deployer)cucumber_compose (compose deployer)examples/cucumber/features/.local and compose (no k8s runner integration in Cucumber yet).This is the shape used by the repo’s smoke features:
+Feature: Testing Framework - Local Runner
+
+ Scenario: Run a local smoke scenario (tx + DA + liveness)
+ Given deployer is "local"
+ And topology has 1 validators and 1 executors
+ And run duration is 60 seconds
+ And wallets total funds is 1000000000 split across 50 users
+ And transactions rate is 1 per block
+ And data availability channel rate is 1 per block and blob rate is 1 per block
+ And expect consensus liveness
+ When run scenario
+ Then scenario should succeed
+Local runner smoke:
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin cucumber_host
+Compose runner smoke:
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin cucumber_compose
+Topology / runner selection:
+Given deployer is "local"|"compose"Given topology has <validators> validators and <executors> executorsRun configuration:
+Given run duration is <seconds> secondsGiven wallets total funds is <funds> split across <users> usersWorkloads:
+Given transactions rate is <rate> per blockGiven transactions rate is <rate> per block using <users> usersGiven data availability channel rate is <channel_rate> per block and blob rate is <blob_rate> per blockExpectations:
+Given expect consensus livenessGiven consensus liveness lag allowance is <blocks>Execution + assertion:
+When run scenarioThen scenario should succeedPOL_PROOF_DEV_MODE=true for practical runs).A minimal, end-to-end illustration of adding a custom workload and matching +expectation. This shows the shape of the traits and where to plug into the +framework; expand the logic to fit your real test.
+Key ideas:
+RunContext.use async_trait::async_trait;
+use testing_framework_core::{
+ scenario::{DynError, Expectation, RunContext, RunMetrics, Workload},
+ topology::generation::GeneratedTopology,
+};
+
+pub struct ReachabilityWorkload {
+ target_idx: usize,
+}
+
+impl ReachabilityWorkload {
+ pub fn new(target_idx: usize) -> Self {
+ Self { target_idx }
+ }
+}
+
+#[async_trait]
+impl Workload for ReachabilityWorkload {
+ fn name(&self) -> &str {
+ "reachability_workload"
+ }
+
+ fn expectations(&self) -> Vec<Box<dyn Expectation>> {
+ vec![Box::new(
+ crate::custom_workload_example_expectation::ReachabilityExpectation::new(
+ self.target_idx,
+ ),
+ )]
+ }
+
+ fn init(
+ &mut self,
+ topology: &GeneratedTopology,
+ _run_metrics: &RunMetrics,
+ ) -> Result<(), DynError> {
+ if topology.validators().get(self.target_idx).is_none() {
+ return Err(Box::new(std::io::Error::new(
+ std::io::ErrorKind::Other,
+ "no validator at requested index",
+ )));
+ }
+ Ok(())
+ }
+
+ async fn start(&self, ctx: &RunContext) -> Result<(), DynError> {
+ let client = ctx
+ .node_clients()
+ .validator_clients()
+ .get(self.target_idx)
+ .ok_or_else(|| {
+ Box::new(std::io::Error::new(
+ std::io::ErrorKind::Other,
+ "missing target client",
+ )) as DynError
+ })?;
+
+ // Lightweight API call to prove reachability.
+ client
+ .consensus_info()
+ .await
+ .map(|_| ())
+ .map_err(|e| e.into())
+ }
+}Key ideas:
+use async_trait::async_trait;
+use testing_framework_core::scenario::{DynError, Expectation, RunContext};
+
+pub struct ReachabilityExpectation {
+ target_idx: usize,
+}
+
+impl ReachabilityExpectation {
+ pub fn new(target_idx: usize) -> Self {
+ Self { target_idx }
+ }
+}
+
+#[async_trait]
+impl Expectation for ReachabilityExpectation {
+ fn name(&self) -> &str {
+ "target_reachable"
+ }
+
+ async fn evaluate(&mut self, ctx: &RunContext) -> Result<(), DynError> {
+ let client = ctx
+ .node_clients()
+ .validator_clients()
+ .get(self.target_idx)
+ .ok_or_else(|| {
+ Box::new(std::io::Error::new(
+ std::io::ErrorKind::Other,
+ "missing target client",
+ )) as DynError
+ })?;
+
+ client
+ .consensus_info()
+ .await
+ .map(|_| ())
+ .map_err(|e| e.into())
+ }
+}.with_workload(ReachabilityWorkload::new(0))..with_expectation(...) if needed.Quick reference for the scenario builder DSL. All methods are chainable.
+use std::time::Duration;
+
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_compose::ComposeDeployer;
+use testing_framework_runner_k8s::K8sDeployer;
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt};use testing_framework_core::scenario::{Builder, ScenarioBuilder};
+
+pub fn topology() -> Builder<()> {
+ ScenarioBuilder::topology_with(|t| {
+ t.network_star() // Star topology (all connect to seed node)
+ .validators(3) // Number of validator nodes
+ .executors(2) // Number of executor nodes
+ })
+}use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn wallets_plan() -> testing_framework_core::scenario::Scenario<()> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0))
+ .wallets(50) // Seed 50 funded wallet accounts
+ .build()
+}use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn transactions_plan() -> testing_framework_core::scenario::Scenario<()> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0))
+ .wallets(50)
+ .transactions_with(|txs| {
+ txs.rate(5) // 5 transactions per block
+ .users(20) // Use 20 of the seeded wallets
+ }) // Finish transaction workload config
+ .build()
+}use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn da_plan() -> testing_framework_core::scenario::Scenario<()> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(1))
+ .wallets(50)
+ .da_with(|da| {
+ da.channel_rate(1) // number of DA channels to run
+ .blob_rate(2) // target 2 blobs per block (headroom applied)
+ .headroom_percent(20) // optional headroom when sizing channels
+ }) // Finish DA workload config
+ .build()
+}enable_node_control())use std::time::Duration;
+
+use testing_framework_core::scenario::{NodeControlCapability, ScenarioBuilder};
+use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt};
+
+pub fn chaos_plan() -> testing_framework_core::scenario::Scenario<NodeControlCapability> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .enable_node_control() // Enable node control capability
+ .chaos_with(|c| {
+ c.restart() // Random restart chaos
+ .min_delay(Duration::from_secs(30)) // Min time between restarts
+ .max_delay(Duration::from_secs(60)) // Max time between restarts
+ .target_cooldown(Duration::from_secs(45)) // Cooldown after restart
+ .apply() // Required for chaos configuration
+ })
+ .build()
+}use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn expectations_plan() -> testing_framework_core::scenario::Scenario<()> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0))
+ .expect_consensus_liveness() // Assert blocks are produced continuously
+ .build()
+}use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn run_duration_plan() -> testing_framework_core::scenario::Scenario<()> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0))
+ .with_run_duration(Duration::from_secs(120)) // Run for 120 seconds
+ .build()
+}use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn build_plan() -> testing_framework_core::scenario::Scenario<()> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0)).build() // Construct the final Scenario
+}use testing_framework_runner_compose::ComposeDeployer;
+use testing_framework_runner_k8s::K8sDeployer;
+use testing_framework_runner_local::LocalDeployer;
+
+pub fn deployers() {
+ // Local processes
+ let _deployer = LocalDeployer::default();
+
+ // Docker Compose
+ let _deployer = ComposeDeployer::default();
+
+ // Kubernetes
+ let _deployer = K8sDeployer::default();
+}use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub async fn execution() -> Result<()> {
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0))
+ .expect_consensus_liveness()
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub async fn run_test() -> Result<()> {
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .wallets(50)
+ .transactions_with(|txs| {
+ txs.rate(5) // 5 transactions per block
+ .users(20)
+ })
+ .da_with(|da| {
+ da.channel_rate(1) // number of DA channels
+ .blob_rate(2) // target 2 blobs per block
+ .headroom_percent(20) // optional channel headroom
+ })
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(90))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}Complete reference of environment variables used by the testing framework, organized by category.
+These MUST be set for successful test runs:
+| Variable | Required | Default | Effect |
|---|---|---|---|
POL_PROOF_DEV_MODE | YES | — | REQUIRED for all runners. Set to true to use fast dev-mode proving instead of expensive Groth16. Without this, tests will hang/timeout. |
Example:
+export POL_PROOF_DEV_MODE=true
+Or add to your shell profile (~/.bashrc, ~/.zshrc):
# Required for nomos-testing framework
+export POL_PROOF_DEV_MODE=true
+Control which runner to use and the test topology:
+| Variable | Default | Effect |
|---|---|---|
NOMOS_DEMO_VALIDATORS | 1 | Number of validators (all runners) |
NOMOS_DEMO_EXECUTORS | 1 | Number of executors (all runners) |
NOMOS_DEMO_RUN_SECS | 60 | Run duration in seconds (all runners) |
LOCAL_DEMO_VALIDATORS | — | Legacy: Number of validators (host runner only) |
LOCAL_DEMO_EXECUTORS | — | Legacy: Number of executors (host runner only) |
LOCAL_DEMO_RUN_SECS | — | Legacy: Run duration (host runner only) |
COMPOSE_NODE_PAIRS | — | Compose-specific topology format: "validators×executors" (e.g., 3x2) |
Example:
+# Run with 5 validators, 2 executors, for 120 seconds
+NOMOS_DEMO_VALIDATORS=5 \
+NOMOS_DEMO_EXECUTORS=2 \
+NOMOS_DEMO_RUN_SECS=120 \
+scripts/run/run-examples.sh -t 120 -v 5 -e 2 host
+Required for host runner when not using helper scripts:
+| Variable | Required | Default | Effect |
|---|---|---|---|
NOMOS_NODE_BIN | Yes (host) | — | Path to nomos-node binary |
NOMOS_EXECUTOR_BIN | Yes (host) | — | Path to nomos-executor binary |
NOMOS_NODE_PATH | No | — | Path to nomos-node git checkout (dev workflow) |
Example:
+export NOMOS_NODE_BIN=/path/to/nomos-node/target/release/nomos-node
+export NOMOS_EXECUTOR_BIN=/path/to/nomos-node/target/release/nomos-executor
+Required for compose and k8s runners:
+| Variable | Required | Default | Effect |
|---|---|---|---|
NOMOS_TESTNET_IMAGE | Yes (compose/k8s) | logos-blockchain-testing:local | Docker image tag for node containers |
NOMOS_TESTNET_IMAGE_PULL_POLICY | No | IfNotPresent (local) / Always (ECR) | K8s imagePullPolicy used by the runner |
NOMOS_BINARIES_TAR | No | — | Path to prebuilt bundle (.tar.gz) for image build |
NOMOS_SKIP_IMAGE_BUILD | No | 0 | Skip image rebuild (compose/k8s); assumes image already exists |
NOMOS_FORCE_IMAGE_BUILD | No | 0 | Force rebuilding the image even when the script would normally skip it (e.g. non-local k8s) |
Example:
+# Using prebuilt bundle
+export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
+export NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local
+scripts/build/build_test_image.sh
+
+# Using pre-existing image (skip build)
+export NOMOS_SKIP_IMAGE_BUILD=1
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+Circuit asset configuration for DA workloads:
+| Variable | Default | Effect |
|---|---|---|
NOMOS_KZGRS_PARAMS_PATH | testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params | Path to KZG proving key file |
NOMOS_KZG_DIR_REL | testing-framework/assets/stack/kzgrs_test_params | Directory containing KZG assets (relative to workspace root) |
NOMOS_KZG_FILE | kzgrs_test_params | Filename of the proving key within NOMOS_KZG_DIR_REL |
NOMOS_KZG_CONTAINER_PATH | /kzgrs_test_params/kzgrs_test_params | File path where the node expects KZG params inside containers |
NOMOS_KZG_MODE | Runner-specific | K8s only: hostPath (mount from host) or inImage (embed into image) |
NOMOS_KZG_IN_IMAGE_PARAMS_PATH | /opt/nomos/kzg-params/kzgrs_test_params | K8s inImage mode: where the proving key is stored inside the image |
VERSION | From versions.env | Circuit release tag (used by helper scripts) |
NOMOS_CIRCUITS | — | Directory containing fetched circuit bundles (set by scripts/setup/setup-circuits-stack.sh) |
NOMOS_CIRCUITS_VERSION | — | Legacy alias for VERSION (supported by some build scripts) |
NOMOS_CIRCUITS_PLATFORM | Auto-detected | Override circuits platform (e.g. linux-x86_64, macos-aarch64) |
NOMOS_CIRCUITS_HOST_DIR_REL | .tmp/nomos-circuits-host | Output dir for host circuits bundle (relative to repo root) |
NOMOS_CIRCUITS_LINUX_DIR_REL | .tmp/nomos-circuits-linux | Output dir for linux circuits bundle (relative to repo root) |
NOMOS_CIRCUITS_NONINTERACTIVE | 0 | Set to 1 to overwrite outputs without prompting in setup scripts |
NOMOS_CIRCUITS_REBUILD_RAPIDSNARK | 0 | Set to 1 to force rebuilding rapidsnark (host bundle only) |
Example:
+# Use custom circuit assets
+NOMOS_KZGRS_PARAMS_PATH=/custom/path/to/kzgrs_test_params \
+cargo run -p runner-examples --bin local_runner
+Control node log output (not framework runner logs):
+| Variable | Default | Effect |
|---|---|---|
NOMOS_LOG_LEVEL | info | Global log level: error, warn, info, debug, trace |
NOMOS_LOG_FILTER | — | Fine-grained module filtering (e.g., cryptarchia=trace,nomos_da_sampling=debug) |
NOMOS_LOG_DIR | — | Host runner: directory for per-node log files (persistent). Compose/k8s: use cfgsync.yaml for file logging. |
NOMOS_TESTS_KEEP_LOGS | 0 | Keep per-run temporary directories (useful for debugging/CI artifacts) |
NOMOS_TESTS_TRACING | false | Enable debug tracing preset (combine with NOMOS_LOG_DIR unless external tracing backends configured) |
Important: Node logging ignores RUST_LOG; use NOMOS_LOG_LEVEL and NOMOS_LOG_FILTER for node logs.
Example:
+# Debug logging to files
+NOMOS_LOG_DIR=/tmp/test-logs \
+NOMOS_LOG_LEVEL=debug \
+NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug" \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin local_runner
+
+# Inspect logs
+ls /tmp/test-logs/
+# nomos-node-0.2024-12-18T14-30-00.log
+# nomos-node-1.2024-12-18T14-30-00.log
+Common filter targets:
+| Target Prefix | Subsystem |
|---|---|
cryptarchia | Consensus (Cryptarchia) |
nomos_da_sampling | DA sampling service |
nomos_da_dispersal | DA dispersal service |
nomos_da_verifier | DA verification |
nomos_blend | Mix network/privacy layer |
chain_service | Chain service (node APIs/state) |
chain_network | P2P networking |
chain_leader | Leader election |
Optional observability integration:
+| Variable | Default | Effect |
|---|---|---|
NOMOS_METRICS_QUERY_URL | — | Prometheus-compatible base URL for runner to query (e.g., http://localhost:9090) |
NOMOS_METRICS_OTLP_INGEST_URL | — | Full OTLP HTTP ingest URL for node metrics export (e.g., http://localhost:9090/api/v1/otlp/v1/metrics) |
NOMOS_GRAFANA_URL | — | Grafana base URL for printing/logging (e.g., http://localhost:3000) |
NOMOS_OTLP_ENDPOINT | — | OTLP trace endpoint (optional) |
NOMOS_OTLP_METRICS_ENDPOINT | — | OTLP metrics endpoint (optional) |
Example:
+# Enable Prometheus querying
+export NOMOS_METRICS_QUERY_URL=http://localhost:9090
+export NOMOS_METRICS_OTLP_INGEST_URL=http://localhost:9090/api/v1/otlp/v1/metrics
+export NOMOS_GRAFANA_URL=http://localhost:3000
+
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+Variables specific to Docker Compose deployment:
+| Variable | Default | Effect |
|---|---|---|
COMPOSE_RUNNER_HOST | 127.0.0.1 | Host address for port mappings |
COMPOSE_RUNNER_PRESERVE | 0 | Keep containers running after test (for debugging) |
COMPOSE_RUNNER_HTTP_TIMEOUT_SECS | — | Override HTTP readiness timeout (seconds) |
COMPOSE_RUNNER_HOST_GATEWAY | host.docker.internal:host-gateway | Controls extra_hosts entry injected into compose (set to disable to omit) |
TESTNET_RUNNER_PRESERVE | — | Alias for COMPOSE_RUNNER_PRESERVE |
Example:
+# Keep containers after test for debugging
+COMPOSE_RUNNER_PRESERVE=1 \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+
+# Containers remain running
+docker ps --filter "name=nomos-compose-"
+docker logs <container-id>
+Variables specific to Kubernetes deployment:
+| Variable | Default | Effect |
|---|---|---|
K8S_RUNNER_NAMESPACE | Random UUID | Kubernetes namespace (pin for debugging) |
K8S_RUNNER_RELEASE | Random UUID | Helm release name (pin for debugging) |
K8S_RUNNER_NODE_HOST | — | NodePort host resolution for non-local clusters |
K8S_RUNNER_DEBUG | 0 | Log Helm stdout/stderr for install commands |
K8S_RUNNER_PRESERVE | 0 | Keep namespace/release after run (for debugging) |
K8S_RUNNER_DEPLOYMENT_TIMEOUT_SECS | — | Override deployment readiness timeout |
K8S_RUNNER_HTTP_TIMEOUT_SECS | — | Override HTTP readiness timeout (port-forwards) |
K8S_RUNNER_HTTP_PROBE_TIMEOUT_SECS | — | Override HTTP readiness timeout (NodePort probes) |
K8S_RUNNER_PROMETHEUS_HTTP_TIMEOUT_SECS | — | Override Prometheus readiness timeout |
K8S_RUNNER_PROMETHEUS_HTTP_PROBE_TIMEOUT_SECS | — | Override Prometheus NodePort probe timeout |
Example:
+# Pin namespace for debugging
+K8S_RUNNER_NAMESPACE=nomos-test-debug \
+K8S_RUNNER_PRESERVE=1 \
+K8S_RUNNER_DEBUG=1 \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
+
+# Inspect resources
+kubectl get pods -n nomos-test-debug
+kubectl logs -n nomos-test-debug -l nomos/logical-role=validator
+Platform-specific build configuration:
+| Variable | Default | Effect |
|---|---|---|
NOMOS_BUNDLE_DOCKER_PLATFORM | Host arch | Docker platform for bundle builds: linux/arm64 or linux/amd64 (macOS/Windows hosts) |
NOMOS_BIN_PLATFORM | — | Legacy alias for NOMOS_BUNDLE_DOCKER_PLATFORM |
COMPOSE_CIRCUITS_PLATFORM | Host arch | Circuits platform for image builds: linux-aarch64 or linux-x86_64 |
NOMOS_EXTRA_FEATURES | — | Extra cargo features to enable when building bundles (used by scripts/build/build-bundle.sh) |
macOS / Apple Silicon:
+# Native performance (recommended for local testing)
+export NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64
+
+# Or target amd64 (slower via emulation)
+export NOMOS_BUNDLE_DOCKER_PLATFORM=linux/amd64
+Timeout and performance tuning:
+| Variable | Default | Effect |
|---|---|---|
SLOW_TEST_ENV | false | Doubles built-in readiness timeouts (useful in CI / constrained laptops) |
TESTNET_PRINT_ENDPOINTS | 0 | Print TESTNET_ENDPOINTS / TESTNET_PPROF lines during deploy (set automatically by scripts/run/run-examples.sh) |
NOMOS_DISPERSAL_TIMEOUT_SECS | 20 | DA dispersal timeout (seconds) |
NOMOS_RETRY_COOLDOWN_SECS | 3 | Cooldown between retries (seconds) |
NOMOS_GRACE_PERIOD_SECS | 1200 | Grace period before enforcing strict time-based expectations (seconds) |
NOMOS_PRUNE_DURATION_SECS | 30 | Prune step duration (seconds) |
NOMOS_PRUNE_INTERVAL_SECS | 5 | Interval between prune cycles (seconds) |
NOMOS_SHARE_DURATION_SECS | 5 | Share duration (seconds) |
NOMOS_COMMITMENTS_WAIT_SECS | 1 | Commitments wait duration (seconds) |
NOMOS_SDP_TRIGGER_DELAY_SECS | 5 | SDP trigger delay (seconds) |
Example:
+# Increase timeouts for slow environments
+SLOW_TEST_ENV=true \
+scripts/run/run-examples.sh -t 120 -v 5 -e 2 compose
+Node-level configuration passed through to nomos-node/nomos-executor:
+| Variable | Default | Effect |
|---|---|---|
CONSENSUS_SLOT_TIME | — | Consensus slot time (seconds) |
CONSENSUS_ACTIVE_SLOT_COEFF | — | Active slot coefficient (0.0-1.0) |
NOMOS_USE_AUTONAT | Unset | If set, use AutoNAT instead of a static loopback address for libp2p NAT settings |
NOMOS_CFGSYNC_PORT | 4400 | Port used for cfgsync service inside the stack |
NOMOS_TIME_BACKEND | monotonic | Select time backend (used by compose/k8s stack scripts and deployers) |
Example:
+# Faster block production
+CONSENSUS_SLOT_TIME=5 \
+CONSENSUS_ACTIVE_SLOT_COEFF=0.9 \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin local_runner
+Control framework runner process logs (uses RUST_LOG, not NOMOS_*):
| Variable | Default | Effect |
|---|---|---|
RUST_LOG | — | Framework runner log level (e.g., debug, info) |
RUST_BACKTRACE | — | Enable Rust backtraces on panic (1 or full) |
CARGO_TERM_COLOR | — | Cargo output color (always, never, auto) |
Example:
+# Debug framework runner (not nodes)
+RUST_LOG=debug \
+RUST_BACKTRACE=1 \
+cargo run -p runner-examples --bin local_runner
+Variables used by helper scripts (scripts/run/run-examples.sh, etc.):
| Variable | Default | Effect |
|---|---|---|
NOMOS_NODE_REV | From versions.env | nomos-node git revision to build/fetch |
NOMOS_BUNDLE_VERSION | From versions.env | Bundle schema version |
NOMOS_IMAGE_SELECTION | — | Internal: image selection mode set by run-examples.sh (local/ecr/auto) |
NOMOS_NODE_APPLY_PATCHES | 1 | Set to 0 to disable applying local patches when building bundles |
NOMOS_NODE_PATCH_DIR | patches/nomos-node | Patch directory applied to nomos-node checkout during bundle builds |
NOMOS_NODE_PATCH_LEVEL | — | Patch application level (all or an integer) for bundle builds |
POL_PROOF_DEV_MODE=true \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
+POL_PROOF_DEV_MODE=true \
+NOMOS_LOG_DIR=/tmp/logs \
+NOMOS_LOG_LEVEL=debug \
+NOMOS_LOG_FILTER="cryptarchia=trace" \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
+POL_PROOF_DEV_MODE=true \
+NOMOS_METRICS_QUERY_URL=http://localhost:9090 \
+NOMOS_GRAFANA_URL=http://localhost:3000 \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+POL_PROOF_DEV_MODE=true \
+K8S_RUNNER_NAMESPACE=nomos-debug \
+K8S_RUNNER_DEBUG=1 \
+K8S_RUNNER_PRESERVE=1 \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
+env:
+ POL_PROOF_DEV_MODE: true
+ RUST_BACKTRACE: 1
+ NOMOS_TESTS_KEEP_LOGS: 1
+++When should I read this? Skim now to see what's possible, revisit later when you need load testing, chaos scenarios, or custom extensions. Start with basic examples first.
+
Realistic advanced scenarios demonstrating framework capabilities for production testing.
+Adapt from Complete Source:
+| Example | Topology | Workloads | Deployer | Key Feature |
|---|---|---|---|---|
| Load Progression | 3 validators + 2 executors | Increasing tx rate | Compose | Dynamic load testing |
| Sustained Load | 4 validators + 2 executors | High tx + DA rate | Compose | Stress testing |
| Aggressive Chaos | 4 validators + 2 executors | Frequent restarts + traffic | Compose | Resilience validation |
Test consensus under progressively increasing transaction load:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_compose::ComposeDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub async fn load_progression_test() -> Result<()> {
+ for rate in [5, 10, 20, 30] {
+ println!("Testing with rate: {}", rate);
+
+ let mut plan =
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .wallets(50)
+ .transactions_with(|txs| txs.rate(rate).users(20))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+
+ let deployer = ComposeDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+ }
+
+ Ok(())
+}When to use: Finding the maximum sustainable transaction rate for a given topology.
+Run high transaction and DA load for extended duration:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_compose::ComposeDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub async fn sustained_load_test() -> Result<()> {
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(4).executors(2))
+ .wallets(100)
+ .transactions_with(|txs| txs.rate(15).users(50))
+ .da_with(|da| da.channel_rate(2).blob_rate(3))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(300))
+ .build();
+
+ let deployer = ComposeDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}When to use: Validating stability under continuous high load over extended periods.
+Frequent node restarts with active traffic:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_compose::ComposeDeployer;
+use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt};
+
+pub async fn aggressive_chaos_test() -> Result<()> {
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(4).executors(2))
+ .enable_node_control()
+ .wallets(50)
+ .transactions_with(|txs| txs.rate(10).users(20))
+ .chaos_with(|c| {
+ c.restart()
+ .min_delay(Duration::from_secs(10))
+ .max_delay(Duration::from_secs(20))
+ .target_cooldown(Duration::from_secs(15))
+ .apply()
+ })
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(180))
+ .build();
+
+ let deployer = ComposeDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}When to use: Validating recovery and liveness under aggressive failure conditions.
+Note: Requires ComposeDeployer for node control support.
These scenarios require custom implementations but demonstrate framework extensibility:
+Concept: Submit the same batch of independent transactions to different nodes in randomized order/offsets, then verify all transactions are included and final state matches across nodes.
+Requirements:
+ctx.node_clients(), with randomized submission order and timing offsets per nodeWhy useful: Exercises mempool propagation, proposer fairness, and transaction inclusion guarantees under realistic race conditions. Tests that the protocol maintains consistency regardless of which node receives transactions first.
+Implementation notes: Requires both a custom Workload implementation (to submit same transactions to multiple nodes with jitter) and a custom Expectation implementation (to verify inclusion and state consistency).
Concept: Drive different transaction subsets into different validators (or differing arrival orders) to create temporary mempool divergence, then verify mempools/blocks converge to contain the union (no permanent divergence).
+Requirements:
+ctx.node_clients() with disjoint or jittered transaction batchesExpectations:
+Why useful: Exercises mempool gossip and reconciliation under uneven input or latency. Ensures no node "drops" transactions seen elsewhere, validating that mempool synchronization mechanisms correctly propagate transactions across the network even when they arrive at different nodes in different orders.
+Implementation notes: Requires both a custom Workload implementation (to inject disjoint/jittered batches per node) and a custom Expectation implementation (to verify mempool convergence or block inclusion). Uses existing ctx.node_clients() capability—no new infrastructure needed.
Concept: Ramp transaction load over time to observe mempool growth, fee prioritization/eviction, and block saturation behavior, detecting performance regressions and ensuring backpressure/eviction work under increasing load.
+Requirements:
+Expectations:
+Why useful: Detects performance regressions in mempool management. Ensures backpressure and eviction mechanisms work correctly under increasing load, preventing memory exhaustion or unbounded growth. Validates that fee prioritization correctly selects high-value transactions when mempool is full.
+Implementation notes: Can be built with current workload model (ramping rate). Requires custom Expectation implementation that reads mempool metrics (via node HTTP APIs or Prometheus) and monitors throughput to judge behavior. No new infrastructure needed—uses existing observability capabilities.
Concept: Submit malformed transactions and verify they're rejected properly.
+Implementation approach:
+Why useful: Ensures mempool doesn't crash or include invalid transactions under fuzzing.
+Concept: Test consensus robustness under skewed gossip delays by partitioning nodes into latency tiers (tier A ≈10ms, tier B ≈100ms, tier C ≈300ms) and observing propagation lag, fork rate, and eventual convergence.
+Requirements:
+netem/iptables in compose; NetworkPolicy + netem sidecar in k8sExpectations:
+Why useful: Real networks have heterogeneous latency. This stress-tests proposer selection and fork resolution when some peers are "far" (high latency), validating that consensus remains safe and live under realistic network conditions.
+Current blocker: Runner support for per-group delay injection (network delay via netem/iptables) is not present today. Would require new chaos plumbing in compose/k8s deployers to inject network delays per node group.
Concept: Spin up a custom workload/sidecar that runs a libp2p host, joins the cluster's gossip mesh, and publishes a high rate of syntactically valid but useless/stale messages to selected topics, testing gossip backpressure, scoring, and queue handling under a "malicious" peer.
+Requirements:
+Expectations:
+Why useful: Tests Byzantine behavior (malicious peer) which is critical for consensus protocol robustness. More realistic than RPC spam since it uses the actual gossip protocol. Validates that gossip backpressure, peer scoring, and queue management correctly handle adversarial peers without disrupting consensus.
+Current blocker: Requires adding gossip-capable helper (libp2p integration) to the framework. Would need a custom workload/sidecar implementation that can join the gossip mesh and inject messages. The rest of the scenario can use existing runners/workloads.
+Concept: Test consensus recovery after network partitions.
+Requirements:
+block_peer() / unblock_peer() methods in NodeControlHandleWhy useful: Tests the most realistic failure mode in distributed systems.
+Current blocker: Node control doesn't yet support network-level actions (only process restarts).
+Concept: Test consensus and timestamp handling when nodes run with skewed clocks (e.g., +1s, −1s, +200ms jitter) to surface timestamp validation issues, reorg sensitivity, and clock drift handling.
+Requirements:
+Expectations:
+Why useful: Clock skew is a common real-world issue in distributed systems. This validates that consensus correctly handles timestamp validation and maintains safety/liveness when nodes have different clock offsets, preventing timestamp-based attacks or failures.
+Current blocker: Runner ability to skew per-node clocks (e.g., privileged containers with libfaketime/chrony or time-offset netns) is not available today. Would require a new chaos/time-skew hook in deployers to inject clock offsets per node.
Concept: Verify block production intervals stay within expected bounds.
+Implementation approach:
+BlockFeed(slot_duration * active_slot_coeff) ± toleranceWhy useful: Validates consensus timing under various loads.
+Concept: Nodes join and leave mid-run (new identities/addresses added; some nodes permanently removed) to exercise peer discovery, bootstrapping, reputation, and load balancing under churn.
+Requirements:
+Expectations:
+Why useful: Real networks experience churn (nodes joining/leaving). Unlike restarts (which preserve topology), churn changes the actual topology size and peer set, testing how the protocol handles dynamic membership. This exercises peer discovery, bootstrapping, reputation systems, and load balancing under realistic conditions.
+Current blocker: Runner support for dynamic node addition/removal at runtime is not available today. Chaos today only restarts existing nodes; churn would require the ability to spin up new nodes with fresh identities/addresses, update peer lists/bootstraps dynamically, and permanently remove nodes. Would need new topology management capabilities in deployers.
+Concept: Adversarial workload floods node HTTP/WS APIs with high QPS and malformed/bursty requests; expectation checks nodes remain responsive or rate-limit without harming consensus.
+Requirements:
+Expectations:
+Why useful: Validates API hardening under abuse and ensures control/telemetry endpoints don't destabilize the node. Tests that API abuse is properly isolated from consensus operations, preventing DoS attacks on API endpoints from affecting blockchain functionality.
+Implementation notes: Requires custom Workload implementation that directs high-QPS traffic to node APIs (via ctx.node_clients() or direct HTTP clients) and custom Expectation implementation that monitors API responsiveness metrics and consensus liveness. Uses existing node API access—no new infrastructure needed.
Concept: Track wallet balances and verify state consistency.
+Description: After transaction workload completes, query all wallet balances via node API and verify total supply is conserved. Requires tracking initial state, submitted transactions, and final balances. Validates that the ledger maintains correctness under load (no funds lost or created). This is a state assertion expectation that checks correctness, not just liveness.
+ +Concrete scenario shapes that illustrate how to combine topologies, workloads, +and expectations.
+View Complete Source Code:
+Runnable examples: The repo includes complete binaries in examples/src/bin/:
local_runner.rs — Host processes (local)compose_runner.rs — Docker Compose (requires image built)k8s_runner.rs — Kubernetes (requires cluster access and image loaded)Recommended: Use scripts/run/run-examples.sh -t <duration> -v <validators> -e <executors> <mode> where mode is host, compose, or k8s.
Alternative: Direct cargo run: POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin <name>
All runners require POL_PROOF_DEV_MODE=true to avoid expensive proof generation.
Code patterns below show how to build scenarios. Wrap these in #[tokio::test] functions for integration tests, or #[tokio::main] for binaries.
Minimal test that validates basic block production:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub async fn simple_consensus() -> Result<()> {
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(0))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(30))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}When to use: smoke tests for consensus on minimal hardware.
+Test consensus under transaction load:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub async fn transaction_workload() -> Result<()> {
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(2).executors(0))
+ .wallets(20)
+ .transactions_with(|txs| txs.rate(5).users(10))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}When to use: validate transaction submission and inclusion.
+Combined test stressing both transaction and DA layers:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub async fn da_and_transactions() -> Result<()> {
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .wallets(30)
+ .transactions_with(|txs| txs.rate(5).users(15))
+ .da_with(|da| da.channel_rate(2).blob_rate(2))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(90))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}When to use: end-to-end coverage of transaction and DA layers.
+Test system resilience under node restarts:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_compose::ComposeDeployer;
+use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt};
+
+pub async fn chaos_resilience() -> Result<()> {
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(4).executors(2))
+ .enable_node_control()
+ .wallets(20)
+ .transactions_with(|txs| txs.rate(3).users(10))
+ .chaos_with(|c| {
+ c.restart()
+ .min_delay(Duration::from_secs(20))
+ .max_delay(Duration::from_secs(40))
+ .target_cooldown(Duration::from_secs(30))
+ .apply()
+ })
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(120))
+ .build();
+
+ let deployer = ComposeDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}When to use: resilience validation and operational readiness drills.
+Note: Chaos tests require ComposeDeployer or another runner with node control support.
This guide shows how to extend the framework with custom workloads, expectations, runners, and topology helpers. Each section includes the trait outline and a minimal code example.
+Steps:
+testing_framework_core::scenario::Workloadinit to derive inputs from topology/metrics; fail fast if prerequisites missingstart to drive async traffic using RunContext clientstesting-framework/workflows and optionally add a DSL helperTrait outline:
+use async_trait::async_trait;
+use testing_framework_core::scenario::{
+ DynError, Expectation, RunContext, RunMetrics, Workload,
+};
+use testing_framework_core::topology::generation::GeneratedTopology;
+
+struct MyExpectation;
+
+#[async_trait]
+impl Expectation for MyExpectation {
+ fn name(&self) -> &str {
+ "my_expectation"
+ }
+
+ async fn evaluate(&mut self, _ctx: &RunContext) -> Result<(), DynError> {
+ Ok(())
+ }
+}
+
+pub struct MyWorkload {
+ // Configuration fields
+ target_rate: u64,
+}
+
+impl MyWorkload {
+ pub fn new(target_rate: u64) -> Self {
+ Self { target_rate }
+ }
+}
+
+#[async_trait]
+impl Workload for MyWorkload {
+ fn name(&self) -> &str {
+ "my_workload"
+ }
+
+ fn expectations(&self) -> Vec<Box<dyn Expectation>> {
+ // Return bundled expectations that should run with this workload
+ vec![Box::new(MyExpectation)]
+ }
+
+ fn init(
+ &mut self,
+ topology: &GeneratedTopology,
+ _run_metrics: &RunMetrics,
+ ) -> Result<(), DynError> {
+ // Validate prerequisites (e.g., enough nodes, wallet data present)
+ if topology.validators().is_empty() {
+ return Err("no validators available".into());
+ }
+ Ok(())
+ }
+
+ async fn start(&self, ctx: &RunContext) -> Result<(), DynError> {
+ // Drive async activity: submit transactions, query nodes, etc.
+ let clients = ctx.node_clients().validator_clients();
+
+ for client in clients {
+ let info = client.consensus_info().await?;
+ tracing::info!(height = info.height, "workload queried node");
+ }
+
+ Ok(())
+ }
+}Key points:
+name() identifies the workload in logsexpectations() bundles default checks (can be empty)init() validates topology before run startsstart() executes concurrently with other workloads; it should complete before run duration expiresSee Example: New Workload & Expectation for a complete, runnable example.
+Steps:
+testing_framework_core::scenario::Expectationstart_capture to snapshot baseline metrics (optional)evaluate to assert outcomes after workloads finishtesting-framework/workflows if reusableTrait outline:
+use async_trait::async_trait;
+use testing_framework_core::scenario::{DynError, Expectation, RunContext};
+
+pub struct MyExpectation {
+ expected_value: u64,
+ captured_baseline: Option<u64>,
+}
+
+impl MyExpectation {
+ pub fn new(expected_value: u64) -> Self {
+ Self {
+ expected_value,
+ captured_baseline: None,
+ }
+ }
+}
+
+#[async_trait]
+impl Expectation for MyExpectation {
+ fn name(&self) -> &str {
+ "my_expectation"
+ }
+
+ async fn start_capture(&mut self, ctx: &RunContext) -> Result<(), DynError> {
+ // Optional: capture baseline state before workloads start
+ let client = ctx.node_clients().validator_clients().first()
+ .ok_or("no validators")?;
+
+ let info = client.consensus_info().await?;
+ self.captured_baseline = Some(info.height);
+
+ tracing::info!(baseline = self.captured_baseline, "captured baseline");
+ Ok(())
+ }
+
+ async fn evaluate(&mut self, ctx: &RunContext) -> Result<(), DynError> {
+ // Assert the expected condition holds after workloads finish
+ let client = ctx.node_clients().validator_clients().first()
+ .ok_or("no validators")?;
+
+ let info = client.consensus_info().await?;
+ let final_height = info.height;
+
+ let baseline = self.captured_baseline.unwrap_or(0);
+ let delta = final_height.saturating_sub(baseline);
+
+ if delta < self.expected_value {
+ return Err(format!(
+ "expected at least {} blocks, got {}",
+ self.expected_value, delta
+ ).into());
+ }
+
+ tracing::info!(delta, "expectation passed");
+ Ok(())
+ }
+}Key points:
+name() identifies the expectation in logsstart_capture() runs before workloads start (optional)evaluate() runs after workloads finish; return descriptive errorsSteps:
+testing_framework_core::scenario::Deployer<Caps> for your capability typeRunnerNodeClients and spawn a BlockFeedRunContext and provide a CleanupGuard for teardownTrait outline:
+use async_trait::async_trait;
+use testing_framework_core::scenario::{
+ CleanupGuard, Deployer, DynError, Metrics, NodeClients, RunContext, Runner, Scenario,
+ spawn_block_feed,
+};
+use testing_framework_core::topology::deployment::Topology;
+
+pub struct MyDeployer {
+ // Configuration: cluster connection details, etc.
+}
+
+impl MyDeployer {
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+#[async_trait]
+impl Deployer<()> for MyDeployer {
+ type Error = DynError;
+
+ async fn deploy(&self, scenario: &Scenario<()>) -> Result<Runner, Self::Error> {
+ // 1. Launch nodes using scenario.topology()
+ // 2. Wait for readiness (e.g., consensus info endpoint responds)
+ // 3. Build NodeClients for validators/executors
+ // 4. Spawn a block feed for expectations (optional but recommended)
+ // 5. Create NodeControlHandle if you support restarts (optional)
+ // 6. Return a Runner wrapping RunContext + CleanupGuard
+
+ tracing::info!("deploying scenario with MyDeployer");
+
+ let topology: Option<Topology> = None; // Some(topology) if you spawned one
+ let node_clients = NodeClients::default(); // Or NodeClients::from_topology(...)
+
+ let client = node_clients
+ .any_client()
+ .ok_or("no api clients available")?
+ .clone();
+ let (block_feed, block_feed_guard) = spawn_block_feed(client).await?;
+
+ let telemetry = Metrics::empty(); // or Metrics::from_prometheus(...)
+ let node_control = None; // or Some(Arc<dyn NodeControlHandle>)
+
+ let context = RunContext::new(
+ scenario.topology().clone(),
+ topology,
+ node_clients,
+ scenario.duration(),
+ telemetry,
+ block_feed,
+ node_control,
+ );
+
+ // If you also have other resources to clean up (containers/pods/etc),
+ // wrap them in your own CleanupGuard implementation and call
+ // CleanupGuard::cleanup(Box::new(block_feed_guard)) inside it.
+ Ok(Runner::new(context, Some(Box::new(block_feed_guard))))
+ }
+}Key points:
+deploy() must return a fully prepared RunnerCleanupGuard to tear down resources on failure (and on RunHandle drop)NodeControlHandle via RunContextSteps:
+testing_framework_core::topology::config::TopologyBuilder with new layoutsExample:
+use testing_framework_core::topology::{
+ config::TopologyBuilder,
+ configs::network::Libp2pNetworkLayout,
+};
+
+pub trait TopologyBuilderExt {
+ fn network_full(self) -> Self;
+}
+
+impl TopologyBuilderExt for TopologyBuilder {
+ fn network_full(self) -> Self {
+ self.with_network_layout(Libp2pNetworkLayout::Full)
+ }
+}Key points:
+&mut Self)To expose your custom workload through the high-level DSL, add a trait extension:
+use async_trait::async_trait;
+use testing_framework_core::scenario::{DynError, RunContext, ScenarioBuilder, Workload};
+
+#[derive(Default)]
+pub struct MyWorkloadBuilder {
+ target_rate: u64,
+ some_option: bool,
+}
+
+impl MyWorkloadBuilder {
+ pub const fn target_rate(mut self, target_rate: u64) -> Self {
+ self.target_rate = target_rate;
+ self
+ }
+
+ pub const fn some_option(mut self, some_option: bool) -> Self {
+ self.some_option = some_option;
+ self
+ }
+
+ pub const fn build(self) -> MyWorkload {
+ MyWorkload {
+ target_rate: self.target_rate,
+ some_option: self.some_option,
+ }
+ }
+}
+
+pub struct MyWorkload {
+ target_rate: u64,
+ some_option: bool,
+}
+
+#[async_trait]
+impl Workload for MyWorkload {
+ fn name(&self) -> &str {
+ "my_workload"
+ }
+
+ async fn start(&self, _ctx: &RunContext) -> Result<(), DynError> {
+ Ok(())
+ }
+}
+
+pub trait MyWorkloadDsl {
+ fn my_workload_with(
+ self,
+ f: impl FnOnce(MyWorkloadBuilder) -> MyWorkloadBuilder,
+ ) -> Self;
+}
+
+impl MyWorkloadDsl for ScenarioBuilder {
+ fn my_workload_with(
+ self,
+ f: impl FnOnce(MyWorkloadBuilder) -> MyWorkloadBuilder,
+ ) -> Self {
+ let builder = f(MyWorkloadBuilder::default());
+ self.with_workload(builder.build())
+ }
+}Users can then call:
+ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(1))
+ .my_workload_with(|w| {
+ w.target_rate(10)
+ .some_option(true)
+ })
+ .build()Why block-oriented timing?
+Slots advance at a fixed rate (NTP-synchronized, 2s by default), so reasoning
+about blocks and consensus intervals keeps assertions aligned with protocol
+behavior rather than arbitrary wall-clock durations.
Can I reuse the same scenario across runners?
+Yes. The plan stays the same; swap runners (local, compose, k8s) to target
+different environments.
When should I enable chaos workloads?
+Only when testing resilience or operational recovery; keep functional smoke
+tests deterministic.
How long should runs be?
+The framework enforces a minimum of 2× slot duration (4 seconds with default 2s slots), but practical recommendations:
Very short runs (< 30s) risk false confidence—one or two lucky blocks don't prove liveness.
+Do I always need seeded wallets?
+Only for transaction scenarios. Data-availability or pure chaos scenarios may
+not require them, but liveness checks still need validators producing blocks.
What if expectations fail but workloads “look fine”?
+Trust expectations first—they capture the intended success criteria. Use the
+observability signals and runner logs to pinpoint why the system missed the
+target.
with_run_duration(). Framework auto-extends to at least 2× slot duration.Declarative, multi-node blockchain testing for the Logos network
+The Logos Testing Framework enables you to test consensus, data availability, and transaction workloads across local processes, Docker Compose, and Kubernetes deployments—all with a unified scenario API.
+ +Everything in this framework is a Scenario.
+A Scenario is a controlled experiment over time, composed of:
+This single abstraction makes tests declarative, portable, and composable.
+flowchart LR
+ Build[Define Scenario] --> Deploy[Deploy Topology]
+ Deploy --> Execute[Run Workloads]
+ Execute --> Evaluate[Check Expectations]
+
+ style Build fill:#e1f5ff
+ style Deploy fill:#fff4e1
+ style Execute fill:#ffe1f5
+ style Evaluate fill:#e1ffe1
+Declarative API
+Multiple Deployment Modes
+Built-in Workloads
+Comprehensive Observability
+use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_core::scenario::Deployer as _;
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+#[tokio::main]
+async fn main() -> anyhow::Result<()> {
+ let mut scenario = ScenarioBuilder::topology_with(|t| {
+ t.network_star()
+ .validators(3)
+ .executors(1)
+ })
+ .transactions_with(|tx| tx.rate(10).users(5))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&scenario).await?;
+ runner.run(&mut scenario).await?;
+
+ Ok(())
+}Start with the Quickstart Guide for a hands-on introduction that gets you running tests in minutes.
+Explore the User Guide to learn about authoring scenarios, workloads, expectations, and deployment strategies.
+Jump to Operations & Deployment for prerequisites, environment configuration, and continuous integration patterns.
+Check the Developer Reference to implement custom workloads, expectations, and runners.
+Logos is a modular blockchain protocol composed of validators, executors, and a data-availability (DA) subsystem:
+These roles interact tightly, which is why meaningful testing must be performed in multi-node environments that include real networking, timing, and DA interaction.
+The Logos Testing Framework provides the infrastructure to orchestrate these multi-node scenarios reliably across development, CI, and production-like environments.
+Learn more about the protocol: Logos Project Documentation
+| Section | Description |
|---|---|
| Foundations | Architecture, philosophy, and design principles |
| User Guide | Writing and running scenarios, workloads, and expectations |
| Developer Reference | Extending the framework with custom components |
| Operations & Deployment | Setup, CI integration, and environment configuration |
| Appendix | Quick reference, troubleshooting, FAQ, and glossary |
Ready to start? Head to the Quickstart
+ +High-level roles of the crates that make up the framework:
+Configs (testing-framework/configs/): Prepares reusable configuration primitives for nodes, networking, tracing, data availability, and wallets, shared by all scenarios and runners. Includes topology generation and circuit asset resolution.
Core scenario orchestration (testing-framework/core/): Houses the topology and scenario model, runtime coordination, node clients, and readiness/health probes. Defines Deployer and Runner traits, ScenarioBuilder, and RunContext.
Workflows (testing-framework/workflows/): Packages workloads (transaction, DA, chaos) and expectations (consensus liveness) into reusable building blocks. Offers fluent DSL extensions (ScenarioBuilderExt, ChaosBuilderExt).
Runners (testing-framework/runners/{local,compose,k8s}/): Implements deployment backends (local host, Docker Compose, Kubernetes) that all consume the same scenario plan. Each provides a Deployer implementation (LocalDeployer, ComposeDeployer, K8sDeployer).
Runner Examples (crate name: runner-examples, path: examples/): Runnable binaries demonstrating framework usage and serving as living documentation. These are the primary entry point for running scenarios (examples/src/bin/local_runner.rs, examples/src/bin/compose_runner.rs, examples/src/bin/k8s_runner.rs).
| What You're Adding | Where It Goes | Examples |
|---|---|---|
| Node config parameter | testing-framework/configs/src/topology/configs/ | Slot duration, log levels, DA params |
| Topology feature | testing-framework/core/src/topology/ | New network layouts, node roles |
| Scenario capability | testing-framework/core/src/scenario/ | New capabilities, context methods |
| Workload | testing-framework/workflows/src/workloads/ | New traffic generators |
| Expectation | testing-framework/workflows/src/expectations/ | New success criteria |
| Builder API | testing-framework/workflows/src/builder/ | DSL extensions, fluent methods |
| Deployer | testing-framework/runners/ | New deployment backends |
| Example scenario | examples/src/bin/ | Demonstration binaries |
testing-framework/workflows/src/workloads/your_workload.rs:use async_trait::async_trait;
+use testing_framework_core::scenario::{DynError, RunContext, Workload};
+
+pub struct YourWorkload;
+
+#[async_trait]
+impl Workload for YourWorkload {
+ fn name(&self) -> &'static str {
+ "your_workload"
+ }
+
+ async fn start(&self, _ctx: &RunContext) -> Result<(), DynError> {
+ // implementation
+ Ok(())
+ }
+}testing-framework/workflows/src/builder/mod.rs:pub struct YourWorkloadBuilder;
+
+impl YourWorkloadBuilder {
+ pub fn some_config(self) -> Self {
+ self
+ }
+}
+
+pub trait ScenarioBuilderExt: Sized {
+ fn your_workload(self) -> YourWorkloadBuilder;
+}examples/src/bin/your_scenario.rs:use testing_framework_core::scenario::ScenarioBuilder;
+
+pub struct YourWorkloadBuilder;
+
+impl YourWorkloadBuilder {
+ pub fn some_config(self) -> Self {
+ self
+ }
+}
+
+pub trait YourWorkloadDslExt: Sized {
+ fn your_workload_with<F>(self, configurator: F) -> Self
+ where
+ F: FnOnce(YourWorkloadBuilder) -> YourWorkloadBuilder;
+}
+
+impl<Caps> YourWorkloadDslExt for testing_framework_core::scenario::Builder<Caps> {
+ fn your_workload_with<F>(self, configurator: F) -> Self
+ where
+ F: FnOnce(YourWorkloadBuilder) -> YourWorkloadBuilder,
+ {
+ let _ = configurator(YourWorkloadBuilder);
+ self
+ }
+}
+
+pub fn use_in_examples() {
+ let _plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(0))
+ .your_workload_with(|w| w.some_config())
+ .build();
+}testing-framework/workflows/src/expectations/your_expectation.rs:use async_trait::async_trait;
+use testing_framework_core::scenario::{DynError, Expectation, RunContext};
+
+pub struct YourExpectation;
+
+#[async_trait]
+impl Expectation for YourExpectation {
+ fn name(&self) -> &'static str {
+ "your_expectation"
+ }
+
+ async fn evaluate(&mut self, _ctx: &RunContext) -> Result<(), DynError> {
+ // implementation
+ Ok(())
+ }
+}testing-framework/workflows/src/builder/mod.rs:use testing_framework_core::scenario::ScenarioBuilder;
+
+pub trait YourExpectationDslExt: Sized {
+ fn expect_your_condition(self) -> Self;
+}
+
+impl<Caps> YourExpectationDslExt for testing_framework_core::scenario::Builder<Caps> {
+ fn expect_your_condition(self) -> Self {
+ self
+ }
+}
+
+pub fn use_in_examples() {
+ let _plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(0))
+ .expect_your_condition()
+ .build();
+}Deployer trait in testing-framework/runners/your_runner/src/deployer.rs:use async_trait::async_trait;
+use testing_framework_core::scenario::{Deployer, Runner, Scenario};
+
+#[derive(Debug)]
+pub struct YourError;
+
+pub struct YourDeployer;
+
+#[async_trait]
+impl Deployer for YourDeployer {
+ type Error = YourError;
+
+ async fn deploy(&self, _scenario: &Scenario<()>) -> Result<Runner, Self::Error> {
+ // Provision infrastructure
+ // Wait for readiness
+ // Return Runner
+ todo!()
+ }
+}Provide cleanup and handle node control if supported.
+Add example in examples/src/bin/your_runner.rs.
For detailed examples, see Extending the Framework and Custom Workload Example.
+ +The Logos Testing Framework is a purpose-built toolkit for exercising Logos in +realistic, multi-node environments. It solves the gap between small, isolated +tests and full-system validation by letting teams describe a cluster layout, +drive meaningful traffic, and assert the outcomes in one coherent plan.
+It is for protocol engineers, infrastructure operators, and QA teams who need +repeatable confidence that validators, executors, and data-availability +components work together under network and timing constraints.
+Multi-node integration testing is required because many Logos behaviors—block +progress, data availability, liveness under churn—only emerge when several +roles interact over real networking and time. This framework makes those checks +declarative, observable, and portable across environments.
+Here's the conceptual shape of every test you'll write:
+// 1. Define the cluster
+let scenario = ScenarioBuilder::topology_with(|t| {
+ t.network_star()
+ .validators(3)
+ .executors(2)
+})
+// 2. Add workloads (traffic)
+.transactions_with(|tx| tx.rate(10).users(5))
+.da_with(|da| da.channel_rate(2).blob_rate(2))
+
+// 3. Define success criteria
+.expect_consensus_liveness()
+
+// 4. Set experiment duration
+.with_run_duration(Duration::from_secs(60))
+.build();
+
+// 5. Deploy and run
+let runner = deployer.deploy(&scenario).await?;
+runner.run(&mut scenario).await?;This pattern—topology, workloads, expectations, duration—repeats across all scenarios in this book.
+Learn more: For protocol-level documentation and node internals, see the Logos Project Documentation.
+ +Comprehensive guide to log collection, metrics, and debugging across all runners.
+Critical distinction: Node logs and framework logs use different configuration mechanisms.
+| Component | Controlled By | Purpose |
|---|---|---|
Framework binaries (cargo run -p runner-examples --bin local_runner) | RUST_LOG | Runner orchestration, deployment logs |
| Node processes (validators, executors spawned by runner) | NOMOS_LOG_LEVEL, NOMOS_LOG_FILTER (+ NOMOS_LOG_DIR on host runner) | Consensus, DA, mempool, network logs |
Common mistake: Setting RUST_LOG=debug only increases verbosity of the runner binary itself. Node logs remain at their default level unless you also set NOMOS_LOG_LEVEL=debug.
Example:
+# This only makes the RUNNER verbose, not the nodes:
+RUST_LOG=debug cargo run -p runner-examples --bin local_runner
+
+# This makes the NODES verbose:
+NOMOS_LOG_LEVEL=debug cargo run -p runner-examples --bin local_runner
+
+# Both verbose (typically not needed):
+RUST_LOG=debug NOMOS_LOG_LEVEL=debug cargo run -p runner-examples --bin local_runner
+See Environment Variables Reference for complete details. Quick summary:
+| Variable | Default | Effect |
|---|---|---|
NOMOS_LOG_DIR | None (console only) | Host runner: directory for per-node log files. Compose/k8s: use cfgsync.yaml |
NOMOS_LOG_LEVEL | info | Global log level: error, warn, info, debug, trace |
NOMOS_LOG_FILTER | None | Fine-grained target filtering (e.g., cryptarchia=trace,nomos_da_sampling=debug) |
NOMOS_TESTS_TRACING | false | Enable debug tracing preset |
NOMOS_OTLP_ENDPOINT | None | OTLP trace endpoint (optional) |
NOMOS_OTLP_METRICS_ENDPOINT | None | OTLP metrics endpoint (optional) |
Example: Full debug logging to files:
+NOMOS_TESTS_TRACING=true \
+NOMOS_LOG_DIR=/tmp/test-logs \
+NOMOS_LOG_LEVEL=debug \
+NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug,nomos_da_dispersal=debug,nomos_da_verifier=debug" \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin local_runner
+When NOMOS_LOG_DIR is set, each node writes logs to separate files:
File naming pattern:
+nomos-node-0, nomos-node-1, etc. (may include timestamp suffix)nomos-executor-0, nomos-executor-1, etc. (may include timestamp suffix)Example filenames:
+nomos-node-0.2024-12-18T14-30-00.lognomos-node-1.2024-12-18T14-30-00.lognomos-executor-0.2024-12-18T14-30-00.logLocal runner note: The local runner uses per-run temporary directories under the current working directory and removes them after the run unless NOMOS_TESTS_KEEP_LOGS=1. Use NOMOS_LOG_DIR=/path/to/logs to write per-node log files to a stable location.
Common target prefixes for NOMOS_LOG_FILTER:
| Target Prefix | Subsystem |
|---|---|
cryptarchia | Consensus (Cryptarchia) |
nomos_da_sampling | DA sampling service |
nomos_da_dispersal | DA dispersal service |
nomos_da_verifier | DA verification |
nomos_blend | Mix network/privacy layer |
chain_service | Chain service (node APIs/state) |
chain_network | P2P networking |
chain_leader | Leader election |
Example filter:
+NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug,chain_service=info,chain_network=info"
+Default (temporary directories, auto-cleanup):
+POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+# Logs written to temporary directories in working directory
+# Automatically cleaned up after test completes
+Persistent file output:
+NOMOS_LOG_DIR=/tmp/local-logs \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin local_runner
+
+# After test completes:
+ls /tmp/local-logs/
+# Files with prefix: nomos-node-0*, nomos-node-1*, nomos-executor-0*
+# May include timestamps in filename
+Tip: Use NOMOS_LOG_DIR for persistent per-node log files, and NOMOS_TESTS_KEEP_LOGS=1 if you want to keep the per-run temporary directories (configs/state) for post-mortem inspection.
Via Docker logs (default, recommended):
+# List containers (note the UUID prefix in names)
+docker ps --filter "name=nomos-compose-"
+
+# Stream logs from specific container
+docker logs -f <container-id-or-name>
+
+# Or use name pattern matching:
+docker logs -f $(docker ps --filter "name=nomos-compose-.*-validator-0" -q | head -1)
+
+# Show last 100 lines
+docker logs --tail 100 <container-id>
+Via file collection (advanced):
+To write per-node log files inside containers, set tracing_settings.logger: !File in testing-framework/assets/stack/cfgsync.yaml (and ensure the directory is writable). To access them, you must either:
# Ensure cfgsync.yaml is configured to log to /logs
+NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin compose_runner
+
+# After test, copy files from containers:
+docker ps --filter "name=nomos-compose-"
+docker cp <container-id>:/logs/node* /tmp/
+volumes:
+ - /tmp/host-logs:/logs # Add to docker-compose.yml.tera
+Recommendation: Use docker logs by default. File collection inside containers is complex and rarely needed.
Keep containers for debugging:
+COMPOSE_RUNNER_PRESERVE=1 \
+NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
+cargo run -p runner-examples --bin compose_runner
+# Containers remain running after test—inspect with docker logs or docker exec
+Compose debugging variables:
+COMPOSE_RUNNER_HOST=127.0.0.1 — host used for readiness probesCOMPOSE_RUNNER_HOST_GATEWAY=host.docker.internal:host-gateway — controls extra_hosts entry (set to disable to omit)TESTNET_RUNNER_PRESERVE=1 — alias for COMPOSE_RUNNER_PRESERVE=1COMPOSE_RUNNER_HTTP_TIMEOUT_SECS=<secs> — override HTTP readiness timeoutNote: Container names follow pattern nomos-compose-{uuid}-validator-{index}-1 where {uuid} changes per run.
Via kubectl logs (use label selectors):
+# List pods
+kubectl get pods
+
+# Stream logs using label selectors (recommended)
+# Helm chart labels:
+# - nomos/logical-role=validator|executor
+# - nomos/validator-index / nomos/executor-index
+kubectl logs -l nomos/logical-role=validator -f
+kubectl logs -l nomos/logical-role=executor -f
+
+# Stream logs from specific pod
+kubectl logs -f nomos-validator-0
+
+# Previous logs from crashed pods
+kubectl logs --previous -l nomos/logical-role=validator
+Download logs for offline analysis:
+# Using label selectors
+kubectl logs -l nomos/logical-role=validator --tail=1000 > all-validators.log
+kubectl logs -l nomos/logical-role=executor --tail=1000 > all-executors.log
+
+# Specific pods
+kubectl logs nomos-validator-0 > validator-0.log
+kubectl logs nomos-executor-1 > executor-1.log
+K8s debugging variables:
+K8S_RUNNER_DEBUG=1 — logs Helm stdout/stderr for install commandsK8S_RUNNER_PRESERVE=1 — keep namespace/release after runK8S_RUNNER_NODE_HOST=<ip|hostname> — override NodePort host resolutionK8S_RUNNER_NAMESPACE=<name> / K8S_RUNNER_RELEASE=<name> — pin namespace/release (useful for debugging)Specify namespace (if not using default):
+kubectl logs -n my-namespace -l nomos/logical-role=validator -f
+Note: K8s runner is optimized for local clusters (Docker Desktop K8s, minikube, kind). Remote clusters require additional setup.
+OTLP exporters are optional. If you see errors about unreachable OTLP endpoints, it's safe to ignore them unless you're actively collecting traces/metrics.
+To enable OTLP:
+NOMOS_OTLP_ENDPOINT=http://localhost:4317 \
+NOMOS_OTLP_METRICS_ENDPOINT=http://localhost:4318 \
+cargo run -p runner-examples --bin local_runner
+To silence OTLP errors: Simply leave these variables unset (the default).
+Runners expose metrics and node HTTP endpoints for expectation code and debugging.
+scripts/setup/setup-observability.sh:
+scripts/setup/setup-observability.sh compose up then scripts/setup/setup-observability.sh compose envscripts/setup/setup-observability.sh k8s install then scripts/setup/setup-observability.sh k8s envNOMOS_METRICS_QUERY_URL (PromQL base URL) to enable ctx.telemetry() queriesctx.telemetry().prometheus().map(|p| p.base_url())Example:
+# Start observability stack (Compose)
+scripts/setup/setup-observability.sh compose up
+
+# Get environment variables
+eval $(scripts/setup/setup-observability.sh compose env)
+
+# Run scenario with metrics
+POL_PROOF_DEV_MODE=true \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+scripts/setup/setup-observability.sh can)NOMOS_GRAFANA_URL, the deployer prints it in TESTNET_ENDPOINTStesting-framework/assets/stack/monitoring/grafana/dashboards/ (the bundled stack auto-provisions them)Example:
+# Bring up the bundled Prometheus+Grafana stack (optional)
+scripts/setup/setup-observability.sh compose up
+eval $(scripts/setup/setup-observability.sh compose env)
+
+export NOMOS_GRAFANA_URL=http://localhost:3000
+POL_PROOF_DEV_MODE=true scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+Default bundled Grafana login: admin / admin (see scripts/observability/compose/docker-compose.yml).
ctx.node_clients().validator_clients().get(0)testing-framework/core/src/nodes/api_client.rs for available methodsExample usage in expectations:
+use testing_framework_core::scenario::{DynError, RunContext};
+
+async fn evaluate(ctx: &RunContext) -> Result<(), DynError> {
+ let client = &ctx.node_clients().validator_clients()[0];
+
+ let info = client.consensus_info().await?;
+ tracing::info!(height = info.height, "consensus info from validator 0");
+
+ Ok(())
+}flowchart TD
+ Expose[Runner exposes endpoints/ports] --> Collect[Runtime collects block/health signals]
+ Collect --> Consume[Expectations consume signals<br/>decide pass/fail]
+ Consume --> Inspect[Operators inspect logs/metrics<br/>when failures arise]
+NOMOS_LOG_DIR=/tmp/logs \
+NOMOS_LOG_LEVEL=debug \
+NOMOS_LOG_FILTER="cryptarchia=trace" \
+POL_PROOF_DEV_MODE=true \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
+# Start observability stack
+scripts/setup/setup-observability.sh compose up
+eval $(scripts/setup/setup-observability.sh compose env)
+
+# Run with metrics
+POL_PROOF_DEV_MODE=true \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+
+# Access Grafana at http://localhost:3000
+K8S_RUNNER_NAMESPACE=nomos-debug \
+K8S_RUNNER_DEBUG=1 \
+K8S_RUNNER_PRESERVE=1 \
+POL_PROOF_DEV_MODE=true \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
+
+# Inspect logs
+kubectl logs -n nomos-debug -l nomos/logical-role=validator
+The deployer supplies a RunContext that workloads and expectations share. It
+provides:
GeneratedTopology)NodeClients / ClusterClient) for HTTP/RPC callsRunMetrics, Metrics) and block feedNodeControlHandle for managing nodesThe BlockFeed is a broadcast stream of block observations that allows workloads and expectations to monitor blockchain progress in real-time. It polls a validator node continuously and broadcasts new blocks to all subscribers.
Real-time block stream:
+BlockRecord notifications as blocks are producedHeaderId) and full block payloadBlock statistics:
+block_feed.stats().total_transactions()Broadcast semantics:
+BlockFeed is available through RunContext:
let block_feed = ctx.block_feed();Expectations typically use BlockFeed to verify block production and inclusion of transactions/data.
+Example: Counting blocks during a run
+use std::sync::{
+ Arc,
+ atomic::{AtomicU64, Ordering},
+};
+
+use async_trait::async_trait;
+use testing_framework_core::scenario::{DynError, Expectation, RunContext};
+
+struct MinimumBlocksExpectation {
+ min_blocks: u64,
+ captured_blocks: Option<Arc<AtomicU64>>,
+}
+
+#[async_trait]
+impl Expectation for MinimumBlocksExpectation {
+ fn name(&self) -> &'static str {
+ "minimum_blocks"
+ }
+
+ async fn start_capture(&mut self, ctx: &RunContext) -> Result<(), DynError> {
+ let block_count = Arc::new(AtomicU64::new(0));
+ let block_count_task = Arc::clone(&block_count);
+
+ // Subscribe to block feed
+ let mut receiver = ctx.block_feed().subscribe();
+
+ // Spawn a task to count blocks
+ tokio::spawn(async move {
+ loop {
+ match receiver.recv().await {
+ Ok(_record) => {
+ block_count_task.fetch_add(1, Ordering::Relaxed);
+ }
+ Err(tokio::sync::broadcast::error::RecvError::Lagged(skipped)) => {
+ tracing::debug!(skipped, "receiver lagged, skipping blocks");
+ }
+ Err(tokio::sync::broadcast::error::RecvError::Closed) => {
+ tracing::debug!("block feed closed");
+ break;
+ }
+ }
+ }
+ });
+
+ self.captured_blocks = Some(block_count);
+ Ok(())
+ }
+
+ async fn evaluate(&mut self, ctx: &RunContext) -> Result<(), DynError> {
+ let blocks = self.captured_blocks
+ .as_ref()
+ .expect("start_capture must be called first")
+ .load(Ordering::Relaxed);
+
+ if blocks < self.min_blocks {
+ return Err(format!(
+ "expected at least {} blocks, observed {}",
+ self.min_blocks, blocks
+ ).into());
+ }
+
+ tracing::info!(blocks, min = self.min_blocks, "minimum blocks expectation passed");
+ Ok(())
+ }
+}Example: Inspecting block contents
+use testing_framework_core::scenario::{DynError, RunContext};
+
+async fn start_capture(ctx: &RunContext) -> Result<(), DynError> {
+ let mut receiver = ctx.block_feed().subscribe();
+
+ tokio::spawn(async move {
+ loop {
+ match receiver.recv().await {
+ Ok(record) => {
+ // Access block header
+ let header_id = &record.header;
+
+ // Access full block
+ let tx_count = record.block.transactions().len();
+
+ tracing::debug!(
+ ?header_id,
+ tx_count,
+ "observed block"
+ );
+
+ // Process transactions, DA blobs, etc.
+ }
+ Err(tokio::sync::broadcast::error::RecvError::Closed) => break,
+ Err(_) => continue,
+ }
+ }
+ });
+
+ Ok(())
+}Workloads can use BlockFeed to coordinate timing or wait for specific conditions before proceeding.
+Example: Wait for N blocks before starting
+use async_trait::async_trait;
+use testing_framework_core::scenario::{DynError, RunContext, Workload};
+
+struct DelayedWorkload {
+ wait_blocks: usize,
+}
+
+#[async_trait]
+impl Workload for DelayedWorkload {
+ fn name(&self) -> &str {
+ "delayed_workload"
+ }
+
+ async fn start(&self, ctx: &RunContext) -> Result<(), DynError> {
+ tracing::info!(wait_blocks = self.wait_blocks, "waiting for blocks before starting");
+
+ // Subscribe to block feed
+ let mut receiver = ctx.block_feed().subscribe();
+ let mut count = 0;
+
+ // Wait for N blocks
+ while count < self.wait_blocks {
+ match receiver.recv().await {
+ Ok(_) => count += 1,
+ Err(tokio::sync::broadcast::error::RecvError::Lagged(_)) => continue,
+ Err(tokio::sync::broadcast::error::RecvError::Closed) => {
+ return Err("block feed closed before reaching target".into());
+ }
+ }
+ }
+
+ tracing::info!("warmup complete, starting actual workload");
+
+ // Now do the actual work
+ // ...
+
+ Ok(())
+ }
+}Example: Rate limiting based on block production
+use testing_framework_core::scenario::{DynError, RunContext};
+
+async fn generate_request() -> Option<()> {
+ None
+}
+
+async fn start(ctx: &RunContext) -> Result<(), DynError> {
+ let clients = ctx.node_clients().validator_clients();
+ let mut receiver = ctx.block_feed().subscribe();
+ let mut pending_requests: Vec<()> = Vec::new();
+
+ loop {
+ tokio::select! {
+ // Issue a batch on each new block.
+ Ok(_record) = receiver.recv() => {
+ if !pending_requests.is_empty() {
+ tracing::debug!(count = pending_requests.len(), "issuing requests on new block");
+ for _req in pending_requests.drain(..) {
+ let _info = clients[0].consensus_info().await?;
+ }
+ }
+ }
+
+ // Generate work continuously.
+ Some(req) = generate_request() => {
+ pending_requests.push(req);
+ }
+ }
+ }
+}Use BlockFeed when:
+Use direct polling when:
+Example direct polling in expectations:
+use testing_framework_core::scenario::{DynError, RunContext};
+
+async fn evaluate(ctx: &RunContext) -> Result<(), DynError> {
+ let client = &ctx.node_clients().validator_clients()[0];
+
+ // Poll current height once
+ let info = client.consensus_info().await?;
+ tracing::info!(height = info.height, "final block height");
+
+ // This is simpler than BlockFeed for one-time checks
+ Ok(())
+}Access aggregated statistics without subscribing to the feed:
+use testing_framework_core::scenario::{DynError, RunContext};
+
+async fn evaluate(ctx: &RunContext, expected_min: u64) -> Result<(), DynError> {
+ let stats = ctx.block_feed().stats();
+ let total_txs = stats.total_transactions();
+
+ tracing::info!(total_txs, "transactions observed across all blocks");
+
+ if total_txs < expected_min {
+ return Err(format!(
+ "expected at least {} transactions, observed {}",
+ expected_min, total_txs
+ ).into());
+ }
+
+ Ok(())
+}Subscription timing:
+start_capture() for expectationsstart() for workloadsLagged receivers:
+RecvError::Lagged(skipped) gracefullyFeed lifetime:
+Performance:
+The framework's built-in expectations use BlockFeed extensively:
+ConsensusLiveness: Doesn't directly subscribe but uses block feed stats to verify progressDataAvailabilityExpectation: Subscribes to inspect DA blobs in each block and track inscription/dispersalTransactionInclusion: Subscribes to find specific transactions in blocksSee Examples and Workloads & Expectations for more patterns.
+The framework currently supports process-level chaos (node restarts) for +resilience testing:
+Supported:
+restart_validator)restart_executor).chaos().restart()Not Yet Supported:
+For network partition testing, see Extension Ideas
+which describes the proposed block_peer/unblock_peer API (not yet implemented).
Check for control support and use it conditionally:
+use async_trait::async_trait;
+use testing_framework_core::scenario::{DynError, RunContext, Workload};
+
+struct RestartWorkload;
+
+#[async_trait]
+impl Workload for RestartWorkload {
+ fn name(&self) -> &str {
+ "restart_workload"
+ }
+
+ async fn start(&self, ctx: &RunContext) -> Result<(), DynError> {
+ if let Some(control) = ctx.node_control() {
+ // Restart the first validator (index 0) if supported.
+ control.restart_validator(0).await?;
+ }
+ Ok(())
+ }
+}When chaos workloads need control, require enable_node_control() in the
+scenario builder and deploy with a runner that supports it.
The NodeControlHandle trait currently provides:
use async_trait::async_trait;
+use testing_framework_core::scenario::DynError;
+
+#[async_trait]
+pub trait NodeControlHandle: Send + Sync {
+ async fn restart_validator(&self, index: usize) -> Result<(), DynError>;
+ async fn restart_executor(&self, index: usize) -> Result<(), DynError>;
+}Future extensions may include peer blocking/unblocking or other control +operations. For now, focus on restart-based chaos patterns as shown in the +chaos workload examples.
+NodeControlHandle.Operational readiness focuses on prerequisites, environment fit, and clear signals that ensure your test scenarios run reliably across different deployment targets.
+Prerequisites:
+versions.env file at repository root (required by helper scripts)nomos-node, nomos-executor) available or built on demandArtifacts:
+Environment Configuration:
+POL_PROOF_DEV_MODE=true is REQUIRED for all runners to avoid expensive proof generationNOMOS_LOG_* variablesReadiness & Health:
+The framework is intentionally runner-agnostic: the same scenario plan runs across all deployment targets. Understanding which operational concerns apply to each runner helps you choose the right fit.
+| Concern | Host | Compose | Kubernetes |
|---|---|---|---|
| Topology | Full support | Full support | Full support |
| Workloads | All workloads | All workloads | All workloads |
| Expectations | All expectations | All expectations | All expectations |
| Chaos / Node Control | Not supported | Supported | Not yet |
| Metrics / Observability | Manual setup | External stack | Cluster-wide |
| Log Collection | Temp files | Container logs | Pod logs |
| Isolation | Process-level | Container | Pod + namespace |
| Setup Time | < 1 min | 2-5 min | 5-10 min |
| CI Recommended? | Smoke tests | Primary | Large-scale only |
Key insight: Operational concerns (prerequisites, environment variables) are largely consistent across runners, while deployment-specific concerns (isolation, chaos support) vary by backend.
+flowchart LR
+ Setup[Prerequisites & Setup] --> Run[Run Scenarios]
+ Run --> Monitor[Monitor & Observe]
+ Monitor --> Debug{Success?}
+ Debug -->|No| Triage[Failure Triage]
+ Triage --> Setup
+ Debug -->|Yes| Done[Complete]
+This Operations & Deployment section covers:
+Philosophy: Treat operational hygiene—assets present, prerequisites satisfied, observability reachable—as the first step to reliable scenario outcomes.
+ +Operational readiness focuses on prerequisites, environment fit, and clear +signals:
+versions.env file at repository root (required by helper scripts; defines VERSION, NOMOS_NODE_REV, NOMOS_BUNDLE_VERSION)nomos-node checkout available, or use scripts/run-examples.sh which clones/builds on demandbuild-binaries workflowtesting-framework/assets/stack/kzgrs_test_params. Fetch them via
+scripts/setup-nomos-circuits.sh or override the path with NOMOS_KZGRS_PARAMS_PATH.POL_PROOF_DEV_MODE=true is required for all runners
+(local, compose, k8s) unless you want expensive Groth16 proof generation that
+will cause tests to timeout. Configure logging via NOMOS_LOG_DIR, NOMOS_LOG_LEVEL,
+and NOMOS_LOG_FILTER (see Logging and Observability
+for details). Note that nodes ignore RUST_LOG and only respond to NOMOS_* variables.Treat operational hygiene—assets present, prerequisites satisfied, observability +reachable—as the first step to reliable scenario outcomes.
+This section consolidates all environment variables used by the framework, helper scripts, and node processes.
+| Variable | Default | Effect | Used By | Required? |
|---|---|---|---|---|
POL_PROOF_DEV_MODE | (none) | When set to true or 1, disables expensive Groth16 proof generation. Critical for testing to avoid timeouts. | All runners | Yes |
VERSION | (none) | Framework version identifier, read from versions.env | Helper scripts | Yes (in versions.env) |
NOMOS_NODE_REV | (none) | Git revision of nomos-node to build/checkout | Helper scripts | Yes (in versions.env) |
NOMOS_BUNDLE_VERSION | (none) | Version tag for prebuilt binary bundles | Helper scripts | Yes (in versions.env) |
| Variable | Default | Effect | Used By | Required? |
|---|---|---|---|---|
NOMOS_NODE_BIN | testing-framework/assets/stack/bin/nomos-node | Path to nomos-node binary | Local runner | No (uses default) |
NOMOS_EXECUTOR_BIN | testing-framework/assets/stack/bin/nomos-executor | Path to nomos-executor binary | Local runner | No (uses default) |
NOMOS_CLI_BIN | testing-framework/assets/stack/bin/nomos-cli | Path to nomos-cli binary | Helper scripts | No (uses default) |
NOMOS_KZGRS_PARAMS_PATH | testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params | Path to KZG circuit parameters file (note: path includes filename twice—directory contains file kzgrs_test_params) | All runners (for DA workloads) | No (uses default) |
NOMOS_NODE_PATH | (none) | Use local nomos-node checkout at this path (skips git clone/fetch) | build-bundle.sh | No |
NOMOS_BINARIES_TAR | .tmp/nomos-binaries-{platform}-{version}.tar.gz | Path to prebuilt binaries tarball | run-examples.sh | No (builds if missing) |
| Variable | Default | Effect | Used By | Required? |
|---|---|---|---|---|
NOMOS_TESTNET_IMAGE | logos-blockchain-testing:local | Docker image name for compose/k8s runners | Compose, K8s runners | No (uses default) |
NOMOS_BUNDLE_DOCKER_PLATFORM | linux/amd64 (or linux/arm64 on Apple Silicon Docker Desktop) | Docker platform for Linux binary builds when running on non-Linux host | build-bundle.sh, build_test_image.sh | No (auto-detected) |
NOMOS_SKIP_IMAGE_BUILD | 0 | When set to 1, skips Docker image build step | run-examples.sh compose/k8s | No |
Important: Node processes ignore RUST_LOG and only respond to NOMOS_* variables.
| Variable | Default | Effect | Used By | Required? |
|---|---|---|---|---|
NOMOS_LOG_DIR | (varies by runner) | Directory for per-node log files. Local runner: temp directory (cleaned up). Compose: /tmp/node-logs in container. Set to persist logs. | All runners | No |
NOMOS_LOG_LEVEL | info | Global log level for node processes (error, warn, info, debug, trace) | Node processes | No |
NOMOS_LOG_FILTER | (none) | Fine-grained logging filter (e.g., nomos_core=debug,consensus=trace) | Node processes | No |
NOMOS_TESTS_TRACING | (none) | When set to true, enables tracing output to console for local runner | Local runner | No |
NOMOS_TESTS_KEEP_LOGS | (none) | When set to 1, prevents cleanup of temporary log directories | Local runner | No |
Note: Runner binaries (not node processes) use RUST_LOG for framework orchestration logs.
| Variable | Default | Effect | Used By | Required? |
|---|---|---|---|---|
RUST_LOG | (none) | Controls logging for runner binaries and framework code (e.g., testing_framework=debug) | Runner binaries | No |
| Variable | Default | Effect | Used By | Required? |
|---|---|---|---|---|
NOMOS_METRICS_QUERY_URL | (none) | Prometheus-compatible PromQL API base URL for querying metrics | Expectations (optional) | No |
NOMOS_METRICS_OTLP_INGEST_URL | (none) | OTLP ingest endpoint for metrics (e.g., for external Tempo/Loki) | Compose runner | No |
NOMOS_GRAFANA_URL | (none) | Grafana dashboard URL (printed in TESTNET_ENDPOINTS for convenience) | Compose runner | No |
NOMOS_OTLP_ENDPOINT | (none) | OpenTelemetry Protocol endpoint for traces (e.g., http://localhost:4317) | Node processes | No |
NOMOS_OTLP_METRICS_ENDPOINT | (none) | OpenTelemetry Protocol endpoint for metrics (e.g., http://localhost:4318) | Node processes | No |
| Variable | Default | Effect | Used By | Required? |
|---|---|---|---|---|
NOMOS_DEMO_VALIDATORS | (none) | Number of validators (used by run-examples.sh -v flag) | run-examples.sh | No (use -v flag) |
NOMOS_DEMO_EXECUTORS | (none) | Number of executors (used by run-examples.sh -e flag) | run-examples.sh | No (use -e flag) |
NOMOS_DEMO_RUN_SECONDS | (none) | Run duration in seconds (used by run-examples.sh -t flag) | run-examples.sh | No (use -t flag) |
| Variable | Default | Effect | Used By | Required? |
|---|---|---|---|---|
NOMOS_K8S_NAMESPACE | default | Kubernetes namespace for deployments | K8s runner | No |
NOMOS_K8S_CONTEXT | (current context) | Kubernetes context to use | K8s runner | No |
NOMOS_K8S_IMAGE_PULL_POLICY | IfNotPresent | Image pull policy (Always, IfNotPresent, Never) | K8s runner | No |
Running local smoke test:
+POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+Running with custom log directory:
+POL_PROOF_DEV_MODE=true \
+NOMOS_LOG_DIR=/tmp/my-test-logs \
+NOMOS_TESTS_KEEP_LOGS=1 \
+cargo run -p runner-examples --bin local_runner
+Running compose with observability:
+POL_PROOF_DEV_MODE=true \
+NOMOS_METRICS_QUERY_URL=http://localhost:9090 \
+NOMOS_GRAFANA_URL=http://localhost:3000 \
+scripts/run-examples.sh -t 60 -v 2 -e 1 compose
+Running with debug logging:
+POL_PROOF_DEV_MODE=true \
+NOMOS_LOG_LEVEL=debug \
+NOMOS_LOG_FILTER="consensus=trace,da=debug" \
+RUST_LOG=testing_framework=debug \
+cargo run -p runner-examples --bin local_runner
+Building for Apple Silicon:
+NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64 scripts/build_test_image.sh
+Both LocalDeployer and ComposeDeployer work in CI environments:
+LocalDeployer in CI:
+ComposeDeployer in CI (recommended):
+Here's a production-ready GitHub Actions workflow demonstrating host and compose runners with proper artifact caching, log collection, and Cucumber test integration:
+name: Integration Tests
+
+on:
+ push:
+ branches: [main, develop]
+ pull_request:
+ branches: [main]
+
+env:
+ CARGO_TERM_COLOR: always
+ RUST_BACKTRACE: 1
+
+jobs:
+ # Host runner: Fast smoke tests
+ test-host:
+ name: Host Runner Tests (Smoke)
+ runs-on: ubuntu-latest
+ timeout-minutes: 15
+
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v4
+
+ - name: Install Rust nightly
+ uses: dtolnay/rust-toolchain@nightly
+ with:
+ components: rustfmt, clippy
+
+ - name: Cache Rust dependencies
+ uses: Swatinem/rust-cache@v2
+ with:
+ key: host-runner-${{ hashFiles('**/Cargo.lock') }}
+
+ - name: Cache binary bundle
+ id: cache-bundle
+ uses: actions/cache@v4
+ with:
+ path: .tmp/nomos-binaries-host-*.tar.gz
+ key: nomos-binaries-host-${{ hashFiles('versions.env') }}
+
+ - name: Build binaries (if not cached)
+ if: steps.cache-bundle.outputs.cache-hit != 'true'
+ run: |
+ scripts/build-bundle.sh --platform host --output .tmp/bundle.tar.gz
+ tar -xzf .tmp/bundle.tar.gz
+ mkdir -p testing-framework/assets/stack/bin
+ cp artifacts/nomos-node artifacts/nomos-executor testing-framework/assets/stack/bin/
+
+ - name: Extract cached binaries
+ if: steps.cache-bundle.outputs.cache-hit == 'true'
+ run: |
+ tar -xzf .tmp/nomos-binaries-host-*.tar.gz
+ mkdir -p testing-framework/assets/stack/bin
+ cp artifacts/nomos-node artifacts/nomos-executor testing-framework/assets/stack/bin/
+
+ - name: Run host smoke tests
+ env:
+ POL_PROOF_DEV_MODE: true
+ NOMOS_TESTS_KEEP_LOGS: 1
+ NOMOS_LOG_DIR: ${{ github.workspace }}/.tmp/host-logs
+ run: |
+ cargo test -p runner-examples --bin local_runner -- --nocapture
+
+ - name: Run Cucumber host tests
+ env:
+ POL_PROOF_DEV_MODE: true
+ run: |
+ cargo test -p runner-examples --bin cucumber_host -- --nocapture
+
+ - name: Upload logs on failure
+ if: failure()
+ uses: actions/upload-artifact@v4
+ with:
+ name: host-runner-logs
+ path: .tmp/host-logs/
+ retention-days: 7
+
+ # Compose runner: Full integration tests
+ test-compose:
+ name: Compose Runner Tests
+ runs-on: ubuntu-latest
+ timeout-minutes: 30
+
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v4
+
+ - name: Set up Docker Buildx
+ uses: docker/setup-buildx-action@v3
+
+ - name: Install Rust nightly
+ uses: dtolnay/rust-toolchain@nightly
+
+ - name: Cache Rust dependencies
+ uses: Swatinem/rust-cache@v2
+ with:
+ key: compose-runner-${{ hashFiles('**/Cargo.lock') }}
+
+ - name: Cache Linux binary bundle
+ id: cache-linux-bundle
+ uses: actions/cache@v4
+ with:
+ path: .tmp/nomos-binaries-linux-*.tar.gz
+ key: nomos-binaries-linux-${{ hashFiles('versions.env') }}
+
+ - name: Build Linux binaries (if not cached)
+ if: steps.cache-linux-bundle.outputs.cache-hit != 'true'
+ run: |
+ scripts/build-bundle.sh --platform linux --output .tmp/linux-bundle.tar.gz
+
+ - name: Cache Docker image layers
+ uses: actions/cache@v4
+ with:
+ path: /tmp/.buildx-cache
+ key: docker-buildx-${{ hashFiles('versions.env', 'testing-framework/assets/stack/**') }}
+ restore-keys: |
+ docker-buildx-
+
+ - name: Build Docker image
+ env:
+ NOMOS_BINARIES_TAR: .tmp/nomos-binaries-linux-*.tar.gz
+ run: |
+ scripts/build_test_image.sh
+
+ - name: Run compose smoke tests
+ env:
+ POL_PROOF_DEV_MODE: true
+ run: |
+ cargo test -p runner-examples --bin compose_runner -- --nocapture
+
+ - name: Run Cucumber compose tests
+ env:
+ POL_PROOF_DEV_MODE: true
+ run: |
+ cargo test -p runner-examples --bin cucumber_compose -- --nocapture
+
+ - name: Collect container logs on failure
+ if: failure()
+ run: |
+ mkdir -p .tmp/compose-logs
+ for container in $(docker ps -a --filter "name=nomos-compose" --format "{{.Names}}"); do
+ docker logs "$container" > ".tmp/compose-logs/${container}.log" 2>&1 || true
+ done
+
+ - name: Upload container logs
+ if: failure()
+ uses: actions/upload-artifact@v4
+ with:
+ name: compose-runner-logs
+ path: .tmp/compose-logs/
+ retention-days: 7
+
+ - name: Cleanup containers
+ if: always()
+ run: |
+ docker compose down --volumes --remove-orphans || true
+
+ # Combined status check (required for branch protection)
+ integration-tests:
+ name: Integration Tests Status
+ runs-on: ubuntu-latest
+ needs: [test-host, test-compose]
+ if: always()
+
+ steps:
+ - name: Check all jobs succeeded
+ run: |
+ if [ "${{ needs.test-host.result }}" != "success" ] || \
+ [ "${{ needs.test-compose.result }}" != "success" ]; then
+ echo "One or more integration test jobs failed"
+ exit 1
+ fi
+Key Features:
+Cargo.lockversions.envCustomization:
+# Add matrix for multiple topologies:
+strategy:
+ matrix:
+ topology:
+ - { validators: 1, executors: 1, duration: 60 }
+ - { validators: 2, executors: 1, duration: 90 }
+ - { validators: 3, executors: 2, duration: 120 }
+
+# Or add scheduled nightly stress tests:
+on:
+ schedule:
+ - cron: '0 2 * * *' # 2 AM daily
+See also: .github/workflows/lint.yml for current CI examples.
The framework provides three runner modes: host (local processes), compose (Docker Compose), and k8s (Kubernetes).
+Recommended: Use scripts/run-examples.sh for all modes:
# Host mode (local processes)
+scripts/run-examples.sh -t 60 -v 1 -e 1 host
+
+# Compose mode (Docker Compose)
+scripts/run-examples.sh -t 60 -v 1 -e 1 compose
+
+# K8s mode (Kubernetes)
+scripts/run-examples.sh -t 60 -v 1 -e 1 k8s
+This script handles circuit setup, binary building/bundling, (local) image building, and execution.
+Note: for k8s runs against non-local clusters (e.g. EKS), the cluster pulls images from a registry,
+so a local docker build is not used. In that case, build + push your image separately (see
+scripts/build_test_image.sh) and set NOMOS_TESTNET_IMAGE to the pushed reference.
For a small “does everything still run?” matrix (including --no-image-build variants where relevant), use:
scripts/run-test-matrix.sh -t 120 -v 1 -e 1
+This is useful after making runner/image/script changes, and it forwards --metrics-* options through to scripts/run-examples.sh.
Environment overrides:
+VERSION=v0.3.1 — Circuit versionNOMOS_NODE_REV=<commit> — nomos-node git revisionNOMOS_BINARIES_TAR=path/to/bundle.tar.gz — Use prebuilt bundleNOMOS_SKIP_IMAGE_BUILD=1 — Skip image rebuild (compose/k8s)NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64|linux/amd64 — Docker platform used when building a Linux bundle on non-Linux hosts (macOS/Windows)COMPOSE_CIRCUITS_PLATFORM=linux-aarch64|linux-x86_64 — Circuits platform used when building the compose/k8s image (defaults based on host arch)SLOW_TEST_ENV=true — Doubles built-in readiness timeouts (useful in slower CI / constrained laptops)TESTNET_PRINT_ENDPOINTS=1 — Print TESTNET_ENDPOINTS / TESTNET_PPROF lines during deploy (set automatically by scripts/run-examples.sh)COMPOSE_RUNNER_HTTP_TIMEOUT_SECS=<secs> — Override compose node HTTP readiness timeoutK8S_RUNNER_DEPLOYMENT_TIMEOUT_SECS=<secs> — Override k8s deployment readiness timeoutK8S_RUNNER_HTTP_TIMEOUT_SECS=<secs> — Override k8s HTTP readiness timeout for port-forwardsK8S_RUNNER_HTTP_PROBE_TIMEOUT_SECS=<secs> — Override k8s HTTP readiness timeout for NodePort probesK8S_RUNNER_PROMETHEUS_HTTP_TIMEOUT_SECS=<secs> — Override k8s Prometheus readiness timeoutK8S_RUNNER_PROMETHEUS_HTTP_PROBE_TIMEOUT_SECS=<secs> — Override k8s Prometheus NodePort probe timeoutnomos-node Revision (Dev Workflow)The repo pins a nomos-node revision in versions.env for reproducible builds. To update it (or point to a local checkout), use the helper script:
# Pin to a new git revision (updates versions.env + Cargo.toml git revs)
+scripts/update-nomos-rev.sh --rev <git_sha>
+
+# Use a local nomos-node checkout instead (for development)
+scripts/update-nomos-rev.sh --path /path/to/nomos-node
+
+# If Cargo.toml was marked skip-worktree, clear it
+scripts/update-nomos-rev.sh --unskip-worktree
+Notes:
+NOMOS_NODE_PATH values; prefer --rev for shared history/CI.Cargo.lock to update on the next cargo build/cargo test.If you hit Docker build failures, mysterious I/O errors, or are running out of disk space:
+scripts/clean.sh
+For extra Docker cache cleanup:
+scripts/clean.sh --docker
+For manual control, you can run the local_runner binary directly:
POL_PROOF_DEV_MODE=true \
+NOMOS_NODE_BIN=/path/to/nomos-node \
+NOMOS_EXECUTOR_BIN=/path/to/nomos-executor \
+cargo run -p runner-examples --bin local_runner
+Environment variables:
+NOMOS_DEMO_VALIDATORS=3 — Number of validators (default: 1, or use legacy LOCAL_DEMO_VALIDATORS)NOMOS_DEMO_EXECUTORS=2 — Number of executors (default: 1, or use legacy LOCAL_DEMO_EXECUTORS)NOMOS_DEMO_RUN_SECS=120 — Run duration in seconds (default: 60, or use legacy LOCAL_DEMO_RUN_SECS)NOMOS_NODE_BIN / NOMOS_EXECUTOR_BIN — Paths to binaries (required for direct run)NOMOS_LOG_DIR=/tmp/logs — Directory for per-node log files (host runner). For compose/k8s, set tracing_settings.logger: !File in testing-framework/assets/stack/cfgsync.yaml.NOMOS_TESTS_KEEP_LOGS=1 — Keep per-run temporary directories (useful for debugging/CI artifacts)NOMOS_TESTS_TRACING=true — Enable the debug tracing preset (optional; combine with NOMOS_LOG_DIR unless you have external tracing backends configured)NOMOS_LOG_LEVEL=debug — Set log level (default: info)NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug" — Fine-grained module filteringNote: Requires circuit assets and host binaries. Use scripts/run-examples.sh host to handle setup automatically.
For manual control, you can run the compose_runner binary directly. Compose requires a Docker image with embedded assets.
Recommended setup: Use a prebuilt bundle:
+# Build a Linux bundle (includes binaries + circuits)
+scripts/build-bundle.sh --platform linux
+# Creates .tmp/nomos-binaries-linux-v0.3.1.tar.gz
+
+# Build image (embeds bundle assets)
+export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
+scripts/build_test_image.sh
+
+# Run
+NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin compose_runner
+Platform note (macOS / Apple silicon):
+linux/arm64 engine. If Linux bundle builds are slow/unstable when producing .tmp/nomos-binaries-linux-*.tar.gz, prefer NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64 for local compose/k8s runs.NOMOS_BUNDLE_DOCKER_PLATFORM=linux/amd64 and expect slower builds via emulation.Alternative: Manual circuit/image setup (rebuilds during image build):
+# Fetch and copy circuits
+scripts/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits
+cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/
+
+# Build image
+scripts/build_test_image.sh
+
+# Run
+NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin compose_runner
+Environment variables:
+NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local — Image tag (required, must match built image)POL_PROOF_DEV_MODE=true — Required for all runnersNOMOS_DEMO_VALIDATORS=3 / NOMOS_DEMO_EXECUTORS=2 / NOMOS_DEMO_RUN_SECS=120 — Topology overridesCOMPOSE_NODE_PAIRS=1x1 — Alternative topology format: "validators×executors"NOMOS_METRICS_QUERY_URL — Prometheus-compatible base URL for the runner process to query (optional)NOMOS_METRICS_OTLP_INGEST_URL — Full OTLP HTTP ingest URL for node metrics export (optional)NOMOS_GRAFANA_URL — Grafana base URL for printing/logging (optional)COMPOSE_RUNNER_HOST=127.0.0.1 — Host address for port mappingsCOMPOSE_RUNNER_PRESERVE=1 — Keep containers running after testNOMOS_LOG_LEVEL=debug / NOMOS_LOG_FILTER=... — Control node log verbosity (stdout/stderr)testing-framework/assets/stack/cfgsync.yaml (tracing_settings.logger) — Switch node logs between stdout/stderr and file outputCompose-specific features:
+.enable_node_control() + chaos workloads)NOMOS_METRICS_* / NOMOS_GRAFANA_URL to enable telemetry links and querying
+scripts/setup-observability.sh compose up then scripts/setup-observability.sh compose envImportant:
+/kzgrs_test_params/kzgrs_test_params (note the repeated filename)scripts/run-examples.sh compose to handle all setup automaticallyFor manual control, you can run the k8s_runner binary directly. K8s requires the same image setup as Compose.
Prerequisites:
+kubectl configuredBuild and load image:
+# Build image with bundle (recommended)
+scripts/build-bundle.sh --platform linux
+export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
+scripts/build_test_image.sh
+
+# Load into cluster
+export NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local
+kind load docker-image logos-blockchain-testing:local # For kind
+# OR: minikube image load logos-blockchain-testing:local # For minikube
+# OR: docker push your-registry/logos-blockchain-testing:local # For remote
+Run the example:
+export NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local
+export POL_PROOF_DEV_MODE=true
+cargo run -p runner-examples --bin k8s_runner
+Environment variables:
+NOMOS_TESTNET_IMAGE — Image tag (required)POL_PROOF_DEV_MODE=true — Required for all runnersNOMOS_DEMO_VALIDATORS / NOMOS_DEMO_EXECUTORS / NOMOS_DEMO_RUN_SECS — Topology overridesNOMOS_METRICS_QUERY_URL — Prometheus-compatible base URL for the runner process to query (PromQL)NOMOS_METRICS_OTLP_INGEST_URL — Full OTLP HTTP ingest URL for node metrics export (optional)NOMOS_GRAFANA_URL — Grafana base URL for printing/logging (optional)Metrics + Grafana (optional):
+export NOMOS_METRICS_QUERY_URL=http://your-prometheus:9090
+# Prometheus OTLP receiver example:
+export NOMOS_METRICS_OTLP_INGEST_URL=http://your-prometheus:9090/api/v1/otlp/v1/metrics
+# Optional: print a Grafana link in TESTNET_ENDPOINTS
+export NOMOS_GRAFANA_URL=http://your-grafana:3000
+cargo run -p runner-examples --bin k8s_runner
+Notes:
+NOMOS_METRICS_QUERY_URL must be reachable from the runner process (often via kubectl port-forward).NOMOS_METRICS_OTLP_INGEST_URL must be reachable from nodes (pods/containers) and is backend-specific (Prometheus vs VictoriaMetrics paths differ).
+scripts/setup-observability.sh k8s install then scripts/setup-observability.sh k8s env (optional dashboards: scripts/setup-observability.sh k8s dashboards)Via scripts/run-examples.sh (optional):
scripts/run-examples.sh -t 60 -v 1 -e 1 k8s \
+ --metrics-query-url http://your-prometheus:9090 \
+ --metrics-otlp-ingest-url http://your-prometheus:9090/api/v1/otlp/v1/metrics
+In code (optional):
++#![allow(unused)] +fn main() { +use testing_framework_core::scenario::ScenarioBuilder; +use testing_framework_workflows::ObservabilityBuilderExt as _; + +let plan = ScenarioBuilder::with_node_counts(1, 1) + .with_metrics_query_url_str("http://your-prometheus:9090") + .with_metrics_otlp_ingest_url_str("http://your-prometheus:9090/api/v1/otlp/v1/metrics") + .build(); +}
Important:
+testing-framework/assets/stack/kzgrs_test_params as a hostPath volume with file /kzgrs_test_params/kzgrs_test_params inside pods.enable_node_control()) will failscripts/run-examples.sh k8s to handle all setup automaticallyDA workloads require KZG cryptographic parameters for polynomial commitment schemes.
+Default path: testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params
Note the repeated filename: the directory kzgrs_test_params/ contains a file named kzgrs_test_params. This is the actual proving key file.
Container path (compose/k8s): /kzgrs_test_params/kzgrs_test_params
Override: Set NOMOS_KZGRS_PARAMS_PATH to use a custom location (must point to the file):
NOMOS_KZGRS_PARAMS_PATH=/path/to/custom/params cargo run -p runner-examples --bin local_runner
+The system uses KZG assets in two distinct ways:
+| Concept | Used by | Meaning |
|---|---|---|
| KZG directory | deployers/scripts | A directory that contains the KZG file (and related artifacts). Defaults to testing-framework/assets/stack/kzgrs_test_params and is controlled by NOMOS_KZG_DIR_REL (relative to the workspace root). |
| KZG file path | node processes | A single file path passed to nodes via NOMOS_KZGRS_PARAMS_PATH (inside containers/pods this is typically /kzgrs_test_params/kzgrs_test_params). |
Option 1: Use helper script (recommended):
+# From the repository root
+chmod +x scripts/setup-nomos-circuits.sh
+scripts/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits
+
+# Copy to default location
+cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/
+Option 2: Build locally (advanced):
+# This repository does not provide a `make kzgrs_test_params` target.
+# If you need to regenerate KZG params from source, follow upstream tooling
+# instructions (unspecified here) or use the helper scripts above to fetch a
+# known-good bundle.
+The CI automatically fetches and places assets:
+- name: Install circuits for host build
+ run: |
+ scripts/setup-nomos-circuits.sh v0.3.1 "$TMPDIR/nomos-circuits"
+ cp -a "$TMPDIR/nomos-circuits"/. testing-framework/assets/stack/kzgrs_test_params/
+| Runner | When Required |
|---|---|
| Local | Always (for DA workloads) |
| Compose | During image build (baked into NOMOS_TESTNET_IMAGE) |
| K8s | During image build + deployed to cluster via hostPath volume |
Error without assets:
+Error: missing KZG parameters at testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params
+If you see this error, the file kzgrs_test_params is missing from the directory. Use scripts/run-examples.sh or scripts/setup-nomos-circuits.sh to fetch it.
Critical distinction: Node logs and framework logs use different configuration mechanisms.
+| Component | Controlled By | Purpose |
|---|---|---|
Framework binaries (cargo run -p runner-examples --bin local_runner) | RUST_LOG | Runner orchestration, deployment logs |
| Node processes (validators, executors spawned by runner) | NOMOS_LOG_LEVEL, NOMOS_LOG_FILTER (+ NOMOS_LOG_DIR on host runner) | Consensus, DA, mempool, network logs |
Common mistake: Setting RUST_LOG=debug only increases verbosity of the runner binary itself. Node logs remain at their default level unless you also set NOMOS_LOG_LEVEL=debug.
Example:
+# This only makes the RUNNER verbose, not the nodes:
+RUST_LOG=debug cargo run -p runner-examples --bin local_runner
+
+# This makes the NODES verbose:
+NOMOS_LOG_LEVEL=debug cargo run -p runner-examples --bin local_runner
+
+# Both verbose (typically not needed):
+RUST_LOG=debug NOMOS_LOG_LEVEL=debug cargo run -p runner-examples --bin local_runner
+| Variable | Default | Effect |
|---|---|---|
NOMOS_LOG_DIR | None (console only) | Host runner: directory for per-node log files. Compose/k8s: use testing-framework/assets/stack/cfgsync.yaml (tracing_settings.logger: !File) and mount a writable directory. |
NOMOS_LOG_LEVEL | info | Global log level: error, warn, info, debug, trace |
NOMOS_LOG_FILTER | None | Fine-grained target filtering (e.g., cryptarchia=trace,nomos_da_sampling=debug) |
NOMOS_TESTS_TRACING | false | Enable the debug tracing preset (optional; combine with NOMOS_LOG_DIR unless you have external tracing backends configured) |
NOMOS_OTLP_ENDPOINT | None | OTLP trace endpoint (optional, disables OTLP noise if unset) |
NOMOS_OTLP_METRICS_ENDPOINT | None | OTLP metrics endpoint (optional) |
Example: Full debug logging to files:
+NOMOS_TESTS_TRACING=true \
+NOMOS_LOG_DIR=/tmp/test-logs \
+NOMOS_LOG_LEVEL=debug \
+NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug,nomos_da_dispersal=debug,nomos_da_verifier=debug,nomos_blend=debug,chain_service=info,chain_network=info,chain_leader=info" \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin local_runner
+When NOMOS_LOG_DIR is set, each node writes logs to separate files:
File naming pattern:
+nomos-node-0, nomos-node-1, etc. (may include timestamp suffix)nomos-executor-0, nomos-executor-1, etc. (may include timestamp suffix)Local runner note: The local runner uses per-run temporary directories under the current working directory and removes them after the run unless NOMOS_TESTS_KEEP_LOGS=1. Use NOMOS_LOG_DIR=/path/to/logs to write per-node log files to a stable location.
Common target prefixes for NOMOS_LOG_FILTER:
| Target Prefix | Subsystem |
|---|---|
cryptarchia | Consensus (Cryptarchia) |
nomos_da_sampling | DA sampling service |
nomos_da_dispersal | DA dispersal service |
nomos_da_verifier | DA verification |
nomos_blend | Mix network/privacy layer |
chain_service | Chain service (node APIs/state) |
chain_network | P2P networking |
chain_leader | Leader election |
Example filter:
+NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug,chain_service=info,chain_network=info,chain_leader=info"
+Default (temporary directories, auto-cleanup):
+POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+# Logs written to temporary directories in working directory
+# Automatically cleaned up after test completes
+Persistent file output:
+NOMOS_LOG_DIR=/tmp/local-logs \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin local_runner
+
+# After test completes:
+ls /tmp/local-logs/
+# Files with prefix: nomos-node-0*, nomos-node-1*, nomos-executor-0*
+# May include timestamps in filename
+Tip: Use NOMOS_LOG_DIR for persistent per-node log files, and NOMOS_TESTS_KEEP_LOGS=1 if you want to keep the per-run temporary directories (configs/state) for post-mortem inspection.
Via Docker logs (default, recommended):
+# List containers (note the UUID prefix in names)
+docker ps --filter "name=nomos-compose-"
+
+# Stream logs from specific container
+docker logs -f <container-id-or-name>
+
+# Or use name pattern matching:
+docker logs -f $(docker ps --filter "name=nomos-compose-.*-validator-0" -q | head -1)
+Via file collection (advanced):
+To write per-node log files inside containers, set tracing_settings.logger: !File in testing-framework/assets/stack/cfgsync.yaml (and ensure the directory is writable). To access them, you must either:
# Ensure `testing-framework/assets/stack/cfgsync.yaml` is configured to log to `/logs`
+# via `tracing_settings.logger: !File`.
+NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin compose_runner
+
+# After test, copy files from containers:
+docker ps --filter "name=nomos-compose-"
+docker cp <container-id>:/logs/node* /tmp/
+volumes:
+ - /tmp/host-logs:/logs # Add to docker-compose.yml.tera
+Recommendation: Use docker logs by default. File collection inside containers is complex and rarely needed.
Keep containers for debugging:
+COMPOSE_RUNNER_PRESERVE=1 \
+NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
+cargo run -p runner-examples --bin compose_runner
+# Containers remain running after test—inspect with docker logs or docker exec
+Compose networking/debug knobs:
+COMPOSE_RUNNER_HOST=127.0.0.1 — host used for readiness probes (override for remote Docker daemons / VM networking)COMPOSE_RUNNER_HOST_GATEWAY=host.docker.internal:host-gateway — controls the extra_hosts entry injected into compose (set to disable to omit)TESTNET_RUNNER_PRESERVE=1 — alias for COMPOSE_RUNNER_PRESERVE=1COMPOSE_RUNNER_HTTP_TIMEOUT_SECS=<secs> — override compose node HTTP readiness timeoutNote: Container names follow pattern nomos-compose-{uuid}-validator-{index}-1 where {uuid} changes per run.
Via kubectl logs (use label selectors):
+# List pods
+kubectl get pods
+
+# Stream logs using label selectors (recommended)
+# Helm chart labels:
+# - nomos/logical-role=validator|executor
+# - nomos/validator-index / nomos/executor-index
+kubectl logs -l nomos/logical-role=validator -f
+kubectl logs -l nomos/logical-role=executor -f
+
+# Stream logs from specific pod
+kubectl logs -f nomos-validator-0
+
+# Previous logs from crashed pods
+kubectl logs --previous -l nomos/logical-role=validator
+Download logs for offline analysis:
+# Using label selectors
+kubectl logs -l nomos/logical-role=validator --tail=1000 > all-validators.log
+kubectl logs -l nomos/logical-role=executor --tail=1000 > all-executors.log
+
+# Specific pods
+kubectl logs nomos-validator-0 > validator-0.log
+kubectl logs nomos-executor-1 > executor-1.log
+K8s environment notes:
+logos-blockchain-testing:local must be available on the cluster’s nodes (Docker Desktop shares the local daemon; kind/minikube often requires an explicit image load step).hostPath to your workspace path; this typically won’t work on remote/managed clusters without replacing it with a PV/CSI volume or baking the params into an image.K8S_RUNNER_DEBUG=1 — logs Helm stdout/stderr for install commands.K8S_RUNNER_PRESERVE=1 — keep the namespace/release after the run.K8S_RUNNER_NODE_HOST=<ip|hostname> — override NodePort host resolution for non-local clusters.K8S_RUNNER_NAMESPACE=<name> / K8S_RUNNER_RELEASE=<name> — pin namespace/release instead of random IDs (useful for debugging)Specify namespace (if not using default):
+kubectl logs -n my-namespace -l nomos/logical-role=validator -f
+OTLP exporters are optional. If you see errors about unreachable OTLP endpoints, it's safe to ignore them unless you're actively collecting traces/metrics.
+To enable OTLP:
+NOMOS_OTLP_ENDPOINT=http://localhost:4317 \
+NOMOS_OTLP_METRICS_ENDPOINT=http://localhost:4318 \
+cargo run -p runner-examples --bin local_runner
+To silence OTLP errors: Simply leave these variables unset (the default).
+Runners expose metrics and node HTTP endpoints for expectation code and debugging:
+Prometheus-compatible metrics querying (optional):
+scripts/setup-observability.sh:
+scripts/setup-observability.sh compose up then scripts/setup-observability.sh compose envscripts/setup-observability.sh k8s install then scripts/setup-observability.sh k8s envNOMOS_METRICS_QUERY_URL (PromQL base URL) to enable ctx.telemetry() queries.ctx.telemetry().prometheus().map(|p| p.base_url())Grafana (optional):
+scripts/setup-observability.sh can).NOMOS_GRAFANA_URL, the deployer prints it in TESTNET_ENDPOINTS.testing-framework/assets/stack/monitoring/grafana/dashboards/ for import into your Grafana.Node APIs:
+ctx.node_clients().validator_clients().get(0)testing-framework/core/src/nodes/api_client.rs for available methodsflowchart TD
+ Expose[Runner exposes endpoints/ports] --> Collect[Runtime collects block/health signals]
+ Collect --> Consume[Expectations consume signals<br/>decide pass/fail]
+ Consume --> Inspect[Operators inspect logs/metrics<br/>when failures arise]
+Conceptual chapters that establish the mental model for the framework and how +it approaches multi-node testing.
+ +Practical guidance for shaping scenarios, combining workloads and expectations, +and running them across different environments.
+ +Deep dives for contributors who extend the framework, evolve its abstractions, +or maintain the crate set.
+ +This section covers operational aspects of running the testing framework: prerequisites, deployment configuration, continuous integration, and observability.
+This section is organized for progressive depth:
+Operational Hygiene: Assets present, prerequisites satisfied, observability reachable
+Environment Fit: Choose the right deployment target based on isolation, reproducibility, and resource needs
+Clear Signals: Verify runners report node readiness before starting workloads
+Failure Triage: Map failures to specific causes—missing prerequisites, platform issues, or unmet expectations
+Ready to get started? Begin with Operations Overview →
+ +Quick reference materials, troubleshooting guides, and supplementary information.
+This section complements the main documentation with practical reference materials that you'll return to frequently during development and operations.
+Jump to:
+ + +Quick reference materials, troubleshooting guides, and supplementary information.
+This section complements the main documentation with practical reference materials that you'll return to frequently during development and operations.
+Jump to:
+ + +This page covers everything you need before running your first scenario.
+versions.env (Required)All helper scripts require a versions.env file at the repository root:
VERSION=v0.3.1
+NOMOS_NODE_REV=abc123def456789
+NOMOS_BUNDLE_VERSION=v1
+What it defines:
+VERSION — Circuit release tag for KZG parametersNOMOS_NODE_REV — Git revision of nomos-node to build/fetchNOMOS_BUNDLE_VERSION — Bundle schema versionWhere it's used:
+scripts/run/run-examples.shscripts/build/build-bundle.shscripts/setup/setup-nomos-circuits.shError if missing:
+ERROR: versions.env not found at repository root
+This file is required and should define:
+ VERSION=<circuit release tag>
+ NOMOS_NODE_REV=<nomos-node git revision>
+ NOMOS_BUNDLE_VERSION=<bundle schema version>
+Fix: Ensure you're in the repository root. The file should already exist in the checked-out repo.
+Scenarios need compiled nomos-node and nomos-executor binaries.
scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
+This automatically:
+NOMOS_NODE_BIN / NOMOS_EXECUTOR_BINIf you have a sibling nomos-node checkout:
cd ../nomos-node
+cargo build --release --bin nomos-node --bin nomos-executor
+
+# Set environment variables
+export NOMOS_NODE_BIN=$PWD/target/release/nomos-node
+export NOMOS_EXECUTOR_BIN=$PWD/target/release/nomos-executor
+
+# Return to testing framework
+cd ../nomos-testing
+CI workflows use prebuilt artifacts:
+- name: Download nomos binaries
+ uses: actions/download-artifact@v3
+ with:
+ name: nomos-binaries-linux
+ path: .tmp/
+
+- name: Extract bundle
+ run: |
+ tar -xzf .tmp/nomos-binaries-linux-*.tar.gz -C .tmp/
+ export NOMOS_NODE_BIN=$PWD/.tmp/nomos-node
+ export NOMOS_EXECUTOR_BIN=$PWD/.tmp/nomos-executor
+Data Availability (DA) workloads require KZG cryptographic parameters.
+Default path: testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params
Note: The directory kzgrs_test_params/ contains a file named kzgrs_test_params. This is the proving key file (~120MB).
Container path (compose/k8s): /kzgrs_test_params/kzgrs_test_params
Option 1: Use helper script (recommended):
+# Fetch circuits
+scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits
+
+# Copy to default location
+mkdir -p testing-framework/assets/stack/kzgrs_test_params
+cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/
+
+# Verify (should be ~120MB)
+ls -lh testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params
+Option 2: Let run-examples.sh handle it:
scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
+This automatically fetches and places assets.
+Set NOMOS_KZGRS_PARAMS_PATH to use a custom location:
NOMOS_KZGRS_PARAMS_PATH=/custom/path/to/kzgrs_test_params \
+cargo run -p runner-examples --bin local_runner
+| Runner | When Required |
|---|---|
| Host (local) | Always (for DA workloads) |
| Compose | During image build (baked into image) |
| K8s | During image build + mounted via hostPath |
Error without assets:
+Error: Custom { kind: NotFound, error: "Circuit file not found at: testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params" }
+Requires:
+No Docker required.
+Best for:
+Requires:
+logos-blockchain-testing:localPlatform notes (macOS / Apple silicon):
+NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64 for native performancelinux/amd64 only if targeting amd64 environments (slower via emulation)Best for:
+Requires:
+kubectl configuredLocal cluster setup:
+# Docker Desktop: Enable Kubernetes in settings
+
+# OR: Use kind
+kind create cluster
+kind load docker-image logos-blockchain-testing:local
+
+# OR: Use minikube
+minikube start
+minikube image load logos-blockchain-testing:local
+Remote cluster: Push image to registry and set NOMOS_TESTNET_IMAGE.
Best for:
+POL_PROOF_DEV_MODE=true is REQUIRED for ALL runners!
Without this, proof generation uses expensive Groth16 proving, causing:
+Always set:
+POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+POL_PROOF_DEV_MODE=true scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+# etc.
+Or add to your shell profile:
+# ~/.bashrc or ~/.zshrc
+export POL_PROOF_DEV_MODE=true
+Run this checklist before your first scenario:
+# 1. Verify versions.env exists
+cat versions.env
+
+# 2. Check circuit assets (for DA workloads)
+ls -lh testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params
+
+# 3. Verify POL_PROOF_DEV_MODE is set
+echo $POL_PROOF_DEV_MODE # Should print: true
+
+# 4. For compose/k8s: verify Docker is running
+docker ps
+
+# 5. For compose/k8s: verify image exists
+docker images | grep logos-blockchain-testing
+
+# 6. For host runner: verify node binaries (if not using scripts)
+$NOMOS_NODE_BIN --version
+$NOMOS_EXECUTOR_BIN --version
+The easiest path is to let the helper scripts handle everything:
+# Host runner
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
+
+# Compose runner
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+
+# K8s runner
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
+These scripts:
+versions.env existsNext Steps:
+Declarative, multi-node blockchain testing for the Logos network
+The Logos Testing Framework enables you to test consensus, data availability, and transaction workloads across local processes, Docker Compose, and Kubernetes deployments—all with a unified scenario API.
+ +Everything in this framework is a Scenario.
+A Scenario is a controlled experiment over time, composed of:
+This single abstraction makes tests declarative, portable, and composable.
+flowchart LR
+ Build[Define Scenario] --> Deploy[Deploy Topology]
+ Deploy --> Execute[Run Workloads]
+ Execute --> Evaluate[Check Expectations]
+
+ style Build fill:#e1f5ff
+ style Deploy fill:#fff4e1
+ style Execute fill:#ffe1f5
+ style Evaluate fill:#e1ffe1
+Declarative API
+Multiple Deployment Modes
+Built-in Workloads
+Comprehensive Observability
+use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_core::scenario::Deployer as _;
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+#[tokio::main]
+async fn main() -> anyhow::Result<()> {
+ let mut scenario = ScenarioBuilder::topology_with(|t| {
+ t.network_star()
+ .validators(3)
+ .executors(1)
+ })
+ .transactions_with(|tx| tx.rate(10).users(5))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&scenario).await?;
+ runner.run(&mut scenario).await?;
+
+ Ok(())
+}Start with the Quickstart Guide for a hands-on introduction that gets you running tests in minutes.
+Explore the User Guide to learn about authoring scenarios, workloads, expectations, and deployment strategies.
+Jump to Operations & Deployment for prerequisites, environment configuration, and continuous integration patterns.
+Check the Developer Reference to implement custom workloads, expectations, and runners.
+Logos is a modular blockchain protocol composed of validators, executors, and a data-availability (DA) subsystem:
+These roles interact tightly, which is why meaningful testing must be performed in multi-node environments that include real networking, timing, and DA interaction.
+The Logos Testing Framework provides the infrastructure to orchestrate these multi-node scenarios reliably across development, CI, and production-like environments.
+Learn more about the protocol: Logos Project Documentation
+| Section | Description |
|---|---|
| Foundations | Architecture, philosophy, and design principles |
| User Guide | Writing and running scenarios, workloads, and expectations |
| Developer Reference | Extending the framework with custom components |
| Operations & Deployment | Setup, CI integration, and environment configuration |
| Appendix | Quick reference, troubleshooting, FAQ, and glossary |
Ready to start? Head to the Quickstart
+This book gives you a clear mental model for Logos multi-node testing, shows how +to author scenarios that pair realistic workloads with explicit expectations, +and guides you to run them across local, containerized, and cluster environments +without changing the plan.
+Understand the Framework
+Author and Run Scenarios
+Operate in Production
+Extend the Framework
+Beginner (0-2 hours)
+Intermediate (2-8 hours)
+Advanced (8+ hours)
+nomos-node) for protocol documentation.Get a working example running quickly.
+NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64 to avoid slow/fragile amd64 emulation buildsversions.env file at repository root (defines VERSION, NOMOS_NODE_REV, NOMOS_BUNDLE_VERSION)Note: nomos-node binaries are built automatically on demand or can be provided via prebuilt bundles.
Important: The versions.env file is required by helper scripts. If missing, the scripts will fail with an error. The file should already exist in the repository root.
The framework ships with runnable example binaries in examples/src/bin/.
Recommended: Use the convenience script:
+# From the logos-blockchain-testing directory
+scripts/run/run-examples.sh -t 60 -v 1 -e 1 host
+This handles circuit setup, binary building, and runs a complete scenario: 1 validator + 1 executor, mixed transaction + DA workload (5 tx/block + 1 channel + 1 blob), 60s duration.
+Alternative: Direct cargo run (requires manual setup):
+# Requires circuits in place and NOMOS_NODE_BIN/NOMOS_EXECUTOR_BIN set
+POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+Core API Pattern (simplified example):
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub async fn run_local_demo() -> Result<()> {
+ // Define the scenario (1 validator + 1 executor, tx + DA workload)
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(1))
+ .wallets(1_000)
+ .transactions_with(|txs| {
+ txs.rate(5) // 5 transactions per block
+ .users(500) // use 500 of the seeded wallets
+ })
+ .da_with(|da| {
+ da.channel_rate(1) // 1 channel
+ .blob_rate(1) // target 1 blob per block
+ .headroom_percent(20) // default headroom when sizing channels
+ })
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+
+ // Deploy and run
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}Note: The examples are binaries with #[tokio::main], not test functions. If you want to write integration tests, wrap this pattern in #[tokio::test] functions in your own test suite.
Important: POL_PROOF_DEV_MODE=true disables expensive Groth16 zero-knowledge proof generation for leader election. Without it, proof generation is CPU-intensive and tests will timeout. This is required for all runners (local, compose, k8s) for practical testing. Never use in production.
What you should see:
+NOMOS_TESTS_KEEP_LOGS=1)NOMOS_LOG_DIR=/path/to/logs (files will have prefix like nomos-node-0*, may include timestamps)Let's unpack the code:
+use testing_framework_core::scenario::ScenarioBuilder;
+
+pub fn step_1_topology() -> testing_framework_core::scenario::Builder<()> {
+ ScenarioBuilder::topology_with(|t| {
+ t.network_star() // Star topology: all nodes connect to seed
+ .validators(1) // 1 validator node
+ .executors(1) // 1 executor node (validator + DA dispersal)
+ })
+}This defines what your test network looks like.
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn step_2_wallets() -> testing_framework_core::scenario::Builder<()> {
+ ScenarioBuilder::with_node_counts(1, 1).wallets(1_000) // Seed 1,000 funded wallet accounts
+}Provides funded accounts for transaction submission.
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn step_3_workloads() -> testing_framework_core::scenario::Builder<()> {
+ ScenarioBuilder::with_node_counts(1, 1)
+ .wallets(1_000)
+ .transactions_with(|txs| {
+ txs.rate(5) // 5 transactions per block
+ .users(500) // Use 500 of the 1,000 wallets
+ })
+ .da_with(|da| {
+ da.channel_rate(1) // 1 DA channel (more spawned with headroom)
+ .blob_rate(1) // target 1 blob per block
+ .headroom_percent(20) // default headroom when sizing channels
+ })
+}Generates both transaction and DA traffic to stress both subsystems.
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn step_4_expectation() -> testing_framework_core::scenario::Builder<()> {
+ ScenarioBuilder::with_node_counts(1, 1).expect_consensus_liveness() // This says what success means: blocks must be produced continuously.
+}This says what success means: blocks must be produced continuously.
+use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+
+pub fn step_5_run_duration() -> testing_framework_core::scenario::Builder<()> {
+ ScenarioBuilder::with_node_counts(1, 1).with_run_duration(Duration::from_secs(60))
+}Run for 60 seconds (~27 blocks with default 2s slots, 0.9 coefficient). Framework ensures this is at least 2× the consensus slot duration. Adjust consensus timing via CONSENSUS_SLOT_TIME and CONSENSUS_ACTIVE_SLOT_COEFF.
use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+
+pub async fn step_6_deploy_and_execute() -> Result<()> {
+ let mut plan = ScenarioBuilder::with_node_counts(1, 1).build();
+
+ let deployer = LocalDeployer::default(); // Use local process deployer
+ let runner = deployer.deploy(&plan).await?; // Provision infrastructure
+ let _handle = runner.run(&mut plan).await?; // Execute workloads & expectations
+
+ Ok(())
+}Deployer provisions the infrastructure. Runner orchestrates execution.
+With run-examples.sh (recommended):
+# Scale up to 3 validators + 2 executors, run for 2 minutes
+scripts/run/run-examples.sh -t 120 -v 3 -e 2 host
+With direct cargo run:
+# Uses NOMOS_DEMO_* env vars (or legacy *_DEMO_* vars)
+NOMOS_DEMO_VALIDATORS=3 \
+NOMOS_DEMO_EXECUTORS=2 \
+NOMOS_DEMO_RUN_SECS=120 \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin local_runner
+Use the same API with a different deployer for reproducible containerized environment.
+Recommended: Use the convenience script (handles everything):
+scripts/run/run-examples.sh -t 60 -v 1 -e 1 compose
+This automatically:
+testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params)NOMOS_BINARIES_TAR if available)Alternative: Direct cargo run with manual setup:
+# Option 1: Use prebuilt bundle (recommended for compose/k8s)
+scripts/build/build-bundle.sh --platform linux # Creates .tmp/nomos-binaries-linux-v0.3.1.tar.gz
+export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
+
+# Option 2: Manual circuit/image setup (rebuilds during image build)
+scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits
+cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/
+scripts/build/build_test_image.sh
+
+# Run with Compose
+NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin compose_runner
+Benefit: Reproducible containerized environment (Dockerized nodes, repeatable deployments).
+Optional: Prometheus + Grafana
+The runner can integrate with external observability endpoints. For a ready-to-run local stack:
+scripts/setup/setup-observability.sh compose up
+eval "$(scripts/setup/setup-observability.sh compose env)"
+Then run your compose scenario as usual (the environment variables enable PromQL querying and node OTLP metrics export).
+Note: Compose expects KZG parameters at /kzgrs_test_params/kzgrs_test_params inside containers (the directory name is repeated as the filename).
In code: Just swap the deployer:
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_compose::ComposeDeployer;
+
+pub async fn run_with_compose_deployer() -> Result<()> {
+ // ... same scenario definition ...
+ let mut plan = ScenarioBuilder::with_node_counts(1, 1).build();
+
+ let deployer = ComposeDeployer::default(); // Use Docker Compose
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}Now that you have a working test:
+Conceptual chapters that establish the mental model for the framework and how +it approaches multi-node testing.
+The Logos Testing Framework is a purpose-built toolkit for exercising Logos in +realistic, multi-node environments. It solves the gap between small, isolated +tests and full-system validation by letting teams describe a cluster layout, +drive meaningful traffic, and assert the outcomes in one coherent plan.
+It is for protocol engineers, infrastructure operators, and QA teams who need +repeatable confidence that validators, executors, and data-availability +components work together under network and timing constraints.
+Multi-node integration testing is required because many Logos behaviors—block +progress, data availability, liveness under churn—only emerge when several +roles interact over real networking and time. This framework makes those checks +declarative, observable, and portable across environments.
+Here's the conceptual shape of every test you'll write:
+// 1. Define the cluster
+let scenario = ScenarioBuilder::topology_with(|t| {
+ t.network_star()
+ .validators(3)
+ .executors(2)
+})
+// 2. Add workloads (traffic)
+.transactions_with(|tx| tx.rate(10).users(5))
+.da_with(|da| da.channel_rate(2).blob_rate(2))
+
+// 3. Define success criteria
+.expect_consensus_liveness()
+
+// 4. Set experiment duration
+.with_run_duration(Duration::from_secs(60))
+.build();
+
+// 5. Deploy and run
+let runner = deployer.deploy(&scenario).await?;
+runner.run(&mut scenario).await?;This pattern—topology, workloads, expectations, duration—repeats across all scenarios in this book.
+Learn more: For protocol-level documentation and node internals, see the Logos Project Documentation.
+The framework follows a clear flow: Topology → Scenario → Deployer → Runner → Workloads → Expectations.
+flowchart LR
+ A(Topology<br/>shape cluster) --> B(Scenario<br/>plan)
+ B --> C(Deployer<br/>provision & readiness)
+ C --> D(Runner<br/>orchestrate execution)
+ D --> E(Workloads<br/>drive traffic)
+ E --> F(Expectations<br/>verify outcomes)
+flowchart TB
+ subgraph Examples["Runner Examples"]
+ LocalBin[local_runner.rs]
+ ComposeBin[compose_runner.rs]
+ K8sBin[k8s_runner.rs]
+ CucumberBin[cucumber_*.rs]
+ end
+
+ subgraph Workflows["Workflows (Batteries Included)"]
+ DSL[ScenarioBuilderExt<br/>Fluent API]
+ TxWorkload[Transaction Workload]
+ DAWorkload[DA Workload]
+ ChaosWorkload[Chaos Workload]
+ Expectations[Built-in Expectations]
+ end
+
+ subgraph Core["Core Framework"]
+ ScenarioModel[Scenario Model]
+ Traits[Deployer + Runner Traits]
+ BlockFeed[BlockFeed]
+ NodeClients[Node Clients]
+ Topology[Topology Generation]
+ end
+
+ subgraph Deployers["Runner Implementations"]
+ LocalDeployer[LocalDeployer]
+ ComposeDeployer[ComposeDeployer]
+ K8sDeployer[K8sDeployer]
+ end
+
+ subgraph Support["Supporting Crates"]
+ Configs[Configs & Topology]
+ Nodes[Node API Clients]
+ Cucumber[Cucumber Extensions]
+ end
+
+ Examples --> Workflows
+ Examples --> Deployers
+ Workflows --> Core
+ Deployers --> Core
+ Deployers --> Support
+ Core --> Support
+ Workflows --> Support
+
+ style Examples fill:#e1f5ff
+ style Workflows fill:#e1ffe1
+ style Core fill:#fff4e1
+ style Deployers fill:#ffe1f5
+ style Support fill:#f0f0f0
+Runner Examples (Entry Points)
+Workflows (High-Level API)
+ScenarioBuilderExt trait provides fluent DSLCore Framework (Foundation)
+Scenario model and lifecycle orchestrationDeployer and Runner traits (extension points)BlockFeed for real-time block observationRunContext providing node clients and metricsRunner Implementations
+LocalDeployer - spawns processes on hostComposeDeployer - orchestrates Docker ComposeK8sDeployer - deploys to Kubernetes clusterDeployer traitSupporting Crates
+configs - Topology configuration and generationnodes - HTTP/RPC client for node APIscucumber - BDD/Gherkin integrationflowchart LR
+ Custom[Your Code] -.implements.-> Workload[Workload Trait]
+ Custom -.implements.-> Expectation[Expectation Trait]
+ Custom -.implements.-> Deployer[Deployer Trait]
+
+ Workload --> Core[Core Framework]
+ Expectation --> Core
+ Deployer --> Core
+
+ style Custom fill:#ffe1f5
+ style Core fill:#fff4e1
+Extend by implementing:
+Workload - Custom traffic generation patternsExpectation - Custom success criteriaDeployer - Support for new deployment targetsSee Extending the Framework for details.
+Each layer has a narrow responsibility so that cluster shape, deployment choice, +traffic generation, and health checks can evolve independently while fitting +together predictably.
+The framework is consumed via runnable example binaries in examples/src/bin/:
local_runner.rs — Spawns nodes as host processescompose_runner.rs — Deploys via Docker Compose (requires NOMOS_TESTNET_IMAGE built)k8s_runner.rs — Deploys via Kubernetes Helm (requires cluster + image)Recommended: Use the convenience script:
+scripts/run/run-examples.sh -t <duration> -v <validators> -e <executors> <mode>
+# mode: host, compose, or k8s
+This handles circuit setup, binary building/bundling, image building, and execution.
+Alternative: Direct cargo run (requires manual setup):
+POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin <name>
+Important: All runners require POL_PROOF_DEV_MODE=true to avoid expensive Groth16 proof generation that causes timeouts.
These binaries use the framework API (ScenarioBuilder) to construct and execute scenarios.
Scenarios are defined using a fluent builder pattern:
+use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn scenario_plan() -> testing_framework_core::scenario::Scenario<()> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .wallets(50)
+ .transactions_with(|txs| txs.rate(5).users(20))
+ .da_with(|da| da.channel_rate(1).blob_rate(2))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(90))
+ .build()
+}Key API Points:
+.topology_with(|t| { t.validators(N).executors(M) }) closure pattern_with closures (transactions_with, da_with, chaos_with).enable_node_control() and a compatible runnerThree deployer implementations:
+| Deployer | Backend | Prerequisites | Node Control |
|---|---|---|---|
LocalDeployer | Host processes | Binaries (built on demand or via bundle) | No |
ComposeDeployer | Docker Compose | Image with embedded assets/binaries | Yes |
K8sDeployer | Kubernetes Helm | Cluster + image loaded | Not yet |
Compose-specific features:
+NOMOS_METRICS_QUERY_URL / NOMOS_METRICS_OTLP_INGEST_URL / NOMOS_GRAFANA_URL as needed)NOMOS_OTLP_ENDPOINT, NOMOS_OTLP_METRICS_ENDPOINT)Built via scripts/build/build_test_image.sh:
testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_paramsrun_nomos_node.sh, run_nomos_executor.shNOMOS_TESTNET_IMAGE (default: logos-blockchain-testing:local)scripts/build/build-bundle.sh --platform linux and set NOMOS_BINARIES_TAR before building imageKZG parameters required for DA workloads:
+testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params (note repeated filename—directory contains file kzgrs_test_params)/kzgrs_test_params/kzgrs_test_params (for compose/k8s)NOMOS_KZGRS_PARAMS_PATH=/custom/path/to/file (must point to file)scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/circuits or use scripts/run/run-examples.shTemplates and configs in testing-framework/runners/compose/assets/:
docker-compose.yml.tera — Stack template (validators, executors)testing-framework/assets/stack/cfgsync.yamltesting-framework/assets/stack/monitoring/Two separate logging pipelines:
+| Component | Configuration | Output |
|---|---|---|
| Runner binaries | RUST_LOG | Framework orchestration logs |
| Node processes | NOMOS_LOG_LEVEL, NOMOS_LOG_FILTER (+ NOMOS_LOG_DIR on host runner) | Consensus, DA, mempool logs |
Node logging:
+NOMOS_TESTS_TRACING=true + NOMOS_LOG_DIR for persistent files.docker logs). To write per-node files, set tracing_settings.logger: !File in testing-framework/assets/stack/cfgsync.yaml (and mount a writable directory).kubectl logs). To write per-node files, set tracing_settings.logger: !File in testing-framework/assets/stack/cfgsync.yaml (and mount a writable directory).File naming: Per-node files use prefix nomos-node-{index} or nomos-executor-{index} (may include timestamps).
Prometheus-compatible metrics querying (optional):
+NOMOS_METRICS_QUERY_URL.ctx.telemetry().prometheus().map(|p| p.base_url())Grafana dashboards (optional):
+testing-framework/assets/stack/monitoring/grafana/dashboards/ and can be imported into your Grafana.NOMOS_GRAFANA_URL, the deployer prints it in TESTNET_ENDPOINTS.Node APIs:
+ctx.node_clients().validator_clients().get(0)OTLP (optional):
+NOMOS_OTLP_ENDPOINT=http://localhost:4317NOMOS_OTLP_METRICS_ENDPOINT=http://localhost:4318For detailed logging configuration, see Logging & Observability.
+This framework embodies specific principles that shape how you author and run +scenarios. Understanding these principles helps you write effective tests and +interpret results correctly.
+Describe what you want to test, not how to orchestrate it:
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn declarative_over_imperative() {
+ // Good: declarative
+ let _plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(2).executors(1))
+ .transactions_with(|txs| {
+ txs.rate(5) // 5 transactions per block
+ })
+ .expect_consensus_liveness()
+ .build();
+
+ // Bad: imperative (framework doesn't work this way)
+ // spawn_validator(); spawn_executor();
+ // loop { submit_tx(); check_block(); }
+}Why it matters: The framework handles deployment, readiness, and cleanup. +You focus on test intent, not infrastructure orchestration.
+Reason in blocks and consensus intervals, not wall-clock seconds.
+Consensus defaults:
+CONSENSUS_SLOT_TIME)CONSENSUS_ACTIVE_SLOT_COEFF)use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn protocol_time_not_wall_time() {
+ // Good: protocol-oriented thinking
+ let _plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(2).executors(1))
+ .transactions_with(|txs| {
+ txs.rate(5) // 5 transactions per block
+ })
+ .with_run_duration(Duration::from_secs(60)) // Let framework calculate expected blocks
+ .expect_consensus_liveness() // "Did we produce the expected blocks?"
+ .build();
+
+ // Bad: wall-clock assumptions
+ // "I expect exactly 30 blocks in 60 seconds"
+ // This breaks on slow CI where slot timing might drift
+}Why it matters: Slot timing is fixed (2s by default, NTP-synchronized), so the +expected number of blocks is predictable: ~27 blocks in 60s with the default +0.9 active slot coefficient. The framework calculates expected blocks from slot +duration and run window, making assertions protocol-based rather than tied to +specific wall-clock expectations. Assert on "blocks produced relative to slots" +not "blocks produced in exact wall-clock seconds".
+Default scenarios are repeatable:
+Chaos is opt-in:
+use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt};
+
+pub fn determinism_first() {
+ // Separate: functional test (deterministic)
+ let _plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(2).executors(1))
+ .transactions_with(|txs| {
+ txs.rate(5) // 5 transactions per block
+ })
+ .expect_consensus_liveness()
+ .build();
+
+ // Separate: chaos test (introduces randomness)
+ let _chaos_plan =
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .enable_node_control()
+ .chaos_with(|c| {
+ c.restart()
+ .min_delay(Duration::from_secs(30))
+ .max_delay(Duration::from_secs(60))
+ .target_cooldown(Duration::from_secs(45))
+ .apply()
+ })
+ .transactions_with(|txs| {
+ txs.rate(5) // 5 transactions per block
+ })
+ .expect_consensus_liveness()
+ .build();
+}Why it matters: Mixing determinism with chaos creates noisy, hard-to-debug +failures. Separate concerns make failures actionable.
+Prefer user-facing signals over internal state:
+Good checks:
+Avoid internal checks:
+Why it matters: User-facing signals reflect actual system health. +Internal state can be "healthy" while the system is broken from a user +perspective.
+Always run long enough for meaningful block production:
+use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn minimum_run_windows() {
+ // Bad: too short (~2 blocks with default 2s slots, 0.9 coeff)
+ let _too_short = ScenarioBuilder::with_node_counts(1, 0)
+ .with_run_duration(Duration::from_secs(5))
+ .expect_consensus_liveness()
+ .build();
+
+ // Good: enough blocks for assertions (~27 blocks with default 2s slots, 0.9
+ // coeff)
+ let _good = ScenarioBuilder::with_node_counts(1, 0)
+ .with_run_duration(Duration::from_secs(60))
+ .expect_consensus_liveness()
+ .build();
+}Note: Block counts assume default consensus parameters:
+CONSENSUS_SLOT_TIME)CONSENSUS_ACTIVE_SLOT_COEFF)blocks ≈ (duration / slot_duration) × active_slot_coeffIf upstream changes these parameters, adjust your duration expectations accordingly.
+The framework enforces minimum durations (at least 2× slot duration), but be explicit. Very short runs risk false confidence—one lucky block doesn't prove liveness.
+These principles keep scenarios:
+When authoring scenarios, ask: "Does this test the protocol behavior or +my local environment quirks?"
+A scenario progresses through six distinct phases, each with a specific responsibility:
+flowchart TB
+ subgraph Phase1["1. Build Phase"]
+ Build[Define Scenario]
+ BuildDetails["• Declare topology<br/>• Attach workloads<br/>• Add expectations<br/>• Set run duration"]
+ Build --> BuildDetails
+ end
+
+ subgraph Phase2["2. Deploy Phase"]
+ Deploy[Provision Environment]
+ DeployDetails["• Launch nodes<br/>• Wait for readiness<br/>• Establish connectivity<br/>• Return Runner"]
+ Deploy --> DeployDetails
+ end
+
+ subgraph Phase3["3. Capture Phase"]
+ Capture[Baseline Metrics]
+ CaptureDetails["• Snapshot initial state<br/>• Start BlockFeed<br/>• Initialize expectations"]
+ Capture --> CaptureDetails
+ end
+
+ subgraph Phase4["4. Execution Phase"]
+ Execute[Drive Workloads]
+ ExecuteDetails["• Submit transactions<br/>• Disperse DA blobs<br/>• Trigger chaos events<br/>• Run for duration"]
+ Execute --> ExecuteDetails
+ end
+
+ subgraph Phase5["5. Evaluation Phase"]
+ Evaluate[Check Expectations]
+ EvaluateDetails["• Verify liveness<br/>• Check inclusion<br/>• Validate outcomes<br/>• Aggregate results"]
+ Evaluate --> EvaluateDetails
+ end
+
+ subgraph Phase6["6. Cleanup Phase"]
+ Cleanup[Teardown]
+ CleanupDetails["• Stop nodes<br/>• Remove containers<br/>• Collect logs<br/>• Release resources"]
+ Cleanup --> CleanupDetails
+ end
+
+ Phase1 --> Phase2
+ Phase2 --> Phase3
+ Phase3 --> Phase4
+ Phase4 --> Phase5
+ Phase5 --> Phase6
+
+ style Phase1 fill:#e1f5ff
+ style Phase2 fill:#fff4e1
+ style Phase3 fill:#f0ffe1
+ style Phase4 fill:#ffe1f5
+ style Phase5 fill:#e1ffe1
+ style Phase6 fill:#ffe1e1
+Declare a topology, attach workloads and expectations, and set the run window. The plan is the single source of truth for what will happen.
+Key actions:
+Output: Immutable Scenario plan
Hand the plan to a deployer. It provisions the environment on the chosen backend, waits for nodes to signal readiness, and returns a runner.
+Key actions:
+Output: Runner + RunContext (with node clients, metrics, control handles)
Expectations snapshot initial state before workloads begin.
+Key actions:
+Output: Captured state for later comparison
+The runner starts traffic and behaviors for the planned duration.
+Key actions:
+Duration: Controlled by with_run_duration()
Once activity stops (and optional cooldown completes), the runner checks liveness and workload-specific outcomes.
+Key actions:
+Output: Success or detailed failure report
+Tear down resources so successive runs start fresh and do not inherit leaked state.
+Key actions:
+NOMOS_TESTS_KEEP_LOGS=1)Guarantee: Runs even on panic via CleanupGuard
Practical guidance for shaping scenarios, combining workloads and expectations, +and running them across different environments.
+The workspace focuses on multi-node integration testing and sits alongside a
+nomos-node checkout. Its crates separate concerns to keep scenarios
+repeatable and portable:
runner-examples, path: examples/):
+runnable binaries (examples/src/bin/local_runner.rs,
+examples/src/bin/compose_runner.rs, examples/src/bin/k8s_runner.rs) that
+demonstrate complete scenario execution with each deployer.This split keeps configuration, orchestration, reusable traffic patterns, and +deployment adapters loosely coupled while sharing one mental model for tests.
+Directory structure with key paths annotated:
+logos-blockchain-testing/
+├─ testing-framework/ # Core library crates
+│ ├─ configs/ # Node config builders, topology generation, tracing/logging config
+│ ├─ core/ # Scenario model (ScenarioBuilder), runtime (Runner, Deployer), topology, node spawning
+│ ├─ workflows/ # Workloads (transactions, DA, chaos), expectations (liveness), builder DSL extensions
+│ ├─ runners/ # Deployment backends
+│ │ ├─ local/ # LocalDeployer (spawns local processes)
+│ │ ├─ compose/ # ComposeDeployer (Docker Compose + Prometheus)
+│ │ └─ k8s/ # K8sDeployer (Kubernetes Helm)
+│ └─ assets/ # Docker/K8s stack assets
+│ └─ stack/
+│ ├─ kzgrs_test_params/ # KZG circuit parameters directory
+│ │ └─ kzgrs_test_params # Actual proving key file (note repeated name)
+│ ├─ monitoring/ # Prometheus config
+│ ├─ scripts/ # Container entrypoints
+│ └─ cfgsync.yaml # Config sync server template
+│
+├─ examples/ # PRIMARY ENTRY POINT: runnable binaries
+│ └─ src/bin/
+│ ├─ local_runner.rs # Host processes demo (LocalDeployer)
+│ ├─ compose_runner.rs # Docker Compose demo (ComposeDeployer)
+│ └─ k8s_runner.rs # Kubernetes demo (K8sDeployer)
+│
+├─ scripts/ # Helper utilities
+│ ├─ run-examples.sh # Convenience script (handles setup + runs examples)
+│ ├─ build-bundle.sh # Build prebuilt binaries+circuits bundle
+│ ├─ setup-circuits-stack.sh # Fetch KZG parameters (Linux + host)
+│ └─ setup-nomos-circuits.sh # Legacy circuit fetcher
+│
+└─ book/ # This documentation (mdBook)
+testing-framework/Core library crates providing the testing API.
+| Crate | Purpose | Key Exports |
|---|---|---|
configs | Node configuration builders | Topology generation, tracing config |
core | Scenario model & runtime | ScenarioBuilder, Deployer, Runner |
workflows | Workloads & expectations | ScenarioBuilderExt, ChaosBuilderExt |
runners/local | Local process deployer | LocalDeployer |
runners/compose | Docker Compose deployer | ComposeDeployer |
runners/k8s | Kubernetes deployer | K8sDeployer |
testing-framework/assets/stack/Docker/K8s deployment assets:
+kzgrs_test_params/kzgrs_test_params: Circuit parameters file (note repeated name; override via NOMOS_KZGRS_PARAMS_PATH)monitoring/: Prometheus configscripts/: Container entrypointsscripts/Convenience utilities:
+run-examples.sh: All-in-one script for host/compose/k8s modes (recommended)build-bundle.sh: Create prebuilt binaries+circuits bundle for compose/k8sbuild_test_image.sh: Build the compose/k8s Docker image (bakes in assets)setup-circuits-stack.sh: Fetch KZG parameters for both Linux and hostcfgsync.yaml: Configuration sync server templateexamples/ (Start Here!)Runnable binaries demonstrating framework usage:
+local_runner.rs — Local processescompose_runner.rs — Docker Compose (requires NOMOS_TESTNET_IMAGE built)k8s_runner.rs — Kubernetes (requires cluster + image)Run with: POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin <name>
All runners require POL_PROOF_DEV_MODE=true to avoid expensive proof generation.
scripts/Helper utilities:
+setup-nomos-circuits.sh: Fetch KZG parameters from releasesCompose runner includes:
+http://localhost:9090 (metrics scraping)ctx.telemetry().prometheus().map(|p| p.base_url())Logging controlled by:
+NOMOS_LOG_DIR — Write per-node log filesNOMOS_LOG_LEVEL — Global log level (error/warn/info/debug/trace)NOMOS_LOG_FILTER — Target-specific filtering (e.g., cryptarchia=trace,nomos_da_sampling=debug)NOMOS_TESTS_TRACING — Enable file logging for local runnerSee Logging & Observability for details.
+| To Do This | Go Here |
|---|---|
| Run an example | examples/src/bin/ → cargo run -p runner-examples --bin <name> |
| Write a custom scenario | testing-framework/core/ → Implement using ScenarioBuilder |
| Add a new workload | testing-framework/workflows/src/workloads/ → Implement Workload trait |
| Add a new expectation | testing-framework/workflows/src/expectations/ → Implement Expectation trait |
| Modify node configs | testing-framework/configs/src/topology/configs/ |
| Extend builder DSL | testing-framework/workflows/src/builder/ → Add trait methods |
| Add a new deployer | testing-framework/runners/ → Implement Deployer trait |
For detailed guidance, see Internal Crate Reference.
+Creating a scenario is a declarative exercise. This page walks you through the core authoring loop with concrete examples, explains the units and timing model, and shows how to structure scenarios in Rust test suites.
+Every scenario follows the same pattern:
+flowchart LR
+ A[1. Topology] --> B[2. Workloads]
+ B --> C[3. Expectations]
+ C --> D[4. Duration]
+ D --> E[5. Deploy & Run]
+Let's build a minimal consensus liveness test step-by-step.
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+let scenario = ScenarioBuilder::topology_with(|t| {
+ t.network_star() // Star network (one gateway + nodes)
+ .validators(3) // 3 validator nodes
+ .executors(1) // 1 executor node
+})What goes in topology?
+network_star() is currently the only built-in layout)What does NOT go in topology?
+.wallets(20) // Seed funded wallet accounts for transaction workloads
+.transactions_with(|tx| {
+ tx.rate(10) // 10 transactions per block
+ .users(5) // distributed across 5 wallets
+})What goes in workloads?
+Units explained:
+.rate(10) = 10 transactions per block (not per second!).users(5) = use 5 distinct wallet accounts.expect_consensus_liveness()What goes in expectations?
+When do expectations run? +After the duration window ends, during the evaluation phase of the scenario lifecycle.
+use std::time::Duration;
+
+.with_run_duration(Duration::from_secs(60))How long is enough?
+What happens during this window?
+.build();
+
+// Choose a runner
+use testing_framework_core::scenario::Deployer;
+use testing_framework_runner_local::LocalDeployer;
+
+let deployer = LocalDeployer::default();
+let runner = deployer.deploy(&scenario).await?;
+let _result = runner.run(&mut scenario).await?;Putting it all together:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+#[tokio::test]
+async fn hello_consensus_liveness() -> Result<()> {
+ let mut scenario = ScenarioBuilder::topology_with(|t| {
+ t.network_star()
+ .validators(3)
+ .executors(1)
+ })
+ .wallets(20)
+ .transactions_with(|tx| tx.rate(10).users(5))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&scenario).await?;
+ runner.run(&mut scenario).await?;
+
+ Ok(())
+}Run it:
+POL_PROOF_DEV_MODE=true cargo test hello_consensus_liveness
+Wrong mental model: .rate(10) = 10 tx/second
Correct mental model: .rate(10) = 10 tx/block
Why? The blockchain produces blocks at variable rates depending on consensus timing. The framework submits the configured rate per block to ensure predictable load regardless of block time.
+Example:
+.rate(10) → 10 tx/block → 5 tx/second average.rate(10) → 10 tx/block → 2 tx/second average.with_run_duration(Duration::from_secs(60)) means the scenario runs for 60 seconds of real time, not 60 blocks.
How many blocks will be produced? +Depends on consensus timing (slot time, active slot coefficient). Typical: 1-2 seconds per block.
+Rule of thumb:
+// tests/integration_test.rs
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+#[tokio::test]
+async fn test_consensus_liveness() -> Result<()> {
+ let mut scenario = ScenarioBuilder::topology_with(|t| {
+ t.network_star().validators(3).executors(1)
+ })
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(30))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&scenario).await?;
+ runner.run(&mut scenario).await?;
+ Ok(())
+}
+
+#[tokio::test]
+async fn test_transaction_inclusion() -> Result<()> {
+ let mut scenario = ScenarioBuilder::topology_with(|t| {
+ t.network_star().validators(2).executors(1)
+ })
+ .wallets(10)
+ .transactions_with(|tx| tx.rate(5).users(5))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&scenario).await?;
+ runner.run(&mut scenario).await?;
+ Ok(())
+}Extract common topology patterns:
+// tests/helpers.rs
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn minimal_topology() -> ScenarioBuilder {
+ ScenarioBuilder::topology_with(|t| {
+ t.network_star().validators(2).executors(1)
+ })
+}
+
+pub fn production_like_topology() -> ScenarioBuilder {
+ ScenarioBuilder::topology_with(|t| {
+ t.network_star().validators(7).executors(3)
+ })
+}
+
+// tests/consensus_tests.rs
+use std::time::Duration;
+
+use helpers::*;
+
+#[tokio::test]
+async fn small_cluster_liveness() -> anyhow::Result<()> {
+ let mut scenario = minimal_topology()
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(30))
+ .build();
+ // ... deploy and run
+ Ok(())
+}
+
+#[tokio::test]
+async fn large_cluster_liveness() -> anyhow::Result<()> {
+ let mut scenario = production_like_topology()
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+ // ... deploy and run
+ Ok(())
+}Test the same behavior across different scales:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+async fn test_liveness_with_topology(validators: usize, executors: usize) -> Result<()> {
+ let mut scenario = ScenarioBuilder::topology_with(|t| {
+ t.network_star()
+ .validators(validators)
+ .executors(executors)
+ })
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&scenario).await?;
+ runner.run(&mut scenario).await?;
+ Ok(())
+}
+
+#[tokio::test]
+async fn liveness_small() -> Result<()> {
+ test_liveness_with_topology(2, 1).await
+}
+
+#[tokio::test]
+async fn liveness_medium() -> Result<()> {
+ test_liveness_with_topology(5, 2).await
+}
+
+#[tokio::test]
+async fn liveness_large() -> Result<()> {
+ test_liveness_with_topology(10, 3).await
+}Do include:
+.validators(3), .executors(1)).network_star())Don't include:
+Do include:
+.transactions_with(|tx| ...)).da_with(|da| ...)).with_workload(RandomRestartWorkload::new(...)))Don't include:
+Do include:
+.expect_consensus_liveness()).with_expectation(MyExpectation::new()))Don't include:
+test_consensus_survives_validator_restart not test_1.rate(N) is per-block, .with_run_duration() is wall-clockWorkloads describe the activity a scenario generates; expectations describe the signals that must hold when that activity completes. This page is the canonical reference for all built-in workloads and expectations, including configuration knobs, defaults, prerequisites, and debugging guidance.
+flowchart TD
+ I[Inputs<br/>topology + wallets + rates] --> Init[Workload init]
+ Init --> Drive[Drive traffic]
+ Drive --> Collect[Collect signals]
+ Collect --> Eval[Expectations evaluate]
+Key concepts:
+Submits user-level transactions at a configurable rate to exercise transaction processing and inclusion paths.
+Import:
+use testing_framework_workflows::workloads::transaction::Workload;| Parameter | Type | Default | Description |
|---|---|---|---|
rate | u64 | Required | Transactions per block (not per second!) |
users | Option<usize> | All wallets | Number of distinct wallet accounts to use |
use testing_framework_workflows::ScenarioBuilderExt;
+
+ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
+ .wallets(20) // Seed 20 wallet accounts
+ .transactions_with(|tx| {
+ tx.rate(10) // 10 transactions per block
+ .users(5) // Use only 5 of the 20 wallets
+ })
+ .with_run_duration(Duration::from_secs(60))
+ .build();use testing_framework_workflows::workloads::transaction;
+
+let tx_workload = transaction::Workload::with_rate(10)
+ .expect("transaction rate must be non-zero");
+
+ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
+ .wallets(20)
+ .with_workload(tx_workload)
+ .with_run_duration(Duration::from_secs(60))
+ .build();Wallet accounts must be seeded:
+.wallets(N) // Before .transactions_with()The workload will fail during init() if no wallets are configured.
Proof generation must be fast:
+export POL_PROOF_DEV_MODE=true
+Without this, proof generation takes ~30-60 seconds per transaction, causing timeouts.
+Circuit artifacts must be available:
+scripts/run/run-examples.shscripts/setup/setup-circuits-stack.sh (recommended) / scripts/setup/setup-nomos-circuits.shTxInclusionExpectation — Verifies that submitted transactions were included in blocks.
+What it checks:
+N transactions were included on-chain (where N = rate × user count × expected block count)Failure modes:
+Error: Expectation failed: TxInclusionExpectation
+ Expected: >= 600 transactions (10 tx/block × 60 blocks)
+ Observed: 127 transactions
+
+ Possible causes:
+ - POL_PROOF_DEV_MODE not set (proof generation too slow)
+ - Duration too short (nodes still syncing)
+ - Node crashes (check logs for panics/OOM)
+ - Wallet accounts not seeded (check topology config)
+How to debug:
+grep "proof generation" $NOMOS_LOG_DIR/executor-0/*.log
+POL_PROOF_DEV_MODE=true was set.with_run_duration(Duration::from_secs(120)).rate(5) instead of .rate(10)Drives blob and channel activity to exercise data availability paths and storage.
+Import:
+use testing_framework_workflows::workloads::da::Workload;| Parameter | Type | Default | Description |
|---|---|---|---|
blob_rate_per_block | NonZeroU64 | Required | Blobs to publish per block |
channel_rate_per_block | NonZeroU64 | Required | Channels to create per block |
headroom_percent | u64 | 20 | Extra capacity for channel planning (avoids saturation) |
use testing_framework_workflows::ScenarioBuilderExt;
+
+ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .da_with(|da| {
+ da.channel_rate(2) // 2 channels per block
+ .blob_rate(4) // 4 blobs per block
+ })
+ .with_run_duration(Duration::from_secs(120))
+ .build();use std::num::NonZeroU64;
+use testing_framework_workflows::workloads::da;
+
+let da_workload = da::Workload::with_rate(
+ NonZeroU64::new(4).unwrap(), // blob_rate_per_block
+ NonZeroU64::new(2).unwrap(), // channel_rate_per_block
+ 20, // headroom_percent
+);
+
+ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .with_workload(da_workload)
+ .with_run_duration(Duration::from_secs(120))
+ .build();Executors must be present:
+.executors(N) // At least 1 executorDA workload requires executor nodes to handle blob publishing.
+Sufficient duration: +Channel creation and blob publishing are slower than transaction submission. Allow 120+ seconds.
+Circuit artifacts: +Same as transaction workload (POL_PROOF_DEV_MODE, circuits staged).
+DaWorkloadExpectation — Verifies blobs and channels were created and published.
+What it checks:
+N channels were created (where N = channel_rate × expected blocks)M blobs were published (where M = blob_rate × expected blocks × headroom)Failure modes:
+Error: Expectation failed: DaWorkloadExpectation
+ Expected: >= 60 channels (2 channels/block × 30 blocks)
+ Observed: 23 channels
+
+ Possible causes:
+ - Executors crashed or restarted (check executor logs)
+ - Duration too short (channels still being created)
+ - Blob publishing failed (check executor API errors)
+ - Network issues (check validator/executor connectivity)
+How to debug:
+grep "channel\|blob" $NOMOS_LOG_DIR/executor-0/*.log
+grep "panic\|killed" $NOMOS_LOG_DIR/executor-*/*.log
+.with_run_duration(Duration::from_secs(180)).channel_rate(1).blob_rate(2)Triggers controlled node restarts to test resilience and recovery behaviors.
+Import:
+use testing_framework_workflows::workloads::chaos::RandomRestartWorkload;| Parameter | Type | Default | Description |
|---|---|---|---|
min_delay | Duration | Required | Minimum time between restart attempts |
max_delay | Duration | Required | Maximum time between restart attempts |
target_cooldown | Duration | Required | Minimum time before restarting same node again |
include_validators | bool | Required | Whether to restart validators |
include_executors | bool | Required | Whether to restart executors |
use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::{ScenarioBuilderExt, workloads::chaos::RandomRestartWorkload};
+
+let scenario = ScenarioBuilder::topology_with(|t| {
+ t.network_star().validators(3).executors(2)
+})
+.enable_node_control() // REQUIRED for chaos
+.with_workload(RandomRestartWorkload::new(
+ Duration::from_secs(45), // min_delay
+ Duration::from_secs(75), // max_delay
+ Duration::from_secs(120), // target_cooldown
+ true, // include_validators
+ true, // include_executors
+))
+.expect_consensus_liveness()
+.with_run_duration(Duration::from_secs(180))
+.build();Node control must be enabled:
+.enable_node_control()This adds NodeControlCapability to the scenario.
Runner must support node control:
+Sufficient topology:
+Realistic timing:
+None. You must explicitly add expectations (typically .expect_consensus_liveness()).
Why? Chaos workloads are about testing recovery under disruption. The appropriate expectation depends on what you're testing:
+.expect_consensus_liveness()Error: Workload failed: chaos_restart
+ Cause: NodeControlHandle not available
+
+ Possible causes:
+ - Forgot .enable_node_control() in scenario builder
+ - Using local runner (doesn't support node control)
+ - Using k8s runner (doesn't support node control)
+Or:
+Error: Expectation failed: ConsensusLiveness
+ Expected: >= 20 blocks
+ Observed: 8 blocks
+
+ Possible causes:
+ - Restart frequency too high (nodes can't recover)
+ - Consensus timing too slow (increase duration)
+ - Too many validators restarted simultaneously
+ - Nodes crashed after restart (check logs)
+How to debug:
+grep "restarting\|restart complete" $NOMOS_LOG_DIR/*/*.log
+grep "NodeControlHandle" $NOMOS_LOG_DIR/*/*.log
+Duration::from_secs(180)include_validators = false (test executors only).with_run_duration(Duration::from_secs(300))Verifies the system continues to produce blocks during the execution window.
+Import:
+use testing_framework_workflows::ScenarioBuilderExt;ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();N blocks were produced (where N = duration / expected_block_time)Error: Expectation failed: ConsensusLiveness
+ Expected: >= 30 blocks
+ Observed: 3 blocks
+
+ Possible causes:
+ - Nodes crashed or never started (check logs)
+ - Consensus timing misconfigured (CONSENSUS_SLOT_TIME too high)
+ - Insufficient validators (need >= 2 for BFT consensus)
+ - Duration too short (nodes still syncing)
+grep "node started\|listening on" $NOMOS_LOG_DIR/*/*.log
+grep "block.*height" $NOMOS_LOG_DIR/validator-*/*.log
+grep "consensus.*slot\|proposal" $NOMOS_LOG_DIR/validator-*/*.log
+.with_run_duration(Duration::from_secs(120))echo $CONSENSUS_SLOT_TIME $CONSENSUS_ACTIVE_SLOT_COEFFEach workload automatically attaches its own expectation:
+| Workload | Expectation | What It Checks |
|---|---|---|
| Transaction | TxInclusionExpectation | Transactions were included in blocks |
| DA | DaWorkloadExpectation | Blobs and channels were created/published |
| Chaos | (None) | Add .expect_consensus_liveness() explicitly |
These expectations are added automatically when using the DSL (.transactions_with(), .da_with()).
.wallets(20)
+.transactions_with(|tx| tx.rate(10).users(5))| What | Value | Unit |
|---|---|---|
| Rate | 10 | tx/block |
| Users | 5 | wallet accounts |
| Wallets | 20 | total seeded |
.da_with(|da| da.channel_rate(2).blob_rate(4))| What | Value | Unit |
|---|---|---|
| Channel rate | 2 | channels/block |
| Blob rate | 4 | blobs/block |
| Headroom | 20 | percent |
.enable_node_control()
+.with_workload(RandomRestartWorkload::new(
+ Duration::from_secs(45), // min
+ Duration::from_secs(75), // max
+ Duration::from_secs(120), // cooldown
+ true, // validators
+ true, // executors
+))ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .wallets(20)
+ .transactions_with(|tx| tx.rate(5).users(10))
+ .da_with(|da| da.channel_rate(2).blob_rate(2))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(120))
+ .build();All workloads run concurrently. Expectations for each workload run after the execution window ends.
+use testing_framework_core::scenario::Expectation;
+
+struct MyCustomExpectation;
+
+#[async_trait]
+impl Expectation for MyCustomExpectation {
+ async fn evaluate(&self, ctx: &RunContext) -> Result<(), DynError> {
+ // Access BlockFeed, metrics, topology, etc.
+ let block_count = ctx.block_feed()?.count();
+ if block_count < 10 {
+ return Err("Not enough blocks".into());
+ }
+ Ok(())
+ }
+}
+
+ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
+ .with_expectation(MyCustomExpectation)
+ .with_run_duration(Duration::from_secs(60))
+ .build();When a workload or expectation fails:
+$NOMOS_LOG_DIR/*/ or docker compose logs or kubectl logsPOL_PROOF_DEV_MODE, NOMOS_NODE_BIN, etc.++When should I read this? After writing 2-3 scenarios. This page documents patterns that emerge from real usage—come back when you're refactoring or standardizing your test suite.
+
Patterns that keep scenarios readable and reusable:
+These patterns make scenario definitions self-explanatory while staying aligned +with the framework’s block-oriented timing model.
+This page collects proven patterns for authoring, running, and maintaining test scenarios that are reliable, maintainable, and actionable.
+State your intent
+star_topology_3val_2exec vs topology)Keep runs meaningful
+Separate concerns
+Start small, scale up
+Reuse patterns
+Example: Topology preset
+pub fn standard_da_topology() -> GeneratedTopology {
+ TopologyBuilder::new()
+ .network_star()
+ .validators(3)
+ .executors(2)
+ .generate()
+}Example: Shared constants
+pub const STANDARD_TX_RATE: f64 = 10.0;
+pub const STANDARD_DA_CHANNEL_RATE: f64 = 2.0;
+pub const SHORT_RUN_DURATION: Duration = Duration::from_secs(60);
+pub const LONG_RUN_DURATION: Duration = Duration::from_secs(300);Observe first, tune second
+RUST_LOG=debug, NOMOS_LOG_LEVEL=debug) only after initial failureNOMOS_TESTS_KEEP_LOGS=1 to persist logs when debugging failuresUse BlockFeed effectively
+block_feed.stats().total_transactions()) to verify inclusionCollect metrics
+scripts/setup/setup-observability.sh compose up for visualizing node behaviorEnvironment fit
+Runner-specific considerations
+| Runner | When to Use | When to Avoid |
|---|---|---|
| Host | Development iteration, fast feedback | Chaos testing, container-specific issues |
| Compose | CI pipelines, chaos tests, reproducibility | Very large topologies (>10 nodes) |
| K8s | Production-like testing, cluster behaviors | Local development, fast iteration |
Minimal surprises
+versions.env to pin node versions consistently across environmentsUse matrix builds
+strategy:
+ matrix:
+ runner: [host, compose]
+ topology: [small, medium]
+Cache aggressively
+target/)assets/stack/kzgrs_test_params/)Collect logs on failure
+- name: Collect logs on failure
+ if: failure()
+ run: |
+ mkdir -p test-logs
+ find /tmp -name "nomos-*.log" -exec cp {} test-logs/ \;
+- uses: actions/upload-artifact@v3
+ if: failure()
+ with:
+ name: test-logs-${{ matrix.runner }}
+ path: test-logs/
+Time limits
+timeout-minutes: 30See also: CI Integration for complete workflow examples
+DON'T: Run without POL_PROOF_DEV_MODE
+# BAD: Will hang/timeout on proof generation
+cargo run -p runner-examples --bin local_runner
+
+# GOOD: Fast mode for testing
+POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+DON'T: Use tiny durations
+// BAD: Not enough time for blocks to propagate
+.with_run_duration(Duration::from_secs(5))
+
+// GOOD: Allow multiple consensus rounds
+.with_run_duration(Duration::from_secs(60))DON'T: Ignore cleanup failures
+// BAD: Next run inherits leaked state
+runner.run(&mut scenario).await?;
+// forgot to call cleanup or use CleanupGuard
+
+// GOOD: Cleanup via guard (automatic on panic)
+let _cleanup = CleanupGuard::new(runner.clone());
+runner.run(&mut scenario).await?;DON'T: Mix concerns in one scenario
+// BAD: Hard to debug when it fails
+.transactions_with(|tx| tx.rate(50).users(100)) // high load
+.chaos_with(|c| c.restart().min_delay(...)) // AND chaos
+.da_with(|da| da.channel_rate(10).blob_rate(20)) // AND DA stress
+
+// GOOD: Separate tests for each concern
+// Test 1: High transaction load only
+// Test 2: Chaos resilience only
+// Test 3: DA stress onlyDON'T: Hardcode paths or ports
+// BAD: Breaks on different machines
+let path = PathBuf::from("/home/user/circuits/kzgrs_test_params");
+let port = 9000; // might conflict
+
+// GOOD: Use env vars and dynamic allocation
+let path = std::env::var("NOMOS_KZGRS_PARAMS_PATH")
+ .unwrap_or_else(|_| "assets/stack/kzgrs_test_params/kzgrs_test_params".to_string());
+let port = get_available_tcp_port();DON'T: Ignore resource limits
+# BAD: Large topology without checking resources
+scripts/run/run-examples.sh -v 20 -e 10 compose
+# (might OOM or exhaust ulimits)
+
+# GOOD: Scale gradually and monitor resources
+scripts/run/run-examples.sh -v 3 -e 2 compose # start small
+docker stats # monitor resource usage
+# then increase if resources allow
+Minimal viable topology
+Workload rate selection
+Duration guidelines
+| Test Type | Minimum Duration | Typical Duration |
|---|---|---|
| Smoke test | 30s | 60s |
| Integration test | 60s | 120s |
| Load test | 120s | 300s |
| Resilience test | 120s | 300s |
| Soak test | 600s (10m) | 3600s (1h) |
Expectation selection
+| Test Goal | Expectations |
|---|---|
| Basic functionality | expect_consensus_liveness() |
| Transaction handling | expect_consensus_liveness() + custom inclusion check |
| DA correctness | expect_consensus_liveness() + DA dispersal/sampling checks |
| Resilience | expect_consensus_liveness() + recovery time measurement |
Validate scenarios before committing
+Handling flaky tests
+See also:
+Concrete scenario shapes that illustrate how to combine topologies, workloads, +and expectations.
+View Complete Source Code:
+Runnable examples: The repo includes complete binaries in examples/src/bin/:
local_runner.rs — Host processes (local)compose_runner.rs — Docker Compose (requires image built)k8s_runner.rs — Kubernetes (requires cluster access and image loaded)Recommended: Use scripts/run/run-examples.sh -t <duration> -v <validators> -e <executors> <mode> where mode is host, compose, or k8s.
Alternative: Direct cargo run: POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin <name>
All runners require POL_PROOF_DEV_MODE=true to avoid expensive proof generation.
Code patterns below show how to build scenarios. Wrap these in #[tokio::test] functions for integration tests, or #[tokio::main] for binaries.
Minimal test that validates basic block production:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub async fn simple_consensus() -> Result<()> {
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(0))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(30))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}When to use: smoke tests for consensus on minimal hardware.
+Test consensus under transaction load:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub async fn transaction_workload() -> Result<()> {
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(2).executors(0))
+ .wallets(20)
+ .transactions_with(|txs| txs.rate(5).users(10))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}When to use: validate transaction submission and inclusion.
+Combined test stressing both transaction and DA layers:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub async fn da_and_transactions() -> Result<()> {
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .wallets(30)
+ .transactions_with(|txs| txs.rate(5).users(15))
+ .da_with(|da| da.channel_rate(2).blob_rate(2))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(90))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}When to use: end-to-end coverage of transaction and DA layers.
+Test system resilience under node restarts:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_compose::ComposeDeployer;
+use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt};
+
+pub async fn chaos_resilience() -> Result<()> {
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(4).executors(2))
+ .enable_node_control()
+ .wallets(20)
+ .transactions_with(|txs| txs.rate(3).users(10))
+ .chaos_with(|c| {
+ c.restart()
+ .min_delay(Duration::from_secs(20))
+ .max_delay(Duration::from_secs(40))
+ .target_cooldown(Duration::from_secs(30))
+ .apply()
+ })
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(120))
+ .build();
+
+ let deployer = ComposeDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}When to use: resilience validation and operational readiness drills.
+Note: Chaos tests require ComposeDeployer or another runner with node control support.
++When should I read this? Skim now to see what's possible, revisit later when you need load testing, chaos scenarios, or custom extensions. Start with basic examples first.
+
Realistic advanced scenarios demonstrating framework capabilities for production testing.
+Adapt from Complete Source:
+| Example | Topology | Workloads | Deployer | Key Feature |
|---|---|---|---|---|
| Load Progression | 3 validators + 2 executors | Increasing tx rate | Compose | Dynamic load testing |
| Sustained Load | 4 validators + 2 executors | High tx + DA rate | Compose | Stress testing |
| Aggressive Chaos | 4 validators + 2 executors | Frequent restarts + traffic | Compose | Resilience validation |
Test consensus under progressively increasing transaction load:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_compose::ComposeDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub async fn load_progression_test() -> Result<()> {
+ for rate in [5, 10, 20, 30] {
+ println!("Testing with rate: {}", rate);
+
+ let mut plan =
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .wallets(50)
+ .transactions_with(|txs| txs.rate(rate).users(20))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+
+ let deployer = ComposeDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+ }
+
+ Ok(())
+}When to use: Finding the maximum sustainable transaction rate for a given topology.
+Run high transaction and DA load for extended duration:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_compose::ComposeDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub async fn sustained_load_test() -> Result<()> {
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(4).executors(2))
+ .wallets(100)
+ .transactions_with(|txs| txs.rate(15).users(50))
+ .da_with(|da| da.channel_rate(2).blob_rate(3))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(300))
+ .build();
+
+ let deployer = ComposeDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}When to use: Validating stability under continuous high load over extended periods.
+Frequent node restarts with active traffic:
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_compose::ComposeDeployer;
+use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt};
+
+pub async fn aggressive_chaos_test() -> Result<()> {
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(4).executors(2))
+ .enable_node_control()
+ .wallets(50)
+ .transactions_with(|txs| txs.rate(10).users(20))
+ .chaos_with(|c| {
+ c.restart()
+ .min_delay(Duration::from_secs(10))
+ .max_delay(Duration::from_secs(20))
+ .target_cooldown(Duration::from_secs(15))
+ .apply()
+ })
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(180))
+ .build();
+
+ let deployer = ComposeDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}When to use: Validating recovery and liveness under aggressive failure conditions.
+Note: Requires ComposeDeployer for node control support.
These scenarios require custom implementations but demonstrate framework extensibility:
+Concept: Submit the same batch of independent transactions to different nodes in randomized order/offsets, then verify all transactions are included and final state matches across nodes.
+Requirements:
+ctx.node_clients(), with randomized submission order and timing offsets per nodeWhy useful: Exercises mempool propagation, proposer fairness, and transaction inclusion guarantees under realistic race conditions. Tests that the protocol maintains consistency regardless of which node receives transactions first.
+Implementation notes: Requires both a custom Workload implementation (to submit same transactions to multiple nodes with jitter) and a custom Expectation implementation (to verify inclusion and state consistency).
Concept: Drive different transaction subsets into different validators (or differing arrival orders) to create temporary mempool divergence, then verify mempools/blocks converge to contain the union (no permanent divergence).
+Requirements:
+ctx.node_clients() with disjoint or jittered transaction batchesExpectations:
+Why useful: Exercises mempool gossip and reconciliation under uneven input or latency. Ensures no node "drops" transactions seen elsewhere, validating that mempool synchronization mechanisms correctly propagate transactions across the network even when they arrive at different nodes in different orders.
+Implementation notes: Requires both a custom Workload implementation (to inject disjoint/jittered batches per node) and a custom Expectation implementation (to verify mempool convergence or block inclusion). Uses existing ctx.node_clients() capability—no new infrastructure needed.
Concept: Ramp transaction load over time to observe mempool growth, fee prioritization/eviction, and block saturation behavior, detecting performance regressions and ensuring backpressure/eviction work under increasing load.
+Requirements:
+Expectations:
+Why useful: Detects performance regressions in mempool management. Ensures backpressure and eviction mechanisms work correctly under increasing load, preventing memory exhaustion or unbounded growth. Validates that fee prioritization correctly selects high-value transactions when mempool is full.
+Implementation notes: Can be built with current workload model (ramping rate). Requires custom Expectation implementation that reads mempool metrics (via node HTTP APIs or Prometheus) and monitors throughput to judge behavior. No new infrastructure needed—uses existing observability capabilities.
Concept: Submit malformed transactions and verify they're rejected properly.
+Implementation approach:
+Why useful: Ensures mempool doesn't crash or include invalid transactions under fuzzing.
+Concept: Test consensus robustness under skewed gossip delays by partitioning nodes into latency tiers (tier A ≈10ms, tier B ≈100ms, tier C ≈300ms) and observing propagation lag, fork rate, and eventual convergence.
+Requirements:
+netem/iptables in compose; NetworkPolicy + netem sidecar in k8sExpectations:
+Why useful: Real networks have heterogeneous latency. This stress-tests proposer selection and fork resolution when some peers are "far" (high latency), validating that consensus remains safe and live under realistic network conditions.
+Current blocker: Runner support for per-group delay injection (network delay via netem/iptables) is not present today. Would require new chaos plumbing in compose/k8s deployers to inject network delays per node group.
Concept: Spin up a custom workload/sidecar that runs a libp2p host, joins the cluster's gossip mesh, and publishes a high rate of syntactically valid but useless/stale messages to selected topics, testing gossip backpressure, scoring, and queue handling under a "malicious" peer.
+Requirements:
+Expectations:
+Why useful: Tests Byzantine behavior (malicious peer) which is critical for consensus protocol robustness. More realistic than RPC spam since it uses the actual gossip protocol. Validates that gossip backpressure, peer scoring, and queue management correctly handle adversarial peers without disrupting consensus.
+Current blocker: Requires adding gossip-capable helper (libp2p integration) to the framework. Would need a custom workload/sidecar implementation that can join the gossip mesh and inject messages. The rest of the scenario can use existing runners/workloads.
+Concept: Test consensus recovery after network partitions.
+Requirements:
+block_peer() / unblock_peer() methods in NodeControlHandleWhy useful: Tests the most realistic failure mode in distributed systems.
+Current blocker: Node control doesn't yet support network-level actions (only process restarts).
+Concept: Test consensus and timestamp handling when nodes run with skewed clocks (e.g., +1s, −1s, +200ms jitter) to surface timestamp validation issues, reorg sensitivity, and clock drift handling.
+Requirements:
+Expectations:
+Why useful: Clock skew is a common real-world issue in distributed systems. This validates that consensus correctly handles timestamp validation and maintains safety/liveness when nodes have different clock offsets, preventing timestamp-based attacks or failures.
+Current blocker: Runner ability to skew per-node clocks (e.g., privileged containers with libfaketime/chrony or time-offset netns) is not available today. Would require a new chaos/time-skew hook in deployers to inject clock offsets per node.
Concept: Verify block production intervals stay within expected bounds.
+Implementation approach:
+BlockFeed(slot_duration * active_slot_coeff) ± toleranceWhy useful: Validates consensus timing under various loads.
+Concept: Nodes join and leave mid-run (new identities/addresses added; some nodes permanently removed) to exercise peer discovery, bootstrapping, reputation, and load balancing under churn.
+Requirements:
+Expectations:
+Why useful: Real networks experience churn (nodes joining/leaving). Unlike restarts (which preserve topology), churn changes the actual topology size and peer set, testing how the protocol handles dynamic membership. This exercises peer discovery, bootstrapping, reputation systems, and load balancing under realistic conditions.
+Current blocker: Runner support for dynamic node addition/removal at runtime is not available today. Chaos today only restarts existing nodes; churn would require the ability to spin up new nodes with fresh identities/addresses, update peer lists/bootstraps dynamically, and permanently remove nodes. Would need new topology management capabilities in deployers.
+Concept: Adversarial workload floods node HTTP/WS APIs with high QPS and malformed/bursty requests; expectation checks nodes remain responsive or rate-limit without harming consensus.
+Requirements:
+Expectations:
+Why useful: Validates API hardening under abuse and ensures control/telemetry endpoints don't destabilize the node. Tests that API abuse is properly isolated from consensus operations, preventing DoS attacks on API endpoints from affecting blockchain functionality.
+Implementation notes: Requires custom Workload implementation that directs high-QPS traffic to node APIs (via ctx.node_clients() or direct HTTP clients) and custom Expectation implementation that monitors API responsiveness metrics and consensus liveness. Uses existing node API access—no new infrastructure needed.
Concept: Track wallet balances and verify state consistency.
+Description: After transaction workload completes, query all wallet balances via node API and verify total supply is conserved. Requires tracking initial state, submitted transactions, and final balances. Validates that the ledger maintains correctness under load (no funds lost or created). This is a state assertion expectation that checks correctness, not just liveness.
+The Logos testing repo includes a small Cucumber (Gherkin) harness for “smoke” scenarios. It is useful when you want readable acceptance-style checks, but it intentionally exposes a limited surface area compared to Rust scenarios.
+testing-framework/cucumber.examples (crate runner-examples):
+cucumber_host (local/host deployer)cucumber_compose (compose deployer)examples/cucumber/features/.local and compose (no k8s runner integration in Cucumber yet).This is the shape used by the repo’s smoke features:
+Feature: Testing Framework - Local Runner
+
+ Scenario: Run a local smoke scenario (tx + DA + liveness)
+ Given deployer is "local"
+ And topology has 1 validators and 1 executors
+ And run duration is 60 seconds
+ And wallets total funds is 1000000000 split across 50 users
+ And transactions rate is 1 per block
+ And data availability channel rate is 1 per block and blob rate is 1 per block
+ And expect consensus liveness
+ When run scenario
+ Then scenario should succeed
+Local runner smoke:
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin cucumber_host
+Compose runner smoke:
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin cucumber_compose
+Topology / runner selection:
+Given deployer is "local"|"compose"Given topology has <validators> validators and <executors> executorsRun configuration:
+Given run duration is <seconds> secondsGiven wallets total funds is <funds> split across <users> usersWorkloads:
+Given transactions rate is <rate> per blockGiven transactions rate is <rate> per block using <users> usersGiven data availability channel rate is <channel_rate> per block and blob rate is <blob_rate> per blockExpectations:
+Given expect consensus livenessGiven consensus liveness lag allowance is <blocks>Execution + assertion:
+When run scenarioThen scenario should succeedPOL_PROOF_DEV_MODE=true for practical runs).This page focuses on how scenarios are executed (deploy → run → evaluate → cleanup), what artifacts you get back, and how that differs across runners.
+For “just run something that works” commands, see Running Examples.
+When you run a built scenario via a deployer, the run follows the same shape:
+flowchart TD
+ Build[Scenario built] --> Deploy[Deploy]
+ Deploy --> Capture[Capture]
+ Capture --> Execute[Execute]
+ Execute --> Evaluate[Evaluate]
+ Evaluate --> Cleanup[Cleanup]
+use std::time::Duration;
+
+use testing_framework_core::scenario::{Deployer as _, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+async fn run_once() -> anyhow::Result<()> {
+ let mut scenario = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
+ .wallets(20)
+ .transactions_with(|tx| tx.rate(1).users(5))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build()?;
+
+ let runner = LocalDeployer::default().deploy(&scenario).await?;
+ runner.run(&mut scenario).await?;
+
+ Ok(())
+}Notes:
+with_run_duration(...) is wall-clock time, not “number of blocks”..transactions_with(...) rates are per-block.scripts/run/run-examples.sh unless they are embedding the framework in their own test crate.NOMOS_TESTS_KEEP_LOGS=1 and/or NOMOS_LOG_DIR=...Run the built-in local examples:
+POL_PROOF_DEV_MODE=true \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
+docker compose logs (and any node-level log configuration you apply)COMPOSE_RUNNER_PRESERVE=1 to keep the environment up after a runRun the built-in compose examples:
+POL_PROOF_DEV_MODE=true \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+kubectl logs ...K8S_RUNNER_PRESERVE=1 and K8S_RUNNER_NAMESPACE=... to keep resources aroundRun the built-in k8s examples:
+POL_PROOF_DEV_MODE=true \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
+NOMOS_LOG_DIR, NOMOS_LOG_LEVEL, NOMOS_LOG_FILTER (see Logging & Observability)RUST_LOG (runner process only)NOMOS_TESTS_KEEP_LOGS=1COMPOSE_RUNNER_PRESERVE=1K8S_RUNNER_PRESERVE=1Runners turn a scenario plan into a live environment while keeping the plan +unchanged. Choose based on feedback speed, reproducibility, and fidelity. For +environment and operational considerations, see Operations Overview.
+Important: All runners require POL_PROOF_DEV_MODE=true to avoid expensive Groth16 proof generation that causes timeouts.
LocalDeployer).local_runner.rs, script mode: hostRun with: scripts/run/run-examples.sh -t 60 -v 1 -e 1 host
ComposeDeployer).compose_runner.rs, script mode: composeRun with: scripts/run/run-examples.sh -t 60 -v 1 -e 1 compose
K8sDeployer).k8s_runner.rs, script mode: k8sRun with: scripts/run/run-examples.sh -t 60 -v 1 -e 1 k8s
flowchart TB
+ subgraph Host["Host Runner (Local)"]
+ H1["Speed: Fast"]
+ H2["Isolation: Shared host"]
+ H3["Setup: Minimal"]
+ H4["Chaos: Not supported"]
+ H5["CI: Quick smoke tests"]
+ end
+
+ subgraph Compose["Compose Runner (Docker)"]
+ C1["Speed: Medium"]
+ C2["Isolation: Containerized"]
+ C3["Setup: Image build required"]
+ C4["Chaos: Supported"]
+ C5["CI: Recommended"]
+ end
+
+ subgraph K8s["K8s Runner (Cluster)"]
+ K1["Speed: Slower"]
+ K2["Isolation: Pod-level"]
+ K3["Setup: Cluster + image"]
+ K4["Chaos: Not yet supported"]
+ K5["CI: Large-scale tests"]
+ end
+
+ Decision{Choose Based On}
+ Decision -->|Fast iteration| Host
+ Decision -->|Reproducibility| Compose
+ Decision -->|Production-like| K8s
+
+ style Host fill:#e1f5ff
+ style Compose fill:#e1ffe1
+ style K8s fill:#ffe1f5
+| Feature | Host | Compose | K8s |
|---|---|---|---|
| Speed | Fastest | Medium | Slowest |
| Setup Time | < 1 min | 2-5 min | 5-10 min |
| Isolation | Process-level | Container | Pod + namespace |
| Node Control | No | Yes | Not yet |
| Observability | Basic | External stack | Cluster-wide |
| CI Integration | Smoke tests | Recommended | Heavy tests |
| Resource Usage | Low | Medium | High |
| Reproducibility | Environment-dependent | High | Highest |
| Network Fidelity | Localhost only | Virtual network | Real cluster |
| Parallel Runs | Port conflicts | Isolated | Namespace isolation |
flowchart TD
+ Start[Need to run tests?] --> Q1{Local development?}
+ Q1 -->|Yes| Q2{Testing chaos?}
+ Q1 -->|No| Q5{Have cluster access?}
+
+ Q2 -->|Yes| UseCompose[Use Compose]
+ Q2 -->|No| Q3{Need isolation?}
+
+ Q3 -->|Yes| UseCompose
+ Q3 -->|No| UseHost[Use Host]
+
+ Q5 -->|Yes| Q6{Large topology?}
+ Q5 -->|No| Q7{CI pipeline?}
+
+ Q6 -->|Yes| UseK8s[Use K8s]
+ Q6 -->|No| UseCompose
+
+ Q7 -->|Yes| Q8{Docker available?}
+ Q7 -->|No| UseHost
+
+ Q8 -->|Yes| UseCompose
+ Q8 -->|No| UseHost
+
+ style UseHost fill:#e1f5ff
+ style UseCompose fill:#e1ffe1
+ style UseK8s fill:#ffe1f5
+Use Host Runner when:
+Use Compose Runner when:
+Use K8s Runner when:
+The deployer supplies a RunContext that workloads and expectations share. It
+provides:
GeneratedTopology)NodeClients / ClusterClient) for HTTP/RPC callsRunMetrics, Metrics) and block feedNodeControlHandle for managing nodesThe BlockFeed is a broadcast stream of block observations that allows workloads and expectations to monitor blockchain progress in real-time. It polls a validator node continuously and broadcasts new blocks to all subscribers.
Real-time block stream:
+BlockRecord notifications as blocks are producedHeaderId) and full block payloadBlock statistics:
+block_feed.stats().total_transactions()Broadcast semantics:
+BlockFeed is available through RunContext:
let block_feed = ctx.block_feed();Expectations typically use BlockFeed to verify block production and inclusion of transactions/data.
+Example: Counting blocks during a run
+use std::sync::{
+ Arc,
+ atomic::{AtomicU64, Ordering},
+};
+
+use async_trait::async_trait;
+use testing_framework_core::scenario::{DynError, Expectation, RunContext};
+
+struct MinimumBlocksExpectation {
+ min_blocks: u64,
+ captured_blocks: Option<Arc<AtomicU64>>,
+}
+
+#[async_trait]
+impl Expectation for MinimumBlocksExpectation {
+ fn name(&self) -> &'static str {
+ "minimum_blocks"
+ }
+
+ async fn start_capture(&mut self, ctx: &RunContext) -> Result<(), DynError> {
+ let block_count = Arc::new(AtomicU64::new(0));
+ let block_count_task = Arc::clone(&block_count);
+
+ // Subscribe to block feed
+ let mut receiver = ctx.block_feed().subscribe();
+
+ // Spawn a task to count blocks
+ tokio::spawn(async move {
+ loop {
+ match receiver.recv().await {
+ Ok(_record) => {
+ block_count_task.fetch_add(1, Ordering::Relaxed);
+ }
+ Err(tokio::sync::broadcast::error::RecvError::Lagged(skipped)) => {
+ tracing::debug!(skipped, "receiver lagged, skipping blocks");
+ }
+ Err(tokio::sync::broadcast::error::RecvError::Closed) => {
+ tracing::debug!("block feed closed");
+ break;
+ }
+ }
+ }
+ });
+
+ self.captured_blocks = Some(block_count);
+ Ok(())
+ }
+
+ async fn evaluate(&mut self, ctx: &RunContext) -> Result<(), DynError> {
+ let blocks = self.captured_blocks
+ .as_ref()
+ .expect("start_capture must be called first")
+ .load(Ordering::Relaxed);
+
+ if blocks < self.min_blocks {
+ return Err(format!(
+ "expected at least {} blocks, observed {}",
+ self.min_blocks, blocks
+ ).into());
+ }
+
+ tracing::info!(blocks, min = self.min_blocks, "minimum blocks expectation passed");
+ Ok(())
+ }
+}Example: Inspecting block contents
+use testing_framework_core::scenario::{DynError, RunContext};
+
+async fn start_capture(ctx: &RunContext) -> Result<(), DynError> {
+ let mut receiver = ctx.block_feed().subscribe();
+
+ tokio::spawn(async move {
+ loop {
+ match receiver.recv().await {
+ Ok(record) => {
+ // Access block header
+ let header_id = &record.header;
+
+ // Access full block
+ let tx_count = record.block.transactions().len();
+
+ tracing::debug!(
+ ?header_id,
+ tx_count,
+ "observed block"
+ );
+
+ // Process transactions, DA blobs, etc.
+ }
+ Err(tokio::sync::broadcast::error::RecvError::Closed) => break,
+ Err(_) => continue,
+ }
+ }
+ });
+
+ Ok(())
+}Workloads can use BlockFeed to coordinate timing or wait for specific conditions before proceeding.
+Example: Wait for N blocks before starting
+use async_trait::async_trait;
+use testing_framework_core::scenario::{DynError, RunContext, Workload};
+
+struct DelayedWorkload {
+ wait_blocks: usize,
+}
+
+#[async_trait]
+impl Workload for DelayedWorkload {
+ fn name(&self) -> &str {
+ "delayed_workload"
+ }
+
+ async fn start(&self, ctx: &RunContext) -> Result<(), DynError> {
+ tracing::info!(wait_blocks = self.wait_blocks, "waiting for blocks before starting");
+
+ // Subscribe to block feed
+ let mut receiver = ctx.block_feed().subscribe();
+ let mut count = 0;
+
+ // Wait for N blocks
+ while count < self.wait_blocks {
+ match receiver.recv().await {
+ Ok(_) => count += 1,
+ Err(tokio::sync::broadcast::error::RecvError::Lagged(_)) => continue,
+ Err(tokio::sync::broadcast::error::RecvError::Closed) => {
+ return Err("block feed closed before reaching target".into());
+ }
+ }
+ }
+
+ tracing::info!("warmup complete, starting actual workload");
+
+ // Now do the actual work
+ // ...
+
+ Ok(())
+ }
+}Example: Rate limiting based on block production
+use testing_framework_core::scenario::{DynError, RunContext};
+
+async fn generate_request() -> Option<()> {
+ None
+}
+
+async fn start(ctx: &RunContext) -> Result<(), DynError> {
+ let clients = ctx.node_clients().validator_clients();
+ let mut receiver = ctx.block_feed().subscribe();
+ let mut pending_requests: Vec<()> = Vec::new();
+
+ loop {
+ tokio::select! {
+ // Issue a batch on each new block.
+ Ok(_record) = receiver.recv() => {
+ if !pending_requests.is_empty() {
+ tracing::debug!(count = pending_requests.len(), "issuing requests on new block");
+ for _req in pending_requests.drain(..) {
+ let _info = clients[0].consensus_info().await?;
+ }
+ }
+ }
+
+ // Generate work continuously.
+ Some(req) = generate_request() => {
+ pending_requests.push(req);
+ }
+ }
+ }
+}Use BlockFeed when:
+Use direct polling when:
+Example direct polling in expectations:
+use testing_framework_core::scenario::{DynError, RunContext};
+
+async fn evaluate(ctx: &RunContext) -> Result<(), DynError> {
+ let client = &ctx.node_clients().validator_clients()[0];
+
+ // Poll current height once
+ let info = client.consensus_info().await?;
+ tracing::info!(height = info.height, "final block height");
+
+ // This is simpler than BlockFeed for one-time checks
+ Ok(())
+}Access aggregated statistics without subscribing to the feed:
+use testing_framework_core::scenario::{DynError, RunContext};
+
+async fn evaluate(ctx: &RunContext, expected_min: u64) -> Result<(), DynError> {
+ let stats = ctx.block_feed().stats();
+ let total_txs = stats.total_transactions();
+
+ tracing::info!(total_txs, "transactions observed across all blocks");
+
+ if total_txs < expected_min {
+ return Err(format!(
+ "expected at least {} transactions, observed {}",
+ expected_min, total_txs
+ ).into());
+ }
+
+ Ok(())
+}Subscription timing:
+start_capture() for expectationsstart() for workloadsLagged receivers:
+RecvError::Lagged(skipped) gracefullyFeed lifetime:
+Performance:
+The framework's built-in expectations use BlockFeed extensively:
+ConsensusLiveness: Doesn't directly subscribe but uses block feed stats to verify progressDataAvailabilityExpectation: Subscribes to inspect DA blobs in each block and track inscription/dispersalTransactionInclusion: Subscribes to find specific transactions in blocksSee Examples and Workloads & Expectations for more patterns.
+The framework currently supports process-level chaos (node restarts) for +resilience testing:
+Supported:
+restart_validator)restart_executor).chaos().restart()Not Yet Supported:
+For network partition testing, see Extension Ideas
+which describes the proposed block_peer/unblock_peer API (not yet implemented).
Check for control support and use it conditionally:
+use async_trait::async_trait;
+use testing_framework_core::scenario::{DynError, RunContext, Workload};
+
+struct RestartWorkload;
+
+#[async_trait]
+impl Workload for RestartWorkload {
+ fn name(&self) -> &str {
+ "restart_workload"
+ }
+
+ async fn start(&self, ctx: &RunContext) -> Result<(), DynError> {
+ if let Some(control) = ctx.node_control() {
+ // Restart the first validator (index 0) if supported.
+ control.restart_validator(0).await?;
+ }
+ Ok(())
+ }
+}When chaos workloads need control, require enable_node_control() in the
+scenario builder and deploy with a runner that supports it.
The NodeControlHandle trait currently provides:
use async_trait::async_trait;
+use testing_framework_core::scenario::DynError;
+
+#[async_trait]
+pub trait NodeControlHandle: Send + Sync {
+ async fn restart_validator(&self, index: usize) -> Result<(), DynError>;
+ async fn restart_executor(&self, index: usize) -> Result<(), DynError>;
+}Future extensions may include peer blocking/unblocking or other control +operations. For now, focus on restart-based chaos patterns as shown in the +chaos workload examples.
+NodeControlHandle.++When should I read this? You don't need chaos testing to be productive with the framework. Focus on basic scenarios first—chaos is for resilience validation and operational readiness drills once your core tests are stable.
+
Chaos in the framework uses node control to introduce failures and validate
+recovery. The built-in restart workload lives in
+testing_framework_workflows::workloads::chaos::RandomRestartWorkload.
NodeControlCapability (enable_node_control() in the scenario
+builder) and a runner that provides a NodeControlHandle.NodeControlHandle support (e.g., compose) for chaos workloads.use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::{ScenarioBuilderExt, workloads::chaos::RandomRestartWorkload};
+
+pub fn random_restart_plan() -> testing_framework_core::scenario::Scenario<
+ testing_framework_core::scenario::NodeControlCapability,
+> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(2).executors(1))
+ .enable_node_control()
+ .with_workload(RandomRestartWorkload::new(
+ Duration::from_secs(45), // min delay
+ Duration::from_secs(75), // max delay
+ Duration::from_secs(120), // target cooldown
+ true, // include validators
+ true, // include executors
+ ))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(150))
+ .build()
+}This page focuses on cluster manipulation: node control, chaos patterns, and +what the tooling supports today.
+NodeControlHandle (compose runner).Deep dives for contributors who extend the framework, evolve its abstractions, +or maintain the crate set.
+The scenario model defines clear, composable responsibilities:
+Developers extending the model should keep these boundaries strict: topology +describes, scenarios assemble, deployers provision, runners orchestrate, +workloads drive, and expectations judge outcomes. For guidance on adding new +capabilities, see Extending the Framework.
+The framework supports two styles for constructing scenarios:
+.transactions_with(...))Both styles produce the same runtime behavior because they ultimately call the same core builder APIs.
+The DSL is implemented as extension traits (primarily testing_framework_workflows::ScenarioBuilderExt) on the core scenario builder.
use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+let plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .wallets(5)
+ .transactions_with(|txs| txs.rate(5).users(3))
+ .da_with(|da| da.channel_rate(1).blob_rate(1).headroom_percent(20))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();When to use:
+Direct instantiation gives you explicit control over the concrete types you attach:
+use std::{
+ num::{NonZeroU64, NonZeroUsize},
+ time::Duration,
+};
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::{
+ expectations::ConsensusLiveness,
+ workloads::{da, transaction},
+};
+
+let tx_workload = transaction::Workload::with_rate(5)
+ .expect("transaction rate must be non-zero")
+ .with_user_limit(NonZeroUsize::new(3));
+
+let da_workload = da::Workload::with_rate(
+ NonZeroU64::new(1).unwrap(), // blob rate per block
+ NonZeroU64::new(1).unwrap(), // channel rate per block
+ da::Workload::default_headroom_percent(),
+);
+
+let plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .wallets(5)
+ .with_workload(tx_workload)
+ .with_workload(da_workload)
+ .with_expectation(ConsensusLiveness::default())
+ .with_run_duration(Duration::from_secs(60))
+ .build();When to use:
+| High-Level DSL | Low-Level Direct |
|---|---|
.transactions_with(|txs| txs.rate(5).users(3)) | .with_workload(transaction::Workload::with_rate(5).expect(...).with_user_limit(...)) |
.da_with(|da| da.blob_rate(1).channel_rate(1)) | .with_workload(da::Workload::with_rate(...)) |
.expect_consensus_liveness() | .with_expectation(ConsensusLiveness::default()) |
Workloads can bundle expectations by implementing Workload::expectations().
These bundled expectations are attached automatically whenever you call .with_workload(...) (including when you use the DSL), because the core builder expands workload expectations during attachment.
Mixing is common: use the DSL for built-ins, and direct instantiation for custom pieces.
+use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::{ScenarioBuilderExt, workloads::transaction};
+
+let tx_workload = transaction::Workload::with_rate(5)
+ .expect("transaction rate must be non-zero");
+
+let plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .wallets(5)
+ .with_workload(tx_workload) // direct instantiation
+ .expect_consensus_liveness() // DSL
+ .with_run_duration(Duration::from_secs(60))
+ .build();The DSL methods are thin wrappers. For example:
+builder.transactions_with(|txs| txs.rate(5).users(3))
is roughly equivalent to:
+builder.transactions().rate(5).users(3).apply()
DSL method not found
+use testing_framework_workflows::ScenarioBuilderExt;This guide shows how to extend the framework with custom workloads, expectations, runners, and topology helpers. Each section includes the trait outline and a minimal code example.
+Steps:
+testing_framework_core::scenario::Workloadinit to derive inputs from topology/metrics; fail fast if prerequisites missingstart to drive async traffic using RunContext clientstesting-framework/workflows and optionally add a DSL helperTrait outline:
+use async_trait::async_trait;
+use testing_framework_core::scenario::{
+ DynError, Expectation, RunContext, RunMetrics, Workload,
+};
+use testing_framework_core::topology::generation::GeneratedTopology;
+
+struct MyExpectation;
+
+#[async_trait]
+impl Expectation for MyExpectation {
+ fn name(&self) -> &str {
+ "my_expectation"
+ }
+
+ async fn evaluate(&mut self, _ctx: &RunContext) -> Result<(), DynError> {
+ Ok(())
+ }
+}
+
+pub struct MyWorkload {
+ // Configuration fields
+ target_rate: u64,
+}
+
+impl MyWorkload {
+ pub fn new(target_rate: u64) -> Self {
+ Self { target_rate }
+ }
+}
+
+#[async_trait]
+impl Workload for MyWorkload {
+ fn name(&self) -> &str {
+ "my_workload"
+ }
+
+ fn expectations(&self) -> Vec<Box<dyn Expectation>> {
+ // Return bundled expectations that should run with this workload
+ vec![Box::new(MyExpectation)]
+ }
+
+ fn init(
+ &mut self,
+ topology: &GeneratedTopology,
+ _run_metrics: &RunMetrics,
+ ) -> Result<(), DynError> {
+ // Validate prerequisites (e.g., enough nodes, wallet data present)
+ if topology.validators().is_empty() {
+ return Err("no validators available".into());
+ }
+ Ok(())
+ }
+
+ async fn start(&self, ctx: &RunContext) -> Result<(), DynError> {
+ // Drive async activity: submit transactions, query nodes, etc.
+ let clients = ctx.node_clients().validator_clients();
+
+ for client in clients {
+ let info = client.consensus_info().await?;
+ tracing::info!(height = info.height, "workload queried node");
+ }
+
+ Ok(())
+ }
+}Key points:
+name() identifies the workload in logsexpectations() bundles default checks (can be empty)init() validates topology before run startsstart() executes concurrently with other workloads; it should complete before run duration expiresSee Example: New Workload & Expectation for a complete, runnable example.
+Steps:
+testing_framework_core::scenario::Expectationstart_capture to snapshot baseline metrics (optional)evaluate to assert outcomes after workloads finishtesting-framework/workflows if reusableTrait outline:
+use async_trait::async_trait;
+use testing_framework_core::scenario::{DynError, Expectation, RunContext};
+
+pub struct MyExpectation {
+ expected_value: u64,
+ captured_baseline: Option<u64>,
+}
+
+impl MyExpectation {
+ pub fn new(expected_value: u64) -> Self {
+ Self {
+ expected_value,
+ captured_baseline: None,
+ }
+ }
+}
+
+#[async_trait]
+impl Expectation for MyExpectation {
+ fn name(&self) -> &str {
+ "my_expectation"
+ }
+
+ async fn start_capture(&mut self, ctx: &RunContext) -> Result<(), DynError> {
+ // Optional: capture baseline state before workloads start
+ let client = ctx.node_clients().validator_clients().first()
+ .ok_or("no validators")?;
+
+ let info = client.consensus_info().await?;
+ self.captured_baseline = Some(info.height);
+
+ tracing::info!(baseline = self.captured_baseline, "captured baseline");
+ Ok(())
+ }
+
+ async fn evaluate(&mut self, ctx: &RunContext) -> Result<(), DynError> {
+ // Assert the expected condition holds after workloads finish
+ let client = ctx.node_clients().validator_clients().first()
+ .ok_or("no validators")?;
+
+ let info = client.consensus_info().await?;
+ let final_height = info.height;
+
+ let baseline = self.captured_baseline.unwrap_or(0);
+ let delta = final_height.saturating_sub(baseline);
+
+ if delta < self.expected_value {
+ return Err(format!(
+ "expected at least {} blocks, got {}",
+ self.expected_value, delta
+ ).into());
+ }
+
+ tracing::info!(delta, "expectation passed");
+ Ok(())
+ }
+}Key points:
+name() identifies the expectation in logsstart_capture() runs before workloads start (optional)evaluate() runs after workloads finish; return descriptive errorsSteps:
+testing_framework_core::scenario::Deployer<Caps> for your capability typeRunnerNodeClients and spawn a BlockFeedRunContext and provide a CleanupGuard for teardownTrait outline:
+use async_trait::async_trait;
+use testing_framework_core::scenario::{
+ CleanupGuard, Deployer, DynError, Metrics, NodeClients, RunContext, Runner, Scenario,
+ spawn_block_feed,
+};
+use testing_framework_core::topology::deployment::Topology;
+
+pub struct MyDeployer {
+ // Configuration: cluster connection details, etc.
+}
+
+impl MyDeployer {
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+#[async_trait]
+impl Deployer<()> for MyDeployer {
+ type Error = DynError;
+
+ async fn deploy(&self, scenario: &Scenario<()>) -> Result<Runner, Self::Error> {
+ // 1. Launch nodes using scenario.topology()
+ // 2. Wait for readiness (e.g., consensus info endpoint responds)
+ // 3. Build NodeClients for validators/executors
+ // 4. Spawn a block feed for expectations (optional but recommended)
+ // 5. Create NodeControlHandle if you support restarts (optional)
+ // 6. Return a Runner wrapping RunContext + CleanupGuard
+
+ tracing::info!("deploying scenario with MyDeployer");
+
+ let topology: Option<Topology> = None; // Some(topology) if you spawned one
+ let node_clients = NodeClients::default(); // Or NodeClients::from_topology(...)
+
+ let client = node_clients
+ .any_client()
+ .ok_or("no api clients available")?
+ .clone();
+ let (block_feed, block_feed_guard) = spawn_block_feed(client).await?;
+
+ let telemetry = Metrics::empty(); // or Metrics::from_prometheus(...)
+ let node_control = None; // or Some(Arc<dyn NodeControlHandle>)
+
+ let context = RunContext::new(
+ scenario.topology().clone(),
+ topology,
+ node_clients,
+ scenario.duration(),
+ telemetry,
+ block_feed,
+ node_control,
+ );
+
+ // If you also have other resources to clean up (containers/pods/etc),
+ // wrap them in your own CleanupGuard implementation and call
+ // CleanupGuard::cleanup(Box::new(block_feed_guard)) inside it.
+ Ok(Runner::new(context, Some(Box::new(block_feed_guard))))
+ }
+}Key points:
+deploy() must return a fully prepared RunnerCleanupGuard to tear down resources on failure (and on RunHandle drop)NodeControlHandle via RunContextSteps:
+testing_framework_core::topology::config::TopologyBuilder with new layoutsExample:
+use testing_framework_core::topology::{
+ config::TopologyBuilder,
+ configs::network::Libp2pNetworkLayout,
+};
+
+pub trait TopologyBuilderExt {
+ fn network_full(self) -> Self;
+}
+
+impl TopologyBuilderExt for TopologyBuilder {
+ fn network_full(self) -> Self {
+ self.with_network_layout(Libp2pNetworkLayout::Full)
+ }
+}Key points:
+&mut Self)To expose your custom workload through the high-level DSL, add a trait extension:
+use async_trait::async_trait;
+use testing_framework_core::scenario::{DynError, RunContext, ScenarioBuilder, Workload};
+
+#[derive(Default)]
+pub struct MyWorkloadBuilder {
+ target_rate: u64,
+ some_option: bool,
+}
+
+impl MyWorkloadBuilder {
+ pub const fn target_rate(mut self, target_rate: u64) -> Self {
+ self.target_rate = target_rate;
+ self
+ }
+
+ pub const fn some_option(mut self, some_option: bool) -> Self {
+ self.some_option = some_option;
+ self
+ }
+
+ pub const fn build(self) -> MyWorkload {
+ MyWorkload {
+ target_rate: self.target_rate,
+ some_option: self.some_option,
+ }
+ }
+}
+
+pub struct MyWorkload {
+ target_rate: u64,
+ some_option: bool,
+}
+
+#[async_trait]
+impl Workload for MyWorkload {
+ fn name(&self) -> &str {
+ "my_workload"
+ }
+
+ async fn start(&self, _ctx: &RunContext) -> Result<(), DynError> {
+ Ok(())
+ }
+}
+
+pub trait MyWorkloadDsl {
+ fn my_workload_with(
+ self,
+ f: impl FnOnce(MyWorkloadBuilder) -> MyWorkloadBuilder,
+ ) -> Self;
+}
+
+impl MyWorkloadDsl for ScenarioBuilder {
+ fn my_workload_with(
+ self,
+ f: impl FnOnce(MyWorkloadBuilder) -> MyWorkloadBuilder,
+ ) -> Self {
+ let builder = f(MyWorkloadBuilder::default());
+ self.with_workload(builder.build())
+ }
+}Users can then call:
+ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(1))
+ .my_workload_with(|w| {
+ w.target_rate(10)
+ .some_option(true)
+ })
+ .build()A minimal, end-to-end illustration of adding a custom workload and matching +expectation. This shows the shape of the traits and where to plug into the +framework; expand the logic to fit your real test.
+Key ideas:
+RunContext.use async_trait::async_trait;
+use testing_framework_core::{
+ scenario::{DynError, Expectation, RunContext, RunMetrics, Workload},
+ topology::generation::GeneratedTopology,
+};
+
+pub struct ReachabilityWorkload {
+ target_idx: usize,
+}
+
+impl ReachabilityWorkload {
+ pub fn new(target_idx: usize) -> Self {
+ Self { target_idx }
+ }
+}
+
+#[async_trait]
+impl Workload for ReachabilityWorkload {
+ fn name(&self) -> &str {
+ "reachability_workload"
+ }
+
+ fn expectations(&self) -> Vec<Box<dyn Expectation>> {
+ vec![Box::new(
+ crate::custom_workload_example_expectation::ReachabilityExpectation::new(
+ self.target_idx,
+ ),
+ )]
+ }
+
+ fn init(
+ &mut self,
+ topology: &GeneratedTopology,
+ _run_metrics: &RunMetrics,
+ ) -> Result<(), DynError> {
+ if topology.validators().get(self.target_idx).is_none() {
+ return Err(Box::new(std::io::Error::new(
+ std::io::ErrorKind::Other,
+ "no validator at requested index",
+ )));
+ }
+ Ok(())
+ }
+
+ async fn start(&self, ctx: &RunContext) -> Result<(), DynError> {
+ let client = ctx
+ .node_clients()
+ .validator_clients()
+ .get(self.target_idx)
+ .ok_or_else(|| {
+ Box::new(std::io::Error::new(
+ std::io::ErrorKind::Other,
+ "missing target client",
+ )) as DynError
+ })?;
+
+ // Lightweight API call to prove reachability.
+ client
+ .consensus_info()
+ .await
+ .map(|_| ())
+ .map_err(|e| e.into())
+ }
+}Key ideas:
+use async_trait::async_trait;
+use testing_framework_core::scenario::{DynError, Expectation, RunContext};
+
+pub struct ReachabilityExpectation {
+ target_idx: usize,
+}
+
+impl ReachabilityExpectation {
+ pub fn new(target_idx: usize) -> Self {
+ Self { target_idx }
+ }
+}
+
+#[async_trait]
+impl Expectation for ReachabilityExpectation {
+ fn name(&self) -> &str {
+ "target_reachable"
+ }
+
+ async fn evaluate(&mut self, ctx: &RunContext) -> Result<(), DynError> {
+ let client = ctx
+ .node_clients()
+ .validator_clients()
+ .get(self.target_idx)
+ .ok_or_else(|| {
+ Box::new(std::io::Error::new(
+ std::io::ErrorKind::Other,
+ "missing target client",
+ )) as DynError
+ })?;
+
+ client
+ .consensus_info()
+ .await
+ .map(|_| ())
+ .map_err(|e| e.into())
+ }
+}.with_workload(ReachabilityWorkload::new(0))..with_expectation(...) if needed.High-level roles of the crates that make up the framework:
+Configs (testing-framework/configs/): Prepares reusable configuration primitives for nodes, networking, tracing, data availability, and wallets, shared by all scenarios and runners. Includes topology generation and circuit asset resolution.
Core scenario orchestration (testing-framework/core/): Houses the topology and scenario model, runtime coordination, node clients, and readiness/health probes. Defines Deployer and Runner traits, ScenarioBuilder, and RunContext.
Workflows (testing-framework/workflows/): Packages workloads (transaction, DA, chaos) and expectations (consensus liveness) into reusable building blocks. Offers fluent DSL extensions (ScenarioBuilderExt, ChaosBuilderExt).
Runners (testing-framework/runners/{local,compose,k8s}/): Implements deployment backends (local host, Docker Compose, Kubernetes) that all consume the same scenario plan. Each provides a Deployer implementation (LocalDeployer, ComposeDeployer, K8sDeployer).
Runner Examples (crate name: runner-examples, path: examples/): Runnable binaries demonstrating framework usage and serving as living documentation. These are the primary entry point for running scenarios (examples/src/bin/local_runner.rs, examples/src/bin/compose_runner.rs, examples/src/bin/k8s_runner.rs).
| What You're Adding | Where It Goes | Examples |
|---|---|---|
| Node config parameter | testing-framework/configs/src/topology/configs/ | Slot duration, log levels, DA params |
| Topology feature | testing-framework/core/src/topology/ | New network layouts, node roles |
| Scenario capability | testing-framework/core/src/scenario/ | New capabilities, context methods |
| Workload | testing-framework/workflows/src/workloads/ | New traffic generators |
| Expectation | testing-framework/workflows/src/expectations/ | New success criteria |
| Builder API | testing-framework/workflows/src/builder/ | DSL extensions, fluent methods |
| Deployer | testing-framework/runners/ | New deployment backends |
| Example scenario | examples/src/bin/ | Demonstration binaries |
testing-framework/workflows/src/workloads/your_workload.rs:use async_trait::async_trait;
+use testing_framework_core::scenario::{DynError, RunContext, Workload};
+
+pub struct YourWorkload;
+
+#[async_trait]
+impl Workload for YourWorkload {
+ fn name(&self) -> &'static str {
+ "your_workload"
+ }
+
+ async fn start(&self, _ctx: &RunContext) -> Result<(), DynError> {
+ // implementation
+ Ok(())
+ }
+}testing-framework/workflows/src/builder/mod.rs:pub struct YourWorkloadBuilder;
+
+impl YourWorkloadBuilder {
+ pub fn some_config(self) -> Self {
+ self
+ }
+}
+
+pub trait ScenarioBuilderExt: Sized {
+ fn your_workload(self) -> YourWorkloadBuilder;
+}examples/src/bin/your_scenario.rs:use testing_framework_core::scenario::ScenarioBuilder;
+
+pub struct YourWorkloadBuilder;
+
+impl YourWorkloadBuilder {
+ pub fn some_config(self) -> Self {
+ self
+ }
+}
+
+pub trait YourWorkloadDslExt: Sized {
+ fn your_workload_with<F>(self, configurator: F) -> Self
+ where
+ F: FnOnce(YourWorkloadBuilder) -> YourWorkloadBuilder;
+}
+
+impl<Caps> YourWorkloadDslExt for testing_framework_core::scenario::Builder<Caps> {
+ fn your_workload_with<F>(self, configurator: F) -> Self
+ where
+ F: FnOnce(YourWorkloadBuilder) -> YourWorkloadBuilder,
+ {
+ let _ = configurator(YourWorkloadBuilder);
+ self
+ }
+}
+
+pub fn use_in_examples() {
+ let _plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(0))
+ .your_workload_with(|w| w.some_config())
+ .build();
+}testing-framework/workflows/src/expectations/your_expectation.rs:use async_trait::async_trait;
+use testing_framework_core::scenario::{DynError, Expectation, RunContext};
+
+pub struct YourExpectation;
+
+#[async_trait]
+impl Expectation for YourExpectation {
+ fn name(&self) -> &'static str {
+ "your_expectation"
+ }
+
+ async fn evaluate(&mut self, _ctx: &RunContext) -> Result<(), DynError> {
+ // implementation
+ Ok(())
+ }
+}testing-framework/workflows/src/builder/mod.rs:use testing_framework_core::scenario::ScenarioBuilder;
+
+pub trait YourExpectationDslExt: Sized {
+ fn expect_your_condition(self) -> Self;
+}
+
+impl<Caps> YourExpectationDslExt for testing_framework_core::scenario::Builder<Caps> {
+ fn expect_your_condition(self) -> Self {
+ self
+ }
+}
+
+pub fn use_in_examples() {
+ let _plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(0))
+ .expect_your_condition()
+ .build();
+}Deployer trait in testing-framework/runners/your_runner/src/deployer.rs:use async_trait::async_trait;
+use testing_framework_core::scenario::{Deployer, Runner, Scenario};
+
+#[derive(Debug)]
+pub struct YourError;
+
+pub struct YourDeployer;
+
+#[async_trait]
+impl Deployer for YourDeployer {
+ type Error = YourError;
+
+ async fn deploy(&self, _scenario: &Scenario<()>) -> Result<Runner, Self::Error> {
+ // Provision infrastructure
+ // Wait for readiness
+ // Return Runner
+ todo!()
+ }
+}Provide cleanup and handle node control if supported.
+Add example in examples/src/bin/your_runner.rs.
For detailed examples, see Extending the Framework and Custom Workload Example.
+This section covers operational aspects of running the testing framework: prerequisites, deployment configuration, continuous integration, and observability.
+This section is organized for progressive depth:
+Operational Hygiene: Assets present, prerequisites satisfied, observability reachable
+Environment Fit: Choose the right deployment target based on isolation, reproducibility, and resource needs
+Clear Signals: Verify runners report node readiness before starting workloads
+Failure Triage: Map failures to specific causes—missing prerequisites, platform issues, or unmet expectations
+Ready to get started? Begin with Operations Overview →
+Operational readiness focuses on prerequisites, environment fit, and clear signals that ensure your test scenarios run reliably across different deployment targets.
+Prerequisites:
+versions.env file at repository root (required by helper scripts)nomos-node, nomos-executor) available or built on demandArtifacts:
+Environment Configuration:
+POL_PROOF_DEV_MODE=true is REQUIRED for all runners to avoid expensive proof generationNOMOS_LOG_* variablesReadiness & Health:
+The framework is intentionally runner-agnostic: the same scenario plan runs across all deployment targets. Understanding which operational concerns apply to each runner helps you choose the right fit.
+| Concern | Host | Compose | Kubernetes |
|---|---|---|---|
| Topology | Full support | Full support | Full support |
| Workloads | All workloads | All workloads | All workloads |
| Expectations | All expectations | All expectations | All expectations |
| Chaos / Node Control | Not supported | Supported | Not yet |
| Metrics / Observability | Manual setup | External stack | Cluster-wide |
| Log Collection | Temp files | Container logs | Pod logs |
| Isolation | Process-level | Container | Pod + namespace |
| Setup Time | < 1 min | 2-5 min | 5-10 min |
| CI Recommended? | Smoke tests | Primary | Large-scale only |
Key insight: Operational concerns (prerequisites, environment variables) are largely consistent across runners, while deployment-specific concerns (isolation, chaos support) vary by backend.
+flowchart LR
+ Setup[Prerequisites & Setup] --> Run[Run Scenarios]
+ Run --> Monitor[Monitor & Observe]
+ Monitor --> Debug{Success?}
+ Debug -->|No| Triage[Failure Triage]
+ Triage --> Setup
+ Debug -->|Yes| Done[Complete]
+This Operations & Deployment section covers:
+Philosophy: Treat operational hygiene—assets present, prerequisites satisfied, observability reachable—as the first step to reliable scenario outcomes.
+This page covers everything you need before running your first scenario.
+versions.env (Required)All helper scripts require a versions.env file at the repository root:
VERSION=v0.3.1
+NOMOS_NODE_REV=abc123def456789
+NOMOS_BUNDLE_VERSION=v1
+What it defines:
+VERSION — Circuit release tag for KZG parametersNOMOS_NODE_REV — Git revision of nomos-node to build/fetchNOMOS_BUNDLE_VERSION — Bundle schema versionWhere it's used:
+scripts/run/run-examples.shscripts/build/build-bundle.shscripts/setup/setup-nomos-circuits.shError if missing:
+ERROR: versions.env not found at repository root
+This file is required and should define:
+ VERSION=<circuit release tag>
+ NOMOS_NODE_REV=<nomos-node git revision>
+ NOMOS_BUNDLE_VERSION=<bundle schema version>
+Fix: Ensure you're in the repository root. The file should already exist in the checked-out repo.
+Scenarios need compiled nomos-node and nomos-executor binaries.
scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
+This automatically:
+NOMOS_NODE_BIN / NOMOS_EXECUTOR_BINIf you have a sibling nomos-node checkout:
cd ../nomos-node
+cargo build --release --bin nomos-node --bin nomos-executor
+
+# Set environment variables
+export NOMOS_NODE_BIN=$PWD/target/release/nomos-node
+export NOMOS_EXECUTOR_BIN=$PWD/target/release/nomos-executor
+
+# Return to testing framework
+cd ../nomos-testing
+CI workflows use prebuilt artifacts:
+- name: Download nomos binaries
+ uses: actions/download-artifact@v3
+ with:
+ name: nomos-binaries-linux
+ path: .tmp/
+
+- name: Extract bundle
+ run: |
+ tar -xzf .tmp/nomos-binaries-linux-*.tar.gz -C .tmp/
+ export NOMOS_NODE_BIN=$PWD/.tmp/nomos-node
+ export NOMOS_EXECUTOR_BIN=$PWD/.tmp/nomos-executor
+Data Availability (DA) workloads require KZG cryptographic parameters.
+Default path: testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params
Note: The directory kzgrs_test_params/ contains a file named kzgrs_test_params. This is the proving key file (~120MB).
Container path (compose/k8s): /kzgrs_test_params/kzgrs_test_params
Option 1: Use helper script (recommended):
+# Fetch circuits
+scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits
+
+# Copy to default location
+mkdir -p testing-framework/assets/stack/kzgrs_test_params
+cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/
+
+# Verify (should be ~120MB)
+ls -lh testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params
+Option 2: Let run-examples.sh handle it:
scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
+This automatically fetches and places assets.
+Set NOMOS_KZGRS_PARAMS_PATH to use a custom location:
NOMOS_KZGRS_PARAMS_PATH=/custom/path/to/kzgrs_test_params \
+cargo run -p runner-examples --bin local_runner
+| Runner | When Required |
|---|---|
| Host (local) | Always (for DA workloads) |
| Compose | During image build (baked into image) |
| K8s | During image build + mounted via hostPath |
Error without assets:
+Error: Custom { kind: NotFound, error: "Circuit file not found at: testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params" }
+Requires:
+No Docker required.
+Best for:
+Requires:
+logos-blockchain-testing:localPlatform notes (macOS / Apple silicon):
+NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64 for native performancelinux/amd64 only if targeting amd64 environments (slower via emulation)Best for:
+Requires:
+kubectl configuredLocal cluster setup:
+# Docker Desktop: Enable Kubernetes in settings
+
+# OR: Use kind
+kind create cluster
+kind load docker-image logos-blockchain-testing:local
+
+# OR: Use minikube
+minikube start
+minikube image load logos-blockchain-testing:local
+Remote cluster: Push image to registry and set NOMOS_TESTNET_IMAGE.
Best for:
+POL_PROOF_DEV_MODE=true is REQUIRED for ALL runners!
Without this, proof generation uses expensive Groth16 proving, causing:
+Always set:
+POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+POL_PROOF_DEV_MODE=true scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+# etc.
+Or add to your shell profile:
+# ~/.bashrc or ~/.zshrc
+export POL_PROOF_DEV_MODE=true
+Run this checklist before your first scenario:
+# 1. Verify versions.env exists
+cat versions.env
+
+# 2. Check circuit assets (for DA workloads)
+ls -lh testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params
+
+# 3. Verify POL_PROOF_DEV_MODE is set
+echo $POL_PROOF_DEV_MODE # Should print: true
+
+# 4. For compose/k8s: verify Docker is running
+docker ps
+
+# 5. For compose/k8s: verify image exists
+docker images | grep logos-blockchain-testing
+
+# 6. For host runner: verify node binaries (if not using scripts)
+$NOMOS_NODE_BIN --version
+$NOMOS_EXECUTOR_BIN --version
+The easiest path is to let the helper scripts handle everything:
+# Host runner
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
+
+# Compose runner
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+
+# K8s runner
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
+These scripts:
+versions.env existsNext Steps:
+The framework provides three runner modes: host (local processes), compose (Docker Compose), and k8s (Kubernetes).
+Use scripts/run/run-examples.sh for all modes—it handles all setup automatically:
# Host mode (local processes)
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
+
+# Compose mode (Docker Compose)
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+
+# K8s mode (Kubernetes)
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
+Parameters:
+-t 60 — Run duration in seconds-v 3 — Number of validators-e 1 — Number of executorshost|compose|k8s — Deployment modeThis script handles:
+Note: For k8s runs against non-local clusters (e.g. EKS), the cluster pulls images from a registry. In that case, build + push your image separately (see scripts/build/build_test_image.sh) and set NOMOS_TESTNET_IMAGE to the pushed reference.
For a small "does everything still run?" matrix across all runners:
+scripts/run/run-test-matrix.sh -t 120 -v 1 -e 1
+This runs host, compose, and k8s modes with various image-build configurations. Useful after making runner/image/script changes. Forwards --metrics-* options through to scripts/run/run-examples.sh.
Common options:
+--modes host,compose,k8s — Restrict which modes run--no-clean — Skip scripts/ops/clean.sh step--no-bundles — Skip scripts/build/build-bundle.sh (reuses existing .tmp tarballs)--no-image-build — Skip the “rebuild image” variants in the matrix (compose/k8s)--allow-nonzero-progress — Soft-pass expectation failures if logs show non-zero progress (local iteration only)--force-k8s-image-build — Allow the k8s image-build variant even on non-docker-desktop clustersEnvironment overrides:
+VERSION=v0.3.1 — Circuit versionNOMOS_NODE_REV=<commit> — nomos-node git revisionNOMOS_BINARIES_TAR=path/to/bundle.tar.gz — Use prebuilt bundleNOMOS_SKIP_IMAGE_BUILD=1 — Skip image rebuild inside run-examples.sh (compose/k8s)NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64|linux/amd64 — Docker platform for bundle builds (macOS/Windows)COMPOSE_CIRCUITS_PLATFORM=linux-aarch64|linux-x86_64 — Circuits platform for image buildsSLOW_TEST_ENV=true — Doubles built-in readiness timeouts (useful in CI / constrained laptops)TESTNET_PRINT_ENDPOINTS=1 — Print TESTNET_ENDPOINTS / TESTNET_PPROF lines during deployThe repo pins a nomos-node revision in versions.env for reproducible builds. To update it or point to a local checkout:
# Pin to a new git revision (updates versions.env + Cargo.toml git revs)
+scripts/ops/update-nomos-rev.sh --rev <git_sha>
+
+# Use a local nomos-node checkout instead (for development)
+scripts/ops/update-nomos-rev.sh --path /path/to/nomos-node
+
+# If Cargo.toml was marked skip-worktree, clear it
+scripts/ops/update-nomos-rev.sh --unskip-worktree
+Notes:
+NOMOS_NODE_PATH values; prefer --rev for shared history/CICargo.lock to update on the next cargo build/cargo testIf you hit Docker build failures, I/O errors, or disk space issues:
+scripts/ops/clean.sh
+For extra Docker cache cleanup:
+scripts/ops/clean.sh --docker
+For manual control, run the local_runner binary directly:
POL_PROOF_DEV_MODE=true \
+NOMOS_NODE_BIN=/path/to/nomos-node \
+NOMOS_EXECUTOR_BIN=/path/to/nomos-executor \
+cargo run -p runner-examples --bin local_runner
+| Variable | Default | Effect |
|---|---|---|
NOMOS_DEMO_VALIDATORS | 1 | Number of validators (legacy: LOCAL_DEMO_VALIDATORS) |
NOMOS_DEMO_EXECUTORS | 1 | Number of executors (legacy: LOCAL_DEMO_EXECUTORS) |
NOMOS_DEMO_RUN_SECS | 60 | Run duration in seconds (legacy: LOCAL_DEMO_RUN_SECS) |
NOMOS_NODE_BIN | — | Path to nomos-node binary (required) |
NOMOS_EXECUTOR_BIN | — | Path to nomos-executor binary (required) |
NOMOS_LOG_DIR | None | Directory for per-node log files |
NOMOS_TESTS_KEEP_LOGS | 0 | Keep per-run temporary directories (useful for debugging/CI) |
NOMOS_TESTS_TRACING | false | Enable debug tracing preset |
NOMOS_LOG_LEVEL | info | Global log level: error, warn, info, debug, trace |
NOMOS_LOG_FILTER | None | Fine-grained module filtering (e.g., cryptarchia=trace,nomos_da_sampling=debug) |
POL_PROOF_DEV_MODE | — | REQUIRED: Set to true for all runners |
Note: Requires circuit assets and host binaries. Use scripts/run/run-examples.sh host to handle setup automatically.
For manual control, run the compose_runner binary directly. Compose requires a Docker image with embedded assets.
# 1. Build a Linux bundle (includes binaries + circuits)
+scripts/build/build-bundle.sh --platform linux
+# Creates .tmp/nomos-binaries-linux-v0.3.1.tar.gz
+
+# 2. Build image (embeds bundle assets)
+export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
+scripts/build/build_test_image.sh
+
+# 3. Run
+NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin compose_runner
+# Fetch and copy circuits
+scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits
+cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/
+
+# Build image
+scripts/build/build_test_image.sh
+
+# Run
+NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin compose_runner
+linux/arm64 engine by defaultNOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64 (recommended for local testing)NOMOS_BUNDLE_DOCKER_PLATFORM=linux/amd64 (slower via emulation)| Variable | Default | Effect |
|---|---|---|
NOMOS_TESTNET_IMAGE | — | Image tag (required, must match built image) |
POL_PROOF_DEV_MODE | — | REQUIRED: Set to true for all runners |
NOMOS_DEMO_VALIDATORS | 1 | Number of validators |
NOMOS_DEMO_EXECUTORS | 1 | Number of executors |
NOMOS_DEMO_RUN_SECS | 60 | Run duration in seconds |
COMPOSE_NODE_PAIRS | — | Alternative topology format: "validators×executors" (e.g., 3x2) |
NOMOS_METRICS_QUERY_URL | None | Prometheus-compatible base URL for runner to query |
NOMOS_METRICS_OTLP_INGEST_URL | None | Full OTLP HTTP ingest URL for node metrics export |
NOMOS_GRAFANA_URL | None | Grafana base URL for printing/logging |
COMPOSE_RUNNER_HOST | 127.0.0.1 | Host address for port mappings |
COMPOSE_RUNNER_PRESERVE | 0 | Keep containers running after test |
NOMOS_LOG_LEVEL | info | Node log level (stdout/stderr) |
NOMOS_LOG_FILTER | None | Fine-grained module filtering |
Config file option: testing-framework/assets/stack/cfgsync.yaml (tracing_settings.logger) — Switch node logs between stdout/stderr and file output
.enable_node_control() + chaos workloads)NOMOS_METRICS_* / NOMOS_GRAFANA_URL to enable telemetry links and querying
+scripts/setup/setup-observability.sh compose up then scripts/setup/setup-observability.sh compose envImportant:
+/kzgrs_test_params/kzgrs_test_params (note the repeated filename)scripts/run/run-examples.sh compose to handle all setup automaticallyFor manual control, run the k8s_runner binary directly. K8s requires the same image setup as Compose.
kubectl configured# 1. Build image with bundle (recommended)
+scripts/build/build-bundle.sh --platform linux
+export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
+scripts/build/build_test_image.sh
+
+# 2. Load into cluster (choose one)
+export NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local
+
+# For kind:
+kind load docker-image logos-blockchain-testing:local
+
+# For minikube:
+minikube image load logos-blockchain-testing:local
+
+# For remote cluster (push to registry):
+docker tag logos-blockchain-testing:local your-registry/logos-blockchain-testing:latest
+docker push your-registry/logos-blockchain-testing:latest
+export NOMOS_TESTNET_IMAGE=your-registry/logos-blockchain-testing:latest
+export NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local
+export POL_PROOF_DEV_MODE=true
+cargo run -p runner-examples --bin k8s_runner
+| Variable | Default | Effect |
|---|---|---|
NOMOS_TESTNET_IMAGE | — | Image tag (required) |
POL_PROOF_DEV_MODE | — | REQUIRED: Set to true for all runners |
NOMOS_DEMO_VALIDATORS | 1 | Number of validators |
NOMOS_DEMO_EXECUTORS | 1 | Number of executors |
NOMOS_DEMO_RUN_SECS | 60 | Run duration in seconds |
NOMOS_METRICS_QUERY_URL | None | Prometheus-compatible base URL for runner to query (PromQL) |
NOMOS_METRICS_OTLP_INGEST_URL | None | Full OTLP HTTP ingest URL for node metrics export |
NOMOS_GRAFANA_URL | None | Grafana base URL for printing/logging |
K8S_RUNNER_NAMESPACE | Random | Kubernetes namespace (pin for debugging) |
K8S_RUNNER_RELEASE | Random | Helm release name (pin for debugging) |
K8S_RUNNER_NODE_HOST | — | NodePort host resolution for non-local clusters |
K8S_RUNNER_DEBUG | 0 | Log Helm stdout/stderr for install commands |
K8S_RUNNER_PRESERVE | 0 | Keep namespace/release after run (for debugging) |
export NOMOS_METRICS_QUERY_URL=http://your-prometheus:9090
+# Prometheus OTLP receiver example:
+export NOMOS_METRICS_OTLP_INGEST_URL=http://your-prometheus:9090/api/v1/otlp/v1/metrics
+# Optional: print Grafana link in TESTNET_ENDPOINTS
+export NOMOS_GRAFANA_URL=http://your-grafana:3000
+cargo run -p runner-examples --bin k8s_runner
+Notes:
+NOMOS_METRICS_QUERY_URL must be reachable from the runner process (often via kubectl port-forward)NOMOS_METRICS_OTLP_INGEST_URL must be reachable from nodes (pods/containers) and is backend-specific
+scripts/setup/setup-observability.sh k8s install then scripts/setup/setup-observability.sh k8s envscripts/setup/setup-observability.sh k8s dashboardsscripts/run/run-examples.sh (Recommended)scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s \
+ --metrics-query-url http://your-prometheus:9090 \
+ --metrics-otlp-ingest-url http://your-prometheus:9090/api/v1/otlp/v1/metrics
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ObservabilityBuilderExt as _;
+
+let plan = ScenarioBuilder::with_node_counts(1, 1)
+ .with_metrics_query_url_str("http://your-prometheus:9090")
+ .with_metrics_otlp_ingest_url_str("http://your-prometheus:9090/api/v1/otlp/v1/metrics")
+ .build();testing-framework/assets/stack/kzgrs_test_params as a hostPath volume/kzgrs_test_params/kzgrs_test_params.enable_node_control()) will failscripts/run/run-examples.sh k8s to handle all setup automaticallyBoth LocalDeployer and ComposeDeployer work well in CI environments. Choose based on your tradeoffs.
+LocalDeployer (Host Runner):
+ComposeDeployer (Recommended for CI):
+K8sDeployer:
+Existing Examples:
+See .github/workflows/lint.yml (jobs: host_smoke, compose_smoke) for CI examples running the demo scenarios in this repository.
Here's a comprehensive GitHub Actions workflow demonstrating host and compose runners with caching, matrix testing, and log collection:
+name: Testing Framework CI
+
+on:
+ push:
+ branches: [main, develop]
+ pull_request:
+ branches: [main]
+
+env:
+ POL_PROOF_DEV_MODE: true
+ CARGO_TERM_COLOR: always
+ RUST_BACKTRACE: 1
+
+jobs:
+ # Quick smoke test with host runner (no Docker)
+ host_smoke:
+ name: Host Runner Smoke Test
+ runs-on: ubuntu-latest
+ timeout-minutes: 15
+
+ steps:
+ - name: Checkout repository
+ uses: actions/checkout@v4
+
+ - name: Set up Rust toolchain
+ uses: actions-rs/toolchain@v1
+ with:
+ profile: minimal
+ toolchain: nightly
+ override: true
+
+ - name: Cache Rust dependencies
+ uses: actions/cache@v3
+ with:
+ path: |
+ ~/.cargo/bin/
+ ~/.cargo/registry/index/
+ ~/.cargo/registry/cache/
+ ~/.cargo/git/db/
+ target/
+ key: ${{ runner.os }}-cargo-host-${{ hashFiles('**/Cargo.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-cargo-host-
+
+ - name: Cache nomos-node build
+ uses: actions/cache@v3
+ with:
+ path: |
+ ../nomos-node/target/release/nomos-node
+ ../nomos-node/target/release/nomos-executor
+ key: ${{ runner.os }}-nomos-${{ hashFiles('../nomos-node/**/Cargo.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-nomos-
+
+ - name: Run host smoke test
+ run: |
+ # Use run-examples.sh which handles setup automatically
+ scripts/run/run-examples.sh -t 120 -v 3 -e 1 host
+
+ - name: Upload logs on failure
+ if: failure()
+ uses: actions/upload-artifact@v3
+ with:
+ name: host-runner-logs
+ path: |
+ .tmp/
+ *.log
+ retention-days: 7
+
+ # Compose runner matrix (with Docker)
+ compose_matrix:
+ name: Compose Runner (${{ matrix.topology }})
+ runs-on: ubuntu-latest
+ timeout-minutes: 25
+
+ strategy:
+ fail-fast: false
+ matrix:
+ topology:
+ - "3v1e"
+ - "5v1e"
+
+ steps:
+ - name: Checkout repository
+ uses: actions/checkout@v4
+
+ - name: Set up Rust toolchain
+ uses: actions-rs/toolchain@v1
+ with:
+ profile: minimal
+ toolchain: nightly
+ override: true
+
+ - name: Set up Docker Buildx
+ uses: docker/setup-buildx-action@v2
+
+ - name: Cache Rust dependencies
+ uses: actions/cache@v3
+ with:
+ path: |
+ ~/.cargo/bin/
+ ~/.cargo/registry/index/
+ ~/.cargo/registry/cache/
+ ~/.cargo/git/db/
+ target/
+ key: ${{ runner.os }}-cargo-compose-${{ hashFiles('**/Cargo.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-cargo-compose-
+
+ - name: Cache Docker layers
+ uses: actions/cache@v3
+ with:
+ path: /tmp/.buildx-cache
+ key: ${{ runner.os }}-buildx-${{ hashFiles('Dockerfile', 'scripts/build/build_test_image.sh') }}
+ restore-keys: |
+ ${{ runner.os }}-buildx-
+
+ - name: Run compose test
+ env:
+ TOPOLOGY: ${{ matrix.topology }}
+ run: |
+ # Build and run with the specified topology
+ scripts/run/run-examples.sh -t 120 -v ${TOPOLOGY:0:1} -e ${TOPOLOGY:2:1} compose
+
+ - name: Collect Docker logs on failure
+ if: failure()
+ run: |
+ mkdir -p logs
+ for container in $(docker ps -a --filter "name=nomos-compose-" -q); do
+ docker logs $container > logs/$(docker inspect --format='{{.Name}}' $container).log 2>&1
+ done
+
+ - name: Upload logs and artifacts
+ if: failure()
+ uses: actions/upload-artifact@v3
+ with:
+ name: compose-${{ matrix.topology }}-logs
+ path: |
+ logs/
+ .tmp/
+ retention-days: 7
+
+ - name: Clean up Docker resources
+ if: always()
+ run: |
+ docker compose down -v 2>/dev/null || true
+ docker ps -a --filter "name=nomos-compose-" -q | xargs -r docker rm -f
+
+ # Cucumber/BDD integration tests (if enabled)
+ cucumber_tests:
+ name: Cucumber BDD Tests
+ runs-on: ubuntu-latest
+ timeout-minutes: 20
+
+ steps:
+ - name: Checkout repository
+ uses: actions/checkout@v4
+
+ - name: Set up Rust toolchain
+ uses: actions-rs/toolchain@v1
+ with:
+ profile: minimal
+ toolchain: nightly
+ override: true
+
+ - name: Cache dependencies
+ uses: actions/cache@v3
+ with:
+ path: |
+ ~/.cargo/bin/
+ ~/.cargo/registry/index/
+ ~/.cargo/registry/cache/
+ ~/.cargo/git/db/
+ target/
+ key: ${{ runner.os }}-cargo-cucumber-${{ hashFiles('**/Cargo.lock') }}
+ restore-keys: |
+ ${{ runner.os }}-cargo-cucumber-
+
+ - name: Run Cucumber tests
+ run: |
+ # Build prerequisites
+ scripts/build/build-bundle.sh --platform linux
+ export NOMOS_BINARIES_TAR=$(ls -t .tmp/nomos-binaries-linux-*.tar.gz | head -1)
+
+ # Run Cucumber tests (host runner)
+ cargo test -p runner-examples --bin cucumber_host
+
+ - name: Upload test report
+ if: always()
+ uses: actions/upload-artifact@v3
+ with:
+ name: cucumber-report
+ path: |
+ target/cucumber-reports/
+ retention-days: 14
+
+ # Summary job (requires all tests to pass)
+ ci_success:
+ name: CI Success
+ needs: [host_smoke, compose_matrix, cucumber_tests]
+ runs-on: ubuntu-latest
+ if: always()
+
+ steps:
+ - name: Check all jobs
+ run: |
+ if [[ "${{ needs.host_smoke.result }}" != "success" ]] || \
+ [[ "${{ needs.compose_matrix.result }}" != "success" ]] || \
+ [[ "${{ needs.cucumber_tests.result }}" != "success" ]]; then
+ echo "One or more CI jobs failed"
+ exit 1
+ fi
+ echo "All CI jobs passed!"
+3v1e, 5v1e)Topology Matrix:
+Add more topologies for comprehensive testing:
+matrix:
+ topology:
+ - "3v1e"
+ - "5v1e"
+ - "10v2e" # Larger scale
+Timeout Adjustments:
+Increase timeout-minutes for longer-running scenarios or slower environments:
timeout-minutes: 30 # Instead of 15
+Artifact Retention:
+Change retention-days based on your storage needs:
retention-days: 14 # Keep logs for 2 weeks
+Conditional Execution:
+Run expensive tests only on merge to main:
+if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+Always set POL_PROOF_DEV_MODE=true globally in your workflow env:
env:
+ POL_PROOF_DEV_MODE: true # REQUIRED!
+Without this, tests will hang due to expensive proof generation.
+Prefer scripts/run/run-examples.sh which handles all setup automatically:
scripts/run/run-examples.sh -t 120 -v 3 -e 1 host
+This is more reliable than manual cargo run commands.
Cache Rust dependencies, nomos-node builds, and Docker layers to speed up CI:
+- name: Cache Rust dependencies
+ uses: actions/cache@v3
+ with:
+ path: |
+ ~/.cargo/bin/
+ ~/.cargo/registry/index/
+ ~/.cargo/registry/cache/
+ ~/.cargo/git/db/
+ target/
+ key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
+Always upload logs when tests fail for easier debugging:
+- name: Upload logs on failure
+ if: failure()
+ uses: actions/upload-artifact@v3
+ with:
+ name: test-logs
+ path: |
+ .tmp/
+ *.log
+ retention-days: 7
+For large projects, split host/compose/k8s into separate workflow files:
+.github/workflows/test-host.yml — Fast smoke tests.github/workflows/test-compose.yml — Reproducible integration tests.github/workflows/test-k8s.yml — Production-like validation (nightly)K8s tests are slower. Consider running them only on main branch or scheduled:
+on:
+ push:
+ branches: [main]
+ schedule:
+ - cron: '0 2 * * *' # Daily at 2 AM
+SLOW_TEST_ENV=true if needed)docker system prune)Add debug environment variables temporarily:
+env:
+ RUST_LOG: debug
+ NOMOS_LOG_LEVEL: debug
+Set COMPOSE_RUNNER_PRESERVE=1 to keep containers running for inspection:
- name: Run compose test (preserve on failure)
+ env:
+ COMPOSE_RUNNER_PRESERVE: 1
+ run: scripts/run/run-examples.sh -t 120 -v 3 -e 1 compose
+Download uploaded artifacts from the GitHub Actions UI to inspect logs locally.
+Complete reference of environment variables used by the testing framework, organized by category.
+These MUST be set for successful test runs:
+| Variable | Required | Default | Effect |
|---|---|---|---|
POL_PROOF_DEV_MODE | YES | — | REQUIRED for all runners. Set to true to use fast dev-mode proving instead of expensive Groth16. Without this, tests will hang/timeout. |
Example:
+export POL_PROOF_DEV_MODE=true
+Or add to your shell profile (~/.bashrc, ~/.zshrc):
# Required for nomos-testing framework
+export POL_PROOF_DEV_MODE=true
+Control which runner to use and the test topology:
+| Variable | Default | Effect |
|---|---|---|
NOMOS_DEMO_VALIDATORS | 1 | Number of validators (all runners) |
NOMOS_DEMO_EXECUTORS | 1 | Number of executors (all runners) |
NOMOS_DEMO_RUN_SECS | 60 | Run duration in seconds (all runners) |
LOCAL_DEMO_VALIDATORS | — | Legacy: Number of validators (host runner only) |
LOCAL_DEMO_EXECUTORS | — | Legacy: Number of executors (host runner only) |
LOCAL_DEMO_RUN_SECS | — | Legacy: Run duration (host runner only) |
COMPOSE_NODE_PAIRS | — | Compose-specific topology format: "validators×executors" (e.g., 3x2) |
Example:
+# Run with 5 validators, 2 executors, for 120 seconds
+NOMOS_DEMO_VALIDATORS=5 \
+NOMOS_DEMO_EXECUTORS=2 \
+NOMOS_DEMO_RUN_SECS=120 \
+scripts/run/run-examples.sh -t 120 -v 5 -e 2 host
+Required for host runner when not using helper scripts:
+| Variable | Required | Default | Effect |
|---|---|---|---|
NOMOS_NODE_BIN | Yes (host) | — | Path to nomos-node binary |
NOMOS_EXECUTOR_BIN | Yes (host) | — | Path to nomos-executor binary |
NOMOS_NODE_PATH | No | — | Path to nomos-node git checkout (dev workflow) |
Example:
+export NOMOS_NODE_BIN=/path/to/nomos-node/target/release/nomos-node
+export NOMOS_EXECUTOR_BIN=/path/to/nomos-node/target/release/nomos-executor
+Required for compose and k8s runners:
+| Variable | Required | Default | Effect |
|---|---|---|---|
NOMOS_TESTNET_IMAGE | Yes (compose/k8s) | logos-blockchain-testing:local | Docker image tag for node containers |
NOMOS_TESTNET_IMAGE_PULL_POLICY | No | IfNotPresent (local) / Always (ECR) | K8s imagePullPolicy used by the runner |
NOMOS_BINARIES_TAR | No | — | Path to prebuilt bundle (.tar.gz) for image build |
NOMOS_SKIP_IMAGE_BUILD | No | 0 | Skip image rebuild (compose/k8s); assumes image already exists |
NOMOS_FORCE_IMAGE_BUILD | No | 0 | Force rebuilding the image even when the script would normally skip it (e.g. non-local k8s) |
Example:
+# Using prebuilt bundle
+export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
+export NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local
+scripts/build/build_test_image.sh
+
+# Using pre-existing image (skip build)
+export NOMOS_SKIP_IMAGE_BUILD=1
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+Circuit asset configuration for DA workloads:
+| Variable | Default | Effect |
|---|---|---|
NOMOS_KZGRS_PARAMS_PATH | testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params | Path to KZG proving key file |
NOMOS_KZG_DIR_REL | testing-framework/assets/stack/kzgrs_test_params | Directory containing KZG assets (relative to workspace root) |
NOMOS_KZG_FILE | kzgrs_test_params | Filename of the proving key within NOMOS_KZG_DIR_REL |
NOMOS_KZG_CONTAINER_PATH | /kzgrs_test_params/kzgrs_test_params | File path where the node expects KZG params inside containers |
NOMOS_KZG_MODE | Runner-specific | K8s only: hostPath (mount from host) or inImage (embed into image) |
NOMOS_KZG_IN_IMAGE_PARAMS_PATH | /opt/nomos/kzg-params/kzgrs_test_params | K8s inImage mode: where the proving key is stored inside the image |
VERSION | From versions.env | Circuit release tag (used by helper scripts) |
NOMOS_CIRCUITS | — | Directory containing fetched circuit bundles (set by scripts/setup/setup-circuits-stack.sh) |
NOMOS_CIRCUITS_VERSION | — | Legacy alias for VERSION (supported by some build scripts) |
NOMOS_CIRCUITS_PLATFORM | Auto-detected | Override circuits platform (e.g. linux-x86_64, macos-aarch64) |
NOMOS_CIRCUITS_HOST_DIR_REL | .tmp/nomos-circuits-host | Output dir for host circuits bundle (relative to repo root) |
NOMOS_CIRCUITS_LINUX_DIR_REL | .tmp/nomos-circuits-linux | Output dir for linux circuits bundle (relative to repo root) |
NOMOS_CIRCUITS_NONINTERACTIVE | 0 | Set to 1 to overwrite outputs without prompting in setup scripts |
NOMOS_CIRCUITS_REBUILD_RAPIDSNARK | 0 | Set to 1 to force rebuilding rapidsnark (host bundle only) |
Example:
+# Use custom circuit assets
+NOMOS_KZGRS_PARAMS_PATH=/custom/path/to/kzgrs_test_params \
+cargo run -p runner-examples --bin local_runner
+Control node log output (not framework runner logs):
+| Variable | Default | Effect |
|---|---|---|
NOMOS_LOG_LEVEL | info | Global log level: error, warn, info, debug, trace |
NOMOS_LOG_FILTER | — | Fine-grained module filtering (e.g., cryptarchia=trace,nomos_da_sampling=debug) |
NOMOS_LOG_DIR | — | Host runner: directory for per-node log files (persistent). Compose/k8s: use cfgsync.yaml for file logging. |
NOMOS_TESTS_KEEP_LOGS | 0 | Keep per-run temporary directories (useful for debugging/CI artifacts) |
NOMOS_TESTS_TRACING | false | Enable debug tracing preset (combine with NOMOS_LOG_DIR unless external tracing backends configured) |
Important: Node logging ignores RUST_LOG; use NOMOS_LOG_LEVEL and NOMOS_LOG_FILTER for node logs.
Example:
+# Debug logging to files
+NOMOS_LOG_DIR=/tmp/test-logs \
+NOMOS_LOG_LEVEL=debug \
+NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug" \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin local_runner
+
+# Inspect logs
+ls /tmp/test-logs/
+# nomos-node-0.2024-12-18T14-30-00.log
+# nomos-node-1.2024-12-18T14-30-00.log
+Common filter targets:
+| Target Prefix | Subsystem |
|---|---|
cryptarchia | Consensus (Cryptarchia) |
nomos_da_sampling | DA sampling service |
nomos_da_dispersal | DA dispersal service |
nomos_da_verifier | DA verification |
nomos_blend | Mix network/privacy layer |
chain_service | Chain service (node APIs/state) |
chain_network | P2P networking |
chain_leader | Leader election |
Optional observability integration:
+| Variable | Default | Effect |
|---|---|---|
NOMOS_METRICS_QUERY_URL | — | Prometheus-compatible base URL for runner to query (e.g., http://localhost:9090) |
NOMOS_METRICS_OTLP_INGEST_URL | — | Full OTLP HTTP ingest URL for node metrics export (e.g., http://localhost:9090/api/v1/otlp/v1/metrics) |
NOMOS_GRAFANA_URL | — | Grafana base URL for printing/logging (e.g., http://localhost:3000) |
NOMOS_OTLP_ENDPOINT | — | OTLP trace endpoint (optional) |
NOMOS_OTLP_METRICS_ENDPOINT | — | OTLP metrics endpoint (optional) |
Example:
+# Enable Prometheus querying
+export NOMOS_METRICS_QUERY_URL=http://localhost:9090
+export NOMOS_METRICS_OTLP_INGEST_URL=http://localhost:9090/api/v1/otlp/v1/metrics
+export NOMOS_GRAFANA_URL=http://localhost:3000
+
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+Variables specific to Docker Compose deployment:
+| Variable | Default | Effect |
|---|---|---|
COMPOSE_RUNNER_HOST | 127.0.0.1 | Host address for port mappings |
COMPOSE_RUNNER_PRESERVE | 0 | Keep containers running after test (for debugging) |
COMPOSE_RUNNER_HTTP_TIMEOUT_SECS | — | Override HTTP readiness timeout (seconds) |
COMPOSE_RUNNER_HOST_GATEWAY | host.docker.internal:host-gateway | Controls extra_hosts entry injected into compose (set to disable to omit) |
TESTNET_RUNNER_PRESERVE | — | Alias for COMPOSE_RUNNER_PRESERVE |
Example:
+# Keep containers after test for debugging
+COMPOSE_RUNNER_PRESERVE=1 \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+
+# Containers remain running
+docker ps --filter "name=nomos-compose-"
+docker logs <container-id>
+Variables specific to Kubernetes deployment:
+| Variable | Default | Effect |
|---|---|---|
K8S_RUNNER_NAMESPACE | Random UUID | Kubernetes namespace (pin for debugging) |
K8S_RUNNER_RELEASE | Random UUID | Helm release name (pin for debugging) |
K8S_RUNNER_NODE_HOST | — | NodePort host resolution for non-local clusters |
K8S_RUNNER_DEBUG | 0 | Log Helm stdout/stderr for install commands |
K8S_RUNNER_PRESERVE | 0 | Keep namespace/release after run (for debugging) |
K8S_RUNNER_DEPLOYMENT_TIMEOUT_SECS | — | Override deployment readiness timeout |
K8S_RUNNER_HTTP_TIMEOUT_SECS | — | Override HTTP readiness timeout (port-forwards) |
K8S_RUNNER_HTTP_PROBE_TIMEOUT_SECS | — | Override HTTP readiness timeout (NodePort probes) |
K8S_RUNNER_PROMETHEUS_HTTP_TIMEOUT_SECS | — | Override Prometheus readiness timeout |
K8S_RUNNER_PROMETHEUS_HTTP_PROBE_TIMEOUT_SECS | — | Override Prometheus NodePort probe timeout |
Example:
+# Pin namespace for debugging
+K8S_RUNNER_NAMESPACE=nomos-test-debug \
+K8S_RUNNER_PRESERVE=1 \
+K8S_RUNNER_DEBUG=1 \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
+
+# Inspect resources
+kubectl get pods -n nomos-test-debug
+kubectl logs -n nomos-test-debug -l nomos/logical-role=validator
+Platform-specific build configuration:
+| Variable | Default | Effect |
|---|---|---|
NOMOS_BUNDLE_DOCKER_PLATFORM | Host arch | Docker platform for bundle builds: linux/arm64 or linux/amd64 (macOS/Windows hosts) |
NOMOS_BIN_PLATFORM | — | Legacy alias for NOMOS_BUNDLE_DOCKER_PLATFORM |
COMPOSE_CIRCUITS_PLATFORM | Host arch | Circuits platform for image builds: linux-aarch64 or linux-x86_64 |
NOMOS_EXTRA_FEATURES | — | Extra cargo features to enable when building bundles (used by scripts/build/build-bundle.sh) |
macOS / Apple Silicon:
+# Native performance (recommended for local testing)
+export NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64
+
+# Or target amd64 (slower via emulation)
+export NOMOS_BUNDLE_DOCKER_PLATFORM=linux/amd64
+Timeout and performance tuning:
+| Variable | Default | Effect |
|---|---|---|
SLOW_TEST_ENV | false | Doubles built-in readiness timeouts (useful in CI / constrained laptops) |
TESTNET_PRINT_ENDPOINTS | 0 | Print TESTNET_ENDPOINTS / TESTNET_PPROF lines during deploy (set automatically by scripts/run/run-examples.sh) |
NOMOS_DISPERSAL_TIMEOUT_SECS | 20 | DA dispersal timeout (seconds) |
NOMOS_RETRY_COOLDOWN_SECS | 3 | Cooldown between retries (seconds) |
NOMOS_GRACE_PERIOD_SECS | 1200 | Grace period before enforcing strict time-based expectations (seconds) |
NOMOS_PRUNE_DURATION_SECS | 30 | Prune step duration (seconds) |
NOMOS_PRUNE_INTERVAL_SECS | 5 | Interval between prune cycles (seconds) |
NOMOS_SHARE_DURATION_SECS | 5 | Share duration (seconds) |
NOMOS_COMMITMENTS_WAIT_SECS | 1 | Commitments wait duration (seconds) |
NOMOS_SDP_TRIGGER_DELAY_SECS | 5 | SDP trigger delay (seconds) |
Example:
+# Increase timeouts for slow environments
+SLOW_TEST_ENV=true \
+scripts/run/run-examples.sh -t 120 -v 5 -e 2 compose
+Node-level configuration passed through to nomos-node/nomos-executor:
+| Variable | Default | Effect |
|---|---|---|
CONSENSUS_SLOT_TIME | — | Consensus slot time (seconds) |
CONSENSUS_ACTIVE_SLOT_COEFF | — | Active slot coefficient (0.0-1.0) |
NOMOS_USE_AUTONAT | Unset | If set, use AutoNAT instead of a static loopback address for libp2p NAT settings |
NOMOS_CFGSYNC_PORT | 4400 | Port used for cfgsync service inside the stack |
NOMOS_TIME_BACKEND | monotonic | Select time backend (used by compose/k8s stack scripts and deployers) |
Example:
+# Faster block production
+CONSENSUS_SLOT_TIME=5 \
+CONSENSUS_ACTIVE_SLOT_COEFF=0.9 \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin local_runner
+Control framework runner process logs (uses RUST_LOG, not NOMOS_*):
| Variable | Default | Effect |
|---|---|---|
RUST_LOG | — | Framework runner log level (e.g., debug, info) |
RUST_BACKTRACE | — | Enable Rust backtraces on panic (1 or full) |
CARGO_TERM_COLOR | — | Cargo output color (always, never, auto) |
Example:
+# Debug framework runner (not nodes)
+RUST_LOG=debug \
+RUST_BACKTRACE=1 \
+cargo run -p runner-examples --bin local_runner
+Variables used by helper scripts (scripts/run/run-examples.sh, etc.):
| Variable | Default | Effect |
|---|---|---|
NOMOS_NODE_REV | From versions.env | nomos-node git revision to build/fetch |
NOMOS_BUNDLE_VERSION | From versions.env | Bundle schema version |
NOMOS_IMAGE_SELECTION | — | Internal: image selection mode set by run-examples.sh (local/ecr/auto) |
NOMOS_NODE_APPLY_PATCHES | 1 | Set to 0 to disable applying local patches when building bundles |
NOMOS_NODE_PATCH_DIR | patches/nomos-node | Patch directory applied to nomos-node checkout during bundle builds |
NOMOS_NODE_PATCH_LEVEL | — | Patch application level (all or an integer) for bundle builds |
POL_PROOF_DEV_MODE=true \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
+POL_PROOF_DEV_MODE=true \
+NOMOS_LOG_DIR=/tmp/logs \
+NOMOS_LOG_LEVEL=debug \
+NOMOS_LOG_FILTER="cryptarchia=trace" \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
+POL_PROOF_DEV_MODE=true \
+NOMOS_METRICS_QUERY_URL=http://localhost:9090 \
+NOMOS_GRAFANA_URL=http://localhost:3000 \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+POL_PROOF_DEV_MODE=true \
+K8S_RUNNER_NAMESPACE=nomos-debug \
+K8S_RUNNER_DEBUG=1 \
+K8S_RUNNER_PRESERVE=1 \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
+env:
+ POL_PROOF_DEV_MODE: true
+ RUST_BACKTRACE: 1
+ NOMOS_TESTS_KEEP_LOGS: 1
+Comprehensive guide to log collection, metrics, and debugging across all runners.
+Critical distinction: Node logs and framework logs use different configuration mechanisms.
+| Component | Controlled By | Purpose |
|---|---|---|
Framework binaries (cargo run -p runner-examples --bin local_runner) | RUST_LOG | Runner orchestration, deployment logs |
| Node processes (validators, executors spawned by runner) | NOMOS_LOG_LEVEL, NOMOS_LOG_FILTER (+ NOMOS_LOG_DIR on host runner) | Consensus, DA, mempool, network logs |
Common mistake: Setting RUST_LOG=debug only increases verbosity of the runner binary itself. Node logs remain at their default level unless you also set NOMOS_LOG_LEVEL=debug.
Example:
+# This only makes the RUNNER verbose, not the nodes:
+RUST_LOG=debug cargo run -p runner-examples --bin local_runner
+
+# This makes the NODES verbose:
+NOMOS_LOG_LEVEL=debug cargo run -p runner-examples --bin local_runner
+
+# Both verbose (typically not needed):
+RUST_LOG=debug NOMOS_LOG_LEVEL=debug cargo run -p runner-examples --bin local_runner
+See Environment Variables Reference for complete details. Quick summary:
+| Variable | Default | Effect |
|---|---|---|
NOMOS_LOG_DIR | None (console only) | Host runner: directory for per-node log files. Compose/k8s: use cfgsync.yaml |
NOMOS_LOG_LEVEL | info | Global log level: error, warn, info, debug, trace |
NOMOS_LOG_FILTER | None | Fine-grained target filtering (e.g., cryptarchia=trace,nomos_da_sampling=debug) |
NOMOS_TESTS_TRACING | false | Enable debug tracing preset |
NOMOS_OTLP_ENDPOINT | None | OTLP trace endpoint (optional) |
NOMOS_OTLP_METRICS_ENDPOINT | None | OTLP metrics endpoint (optional) |
Example: Full debug logging to files:
+NOMOS_TESTS_TRACING=true \
+NOMOS_LOG_DIR=/tmp/test-logs \
+NOMOS_LOG_LEVEL=debug \
+NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug,nomos_da_dispersal=debug,nomos_da_verifier=debug" \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin local_runner
+When NOMOS_LOG_DIR is set, each node writes logs to separate files:
File naming pattern:
+nomos-node-0, nomos-node-1, etc. (may include timestamp suffix)nomos-executor-0, nomos-executor-1, etc. (may include timestamp suffix)Example filenames:
+nomos-node-0.2024-12-18T14-30-00.lognomos-node-1.2024-12-18T14-30-00.lognomos-executor-0.2024-12-18T14-30-00.logLocal runner note: The local runner uses per-run temporary directories under the current working directory and removes them after the run unless NOMOS_TESTS_KEEP_LOGS=1. Use NOMOS_LOG_DIR=/path/to/logs to write per-node log files to a stable location.
Common target prefixes for NOMOS_LOG_FILTER:
| Target Prefix | Subsystem |
|---|---|
cryptarchia | Consensus (Cryptarchia) |
nomos_da_sampling | DA sampling service |
nomos_da_dispersal | DA dispersal service |
nomos_da_verifier | DA verification |
nomos_blend | Mix network/privacy layer |
chain_service | Chain service (node APIs/state) |
chain_network | P2P networking |
chain_leader | Leader election |
Example filter:
+NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug,chain_service=info,chain_network=info"
+Default (temporary directories, auto-cleanup):
+POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+# Logs written to temporary directories in working directory
+# Automatically cleaned up after test completes
+Persistent file output:
+NOMOS_LOG_DIR=/tmp/local-logs \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin local_runner
+
+# After test completes:
+ls /tmp/local-logs/
+# Files with prefix: nomos-node-0*, nomos-node-1*, nomos-executor-0*
+# May include timestamps in filename
+Tip: Use NOMOS_LOG_DIR for persistent per-node log files, and NOMOS_TESTS_KEEP_LOGS=1 if you want to keep the per-run temporary directories (configs/state) for post-mortem inspection.
Via Docker logs (default, recommended):
+# List containers (note the UUID prefix in names)
+docker ps --filter "name=nomos-compose-"
+
+# Stream logs from specific container
+docker logs -f <container-id-or-name>
+
+# Or use name pattern matching:
+docker logs -f $(docker ps --filter "name=nomos-compose-.*-validator-0" -q | head -1)
+
+# Show last 100 lines
+docker logs --tail 100 <container-id>
+Via file collection (advanced):
+To write per-node log files inside containers, set tracing_settings.logger: !File in testing-framework/assets/stack/cfgsync.yaml (and ensure the directory is writable). To access them, you must either:
# Ensure cfgsync.yaml is configured to log to /logs
+NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin compose_runner
+
+# After test, copy files from containers:
+docker ps --filter "name=nomos-compose-"
+docker cp <container-id>:/logs/node* /tmp/
+volumes:
+ - /tmp/host-logs:/logs # Add to docker-compose.yml.tera
+Recommendation: Use docker logs by default. File collection inside containers is complex and rarely needed.
Keep containers for debugging:
+COMPOSE_RUNNER_PRESERVE=1 \
+NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
+cargo run -p runner-examples --bin compose_runner
+# Containers remain running after test—inspect with docker logs or docker exec
+Compose debugging variables:
+COMPOSE_RUNNER_HOST=127.0.0.1 — host used for readiness probesCOMPOSE_RUNNER_HOST_GATEWAY=host.docker.internal:host-gateway — controls extra_hosts entry (set to disable to omit)TESTNET_RUNNER_PRESERVE=1 — alias for COMPOSE_RUNNER_PRESERVE=1COMPOSE_RUNNER_HTTP_TIMEOUT_SECS=<secs> — override HTTP readiness timeoutNote: Container names follow pattern nomos-compose-{uuid}-validator-{index}-1 where {uuid} changes per run.
Via kubectl logs (use label selectors):
+# List pods
+kubectl get pods
+
+# Stream logs using label selectors (recommended)
+# Helm chart labels:
+# - nomos/logical-role=validator|executor
+# - nomos/validator-index / nomos/executor-index
+kubectl logs -l nomos/logical-role=validator -f
+kubectl logs -l nomos/logical-role=executor -f
+
+# Stream logs from specific pod
+kubectl logs -f nomos-validator-0
+
+# Previous logs from crashed pods
+kubectl logs --previous -l nomos/logical-role=validator
+Download logs for offline analysis:
+# Using label selectors
+kubectl logs -l nomos/logical-role=validator --tail=1000 > all-validators.log
+kubectl logs -l nomos/logical-role=executor --tail=1000 > all-executors.log
+
+# Specific pods
+kubectl logs nomos-validator-0 > validator-0.log
+kubectl logs nomos-executor-1 > executor-1.log
+K8s debugging variables:
+K8S_RUNNER_DEBUG=1 — logs Helm stdout/stderr for install commandsK8S_RUNNER_PRESERVE=1 — keep namespace/release after runK8S_RUNNER_NODE_HOST=<ip|hostname> — override NodePort host resolutionK8S_RUNNER_NAMESPACE=<name> / K8S_RUNNER_RELEASE=<name> — pin namespace/release (useful for debugging)Specify namespace (if not using default):
+kubectl logs -n my-namespace -l nomos/logical-role=validator -f
+Note: K8s runner is optimized for local clusters (Docker Desktop K8s, minikube, kind). Remote clusters require additional setup.
+OTLP exporters are optional. If you see errors about unreachable OTLP endpoints, it's safe to ignore them unless you're actively collecting traces/metrics.
+To enable OTLP:
+NOMOS_OTLP_ENDPOINT=http://localhost:4317 \
+NOMOS_OTLP_METRICS_ENDPOINT=http://localhost:4318 \
+cargo run -p runner-examples --bin local_runner
+To silence OTLP errors: Simply leave these variables unset (the default).
+Runners expose metrics and node HTTP endpoints for expectation code and debugging.
+scripts/setup/setup-observability.sh:
+scripts/setup/setup-observability.sh compose up then scripts/setup/setup-observability.sh compose envscripts/setup/setup-observability.sh k8s install then scripts/setup/setup-observability.sh k8s envNOMOS_METRICS_QUERY_URL (PromQL base URL) to enable ctx.telemetry() queriesctx.telemetry().prometheus().map(|p| p.base_url())Example:
+# Start observability stack (Compose)
+scripts/setup/setup-observability.sh compose up
+
+# Get environment variables
+eval $(scripts/setup/setup-observability.sh compose env)
+
+# Run scenario with metrics
+POL_PROOF_DEV_MODE=true \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+scripts/setup/setup-observability.sh can)NOMOS_GRAFANA_URL, the deployer prints it in TESTNET_ENDPOINTStesting-framework/assets/stack/monitoring/grafana/dashboards/ (the bundled stack auto-provisions them)Example:
+# Bring up the bundled Prometheus+Grafana stack (optional)
+scripts/setup/setup-observability.sh compose up
+eval $(scripts/setup/setup-observability.sh compose env)
+
+export NOMOS_GRAFANA_URL=http://localhost:3000
+POL_PROOF_DEV_MODE=true scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+Default bundled Grafana login: admin / admin (see scripts/observability/compose/docker-compose.yml).
ctx.node_clients().validator_clients().get(0)testing-framework/core/src/nodes/api_client.rs for available methodsExample usage in expectations:
+use testing_framework_core::scenario::{DynError, RunContext};
+
+async fn evaluate(ctx: &RunContext) -> Result<(), DynError> {
+ let client = &ctx.node_clients().validator_clients()[0];
+
+ let info = client.consensus_info().await?;
+ tracing::info!(height = info.height, "consensus info from validator 0");
+
+ Ok(())
+}flowchart TD
+ Expose[Runner exposes endpoints/ports] --> Collect[Runtime collects block/health signals]
+ Collect --> Consume[Expectations consume signals<br/>decide pass/fail]
+ Consume --> Inspect[Operators inspect logs/metrics<br/>when failures arise]
+NOMOS_LOG_DIR=/tmp/logs \
+NOMOS_LOG_LEVEL=debug \
+NOMOS_LOG_FILTER="cryptarchia=trace" \
+POL_PROOF_DEV_MODE=true \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
+# Start observability stack
+scripts/setup/setup-observability.sh compose up
+eval $(scripts/setup/setup-observability.sh compose env)
+
+# Run with metrics
+POL_PROOF_DEV_MODE=true \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+
+# Access Grafana at http://localhost:3000
+K8S_RUNNER_NAMESPACE=nomos-debug \
+K8S_RUNNER_DEBUG=1 \
+K8S_RUNNER_PRESERVE=1 \
+POL_PROOF_DEV_MODE=true \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
+
+# Inspect logs
+kubectl logs -n nomos-debug -l nomos/logical-role=validator
+Quick reference materials, troubleshooting guides, and supplementary information.
+This section complements the main documentation with practical reference materials that you'll return to frequently during development and operations.
+Jump to:
+ +Quick reference for the scenario builder DSL. All methods are chainable.
+use std::time::Duration;
+
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_compose::ComposeDeployer;
+use testing_framework_runner_k8s::K8sDeployer;
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt};use testing_framework_core::scenario::{Builder, ScenarioBuilder};
+
+pub fn topology() -> Builder<()> {
+ ScenarioBuilder::topology_with(|t| {
+ t.network_star() // Star topology (all connect to seed node)
+ .validators(3) // Number of validator nodes
+ .executors(2) // Number of executor nodes
+ })
+}use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn wallets_plan() -> testing_framework_core::scenario::Scenario<()> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0))
+ .wallets(50) // Seed 50 funded wallet accounts
+ .build()
+}use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn transactions_plan() -> testing_framework_core::scenario::Scenario<()> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0))
+ .wallets(50)
+ .transactions_with(|txs| {
+ txs.rate(5) // 5 transactions per block
+ .users(20) // Use 20 of the seeded wallets
+ }) // Finish transaction workload config
+ .build()
+}use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn da_plan() -> testing_framework_core::scenario::Scenario<()> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(1))
+ .wallets(50)
+ .da_with(|da| {
+ da.channel_rate(1) // number of DA channels to run
+ .blob_rate(2) // target 2 blobs per block (headroom applied)
+ .headroom_percent(20) // optional headroom when sizing channels
+ }) // Finish DA workload config
+ .build()
+}enable_node_control())use std::time::Duration;
+
+use testing_framework_core::scenario::{NodeControlCapability, ScenarioBuilder};
+use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt};
+
+pub fn chaos_plan() -> testing_framework_core::scenario::Scenario<NodeControlCapability> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .enable_node_control() // Enable node control capability
+ .chaos_with(|c| {
+ c.restart() // Random restart chaos
+ .min_delay(Duration::from_secs(30)) // Min time between restarts
+ .max_delay(Duration::from_secs(60)) // Max time between restarts
+ .target_cooldown(Duration::from_secs(45)) // Cooldown after restart
+ .apply() // Required for chaos configuration
+ })
+ .build()
+}use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn expectations_plan() -> testing_framework_core::scenario::Scenario<()> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0))
+ .expect_consensus_liveness() // Assert blocks are produced continuously
+ .build()
+}use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn run_duration_plan() -> testing_framework_core::scenario::Scenario<()> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0))
+ .with_run_duration(Duration::from_secs(120)) // Run for 120 seconds
+ .build()
+}use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn build_plan() -> testing_framework_core::scenario::Scenario<()> {
+ ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0)).build() // Construct the final Scenario
+}use testing_framework_runner_compose::ComposeDeployer;
+use testing_framework_runner_k8s::K8sDeployer;
+use testing_framework_runner_local::LocalDeployer;
+
+pub fn deployers() {
+ // Local processes
+ let _deployer = LocalDeployer::default();
+
+ // Docker Compose
+ let _deployer = ComposeDeployer::default();
+
+ // Kubernetes
+ let _deployer = K8sDeployer::default();
+}use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub async fn execution() -> Result<()> {
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(0))
+ .expect_consensus_liveness()
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub async fn run_test() -> Result<()> {
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(2))
+ .wallets(50)
+ .transactions_with(|txs| {
+ txs.rate(5) // 5 transactions per block
+ .users(20)
+ })
+ .da_with(|da| {
+ da.channel_rate(1) // number of DA channels
+ .blob_rate(2) // target 2 blobs per block
+ .headroom_percent(20) // optional channel headroom
+ })
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(90))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}Prerequisites for All Runners:
+versions.env file at repository root (required by helper scripts)POL_PROOF_DEV_MODE=true MUST be set for all runners (host, compose, k8s) to avoid expensive Groth16 proof generation that causes timeoutstesting-framework/assets/stack/kzgrs_test_params/kzgrs_test_params (note the repeated filename) for DA workloadsPlatform/Environment Notes:
+NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64 for local compose/k8s runs to avoid slow/fragile amd64 emulation builds..tmp/, target/, and Docker build cache) before retrying.hostPath and uses a local image tag (logos-blockchain-testing:local). This is intended for local clusters (Docker Desktop / minikube / kind), not remote managed clusters without additional setup.
+scripts/ops/clean.sh (and scripts/ops/clean.sh --docker if needed).scripts/ops/clean.sh --docker-system --dangerous (add --volumes if you also want to prune Docker volumes).Recommended: Use scripts/run/run-examples.sh which handles all setup automatically.
Common symptoms and likely causes:
+POL_PROOF_DEV_MODE=true, missing KZG circuit assets (/kzgrs_test_params/kzgrs_test_params file) for DA workloads, too-short run window, port conflicts, or resource exhaustion—set required env vars, verify assets exist, extend duration, check node logs for startup errors..wallets(N) vs .users(M)), transaction rate exceeding block capacity, or rates exceeding block production speed—reduce rate, increase wallet count, verify wallet setup in logs.This section shows what you'll actually see when common issues occur. Each example includes realistic console output and the fix.
+POL_PROOF_DEV_MODE=true (Most Common!)Symptoms:
+What you'll see:
+$ cargo run -p runner-examples --bin local_runner
+ Finished dev [unoptimized + debuginfo] target(s) in 0.48s
+ Running `target/debug/local_runner`
+[INFO runner_examples::local_runner] Starting local runner scenario
+[INFO testing_framework_runner_local] Launching 3 validators
+[INFO testing_framework_runner_local] Waiting for node readiness...
+(hangs here for 5+ minutes, CPU at 100%)
+thread 'main' panicked at 'readiness timeout expired'
+Root Cause: Groth16 proof generation is extremely slow without dev mode. The system tries to compute real cryptographic proofs, which can take minutes per block.
+Fix:
+POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+Prevention: Set this in your shell profile or .env file so you never forget it.
versions.env FileSymptoms:
+What you'll see:
+$ scripts/run/run-examples.sh -t 60 -v 1 -e 1 host
+ERROR: versions.env not found at repository root
+This file is required and should define:
+ VERSION=<circuit release tag>
+ NOMOS_NODE_REV=<nomos-node git revision>
+ NOMOS_BUNDLE_VERSION=<bundle schema version>
+Root Cause: Helper scripts need versions.env to know which versions to build/fetch.
Fix: Ensure you're in the repository root directory. The versions.env file should already exist—verify it's present:
cat versions.env
+# Should show:
+# VERSION=v0.3.1
+# NOMOS_NODE_REV=abc123def456
+# NOMOS_BUNDLE_VERSION=v1
+Symptoms:
+What you'll see:
+$ POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+[INFO testing_framework_runner_local] Starting DA workload
+[ERROR nomos_da_dispersal] Failed to load KZG parameters
+Error: Custom { kind: NotFound, error: "Circuit file not found at: testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params" }
+thread 'main' panicked at 'workload init failed'
+Root Cause: DA (Data Availability) workloads require KZG cryptographic parameters. The file must exist at: testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params (note the repeated filename).
Fix (recommended):
+# Use run-examples.sh which handles setup automatically
+scripts/run/run-examples.sh -t 60 -v 1 -e 1 host
+Fix (manual):
+# Fetch circuits
+scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits
+
+# Copy to expected location
+mkdir -p testing-framework/assets/stack/kzgrs_test_params
+cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/
+
+# Verify (should be ~120MB)
+ls -lh testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params
+Symptoms:
+nomos-node or nomos-executor binaryNOMOS_NODE_BIN / NOMOS_EXECUTOR_BIN not setWhat you'll see:
+$ POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+[INFO testing_framework_runner_local] Spawning validator 0
+Error: Os { code: 2, kind: NotFound, message: "No such file or directory" }
+thread 'main' panicked at 'failed to spawn nomos-node process'
+Root Cause: The local runner needs compiled nomos-node and nomos-executor binaries, but doesn't know where they are.
Fix (recommended):
+# Use run-examples.sh which builds binaries automatically
+scripts/run/run-examples.sh -t 60 -v 1 -e 1 host
+Fix (manual - set paths explicitly):
+# Build binaries first
+cd ../nomos-node # or wherever your nomos-node checkout is
+cargo build --release --bin nomos-node --bin nomos-executor
+
+# Set environment variables
+export NOMOS_NODE_BIN=$PWD/target/release/nomos-node
+export NOMOS_EXECUTOR_BIN=$PWD/target/release/nomos-executor
+
+# Return to testing framework
+cd ../nomos-testing
+POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+Symptoms:
+What you'll see:
+$ scripts/run/run-examples.sh -t 60 -v 1 -e 1 compose
+[INFO runner_examples::compose_runner] Starting compose deployment
+Error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
+thread 'main' panicked at 'compose deployment failed'
+Root Cause: Docker Desktop isn't running, or your user doesn't have permission to access Docker.
+Fix:
+# macOS: Start Docker Desktop application
+open -a Docker
+
+# Linux: Start Docker daemon
+sudo systemctl start docker
+
+# Verify Docker is working
+docker ps
+
+# If permission denied, add your user to docker group (Linux)
+sudo usermod -aG docker $USER
+# Then log out and log back in
+Symptoms:
+What you'll see:
+$ POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin compose_runner
+[INFO testing_framework_runner_compose] Starting compose deployment
+Error: Failed to pull image 'logos-blockchain-testing:local': No such image
+thread 'main' panicked at 'compose deployment failed'
+Root Cause: The Docker image hasn't been built yet, or was pruned.
+Fix (recommended):
+# Use run-examples.sh which builds the image automatically
+scripts/run/run-examples.sh -t 60 -v 1 -e 1 compose
+Fix (manual):
+# 1. Build Linux bundle
+scripts/build/build-bundle.sh --platform linux
+
+# 2. Set bundle path
+export NOMOS_BINARIES_TAR=$(ls -t .tmp/nomos-binaries-linux-*.tar.gz | head -1)
+
+# 3. Build Docker image
+scripts/build/build_test_image.sh
+
+# 4. Verify image exists
+docker images | grep logos-blockchain-testing
+
+# 5. For kind/minikube: load image into cluster
+kind load docker-image logos-blockchain-testing:local
+# OR: minikube image load logos-blockchain-testing:local
+Symptoms:
+What you'll see:
+$ POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+[INFO testing_framework_runner_local] Launching validator 0 on port 18080
+Error: Os { code: 48, kind: AddrInUse, message: "Address already in use" }
+thread 'main' panicked at 'failed to bind port 18080'
+Root Cause: Previous test didn't clean up properly, or another service is using the port.
+Fix:
+# Find processes using the port
+lsof -i :18080 # macOS/Linux
+netstat -ano | findstr :18080 # Windows
+
+# Kill orphaned nomos processes
+pkill nomos-node
+pkill nomos-executor
+
+# For compose: ensure containers are stopped
+docker compose down
+docker ps -a --filter "name=nomos-compose-" -q | xargs docker rm -f
+
+# Check if port is now free
+lsof -i :18080 # Should return nothing
+For Observability Stack Port Conflicts:
+# Edit ports in observability compose file
+vim scripts/observability/compose/docker-compose.yml
+
+# Change conflicting port mappings:
+# ports:
+# - "9090:9090" # Prometheus - change to "19090:9090" if needed
+# - "3000:3000" # Grafana - change to "13000:3000" if needed
+Symptoms:
+What you'll see:
+$ POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+[INFO testing_framework_workflows] Starting transaction workload with 10 users
+[ERROR testing_framework_workflows] Wallet seeding failed: requested 10 users but only 3 wallets available
+thread 'main' panicked at 'workload init failed: insufficient wallets'
+Root Cause: Topology configured fewer wallets than the workload needs. Transaction workload has .users(M) but topology only has .wallets(N) where N < M.
Fix:
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+let scenario = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
+ .wallets(20) // ← Increase wallet count
+ .transactions_with(|tx| {
+ tx.users(10) // ← Must be ≤ wallets(20)
+ .rate(5)
+ })
+ .build();Symptoms:
+What you'll see:
+$ docker ps --filter "name=nomos-compose-"
+CONTAINER ID STATUS
+abc123def456 Restarting (137) 30 seconds ago # 137 = OOM killed
+
+$ docker logs abc123def456
+[INFO nomos_node] Starting validator
+[INFO consensus] Processing block
+Killed # ← OOM killer terminated the process
+Root Cause: Too many nodes, too much workload traffic, or insufficient Docker resources.
+Fix:
+# 1. Reduce topology size
+# In your scenario:
+# .topology(Topology::preset_3v1e()) # Instead of preset_10v2e()
+
+# 2. Reduce workload rates
+# .workload(TransactionWorkload::new().rate(5.0)) # Instead of rate(100.0)
+
+# 3. Increase Docker resources (Docker Desktop)
+# Settings → Resources → Memory: 8GB minimum (12GB+ recommended for large topologies)
+# Settings → Resources → CPUs: 4+ cores recommended
+
+# 4. Increase file descriptor limits (Linux/macOS)
+ulimit -n 4096
+
+# 5. Close other heavy applications (browsers, IDEs, etc.)
+Symptoms:
+What you'll see:
+$ POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+[INFO runner_examples] Test complete, cleaning up
+[INFO testing_framework_runner_local] Removing temporary directories
+$ ls .tmp/
+# Empty or missing
+Root Cause: Framework cleans up temporary directories by default to avoid disk bloat.
+Fix:
+# Persist logs to a specific directory
+NOMOS_LOG_DIR=/tmp/test-logs \
+NOMOS_TESTS_KEEP_LOGS=1 \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin local_runner
+
+# Logs persist after run
+ls /tmp/test-logs/
+# nomos-node-0.2024-12-18T14-30-00.log
+# nomos-node-1.2024-12-18T14-30-00.log
+# ...
+Symptoms:
+What you'll see:
+$ POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+[INFO testing_framework_core] Starting workloads
+[INFO testing_framework_core] Run window: 10 seconds
+[INFO testing_framework_core] Evaluating expectations
+[ERROR testing_framework_core] Consensus liveness expectation failed: expected min 5 blocks, got 1
+thread 'main' panicked at 'expectations failed'
+Root Cause: Run duration too short for consensus parameters. If CONSENSUS_SLOT_TIME=20s but run duration is only 10s, you can't produce many blocks.
Fix:
+use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+// Increase run duration to allow more blocks.
+let scenario = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(120)) // ← Give more time
+ .build();Or adjust consensus timing (if you control node config):
+# Faster block production (shorter slot time)
+CONSENSUS_SLOT_TIME=5 \
+CONSENSUS_ACTIVE_SLOT_COEFF=0.9 \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin local_runner
+When a test fails, check these in order:
+POL_PROOF_DEV_MODE=true is set (REQUIRED for all runners)versions.env exists at repo roottesting-framework/assets/stack/kzgrs_test_params/kzgrs_test_paramsNOMOS_NODE_BIN / NOMOS_EXECUTOR_BIN set, or using run-examples.sh)logos-blockchain-testing:local exists for compose/k8s)lsof -i :18080, kill orphaned processes).wallets(N) ≥ .users(M))NOMOS_LOG_DIR + NOMOS_TESTS_KEEP_LOGS=1 if needed)Still stuck? Check node logs (see Where to Find Logs) for the actual error.
+| Runner | Default Output | With NOMOS_LOG_DIR + Flags | Access Command |
|---|---|---|---|
| Host (local) | Per-run temporary directories under the current working directory (removed unless NOMOS_TESTS_KEEP_LOGS=1) | Per-node files with prefix nomos-node-{index} (set NOMOS_LOG_DIR) | cat $NOMOS_LOG_DIR/nomos-node-0* |
| Compose | Docker container stdout/stderr | Set tracing_settings.logger: !File in testing-framework/assets/stack/cfgsync.yaml (and mount a writable directory) | docker ps then docker logs <container-id> |
| K8s | Pod stdout/stderr | Set tracing_settings.logger: !File in testing-framework/assets/stack/cfgsync.yaml (and mount a writable directory) | kubectl logs -l nomos/logical-role=validator |
Important Notes:
+NOMOS_TESTS_KEEP_LOGS=1. To write per-node log files to a stable location, set NOMOS_LOG_DIR=/path/to/logs.testing-framework/assets/stack/cfgsync.yaml (tracing_settings.logger). By default, rely on docker logs or kubectl logs.nomos-node-{index}* or nomos-executor-{index}* with timestamps, e.g., nomos-node-0.2024-12-01T10-30-45.log (NOT just .log suffix).nomos-compose-<uuid>-validator-0-1 where <uuid> is randomly generated per runConsole output (default):
+POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner 2>&1 | tee test.log
+Persistent file output:
+NOMOS_LOG_DIR=/tmp/debug-logs \
+NOMOS_LOG_LEVEL=debug \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin local_runner
+
+# Inspect logs (note: filenames include timestamps):
+ls /tmp/debug-logs/
+# Example: nomos-node-0.2024-12-01T10-30-45.log
+tail -f /tmp/debug-logs/nomos-node-0* # Use wildcard to match timestamp
+Stream live logs:
+# List running containers (note the UUID prefix in names)
+docker ps --filter "name=nomos-compose-"
+
+# Find your container ID or name from the list, then:
+docker logs -f <container-id>
+
+# Or filter by name pattern:
+docker logs -f $(docker ps --filter "name=nomos-compose-.*-validator-0" -q | head -1)
+
+# Show last 100 lines
+docker logs --tail 100 <container-id>
+Keep containers for post-mortem debugging:
+COMPOSE_RUNNER_PRESERVE=1 \
+NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin compose_runner
+
+# OR: Use run-examples.sh (handles setup automatically)
+COMPOSE_RUNNER_PRESERVE=1 scripts/run/run-examples.sh -t 60 -v 1 -e 1 compose
+
+# After test failure, containers remain running:
+docker ps --filter "name=nomos-compose-"
+docker exec -it <container-id> /bin/sh
+docker logs <container-id> > debug.log
+Note: Container names follow the pattern nomos-compose-{uuid}-validator-{index}-1 or nomos-compose-{uuid}-executor-{index}-1, where {uuid} is randomly generated per run.
Important: Always verify your namespace and use label selectors instead of assuming pod names.
+Stream pod logs (use label selectors):
+# Check your namespace first
+kubectl config view --minify | grep namespace
+
+# All validator pods (add -n <namespace> if not using default)
+kubectl logs -l nomos/logical-role=validator -f
+
+# All executor pods
+kubectl logs -l nomos/logical-role=executor -f
+
+# Specific pod by name (find exact name first)
+kubectl get pods -l nomos/logical-role=validator # Find the exact pod name
+kubectl logs -f <actual-pod-name> # Then use it
+
+# With explicit namespace
+kubectl logs -n my-namespace -l nomos/logical-role=validator -f
+Download logs from crashed pods:
+# Previous logs from crashed pod
+kubectl get pods -l nomos/logical-role=validator # Find crashed pod name first
+kubectl logs --previous <actual-pod-name> > crashed-validator.log
+
+# Or use label selector for all crashed validators
+for pod in $(kubectl get pods -l nomos/logical-role=validator -o name); do
+ kubectl logs --previous $pod > $(basename $pod)-previous.log 2>&1
+done
+Access logs from all pods:
+# All pods in current namespace
+for pod in $(kubectl get pods -o name); do
+ echo "=== $pod ==="
+ kubectl logs $pod
+done > all-logs.txt
+
+# Or use label selectors (recommended)
+kubectl logs -l nomos/logical-role=validator --tail=500 > validators.log
+kubectl logs -l nomos/logical-role=executor --tail=500 > executors.log
+
+# With explicit namespace
+kubectl logs -n my-namespace -l nomos/logical-role=validator --tail=500 > validators.log
+When a test fails, follow this sequence:
+Start with the test harness output—did expectations fail? Was there a deployment error?
+Look for:
+Ensure all nodes started successfully and became ready before workloads began.
+Commands:
+# Local: check process list
+ps aux | grep nomos
+
+# Compose: check container status (note UUID in names)
+docker ps -a --filter "name=nomos-compose-"
+
+# K8s: check pod status (use label selectors, add -n <namespace> if needed)
+kubectl get pods -l nomos/logical-role=validator
+kubectl get pods -l nomos/logical-role=executor
+kubectl describe pod <actual-pod-name> # Get name from above first
+Focus on the first node that exhibited problems or the node with the highest index (often the last to start).
+Common error patterns:
+versions.env file at repository rootPOL_PROOF_DEV_MODE=true (REQUIRED for all runners)testing-framework/assets/stack/kzgrs_test_params/.wallets(N) or reduce .users(M))If logs are too sparse, increase verbosity:
+NOMOS_LOG_LEVEL=debug \
+NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug" \
+cargo run -p runner-examples --bin local_runner
+If metric updates are polluting your logs (fields like counter.* / gauge.*), move those events to a dedicated tracing target (e.g. target: "nomos_metrics") and set NOMOS_LOG_FILTER="nomos_metrics=off,..." so they don’t get formatted into log output.
If expectations report observability issues:
+Prometheus (Compose):
+curl http://localhost:9090/-/healthy
+Node HTTP APIs:
+curl http://localhost:18080/consensus/info # Adjust port per node
+Run a minimal baseline test (e.g., 2 validators, consensus liveness only). If it passes, the issue is in your workload or topology configuration.
+POL_PROOF_DEV_MODE=true (causes slow proof generation), or missing KZG
+assets for DA workloads.POL_PROOF_DEV_MODE=true is set (REQUIRED for all runners).testing-framework/assets/stack/kzgrs_test_params/ (for DA workloads).with_run_duration() to allow more blocks..wallets(N) count or reduce .users(M) in the transaction
+workload (ensure N ≥ M).enable_node_control() wasn't called..enable_node_control() is called in the scenario before .chaos().POL_PROOF_DEV_MODE=true is set (REQUIRED for all runners—without
+it, proof generation is too slow).run-examples.sh, build-bundle.sh, setup-circuits-stack.sh) require versions.env file at repository root.versions.env file should already exist and contains: VERSION=<circuit release tag>
+ NOMOS_NODE_REV=<nomos-node git revision>
+ NOMOS_BUNDLE_VERSION=<bundle schema version>
+Use the checked-in versions.env at the repository root as the source of truth.
pkill nomos-node), wait for Docker cleanup
+(docker compose down), or restart Docker.scripts/run/run-examples.sh -t 60 -v 1 -e 1 compose
+scripts/build/build-bundle.sh --platform linuxexport NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gzscripts/build/build_test_image.shkind load docker-image logos-blockchain-testing:local, or minikube image load ...), or push to a registry and set NOMOS_TESTNET_IMAGE accordingly.testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params (note repeated filename) must exist. Inside containers, it's at /kzgrs_test_params/kzgrs_test_params.scripts/run/run-examples.sh -t 60 -v 1 -e 1 <mode>
+scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuitscp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/ls -lh testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_paramsFor detailed logging configuration and observability setup, see Logging & Observability.
+Why block-oriented timing?
+Slots advance at a fixed rate (NTP-synchronized, 2s by default), so reasoning
+about blocks and consensus intervals keeps assertions aligned with protocol
+behavior rather than arbitrary wall-clock durations.
Can I reuse the same scenario across runners?
+Yes. The plan stays the same; swap runners (local, compose, k8s) to target
+different environments.
When should I enable chaos workloads?
+Only when testing resilience or operational recovery; keep functional smoke
+tests deterministic.
How long should runs be?
+The framework enforces a minimum of 2× slot duration (4 seconds with default 2s slots), but practical recommendations:
Very short runs (< 30s) risk false confidence—one or two lucky blocks don't prove liveness.
+Do I always need seeded wallets?
+Only for transaction scenarios. Data-availability or pure chaos scenarios may
+not require them, but liveness checks still need validators producing blocks.
What if expectations fail but workloads “look fine”?
+Trust expectations first—they capture the intended success criteria. Use the
+observability signals and runner logs to pinpoint why the system missed the
+target.
with_run_duration(). Framework auto-extends to at least 2× slot duration.Declarative, multi-node blockchain testing for the Logos network
+The Logos Testing Framework enables you to test consensus, data availability, and transaction workloads across local processes, Docker Compose, and Kubernetes deployments—all with a unified scenario API.
+ +Everything in this framework is a Scenario.
+A Scenario is a controlled experiment over time, composed of:
+This single abstraction makes tests declarative, portable, and composable.
+flowchart LR
+ Build[Define Scenario] --> Deploy[Deploy Topology]
+ Deploy --> Execute[Run Workloads]
+ Execute --> Evaluate[Check Expectations]
+
+ style Build fill:#e1f5ff
+ style Deploy fill:#fff4e1
+ style Execute fill:#ffe1f5
+ style Evaluate fill:#e1ffe1
+Declarative API
+Multiple Deployment Modes
+Built-in Workloads
+Comprehensive Observability
+use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_core::scenario::Deployer as _;
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+#[tokio::main]
+async fn main() -> anyhow::Result<()> {
+ let mut scenario = ScenarioBuilder::topology_with(|t| {
+ t.network_star()
+ .validators(3)
+ .executors(1)
+ })
+ .transactions_with(|tx| tx.rate(10).users(5))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&scenario).await?;
+ runner.run(&mut scenario).await?;
+
+ Ok(())
+}Start with the Quickstart Guide for a hands-on introduction that gets you running tests in minutes.
+Explore the User Guide to learn about authoring scenarios, workloads, expectations, and deployment strategies.
+Jump to Operations & Deployment for prerequisites, environment configuration, and continuous integration patterns.
+Check the Developer Reference to implement custom workloads, expectations, and runners.
+Logos is a modular blockchain protocol composed of validators, executors, and a data-availability (DA) subsystem:
+These roles interact tightly, which is why meaningful testing must be performed in multi-node environments that include real networking, timing, and DA interaction.
+The Logos Testing Framework provides the infrastructure to orchestrate these multi-node scenarios reliably across development, CI, and production-like environments.
+Learn more about the protocol: Logos Project Documentation
+| Section | Description |
|---|---|
| Foundations | Architecture, philosophy, and design principles |
| User Guide | Writing and running scenarios, workloads, and expectations |
| Developer Reference | Extending the framework with custom components |
| Operations & Deployment | Setup, CI integration, and environment configuration |
| Appendix | Quick reference, troubleshooting, FAQ, and glossary |
Ready to start? Head to the Quickstart
+ +Get a working example running quickly.
+NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64 to avoid slow/fragile amd64 emulation buildsversions.env file at repository root (defines VERSION, NOMOS_NODE_REV, NOMOS_BUNDLE_VERSION)Note: nomos-node binaries are built automatically on demand or can be provided via prebuilt bundles.
Important: The versions.env file is required by helper scripts. If missing, the scripts will fail with an error. The file should already exist in the repository root.
The framework ships with runnable example binaries in examples/src/bin/.
Recommended: Use the convenience script:
+# From the logos-blockchain-testing directory
+scripts/run/run-examples.sh -t 60 -v 1 -e 1 host
+This handles circuit setup, binary building, and runs a complete scenario: 1 validator + 1 executor, mixed transaction + DA workload (5 tx/block + 1 channel + 1 blob), 60s duration.
+Alternative: Direct cargo run (requires manual setup):
+# Requires circuits in place and NOMOS_NODE_BIN/NOMOS_EXECUTOR_BIN set
+POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
+Core API Pattern (simplified example):
+use std::time::Duration;
+
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub async fn run_local_demo() -> Result<()> {
+ // Define the scenario (1 validator + 1 executor, tx + DA workload)
+ let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().validators(1).executors(1))
+ .wallets(1_000)
+ .transactions_with(|txs| {
+ txs.rate(5) // 5 transactions per block
+ .users(500) // use 500 of the seeded wallets
+ })
+ .da_with(|da| {
+ da.channel_rate(1) // 1 channel
+ .blob_rate(1) // target 1 blob per block
+ .headroom_percent(20) // default headroom when sizing channels
+ })
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build();
+
+ // Deploy and run
+ let deployer = LocalDeployer::default();
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}Note: The examples are binaries with #[tokio::main], not test functions. If you want to write integration tests, wrap this pattern in #[tokio::test] functions in your own test suite.
Important: POL_PROOF_DEV_MODE=true disables expensive Groth16 zero-knowledge proof generation for leader election. Without it, proof generation is CPU-intensive and tests will timeout. This is required for all runners (local, compose, k8s) for practical testing. Never use in production.
What you should see:
+NOMOS_TESTS_KEEP_LOGS=1)NOMOS_LOG_DIR=/path/to/logs (files will have prefix like nomos-node-0*, may include timestamps)Let's unpack the code:
+use testing_framework_core::scenario::ScenarioBuilder;
+
+pub fn step_1_topology() -> testing_framework_core::scenario::Builder<()> {
+ ScenarioBuilder::topology_with(|t| {
+ t.network_star() // Star topology: all nodes connect to seed
+ .validators(1) // 1 validator node
+ .executors(1) // 1 executor node (validator + DA dispersal)
+ })
+}This defines what your test network looks like.
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn step_2_wallets() -> testing_framework_core::scenario::Builder<()> {
+ ScenarioBuilder::with_node_counts(1, 1).wallets(1_000) // Seed 1,000 funded wallet accounts
+}Provides funded accounts for transaction submission.
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn step_3_workloads() -> testing_framework_core::scenario::Builder<()> {
+ ScenarioBuilder::with_node_counts(1, 1)
+ .wallets(1_000)
+ .transactions_with(|txs| {
+ txs.rate(5) // 5 transactions per block
+ .users(500) // Use 500 of the 1,000 wallets
+ })
+ .da_with(|da| {
+ da.channel_rate(1) // 1 DA channel (more spawned with headroom)
+ .blob_rate(1) // target 1 blob per block
+ .headroom_percent(20) // default headroom when sizing channels
+ })
+}Generates both transaction and DA traffic to stress both subsystems.
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+pub fn step_4_expectation() -> testing_framework_core::scenario::Builder<()> {
+ ScenarioBuilder::with_node_counts(1, 1).expect_consensus_liveness() // This says what success means: blocks must be produced continuously.
+}This says what success means: blocks must be produced continuously.
+use std::time::Duration;
+
+use testing_framework_core::scenario::ScenarioBuilder;
+
+pub fn step_5_run_duration() -> testing_framework_core::scenario::Builder<()> {
+ ScenarioBuilder::with_node_counts(1, 1).with_run_duration(Duration::from_secs(60))
+}Run for 60 seconds (~27 blocks with default 2s slots, 0.9 coefficient). Framework ensures this is at least 2× the consensus slot duration. Adjust consensus timing via CONSENSUS_SLOT_TIME and CONSENSUS_ACTIVE_SLOT_COEFF.
use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+
+pub async fn step_6_deploy_and_execute() -> Result<()> {
+ let mut plan = ScenarioBuilder::with_node_counts(1, 1).build();
+
+ let deployer = LocalDeployer::default(); // Use local process deployer
+ let runner = deployer.deploy(&plan).await?; // Provision infrastructure
+ let _handle = runner.run(&mut plan).await?; // Execute workloads & expectations
+
+ Ok(())
+}Deployer provisions the infrastructure. Runner orchestrates execution.
+With run-examples.sh (recommended):
+# Scale up to 3 validators + 2 executors, run for 2 minutes
+scripts/run/run-examples.sh -t 120 -v 3 -e 2 host
+With direct cargo run:
+# Uses NOMOS_DEMO_* env vars (or legacy *_DEMO_* vars)
+NOMOS_DEMO_VALIDATORS=3 \
+NOMOS_DEMO_EXECUTORS=2 \
+NOMOS_DEMO_RUN_SECS=120 \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin local_runner
+Use the same API with a different deployer for reproducible containerized environment.
+Recommended: Use the convenience script (handles everything):
+scripts/run/run-examples.sh -t 60 -v 1 -e 1 compose
+This automatically:
+testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params)NOMOS_BINARIES_TAR if available)Alternative: Direct cargo run with manual setup:
+# Option 1: Use prebuilt bundle (recommended for compose/k8s)
+scripts/build/build-bundle.sh --platform linux # Creates .tmp/nomos-binaries-linux-v0.3.1.tar.gz
+export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
+
+# Option 2: Manual circuit/image setup (rebuilds during image build)
+scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits
+cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/
+scripts/build/build_test_image.sh
+
+# Run with Compose
+NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin compose_runner
+Benefit: Reproducible containerized environment (Dockerized nodes, repeatable deployments).
+Optional: Prometheus + Grafana
+The runner can integrate with external observability endpoints. For a ready-to-run local stack:
+scripts/setup/setup-observability.sh compose up
+eval "$(scripts/setup/setup-observability.sh compose env)"
+Then run your compose scenario as usual (the environment variables enable PromQL querying and node OTLP metrics export).
+Note: Compose expects KZG parameters at /kzgrs_test_params/kzgrs_test_params inside containers (the directory name is repeated as the filename).
In code: Just swap the deployer:
+use anyhow::Result;
+use testing_framework_core::scenario::{Deployer, ScenarioBuilder};
+use testing_framework_runner_compose::ComposeDeployer;
+
+pub async fn run_with_compose_deployer() -> Result<()> {
+ // ... same scenario definition ...
+ let mut plan = ScenarioBuilder::with_node_counts(1, 1).build();
+
+ let deployer = ComposeDeployer::default(); // Use Docker Compose
+ let runner = deployer.deploy(&plan).await?;
+ let _handle = runner.run(&mut plan).await?;
+
+ Ok(())
+}Now that you have a working test:
+Runners turn a scenario plan into a live environment while keeping the plan +unchanged. Choose based on feedback speed, reproducibility, and fidelity. For +environment and operational considerations, see Operations Overview.
+Important: All runners require POL_PROOF_DEV_MODE=true to avoid expensive Groth16 proof generation that causes timeouts.
LocalDeployer).local_runner.rs, script mode: hostRun with: scripts/run/run-examples.sh -t 60 -v 1 -e 1 host
ComposeDeployer).compose_runner.rs, script mode: composeRun with: scripts/run/run-examples.sh -t 60 -v 1 -e 1 compose
K8sDeployer).k8s_runner.rs, script mode: k8sRun with: scripts/run/run-examples.sh -t 60 -v 1 -e 1 k8s
flowchart TB
+ subgraph Host["Host Runner (Local)"]
+ H1["Speed: Fast"]
+ H2["Isolation: Shared host"]
+ H3["Setup: Minimal"]
+ H4["Chaos: Not supported"]
+ H5["CI: Quick smoke tests"]
+ end
+
+ subgraph Compose["Compose Runner (Docker)"]
+ C1["Speed: Medium"]
+ C2["Isolation: Containerized"]
+ C3["Setup: Image build required"]
+ C4["Chaos: Supported"]
+ C5["CI: Recommended"]
+ end
+
+ subgraph K8s["K8s Runner (Cluster)"]
+ K1["Speed: Slower"]
+ K2["Isolation: Pod-level"]
+ K3["Setup: Cluster + image"]
+ K4["Chaos: Not yet supported"]
+ K5["CI: Large-scale tests"]
+ end
+
+ Decision{Choose Based On}
+ Decision -->|Fast iteration| Host
+ Decision -->|Reproducibility| Compose
+ Decision -->|Production-like| K8s
+
+ style Host fill:#e1f5ff
+ style Compose fill:#e1ffe1
+ style K8s fill:#ffe1f5
+| Feature | Host | Compose | K8s |
|---|---|---|---|
| Speed | Fastest | Medium | Slowest |
| Setup Time | < 1 min | 2-5 min | 5-10 min |
| Isolation | Process-level | Container | Pod + namespace |
| Node Control | No | Yes | Not yet |
| Observability | Basic | External stack | Cluster-wide |
| CI Integration | Smoke tests | Recommended | Heavy tests |
| Resource Usage | Low | Medium | High |
| Reproducibility | Environment-dependent | High | Highest |
| Network Fidelity | Localhost only | Virtual network | Real cluster |
| Parallel Runs | Port conflicts | Isolated | Namespace isolation |
flowchart TD
+ Start[Need to run tests?] --> Q1{Local development?}
+ Q1 -->|Yes| Q2{Testing chaos?}
+ Q1 -->|No| Q5{Have cluster access?}
+
+ Q2 -->|Yes| UseCompose[Use Compose]
+ Q2 -->|No| Q3{Need isolation?}
+
+ Q3 -->|Yes| UseCompose
+ Q3 -->|No| UseHost[Use Host]
+
+ Q5 -->|Yes| Q6{Large topology?}
+ Q5 -->|No| Q7{CI pipeline?}
+
+ Q6 -->|Yes| UseK8s[Use K8s]
+ Q6 -->|No| UseCompose
+
+ Q7 -->|Yes| Q8{Docker available?}
+ Q7 -->|No| UseHost
+
+ Q8 -->|Yes| UseCompose
+ Q8 -->|No| UseHost
+
+ style UseHost fill:#e1f5ff
+ style UseCompose fill:#e1ffe1
+ style UseK8s fill:#ffe1f5
+Use Host Runner when:
+Use Compose Runner when:
+Use K8s Runner when:
+The framework provides three runner modes: host (local processes), compose (Docker Compose), and k8s (Kubernetes).
+Use scripts/run/run-examples.sh for all modes—it handles all setup automatically:
# Host mode (local processes)
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
+
+# Compose mode (Docker Compose)
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+
+# K8s mode (Kubernetes)
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
+Parameters:
+-t 60 — Run duration in seconds-v 3 — Number of validators-e 1 — Number of executorshost|compose|k8s — Deployment modeThis script handles:
+Note: For k8s runs against non-local clusters (e.g. EKS), the cluster pulls images from a registry. In that case, build + push your image separately (see scripts/build/build_test_image.sh) and set NOMOS_TESTNET_IMAGE to the pushed reference.
For a small "does everything still run?" matrix across all runners:
+scripts/run/run-test-matrix.sh -t 120 -v 1 -e 1
+This runs host, compose, and k8s modes with various image-build configurations. Useful after making runner/image/script changes. Forwards --metrics-* options through to scripts/run/run-examples.sh.
Common options:
+--modes host,compose,k8s — Restrict which modes run--no-clean — Skip scripts/ops/clean.sh step--no-bundles — Skip scripts/build/build-bundle.sh (reuses existing .tmp tarballs)--no-image-build — Skip the “rebuild image” variants in the matrix (compose/k8s)--allow-nonzero-progress — Soft-pass expectation failures if logs show non-zero progress (local iteration only)--force-k8s-image-build — Allow the k8s image-build variant even on non-docker-desktop clustersEnvironment overrides:
+VERSION=v0.3.1 — Circuit versionNOMOS_NODE_REV=<commit> — nomos-node git revisionNOMOS_BINARIES_TAR=path/to/bundle.tar.gz — Use prebuilt bundleNOMOS_SKIP_IMAGE_BUILD=1 — Skip image rebuild inside run-examples.sh (compose/k8s)NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64|linux/amd64 — Docker platform for bundle builds (macOS/Windows)COMPOSE_CIRCUITS_PLATFORM=linux-aarch64|linux-x86_64 — Circuits platform for image buildsSLOW_TEST_ENV=true — Doubles built-in readiness timeouts (useful in CI / constrained laptops)TESTNET_PRINT_ENDPOINTS=1 — Print TESTNET_ENDPOINTS / TESTNET_PPROF lines during deployThe repo pins a nomos-node revision in versions.env for reproducible builds. To update it or point to a local checkout:
# Pin to a new git revision (updates versions.env + Cargo.toml git revs)
+scripts/ops/update-nomos-rev.sh --rev <git_sha>
+
+# Use a local nomos-node checkout instead (for development)
+scripts/ops/update-nomos-rev.sh --path /path/to/nomos-node
+
+# If Cargo.toml was marked skip-worktree, clear it
+scripts/ops/update-nomos-rev.sh --unskip-worktree
+Notes:
+NOMOS_NODE_PATH values; prefer --rev for shared history/CICargo.lock to update on the next cargo build/cargo testIf you hit Docker build failures, I/O errors, or disk space issues:
+scripts/ops/clean.sh
+For extra Docker cache cleanup:
+scripts/ops/clean.sh --docker
+For manual control, run the local_runner binary directly:
POL_PROOF_DEV_MODE=true \
+NOMOS_NODE_BIN=/path/to/nomos-node \
+NOMOS_EXECUTOR_BIN=/path/to/nomos-executor \
+cargo run -p runner-examples --bin local_runner
+| Variable | Default | Effect |
|---|---|---|
NOMOS_DEMO_VALIDATORS | 1 | Number of validators (legacy: LOCAL_DEMO_VALIDATORS) |
NOMOS_DEMO_EXECUTORS | 1 | Number of executors (legacy: LOCAL_DEMO_EXECUTORS) |
NOMOS_DEMO_RUN_SECS | 60 | Run duration in seconds (legacy: LOCAL_DEMO_RUN_SECS) |
NOMOS_NODE_BIN | — | Path to nomos-node binary (required) |
NOMOS_EXECUTOR_BIN | — | Path to nomos-executor binary (required) |
NOMOS_LOG_DIR | None | Directory for per-node log files |
NOMOS_TESTS_KEEP_LOGS | 0 | Keep per-run temporary directories (useful for debugging/CI) |
NOMOS_TESTS_TRACING | false | Enable debug tracing preset |
NOMOS_LOG_LEVEL | info | Global log level: error, warn, info, debug, trace |
NOMOS_LOG_FILTER | None | Fine-grained module filtering (e.g., cryptarchia=trace,nomos_da_sampling=debug) |
POL_PROOF_DEV_MODE | — | REQUIRED: Set to true for all runners |
Note: Requires circuit assets and host binaries. Use scripts/run/run-examples.sh host to handle setup automatically.
For manual control, run the compose_runner binary directly. Compose requires a Docker image with embedded assets.
# 1. Build a Linux bundle (includes binaries + circuits)
+scripts/build/build-bundle.sh --platform linux
+# Creates .tmp/nomos-binaries-linux-v0.3.1.tar.gz
+
+# 2. Build image (embeds bundle assets)
+export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
+scripts/build/build_test_image.sh
+
+# 3. Run
+NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin compose_runner
+# Fetch and copy circuits
+scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits
+cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/
+
+# Build image
+scripts/build/build_test_image.sh
+
+# Run
+NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
+POL_PROOF_DEV_MODE=true \
+cargo run -p runner-examples --bin compose_runner
+linux/arm64 engine by defaultNOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64 (recommended for local testing)NOMOS_BUNDLE_DOCKER_PLATFORM=linux/amd64 (slower via emulation)| Variable | Default | Effect |
|---|---|---|
NOMOS_TESTNET_IMAGE | — | Image tag (required, must match built image) |
POL_PROOF_DEV_MODE | — | REQUIRED: Set to true for all runners |
NOMOS_DEMO_VALIDATORS | 1 | Number of validators |
NOMOS_DEMO_EXECUTORS | 1 | Number of executors |
NOMOS_DEMO_RUN_SECS | 60 | Run duration in seconds |
COMPOSE_NODE_PAIRS | — | Alternative topology format: "validators×executors" (e.g., 3x2) |
NOMOS_METRICS_QUERY_URL | None | Prometheus-compatible base URL for runner to query |
NOMOS_METRICS_OTLP_INGEST_URL | None | Full OTLP HTTP ingest URL for node metrics export |
NOMOS_GRAFANA_URL | None | Grafana base URL for printing/logging |
COMPOSE_RUNNER_HOST | 127.0.0.1 | Host address for port mappings |
COMPOSE_RUNNER_PRESERVE | 0 | Keep containers running after test |
NOMOS_LOG_LEVEL | info | Node log level (stdout/stderr) |
NOMOS_LOG_FILTER | None | Fine-grained module filtering |
Config file option: testing-framework/assets/stack/cfgsync.yaml (tracing_settings.logger) — Switch node logs between stdout/stderr and file output
.enable_node_control() + chaos workloads)NOMOS_METRICS_* / NOMOS_GRAFANA_URL to enable telemetry links and querying
+scripts/setup/setup-observability.sh compose up then scripts/setup/setup-observability.sh compose envImportant:
+/kzgrs_test_params/kzgrs_test_params (note the repeated filename)scripts/run/run-examples.sh compose to handle all setup automaticallyFor manual control, run the k8s_runner binary directly. K8s requires the same image setup as Compose.
kubectl configured# 1. Build image with bundle (recommended)
+scripts/build/build-bundle.sh --platform linux
+export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
+scripts/build/build_test_image.sh
+
+# 2. Load into cluster (choose one)
+export NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local
+
+# For kind:
+kind load docker-image logos-blockchain-testing:local
+
+# For minikube:
+minikube image load logos-blockchain-testing:local
+
+# For remote cluster (push to registry):
+docker tag logos-blockchain-testing:local your-registry/logos-blockchain-testing:latest
+docker push your-registry/logos-blockchain-testing:latest
+export NOMOS_TESTNET_IMAGE=your-registry/logos-blockchain-testing:latest
+export NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local
+export POL_PROOF_DEV_MODE=true
+cargo run -p runner-examples --bin k8s_runner
+| Variable | Default | Effect |
|---|---|---|
NOMOS_TESTNET_IMAGE | — | Image tag (required) |
POL_PROOF_DEV_MODE | — | REQUIRED: Set to true for all runners |
NOMOS_DEMO_VALIDATORS | 1 | Number of validators |
NOMOS_DEMO_EXECUTORS | 1 | Number of executors |
NOMOS_DEMO_RUN_SECS | 60 | Run duration in seconds |
NOMOS_METRICS_QUERY_URL | None | Prometheus-compatible base URL for runner to query (PromQL) |
NOMOS_METRICS_OTLP_INGEST_URL | None | Full OTLP HTTP ingest URL for node metrics export |
NOMOS_GRAFANA_URL | None | Grafana base URL for printing/logging |
K8S_RUNNER_NAMESPACE | Random | Kubernetes namespace (pin for debugging) |
K8S_RUNNER_RELEASE | Random | Helm release name (pin for debugging) |
K8S_RUNNER_NODE_HOST | — | NodePort host resolution for non-local clusters |
K8S_RUNNER_DEBUG | 0 | Log Helm stdout/stderr for install commands |
K8S_RUNNER_PRESERVE | 0 | Keep namespace/release after run (for debugging) |
export NOMOS_METRICS_QUERY_URL=http://your-prometheus:9090
+# Prometheus OTLP receiver example:
+export NOMOS_METRICS_OTLP_INGEST_URL=http://your-prometheus:9090/api/v1/otlp/v1/metrics
+# Optional: print Grafana link in TESTNET_ENDPOINTS
+export NOMOS_GRAFANA_URL=http://your-grafana:3000
+cargo run -p runner-examples --bin k8s_runner
+Notes:
+NOMOS_METRICS_QUERY_URL must be reachable from the runner process (often via kubectl port-forward)NOMOS_METRICS_OTLP_INGEST_URL must be reachable from nodes (pods/containers) and is backend-specific
+scripts/setup/setup-observability.sh k8s install then scripts/setup/setup-observability.sh k8s envscripts/setup/setup-observability.sh k8s dashboardsscripts/run/run-examples.sh (Recommended)scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s \
+ --metrics-query-url http://your-prometheus:9090 \
+ --metrics-otlp-ingest-url http://your-prometheus:9090/api/v1/otlp/v1/metrics
+use testing_framework_core::scenario::ScenarioBuilder;
+use testing_framework_workflows::ObservabilityBuilderExt as _;
+
+let plan = ScenarioBuilder::with_node_counts(1, 1)
+ .with_metrics_query_url_str("http://your-prometheus:9090")
+ .with_metrics_otlp_ingest_url_str("http://your-prometheus:9090/api/v1/otlp/v1/metrics")
+ .build();testing-framework/assets/stack/kzgrs_test_params as a hostPath volume/kzgrs_test_params/kzgrs_test_params.enable_node_control()) will failscripts/run/run-examples.sh k8s to handle all setup automaticallyThis page focuses on how scenarios are executed (deploy → run → evaluate → cleanup), what artifacts you get back, and how that differs across runners.
+For “just run something that works” commands, see Running Examples.
+When you run a built scenario via a deployer, the run follows the same shape:
+flowchart TD
+ Build[Scenario built] --> Deploy[Deploy]
+ Deploy --> Capture[Capture]
+ Capture --> Execute[Execute]
+ Execute --> Evaluate[Evaluate]
+ Evaluate --> Cleanup[Cleanup]
+use std::time::Duration;
+
+use testing_framework_core::scenario::{Deployer as _, ScenarioBuilder};
+use testing_framework_runner_local::LocalDeployer;
+use testing_framework_workflows::ScenarioBuilderExt;
+
+async fn run_once() -> anyhow::Result<()> {
+ let mut scenario = ScenarioBuilder::topology_with(|t| t.network_star().validators(3).executors(1))
+ .wallets(20)
+ .transactions_with(|tx| tx.rate(1).users(5))
+ .expect_consensus_liveness()
+ .with_run_duration(Duration::from_secs(60))
+ .build()?;
+
+ let runner = LocalDeployer::default().deploy(&scenario).await?;
+ runner.run(&mut scenario).await?;
+
+ Ok(())
+}Notes:
+with_run_duration(...) is wall-clock time, not “number of blocks”..transactions_with(...) rates are per-block.scripts/run/run-examples.sh unless they are embedding the framework in their own test crate.NOMOS_TESTS_KEEP_LOGS=1 and/or NOMOS_LOG_DIR=...Run the built-in local examples:
+POL_PROOF_DEV_MODE=true \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 host
+docker compose logs (and any node-level log configuration you apply)COMPOSE_RUNNER_PRESERVE=1 to keep the environment up after a runRun the built-in compose examples:
+POL_PROOF_DEV_MODE=true \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose
+kubectl logs ...K8S_RUNNER_PRESERVE=1 and K8S_RUNNER_NAMESPACE=... to keep resources aroundRun the built-in k8s examples:
+POL_PROOF_DEV_MODE=true \
+scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
+NOMOS_LOG_DIR, NOMOS_LOG_LEVEL, NOMOS_LOG_FILTER (see Logging & Observability)RUST_LOG (runner process only)NOMOS_TESTS_KEEP_LOGS=1COMPOSE_RUNNER_PRESERVE=1K8S_RUNNER_PRESERVE=1++When should I read this? After writing 2-3 scenarios. This page documents patterns that emerge from real usage—come back when you're refactoring or standardizing your test suite.
+
Patterns that keep scenarios readable and reusable:
+These patterns make scenario definitions self-explanatory while staying aligned +with the framework’s block-oriented timing model.
+ +A scenario progresses through six distinct phases, each with a specific responsibility:
+flowchart TB
+ subgraph Phase1["1. Build Phase"]
+ Build[Define Scenario]
+ BuildDetails["• Declare topology<br/>• Attach workloads<br/>• Add expectations<br/>• Set run duration"]
+ Build --> BuildDetails
+ end
+
+ subgraph Phase2["2. Deploy Phase"]
+ Deploy[Provision Environment]
+ DeployDetails["• Launch nodes<br/>• Wait for readiness<br/>• Establish connectivity<br/>• Return Runner"]
+ Deploy --> DeployDetails
+ end
+
+ subgraph Phase3["3. Capture Phase"]
+ Capture[Baseline Metrics]
+ CaptureDetails["• Snapshot initial state<br/>• Start BlockFeed<br/>• Initialize expectations"]
+ Capture --> CaptureDetails
+ end
+
+ subgraph Phase4["4. Execution Phase"]
+ Execute[Drive Workloads]
+ ExecuteDetails["• Submit transactions<br/>• Disperse DA blobs<br/>• Trigger chaos events<br/>• Run for duration"]
+ Execute --> ExecuteDetails
+ end
+
+ subgraph Phase5["5. Evaluation Phase"]
+ Evaluate[Check Expectations]
+ EvaluateDetails["• Verify liveness<br/>• Check inclusion<br/>• Validate outcomes<br/>• Aggregate results"]
+ Evaluate --> EvaluateDetails
+ end
+
+ subgraph Phase6["6. Cleanup Phase"]
+ Cleanup[Teardown]
+ CleanupDetails["• Stop nodes<br/>• Remove containers<br/>• Collect logs<br/>• Release resources"]
+ Cleanup --> CleanupDetails
+ end
+
+ Phase1 --> Phase2
+ Phase2 --> Phase3
+ Phase3 --> Phase4
+ Phase4 --> Phase5
+ Phase5 --> Phase6
+
+ style Phase1 fill:#e1f5ff
+ style Phase2 fill:#fff4e1
+ style Phase3 fill:#f0ffe1
+ style Phase4 fill:#ffe1f5
+ style Phase5 fill:#e1ffe1
+ style Phase6 fill:#ffe1e1
+Declare a topology, attach workloads and expectations, and set the run window. The plan is the single source of truth for what will happen.
+Key actions:
+Output: Immutable Scenario plan
Hand the plan to a deployer. It provisions the environment on the chosen backend, waits for nodes to signal readiness, and returns a runner.
+Key actions:
+Output: Runner + RunContext (with node clients, metrics, control handles)
Expectations snapshot initial state before workloads begin.
+Key actions:
+Output: Captured state for later comparison
+The runner starts traffic and behaviors for the planned duration.
+Key actions:
+Duration: Controlled by with_run_duration()
Once activity stops (and optional cooldown completes), the runner checks liveness and workload-specific outcomes.
+Key actions:
+Output: Success or detailed failure report
+Tear down resources so successive runs start fresh and do not inherit leaked state.
+Key actions:
+NOMOS_TESTS_KEEP_LOGS=1)Guarantee: Runs even on panic via CleanupGuard
The scenario model defines clear, composable responsibilities:
+Developers extending the model should keep these boundaries strict: topology +describes, scenarios assemble, deployers provision, runners orchestrate, +workloads drive, and expectations judge outcomes. For guidance on adding new +capabilities, see Extending the Framework.
+ +