Testing Framework

System-level testing for networked applications from Rust.

The testing framework deploys and controls processes, containers, and clusters. Tests can run several nodes over network connections for a bounded period. Application-specific configuration and clients stay outside the framework, so the runtime can be used with a key-value store, a Raft cluster, a message queue, or a blockchain.

Get Started

Scenarios

A declarative test is represented by a Scenario containing:

Topology — the system under test (a uniform cluster, a composed application stack, or attached external nodes)
Workloads — traffic and conditions that exercise the system
Expectations — success criteria verified after execution
Duration — the time window for the experiment

flowchart LR
    subgraph SC["Scenario"]
        T["topology<br/><small>the system under test</small>"]:::cl
        W["workloads<br/><small>drive traffic</small>"]:::sc
        EX["expectations<br/><small>verify outcomes</small>"]:::sc
        D["duration<br/><small>the run window</small>"]:::sc
    end
    SC --> RN["Runner<br/><small>deploy · run · evaluate · teardown</small>"]:::sc
    classDef cl stroke:#4a90d9,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

The scenario runtime executes these parts in the same order for each declarative entry pattern. The entry pattern determines how the system is supplied.

Entry Patterns

flowchart LR
    A[Uniform managed cluster]:::cl --> S[Scenario]
    B[AppHost composed stack] --> S
    C[Attached / external nodes]:::cl --> S
    S:::sc --> R[Runner: workloads + expectations]:::sc
    M[ManualCluster] --> I[Imperative orchestration]
    classDef cl stroke:#4a90d9,stroke-width:2.5px;
    classDef hd stroke:#4caf7d,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

Three entry patterns use the scenario runtime:

Uniform managed cluster — the framework generates configs and launches N identical nodes from a topology. See Part IV.
AppHost composed stack — the app layer deploys heterogeneous components (processes, child clusters, in-process services) as one system and exposes typed handles to workloads. See Part II.
Attached and external nodes — the scenario targets clusters you already run, or plain URLs. See Existing and External Clusters.

ManualCluster is the imperative alternative. It provides direct start, stop, restart, and readiness operations without the scenario runner, including for step-driven BDD harnesses.

If you are not sure which to use, read Choosing an Entry Pattern.

Provided APIs

Declarative API

Express tests as topology + workloads + expectations
Reuse the same definition across local, Compose, and Kubernetes deployers
Compose stacks from reusable application deployments

Application layer

Deploy heterogeneous systems as one root AppDeployment
Typed, named handles connect workloads to components
Deterministic cleanup, including on partial-deployment failure

Runtime capabilities

Capability-gated node control: restart nodes from workloads, portably
Continuous observation: snapshots, history, and event streams of application state
Telemetry: metrics, logs, and tracing endpoints

Operations

Binary providers resolve node binaries from paths, env vars, builds, or downloads
Reproducible deployments via seeds
Artifact preservation for post-mortem debugging

Quick Example

use testing_framework_app::{AppHost, AppHostLocalDeployer, AppScenarioBuilderExt as _};
use testing_framework_core::scenario::Deployer as _;

let mut scenario = AppHost::scenario()
    .with_app(KvLocalApp::nodes(3))
    .with_workload(KvAppHostConvergence::new(3))
    .build()?;

let runner = AppHostLocalDeployer::default().deploy(&scenario).await?;
runner.run(&mut scenario).await?;

This deploys a three-node key-value store cluster, runs a convergence workload against it (including a node restart), and tears everything down. The remaining chapters cover each part of this pattern in detail.

View the example apps

The Example Apps

The repository includes small applications under examples/ that exercise the framework APIs:

App	Demonstrates
`kvstore`	Uniform clusters, app hosting, convergence testing, all three deployers
`openraft_kv`	Node control, failover, continuous observation
`multi_app`	Composing heterogeneous stacks with typed handles
`nats`, `redis_streams`	Testing third-party binaries you did not write
`pubsub`, `queue`, `metrics_counter`	Additional workload and expectation patterns

Some chapters also link to adopter repositories. The examples listed in this table run from this workspace.

Documentation Structure

Section	Description
Part I — Mental Model	The core abstractions and how to choose between entry patterns
Part II — Composing Applications	The app layer: deployments, handles, teardown
Part III — Scenario Runtime	Workloads, expectations, capabilities, observation
Part IV — Uniform Clusters	Implementing `Application`, topology, config, manual control
Part V — Deployers and Sources	Local, Compose, Kubernetes, external clusters, binaries
Part VI — Extending	Extension points, crate map, boundaries
Part VII — Operations	Running examples, CI, diagnostics, troubleshooting

Start with the Quickstart.

5.7 KiB Raw Blame History