uniform clustersN nodes of one binarysingle binariesyours or third-partycomposed stacksclusters + processes, wiredalready runningattached / external
two ways to drive the test, using the same deployment code
a scenariodeclarative — the runner drivesoryour own codeimperative — ManualCluster
an Application defines the config, client, and deployment shape for one node kind
The testing framework runs system-level tests against multi-process and multi-node deployments. A test starts the system — as local processes, a Compose project, or a Kubernetes deployment — drives traffic against it, verifies outcomes, and tears everything down, all from ordinary Rust code. The sections below explain the APIs used for each part.
This page summarizes the main concepts and links each one to a full chapter. It uses the **job-processing stack** from `examples/multi_app` throughout: jobs enter a queue cluster, a worker process consumes them, and results are written to a result-store cluster.
```mermaid
flowchart LR
WL["workload enqueue 10 jobs"]:::sc --> Q["queue cluster 2 nodes"]:::cl
Q --> W["worker one process"]:::pr
W --> R["result store 2 nodes"]:::cl
R --> EX["expectation 10 results visible"]:::sc
classDef cl stroke:#4a90d9,stroke-width:2.5px;
classDef pr stroke:#e08a3c,stroke-width:2.5px;
classDef sc stroke:#9b6dd6,stroke-width:2.5px;
```
six terms
A Builder creates a Scenario. A Deployer starts the system. The Runner starts its Workloads and evaluates its Expectations.
describe the test
Scenariodeployment and test plan
Builderassembles the scenario
act and check
Workloadcreates activity
Expectationverifies an outcome
execute it
Deployerstarts the system
Runnerruns it end to end
these six terms appear throughout the examples below
Scenariodeployment and test planwhat to deploy, what activity to run, what to verify, and for how longBuilderassembles the scenariothe chain of with_* callsWorkloadcreates activitycode that runs against the live system: send jobs, restart nodes, cut the networkExpectationverifies an outcomecode that checks the result after the activity: all results present, cluster convergedDeployerstarts the systemfor real, as local processes, a Compose project, or a Kubernetes deploymentRunnerruns it end to endwait until ready, run workloads, evaluate expectations, tear down
This map shows how the concepts on the page relate. Each §N badge links the concept to the section that explains it.
{{#include framework-map.svg}}
Click an empty area to enlarge the map. Drag to pan; press Escape or use Close to return.
the whole test
This is the main body of the multi-app-e2e acceptance test. Run it with cargo test -p multi-app-e2e.
example used throughout: enqueue ten jobs in the examples/multi_app stack and check that ten results are stored
```rust,ignore
let mut scenario = AppHost::scenario() // ①
.with_app(JobStackApp::new()) // ②
.with_run_duration(Duration::from_secs(10)) // ③
.with_workload(EnqueueJobs::new(10)) // ④
.with_expectation(AllJobsCompleted::new(10)) // ⑤
.build()?;
let runner = AppHostLocalDeployer::default()
.deploy(&scenario) // ⑥
.await?;
runner.run(&mut scenario).await?; // ⑦
```
① create the scenario · ② add the stack · ③ set the run limit · ④ add traffic · ⑤ add a check · ⑥ start locally · ⑦ run and clean up
① a scenario with no framework-managed nodes of its own — the composed stack provides the system → section 1
② deploy the stack: two clusters and a process, wired together → section 3
③ the run window (a maximum, not a timer you must fill) → section 4
④ ⑤ create activity, verify outcomes → section 4
⑥ where it runs: local processes here; other backends → section 10
The helper API expresses a partition, random restarts, and a convergence check in one chain. Runnable as cargo run -p queue-examples --bin queue_dsl_demo.
produce400 jobs at 40/s→group A ✂ group Bsplit 20 s, then heal+⚡ random restartsevery 5–15 s→expect convergenceall 5 nodes at 400
each helper adds ordinary workloads, expectations, and the capabilities they require. Tests can also use the explicit API
The next two blocks use a second, simpler system, because chaos reads clearest on a uniform cluster: one five-node queue cluster, no worker or store. The scenario produces jobs against it while restarting random nodes and cutting the network in two, then checks that every node still converges:
```mermaid
flowchart LR
WL["produce 400 jobs at 40/s"]:::sc --> A
subgraph A["partition group A"]
N0["node-0"]:::cl
N1["node-1"]:::cl
end
subgraph B["partition group B"]
N2["node-2"]:::cl
N3["node-3"]:::cl
N4["node-4"]:::cl
end
A -. "✂ split 20s, then heal" .- B
RR["⚡ random restarts every 5–15s"]:::pr -.-> A
RR -.-> B
B --> EX["expect all 5 nodes converge at 400"]:::sc
classDef cl stroke:#4a90d9,stroke-width:2.5px;
classDef pr stroke:#e08a3c,stroke-width:2.5px;
classDef sc stroke:#9b6dd6,stroke-width:2.5px;
```
First in the explicit API, compile-checked:
```rust,ignore
let mut scenario = QueueScenarioBuilder::deployment_with(|_| QueueTopology::new(5))
.enable_node_control() // restarts allowed
.with_network_control() // partitions allowed
.with_workload(
QueueProduceWorkload::new() // steady traffic
.operations(400)
.rate_per_sec(40)
.payload_prefix("soak"),
)
.with_workload(RandomRestartWorkload::new( // random node restarts
Duration::from_secs(5),
Duration::from_secs(15),
Duration::from_secs(10),
))
.with_workload(NetworkPartitionWorkload::new( // split, hold, heal
NetworkPartitionSpec::new(vec![
vec!["node-0", "node-1"],
vec!["node-2", "node-3", "node-4"],
]),
Duration::from_secs(20),
Duration::from_secs(20),
))
.with_expectation(QueueConverges::new(400).timeout(Duration::from_secs(60)))
.with_run_duration(Duration::from_secs(120))
.build()?;
let runner = QueueLocalDeployer::default().deploy(&scenario).await?;
runner.run(&mut scenario).await?;
```
Each helper in the shorter chain adds these same workloads, expectations, and capabilities. Tests can use the explicit API whenever the helpers do not cover what they need.
The framework began with an API sketch in this style. Its current implementation separates that idea into scenarios, workloads, expectations, and deployment backends. The sections below show how those parts fit together.
The numbered sections first explain this test, then cover manual control, state, existing deployments, backends, and observation.
---
## 1 · Mental Model
lines ① and ⑦: scenario contents and runner order.
A scenario records what to deploy, what to run, what to check, and the runtime settings.
the runner executes the same order for uniform clusters, composed stacks, and existing deployments
The framework does not contain queue- or blockchain-specific node logic. An Application supplies the deployment shape, client type, config type, and readiness contract for one node kind. A scenario combines the system to deploy, the test behavior, and the runtime settings.
The framework sees an application only through those four things, and `Application` captures exactly that (deploying is the deployer's job):
```rust,ignore
pub trait Application: Send + Sync + 'static {
type Deployment: DeploymentDescriptor + Clone; // cluster shape
type NodeClient: Clone + Send + Sync; // how tests reach a node
type NodeConfig: Clone + Send + Sync; // per-node config type
}
```
Running a scenario (line ⑦) always follows the one lifecycle shown above.
`Application` and `AppDeployment` answer different questions:
| Concept | Describes | Example |
|---|---|---|
| `Application` | one node kind: topology, client, config, readiness | `QueueEnv` |
| `AppDeployment` | how one component or composed stack is prepared and exposed | `JobStackApp` |
A uniform scenario is parameterized directly by an `Application`. A composed scenario uses `AppDeployment` values, which may provision clusters of several application types plus standalone processes.
Tests can also target nodes the framework did not start. A cluster is *managed* when TF starts and removes it, *attached* when TF connects to it and has some control, or *external* when TF only has clients. The example uses managed clusters. Section 8 shows all three modes.
ManualCluster still starts and cleans up managed nodes. Your test code replaces the scenario runner
Most tests let the runner perform deployment, readiness checks, workloads, evaluation, and teardown. Tests that need step-by-step control can perform those actions directly. This choice is independent of ownership: ManualCluster, for example, gives your code control of the sequence while TF still starts and removes the nodes.
```mermaid
flowchart TD
U["Uniform cluster N identical nodes"]:::cl --> S["Scenario"]:::sc
A["Composed stack the job stack — line ②"]:::sc --> S
X["Attached / external clusters you already run"]:::cl --> S
S --> R["Runner one lifecycle for all three"]:::sc
M["ManualCluster managed nodes, you drive"] -.->|bypasses the runner| C["step-by-step node control"]
classDef cl stroke:#4a90d9,stroke-width:2.5px;
classDef sc stroke:#9b6dd6,stroke-width:2.5px;
```
Bypassing the runner changes who drives the nodes, not who owns them: `ManualCluster` nodes are still framework-managed.
Decision table: which pattern fits which system
| Shape of the system under test | Pattern | Read |
|---|---|---|
| N identical nodes of one binary | Uniform managed cluster | [Part IV](part-iv.md) |
| Several apps composed into one stack | `AppHost` + `AppDeployment` | [Part II](part-ii.md) |
| Already-running nodes you must not deploy | Attached / external sources | [section 8](#8--sources-and-ownership) |
| An external driver dictates every step | `ManualCluster`, or direct `DeployContext` for a composed stack | [section 5](#5--imperative-control) |
Next: what line ② deploys for the job-processing example.
each resource is registered for cleanup as soon as it starts. Teardown stops the worker before its dependencies
JobStackApp implements AppDeployment. Its deploy method starts the two clusters, reads their runtime addresses, then starts the worker with both addresses.
```rust,ignore
async fn deploy(self, ctx: &mut DeployContext) -> Result {
let queue = ctx
.deploy_and_expose(QueueLocalApp::nodes(self.queue_nodes)) // ①
.await?;
let results = ctx
.deploy_and_expose(KvLocalApp::nodes(self.result_nodes))
.await?;
let queue_url = queue.first_client().ok_or("queue cluster has no clients")?.base_url().clone();
let results_url = results.first_client().ok_or("result store has no clients")?.base_url().clone();
let worker = ctx
.deploy_and_expose(JobWorkerApp::new(queue_url, results_url)) // ②
.await?;
let stack = JobStackHandle { queue, results, worker }; // ③
ctx.expose(stack.clone())?;
Ok(stack)
}
```
The aggregate returned to test code contains two uniform-cluster handles and one process handle:
```rust,ignore
struct JobStackHandle {
queue: LocalAppCluster,
results: LocalAppCluster,
worker: LocalProcessHandle,
}
```
① deploy_and_expose starts a child and publishes its handle for test code. Registering a second unnamed handle of the same type returns an error.
② dependencies travel by constructor: the worker receives the URLs of the already-running clusters. The dependency endpoints are passed explicitly.
③ the stack handle contains all three members. A test can retrieve the stack or retrieve an exposed child by type.
The worker is the single-binary member. A `LaunchSpec` declares the process; a readiness closure gates it:
```rust,ignore
let launch = LaunchSpec {
binary: worker_binary_provider().resolve()?, // section 9
args: vec!["--queue-url".to_owned(), queue_url.to_string(), /* … */],
..LaunchSpec::default()
};
let process = LocalProcessApp::new("job-worker", launch, endpoints, client)
.with_readiness(|_, client| async move { client.wait_ready().await });
```
The deployment APIs provide the following lifecycle behavior:
- Managed clusters use their configured HTTP or TCP readiness probe. A process uses its readiness closure. A custom deployment must not return from `deploy` until it is usable.
- Managed resources register for cleanup when they start. Cleanup runs in reverse order, so this example stops the worker before either cluster. If `deploy` fails partway through, resources already started are still removed.
Which lifecycle operations each deployment path provides
| Deployment path | Automatic teardown | Explicit control |
|---|---|---|
| uniform cluster | yes | `start_node`, `stop_node`, `restart_node`, readiness waits |
| `LocalProcessApp` | yes | `start`, `stop`, `restart`, `is_running` |
| custom deployment | when it composes managed adapters (they register with scenario cleanup) | only methods its handle implements |
| external | no | none without an adapter |
Next: how lines ③④⑤ send work through the deployed stack and check the result.
The runner starts workloads, waits for the cooldown, then evaluates expectations.
workload⌁ through handles→deployed stack→expectation⌁ through handles
lines ③④⑤: workloads start concurrently, expectation failures are collected, and duration is a maximum
A workload sends requests or performs other activity against the deployed system. An expectation checks the resulting state. Both receive the scenario's typed handles, but the runner executes them in separate phases.
The scenario registers both objects. `runner.run` calls them at the appropriate phases:
```rust,ignore
let mut scenario = AppHost::scenario()
.with_app(JobStackApp::new())
.with_run_duration(Duration::from_secs(10))
.with_workload(EnqueueJobs::new(10)) // register activity
.with_expectation(AllJobsCompleted::new(10)) // register the check
.build()?;
let runner = AppHostLocalDeployer::default().deploy(&scenario).await?;
runner.run(&mut scenario).await?; // TF invokes both
```
The runner supplies the same `RunContext` to both callbacks. In an `AppHost` scenario, they use it to retrieve the typed handles exposed by `JobStackApp`.
### The Workload
`EnqueueJobs` implements TF's `Workload` trait. During the workload phase, the runner calls `start`; returning an error fails the run.
```rust,ignore
#[async_trait]
impl Workload for EnqueueJobs {
fn name(&self) -> &str {
"enqueue_jobs"
}
async fn start(&self, ctx: &RunContext) -> Result<(), DynError> {
let stack = ctx.require_app::()?;
let queue = stack
.queue()
.first_client()
.ok_or("queue cluster has no clients")?;
for index in 0..self.count {
let response: EnqueueResponse = queue
.post("/queue/enqueue", &EnqueueRequest { payload: job_key(index) })
.await?;
if !response.accepted {
return Err(format!("queue rejected job {index}").into());
}
}
Ok(())
}
}
```
### The Expectation
`AllJobsCompleted` implements TF's `Expectation` trait. After workloads and cooldown, the runner calls `evaluate`; `Ok(())` passes this check and `Err(...)` reports an expectation failure.
```rust,ignore
#[async_trait]
impl Expectation for AllJobsCompleted {
fn name(&self) -> &str {
"all_jobs_completed"
}
async fn evaluate(&mut self, ctx: &RunContext) -> Result<(), DynError> {
let stack = ctx.require_app::()?;
let clients = stack.results().clients();
let deadline = Instant::now() + self.timeout;
while Instant::now() < deadline {
if all_results_are_visible(&clients, self.count).await? {
if !stack.worker().is_running().await {
return Err("job worker stopped before evaluation".into());
}
return Ok(());
}
tokio::time::sleep(Duration::from_millis(200)).await;
}
Err(format!("job results did not converge within {:?}", self.timeout).into())
}
}
```
This expectation polls every result-store node until all ten keys read `completed`, and checks that the worker is still running.
The runner behaves as follows:
- All workloads start **concurrently**; a panic is reported as a workload failure; an error ends the run immediately.
- The duration is a **maximum**: when every workload finishes early, cooldown starts early.
- Expectations have four phases: `init`, `start_capture`, `check_during_capture` (~1 s tick), and `evaluate` after cooldown. Failures **aggregate** rather than short-circuit.
- **Cooldown** is the settle window between traffic and evaluation.
- **Runtime extensions** are typed scenario-lifetime services prepared after readiness; the app layer is one, which is why `with_app` is once per scenario.
### Responsibility Split
| TF behavior | Application or test responsibility |
|---|---|
| Nodes are ready before any workload starts | Readiness paths and probes are correct for your node |
| Workloads run concurrently and panics are reported as failures | Every workload terminates; an unbounded one blocks the run |
| Every expectation evaluates and failures aggregate | Expectations poll with their own deadline instead of assuming fresh state |
| Managed resources release in reverse acquisition order, also on partial failure | Custom adapters register cleanup immediately after acquiring a resource |
| Artifacts survive a panic or an explicit preservation setting | Compose images are built beforehand; env-provider binaries are pointed at real files |
Next: running the same deployments without workloads and expectations.
direct control from a Rust test or an external harness.
Tests can control a uniform cluster or deploy a composed stack directly, without using the scenario runner.
your test→start→call + assert→restart→cleanup
ManualCluster for one uniform cluster · DeployContext for an AppDeployment tree
A BDD harness, debugging tool, or ordinary Rust test can control a uniform cluster through ManualCluster. It can also deploy an existing composed stack through DeployContext. In both cases the test code decides when to call, stop, or restart each component.
Runner-driven and manually driven tests use the same cluster and process deployment code:
```mermaid
flowchart TB
R["runner sequences the test"]:::driver --> SETUP["system setup"]:::setup
U["your Rust code / BDD steps sequence the test"]:::driver --> SETUP
SETUP --> C["one uniform cluster"]:::shape
SETUP --> S["composed stack"]:::shape
S --> CC["uniform child clusters"]:::shape
S --> P["standalone processes"]:::process
C --> N["node bring-up topology → ports + peers → config → binary → start → readiness"]:::engine
CC --> N
P --> SP["process bring-up launch settings → binary → start → readiness"]:::engine
N --> H["running resources clients · lifecycle control · reverse cleanup"]:::runtime
SP --> H
classDef driver stroke:#9b6dd6,stroke-width:2.5px;
classDef setup stroke:#777,stroke-width:2px,stroke-dasharray:4 3;
classDef shape stroke:#4a90d9,stroke-width:2.5px;
classDef process stroke:#e08a3c,stroke-width:2.5px;
classDef engine stroke:#777,stroke-width:2px;
classDef runtime stroke:#4caf7d,stroke-width:2.5px;
```
A uniform cluster can be the whole system or one child of a composed stack. Both use the same node startup path. A composed stack can also contain standalone processes. The runner, ManualCluster, and DeployContext call these shared deployment APIs in different ways.
In API terms, `ManualCluster` reuses the `Application` definition and local cluster implementation for `QueueEnv`. It does not execute `AppDeployment::deploy` or create a `DeployContext`; direct composed-stack deployment is the separate path shown later in this section.
### One Uniform Cluster: `ManualCluster`
This test uses the queue from the same job-stack example. It is a normal async test: TF starts and owns the processes, while the test owns the sequence and assertions.
```rust,ignore
#[tokio::test]
async fn drives_queue_cluster_without_a_scenario() -> Result<(), DynError> {
let cluster = ManualCluster::::from_topology(QueueTopology::new(2));
let node0 = cluster.start_node("node-0").await?.client;
let node1 = cluster.start_node("node-1").await?.client;
cluster.wait_network_ready().await?;
enqueue(&node0, "manual-job").await?;
wait_for_queue_len(&[node0, node1], 1).await?;
cluster.restart_node("node-1").await?;
cluster.wait_node_ready("node-1").await?;
let restarted = cluster
.node_client("node-1")
.ok_or("node-1 client missing after restart")?;
wait_for_queue_len(&[restarted], 1).await?;
Ok(())
}
```
Dropping the cluster stops every child process, including on an early `?` or panic. `StartNodeOptions` adds peer selection, config overrides and patches, persistent or snapshot directories, extra arguments, and per-start timeouts.
### A Composed Stack: Direct `AppDeployment`
The same `JobStackApp` recipe used by `.with_app(...)` can be deployed directly. The returned aggregate exposes every component handle, so ordinary Rust can use and control the queue cluster, result-store cluster, and worker process:
```rust,ignore
let mut deployment =
DeployContext::::new(AppHostTopology, NodeClients::default());
let stack = deployment.deploy(JobStackApp::new()).await?;
assert_eq!(stack.queue().node_count(), 2);
assert_eq!(stack.results().node_count(), 2);
let queue = stack.queue().first_client().ok_or("queue has no clients")?;
let results = stack.results().clients();
let worker = stack.worker().clone();
worker.restart().await?;
enqueue(&queue, "imperative-job").await?;
wait_for_completed_result(&results, "imperative-job").await?;
drop(deployment); // reverse cleanup for the whole stack
assert!(!worker.is_running().await);
```
In this form, `DeployContext` keeps the child `AppDeployment`s, their typed handles, and the cleanup callbacks. Dropping it runs cleanup in reverse order, just as scenario teardown does.
| | Declarative scenario | `ManualCluster` | Direct `AppDeployment` |
|---|---|---|---|
| Who sequences behavior? | TF's runner | Your Rust code or external harness | Your Rust code or external harness |
| System shape | Uniform cluster or composed stack | One uniform cluster | One component or composed stack |
| Reusable definition | `Application`, optionally `AppDeployment` | `Application` | `AppDeployment` and its child apps |
| Test behavior | Workloads and expectations | Client calls, helpers, assertions | Handle calls, helpers, assertions |
| Cleanup owner | Scenario runtime | `ManualCluster` | `DeployContext` |
Manual control is also available without abandoning a scenario. A scenario can opt into node control with `with_node_control()`, and app deployments return `ClusterHandle` / `LocalAppCluster` and `LocalProcessHandle` values with direct lifecycle methods.
Next: how TF assigns ports and how applications produce node configuration.
deployment policy controls readiness, retry, and retained artifacts without changing the application config
TF allocates collision-free ports and prepares each node's peer list. Application code converts those values into the config and command expected by its binary. The local, Compose, or Kubernetes backend delivers the files, starts the binary, and applies the requested readiness and retry policy.
```mermaid
flowchart TB
T["1 · topology the test asks for three queue nodes"]:::input
V["2 · framework prepares node 1 identity · reserved port · peer addresses"]:::framework
C["3 · application builds queue configuration node id · HTTP port · peers · sync interval"]:::app
B["4 · backend launches the node write config file · resolve binary · pass args and environment · start"]:::backend
H["5 · ready running resource typed client · lifecycle control · registered cleanup"]:::runtime
T --> V --> C --> B --> H
POL["deployment policy readiness · retry · retained artifacts"]:::policy -. "governs launch" .-> B
POL -. "gates access" .-> H
classDef input stroke:#777,stroke-width:2px;
classDef framework stroke:#4a90d9,stroke-width:2.5px;
classDef app stroke:#9b6dd6,stroke-width:2.5px;
classDef backend stroke:#777,stroke-width:2px,stroke-dasharray:4 3;
classDef runtime stroke:#4caf7d,stroke-width:2.5px;
classDef policy stroke:#c89b3c,stroke-width:2.5px;
```
### Application-Owned Configuration
The queue's real config builder receives one framework-generated node view plus all peer views and returns the value understood by the queue binary:
```rust,ignore
fn build_cluster_node_config(
node: &ClusterNodeView,
peers: &[ClusterPeerView],
) -> Result {
Ok(QueueNodeConfig {
node_id: node.index() as u64,
http_port: node.network_port(),
peers: peers
.iter()
.map(|peer| QueuePeerInfo {
node_id: peer.index() as u64,
http_address: peer.authority(),
})
.collect(),
sync_interval_ms: 500,
})
}
```
The local adapter then says where the binary comes from, how to serialize that typed config, and which port is its API:
```rust,ignore
fn local_process_spec() -> LocalProcessSpec {
LocalProcessSpec::new("QUEUE_NODE_BIN")
.with_binary_provider(queue_binary_provider())
.with_rust_log("queue_node=info")
}
fn render_local_config(config: &QueueNodeConfig) -> Result, DynError> {
yaml_node_config(config)
}
fn http_api_port(config: &QueueNodeConfig) -> u16 {
config.http_port
}
```
Scenario deployment, `ManualCluster`, and uniform child clusters all call these same application functions.
### Delivering Configuration: Local Files and cfgsync
TF renders the same per-node artifacts for each backend. The local backend writes them directly, while container backends deliver them through cfgsync:
```mermaid
flowchart TB
C["typed per-node configuration"]:::app --> A["rendered per-node artifacts config file + any additional files"]:::artifact
A --> L["local backend write directly into the node working directory"]:::local
A --> S["container backends serve artifacts through cfgsync"]:::container
S --> F["cfgsync client in each container register · fetch · write files"]:::container
L --> N["start node binary"]:::process
F --> N
classDef app stroke:#9b6dd6,stroke-width:2.5px;
classDef artifact stroke:#777,stroke-width:2px,stroke-dasharray:4 3;
classDef local stroke:#4a90d9,stroke-width:2.5px;
classDef container stroke:#c89b3c,stroke-width:2.5px;
classDef process stroke:#e08a3c,stroke-width:2.5px;
```
Locally, TF writes files into the process working directory. Compose and Kubernetes nodes cannot see that host directory. For those backends, a cfgsync server holds each node's artifacts, and a client inside the container fetches and writes them before executing the node. cfgsync only transports generated configuration. It does not preserve application state or create snapshots.
### Test-Side Changes for One Start
Tests normally keep the generated ports and peers and patch only the behavior they care about. The Section 5 manual-cluster test really starts its second node with a faster synchronization interval:
```rust,ignore
let node1 = cluster
.start_node_with(
"node-1",
StartNodeOptions::::default().create_patch(|mut config| {
config.sync_interval_ms = 50;
Ok(config)
}),
)
.await?
.client;
```
Use `config_override` only when the test intends to replace the complete generated config. `config_patch` preserves framework-assigned values unless the callback deliberately changes them.
### Deployment Policy
The node config is passed to the application binary. `DeploymentPolicy` separately controls TF's readiness checks, retry behavior, cleanup, and artifact retention:
```rust,ignore
let policy = DeploymentPolicy {
readiness_enabled: true,
readiness_requirement: HttpReadinessRequirement::AtLeast(2),
retry_policy: Some(RetryPolicy::new(
5,
Duration::from_millis(500),
Duration::from_secs(5),
)),
cleanup_policy: CleanupPolicy::new(true),
..DeploymentPolicy::default()
};
```
For the primary scenario cluster, set this through `.with_deployment_policy(policy)`. A child cluster created by an `AppDeployment` carries policy on its `ClusterRequest`. `deploy_local_cluster(...)` uses the default policy.
Readinessrequirement (all nodes / quorum) + probe (HTTP path or TCP) with retry budgets; SLOW_TEST_ENV doubles timeoutsRetrythe local backend respawns a failed cluster attempt with backoff; Compose and Kubernetes currently do not repeat deploymentArtifactslocal files live in node working directories; container backends receive rendered config through cfgsyncRetentionpreserve_artifacts, TF_KEEP_LOGS, or a panic keep local working directories for post-mortems
Next: what happens to node state during restart and restore.
ordinary restart: same working directory · restore: new working directory seeded from a snapshot
Restartstop and respawn the same node in its existing working directory, including state written therepersist_dirplace that working directory at a stable, findable path rather than a framework temporary directorysnapshot_dircopy saved state into the fresh working directory at spawn — the base for stop → snapshot → restore testsConfigconfig_override replaces the generated per-node config; config_patch transforms itSeedswith_deployment_seed feeds deterministic deployment providers
Next: connecting the same test to clusters that TF did not start.
managed, attached, and external clusters use one request API but provide different levels of control.
You request every cluster through the same API. Available control depends on whether TF started it or connected to it.
manageddeployed and torn down
attachedpartially driven
externalclients only
deploy_cluster(ClusterRequest::…) returns clients in all three modes; only managed clusters always provide full lifecycle control
The job-stack example asks TF to start both clusters. A test can instead connect to an existing deployment. deploy_cluster handles all three cases and returns node clients for each one. Full start, stop, and restart control is guaranteed only when TF manages the nodes.
```rust,ignore
let cluster = ctx.deploy_cluster(ClusterRequest::managed(deployment)).await?;
let attached = ctx.deploy_cluster(ClusterRequest::attached(existing)).await?;
let external = ctx.deploy_cluster(ClusterRequest::external(endpoints)).await?;
```
| | Managed | Attached | External |
|---|---|---|---|
| Clients | ✓ | ✓ | ✓ |
| Node control | ✓ | per backend | — |
| Readiness waits | ✓ | ✓ | — |
| Torn down by the framework | ✓ | — | — |
The scenario builder exposes the same modes through `with_existing_cluster`, `with_external_nodes`, and `with_external_only_nodes`. Workloads and expectations use node clients, so they do not need to change when a test moves from a locally managed cluster to an existing deployment.
Next: how TF finds the binaries it has been asked to start.
the worker_binary_provider() call inside line ② — FallbackBinaryProvider tries sources in order, with a cache and cross-process locking
Every process TF starts needs an executable path. A binary provider can return an explicit path, read one from an environment variable, build the binary locally, or download an artifact. A fallback provider tries several providers in order.
The job worker's real provider chain tries an env var override and falls back to a local build:
```rust,ignore
FallbackBinaryProvider::new([
Arc::new(EnvBinaryProvider::new("MULTI_APP_JOB_WORKER_BIN")),
Arc::new(BuildBinaryProvider {
command: BuildCommand::new("cargo")
.with_args(["build", "-p", "multi-app-job-worker", "--bin", "multi-app-job-worker"]),
output_path: "target/debug/multi-app-job-worker".into(),
working_dir: Some(workspace),
lock_dir: None,
}),
])
```
The available providers are explicit path, environment variable, local build, and checksummed download with post-processing. `FallbackBinaryProvider` chains them, with a resolution cache and cross-process locking.
Next: selecting the local, Compose, or Kubernetes backend.
line ⑥: local, Compose, and Kubernetes deployment.
Uniform scenarios can run locally, with Compose, or on Kubernetes. Backend capabilities currently differ.
localprocesses · full node control · app composition
Composecontainers · cfgsync · restart
KubernetesHelm · cfgsync · manual mode
line ⑥ picks the backend; app composition is local-only today
Line ⑥ selects the local backend. Uniform scenarios can also use the Compose and Kubernetes deployers. The table lists the deployment and control features currently implemented by each backend.
| | Local | Compose | Kubernetes |
|---|---|---|---|
| Node startup | processes + temp dirs | generated compose file | Helm chart + values |
| Config delivery | filesystem | cfgsync artifacts | cfgsync artifacts |
| Node control | full | restart | manual mode only |
| App composition | ✓ | — | — |
| Attach / external | external nodes | ✓ | ✓ |
App composition currently runs only on the local backend. Uniform scenarios run on all three. Local working directories are temporary and removed after a successful run unless `TF_KEEP_LOGS` or `preserve_artifacts` is set. They are also retained after a panic.
Next: reading changing application state during a test.
observation is a runtime extension; telemetry is a backend capability
Continuous observation
An `Observer` polls application state on a cadence; tests read `latest_snapshot()`, `history()`, or `subscribe()` from an `ObservationHandle`. Sources can be dynamic, re-queried as nodes come and go.
Telemetry
Metrics, logs, tracing, and Grafana/OTLP endpoints are configured through the observability capability and environment variables. They serve external monitoring, not test logic.
Continuous observation is implemented as a **runtime extension** (section 4); telemetry is a backend capability configured on the scenario, not an extension.
Next: matching common test cases to the APIs covered above.
common test cases and the APIs normally used for them.
The table below shows which APIs are normally used for each kind of test.
convergencerestart recoverysnapshot restorefailoverchaos under loadload / soakthird-party binarieslive networks
| Test kind | Framework tools | Read |
|---|---|---|
| Convergence / consistency | traffic workload + expectation polling every node client | [Workloads](workloads.md), [Expectations](expectations.md) |
| Recovery across a restart | `restart_node` or process `restart()`; working directories survive restarts | [Imperative Control](#5--imperative-control), [Persistence](persistence.md) |
| Restore from saved state | `snapshot_dir` seeding + an expectation on the restored data | [Persistence](persistence.md) |
| Role failover | find the role through observation, restart it via node control, expect a new holder | [Chaos](chaos.md), [Observation](observation.md) |
| Chaos under load | traffic workload + `RandomRestartWorkload` / the chaos builder in one scenario | [Chaos](chaos.md) |
| Load / soak | bounded traffic workloads paced across the run window | [Workloads](workloads.md) |
| Deployment and config validation | the same uniform scenario per backend, plus readiness policy | [Backends](#10--deployment-backends), [Config](#6--configuration-and-deployment-policy) |
| Behavior of a third-party binary | `LocalProcessApp` + `LaunchSpec` around the unmodified executable | [Section 3](#3--composed-applications-the-job-stack) |
| Against a live network | attached or external sources with unchanged workloads and expectations | [Sources](#8--cluster-sources-and-ownership) |