53 KiB
The Framework in Brief
the framework
The testing framework runs system-level tests against multi-process and multi-node deployments, from ordinary Rust code.
every test has four parts
the system under test can be
two ways to drive the test, using the same deployment code
ManualClusteran Application defines the config, client, and deployment shape for one node kind
The testing framework runs system-level tests against multi-process and multi-node deployments. A test starts the system — as local processes, a Compose project, or a Kubernetes deployment — drives traffic against it, verifies outcomes, and tears everything down, all from ordinary Rust code. The sections below explain the APIs used for each part.
This page summarizes the main concepts and links each one to a full chapter. It uses the job-processing stack from examples/multi_app throughout: jobs enter a queue cluster, a worker process consumes them, and results are written to a result-store cluster.
flowchart LR
WL["workload<br/><small>enqueue 10 jobs</small>"]:::sc --> Q["queue cluster<br/><small>2 nodes</small>"]:::cl
Q --> W["worker<br/><small>one process</small>"]:::pr
W --> R["result store<br/><small>2 nodes</small>"]:::cl
R --> EX["expectation<br/><small>10 results visible</small>"]:::sc
classDef cl stroke:#4a90d9,stroke-width:2.5px;
classDef pr stroke:#e08a3c,stroke-width:2.5px;
classDef sc stroke:#9b6dd6,stroke-width:2.5px;
six terms
A Builder creates a Scenario. A Deployer starts the system. The Runner starts its Workloads and evaluates its Expectations.
these six terms appear throughout the examples below
with_* calls
Workloadcreates activitycode that runs against the live system: send jobs, restart nodes, cut the network
Expectationverifies an outcomecode that checks the result after the activity: all results present, cluster converged
Deployerstarts the systemfor real, as local processes, a Compose project, or a Kubernetes deployment
Runnerruns it end to endwait until ready, run workloads, evaluate expectations, tear down
This map shows how the concepts on the page relate. Each §N badge links the concept to the section that explains it.
{{#include framework-map.svg}}
Click an empty area to enlarge the map. Drag to pan; press Escape or use Close to return.
the whole test
This is the main body of the multi-app-e2e acceptance test. Run it with cargo test -p multi-app-e2e.
example used throughout: enqueue ten jobs in the examples/multi_app stack and check that ten results are stored
let mut scenario = AppHost::scenario() // ①
.with_app(JobStackApp::new()) // ②
.with_run_duration(Duration::from_secs(10)) // ③
.with_workload(EnqueueJobs::new(10)) // ④
.with_expectation(AllJobsCompleted::new(10)) // ⑤
.build()?;
let runner = AppHostLocalDeployer::default()
.deploy(&scenario) // ⑥
.await?;
runner.run(&mut scenario).await?; // ⑦
① create the scenario · ② add the stack · ③ set the run limit · ④ add traffic · ⑤ add a check · ⑥ start locally · ⑦ run and clean up
- ① a scenario with no framework-managed nodes of its own — the composed stack provides the system → section 1
- ② deploy the stack: two clusters and a process, wired together → section 3
- ③ the run window (a maximum, not a timer you must fill) → section 4
- ④ ⑤ create activity, verify outcomes → section 4
- ⑥ where it runs: local processes here; other backends → section 10
- ⑦ the runner order: readiness → workloads → cooldown → evaluate → teardown → section 1
the same builder, further
The helper API expresses a partition, random restarts, and a convergence check in one chain. Runnable as cargo run -p queue-examples --bin queue_dsl_demo.
QueueScenario::nodes(5)
.produce(400).rate_per_sec(40).done()
.restart_nodes_randomly().every_secs(5, 15).done()
.partition(["node-0", "node-1"], ["node-2", "node-3", "node-4"]).hold_secs(20).done()
.expect_converged(400).within_secs(60)
.run_secs(120)
.await?;
each helper adds ordinary workloads, expectations, and the capabilities they require. Tests can also use the explicit API
The next two blocks use a second, simpler system, because chaos reads clearest on a uniform cluster: one five-node queue cluster, no worker or store. The scenario produces jobs against it while restarting random nodes and cutting the network in two, then checks that every node still converges:
flowchart LR
WL["produce<br/><small>400 jobs at 40/s</small>"]:::sc --> A
subgraph A["partition group A"]
N0["node-0"]:::cl
N1["node-1"]:::cl
end
subgraph B["partition group B"]
N2["node-2"]:::cl
N3["node-3"]:::cl
N4["node-4"]:::cl
end
A -. "✂ split 20s, then heal" .- B
RR["⚡ random restarts<br/><small>every 5–15s</small>"]:::pr -.-> A
RR -.-> B
B --> EX["expect<br/><small>all 5 nodes converge at 400</small>"]:::sc
classDef cl stroke:#4a90d9,stroke-width:2.5px;
classDef pr stroke:#e08a3c,stroke-width:2.5px;
classDef sc stroke:#9b6dd6,stroke-width:2.5px;
First in the explicit API, compile-checked:
let mut scenario = QueueScenarioBuilder::deployment_with(|_| QueueTopology::new(5))
.enable_node_control() // restarts allowed
.with_network_control() // partitions allowed
.with_workload(
QueueProduceWorkload::new() // steady traffic
.operations(400)
.rate_per_sec(40)
.payload_prefix("soak"),
)
.with_workload(RandomRestartWorkload::new( // random node restarts
Duration::from_secs(5),
Duration::from_secs(15),
Duration::from_secs(10),
))
.with_workload(NetworkPartitionWorkload::new( // split, hold, heal
NetworkPartitionSpec::new(vec![
vec!["node-0", "node-1"],
vec!["node-2", "node-3", "node-4"],
]),
Duration::from_secs(20),
Duration::from_secs(20),
))
.with_expectation(QueueConverges::new(400).timeout(Duration::from_secs(60)))
.with_run_duration(Duration::from_secs(120))
.build()?;
let runner = QueueLocalDeployer::default().deploy(&scenario).await?;
runner.run(&mut scenario).await?;
Each helper in the shorter chain adds these same workloads, expectations, and capabilities. Tests can use the explicit API whenever the helpers do not cover what they need.
The framework began with an API sketch in this style. Its current implementation separates that idea into scenarios, workloads, expectations, and deployment backends. The sections below show how those parts fit together.
The numbered sections first explain this test, then cover manual control, state, existing deployments, backends, and observation.
1 · Mental Model
lines ① and ⑦: scenario contents and runner order.
A scenario records what to deploy, what to run, what to check, and the runtime settings.
the runner executes the same order for uniform clusters, composed stacks, and existing deployments
The framework does not contain queue- or blockchain-specific node logic. An Application supplies the deployment shape, client type, config type, and readiness contract for one node kind. A scenario combines the system to deploy, the test behavior, and the runtime settings.
The framework sees an application only through those four things, and Application captures exactly that (deploying is the deployer's job):
pub trait Application: Send + Sync + 'static {
type Deployment: DeploymentDescriptor + Clone; // cluster shape
type NodeClient: Clone + Send + Sync; // how tests reach a node
type NodeConfig: Clone + Send + Sync; // per-node config type
}
Running a scenario (line ⑦) always follows the one lifecycle shown above.
Application and AppDeployment answer different questions:
| Concept | Describes | Example |
|---|---|---|
Application |
one node kind: topology, client, config, readiness | QueueEnv |
AppDeployment |
how one component or composed stack is prepared and exposed | JobStackApp |
A uniform scenario is parameterized directly by an Application. A composed scenario uses AppDeployment values, which may provision clusters of several application types plus standalone processes.
Tests can also target nodes the framework did not start. A cluster is managed when TF starts and removes it, attached when TF connects to it and has some control, or external when TF only has clients. The example uses managed clusters. Section 8 shows all three modes.
Next: the available ways to run a test.
Application, AppDeployment, and Environments · Scenario Model and Lifecycle
2 · Entry Patterns
runner-driven scenarios and direct, step-by-step control.
The runner can drive a uniform cluster, a composed stack, or an existing deployment. ManualCluster leaves the test sequence to your code.
ManualCluster still starts and cleans up managed nodes. Your test code replaces the scenario runner
Most tests let the runner perform deployment, readiness checks, workloads, evaluation, and teardown. Tests that need step-by-step control can perform those actions directly. This choice is independent of ownership: ManualCluster, for example, gives your code control of the sequence while TF still starts and removes the nodes.
flowchart TD
U["Uniform cluster<br/><small>N identical nodes</small>"]:::cl --> S["Scenario"]:::sc
A["Composed stack<br/><small>the job stack — line ②</small>"]:::sc --> S
X["Attached / external<br/><small>clusters you already run</small>"]:::cl --> S
S --> R["Runner<br/><small>one lifecycle for all three</small>"]:::sc
M["ManualCluster<br/><small>managed nodes, you drive</small>"] -.->|bypasses the runner| C["step-by-step node control"]
classDef cl stroke:#4a90d9,stroke-width:2.5px;
classDef sc stroke:#9b6dd6,stroke-width:2.5px;
Bypassing the runner changes who drives the nodes, not who owns them: ManualCluster nodes are still framework-managed.
Decision table: which pattern fits which system
| Shape of the system under test | Pattern | Read |
|---|---|---|
| N identical nodes of one binary | Uniform managed cluster | Part IV |
| Several apps composed into one stack | AppHost + AppDeployment |
Part II |
| Already-running nodes you must not deploy | Attached / external sources | section 8 |
| An external driver dictates every step | ManualCluster, or direct DeployContext for a composed stack |
section 5 |
Next: what line ② deploys for the job-processing example.
3 · Composed Applications: the Job Stack
line ②: .with_app(JobStackApp::new()).
JobStackApp starts the queue and result store, passes their addresses to the worker, and returns access to all three.
each resource is registered for cleanup as soon as it starts. Teardown stops the worker before its dependencies
JobStackApp implements AppDeployment. Its deploy method starts the two clusters, reads their runtime addresses, then starts the worker with both addresses.
async fn deploy(self, ctx: &mut DeployContext<AppHostEnv>) -> Result<Self::Handle, DynError> {
let queue = ctx
.deploy_and_expose(QueueLocalApp::nodes(self.queue_nodes)) // ①
.await?;
let results = ctx
.deploy_and_expose(KvLocalApp::nodes(self.result_nodes))
.await?;
let queue_url = queue.first_client().ok_or("queue cluster has no clients")?.base_url().clone();
let results_url = results.first_client().ok_or("result store has no clients")?.base_url().clone();
let worker = ctx
.deploy_and_expose(JobWorkerApp::new(queue_url, results_url)) // ②
.await?;
let stack = JobStackHandle { queue, results, worker }; // ③
ctx.expose(stack.clone())?;
Ok(stack)
}
The aggregate returned to test code contains two uniform-cluster handles and one process handle:
struct JobStackHandle {
queue: LocalAppCluster<QueueEnv>,
results: LocalAppCluster<KvEnv>,
worker: LocalProcessHandle<WorkerClient>,
}
- ①
deploy_and_exposestarts a child and publishes its handle for test code. Registering a second unnamed handle of the same type returns an error. - ② dependencies travel by constructor: the worker receives the URLs of the already-running clusters. The dependency endpoints are passed explicitly.
- ③ the stack handle contains all three members. A test can retrieve the stack or retrieve an exposed child by type.
The worker is the single-binary member. A LaunchSpec declares the process; a readiness closure gates it:
let launch = LaunchSpec {
binary: worker_binary_provider().resolve()?, // section 9
args: vec!["--queue-url".to_owned(), queue_url.to_string(), /* … */],
..LaunchSpec::default()
};
let process = LocalProcessApp::new("job-worker", launch, endpoints, client)
.with_readiness(|_, client| async move { client.wait_ready().await });
The deployment APIs provide the following lifecycle behavior:
- Managed clusters use their configured HTTP or TCP readiness probe. A process uses its readiness closure. A custom deployment must not return from
deployuntil it is usable. - Managed resources register for cleanup when they start. Cleanup runs in reverse order, so this example stops the worker before either cluster. If
deployfails partway through, resources already started are still removed.
Which lifecycle operations each deployment path provides
| Deployment path | Automatic teardown | Explicit control |
|---|---|---|
| uniform cluster | yes | start_node, stop_node, restart_node, readiness waits |
LocalProcessApp |
yes | start, stop, restart, is_running |
| custom deployment | when it composes managed adapters (they register with scenario cleanup) | only methods its handle implements |
| external | no | none without an adapter |
Next: how lines ③④⑤ send work through the deployed stack and check the result.
AppDeployment and DeployContext · One Binary: LocalProcessApp · Handle Ownership and Teardown · Composing Heterogeneous Stacks
4 · Test Behavior
lines ③ ④ ⑤: duration, workload, and expectation.
The runner starts workloads, waits for the cooldown, then evaluates expectations.
lines ③④⑤: workloads start concurrently, expectation failures are collected, and duration is a maximum
A workload sends requests or performs other activity against the deployed system. An expectation checks the resulting state. Both receive the scenario's typed handles, but the runner executes them in separate phases.
The scenario registers both objects. runner.run calls them at the appropriate phases:
let mut scenario = AppHost::scenario()
.with_app(JobStackApp::new())
.with_run_duration(Duration::from_secs(10))
.with_workload(EnqueueJobs::new(10)) // register activity
.with_expectation(AllJobsCompleted::new(10)) // register the check
.build()?;
let runner = AppHostLocalDeployer::default().deploy(&scenario).await?;
runner.run(&mut scenario).await?; // TF invokes both
Workload::start(ctx)→cooldown→Expectation::evaluate(ctx)→cleanupThe runner supplies the same RunContext to both callbacks. In an AppHost scenario, they use it to retrieve the typed handles exposed by JobStackApp.
The Workload
EnqueueJobs implements TF's Workload trait. During the workload phase, the runner calls start; returning an error fails the run.
#[async_trait]
impl Workload<AppHostEnv> for EnqueueJobs {
fn name(&self) -> &str {
"enqueue_jobs"
}
async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
let stack = ctx.require_app::<JobStackHandle>()?;
let queue = stack
.queue()
.first_client()
.ok_or("queue cluster has no clients")?;
for index in 0..self.count {
let response: EnqueueResponse = queue
.post("/queue/enqueue", &EnqueueRequest { payload: job_key(index) })
.await?;
if !response.accepted {
return Err(format!("queue rejected job {index}").into());
}
}
Ok(())
}
}
The Expectation
AllJobsCompleted implements TF's Expectation trait. After workloads and cooldown, the runner calls evaluate; Ok(()) passes this check and Err(...) reports an expectation failure.
#[async_trait]
impl Expectation<AppHostEnv> for AllJobsCompleted {
fn name(&self) -> &str {
"all_jobs_completed"
}
async fn evaluate(&mut self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
let stack = ctx.require_app::<JobStackHandle>()?;
let clients = stack.results().clients();
let deadline = Instant::now() + self.timeout;
while Instant::now() < deadline {
if all_results_are_visible(&clients, self.count).await? {
if !stack.worker().is_running().await {
return Err("job worker stopped before evaluation".into());
}
return Ok(());
}
tokio::time::sleep(Duration::from_millis(200)).await;
}
Err(format!("job results did not converge within {:?}", self.timeout).into())
}
}
This expectation polls every result-store node until all ten keys read completed, and checks that the worker is still running.
The runner behaves as follows:
- All workloads start concurrently; a panic is reported as a workload failure; an error ends the run immediately.
- The duration is a maximum: when every workload finishes early, cooldown starts early.
- Expectations have four phases:
init,start_capture,check_during_capture(~1 s tick), andevaluateafter cooldown. Failures aggregate rather than short-circuit. - Cooldown is the settle window between traffic and evaluation.
- Runtime extensions are typed scenario-lifetime services prepared after readiness; the app layer is one, which is why
with_appis once per scenario.
Responsibility Split
| TF behavior | Application or test responsibility |
|---|---|
| Nodes are ready before any workload starts | Readiness paths and probes are correct for your node |
| Workloads run concurrently and panics are reported as failures | Every workload terminates; an unbounded one blocks the run |
| Every expectation evaluates and failures aggregate | Expectations poll with their own deadline instead of assuming fresh state |
| Managed resources release in reverse acquisition order, also on partial failure | Custom adapters register cleanup immediately after acquiring a resource |
| Artifacts survive a panic or an explicit preservation setting | Compose images are built beforehand; env-provider binaries are pointed at real files |
Next: running the same deployments without workloads and expectations.
Workloads and Concurrency · Expectations and Evaluation · Runtime Extensions
5 · Imperative Control
direct control from a Rust test or an external harness.
Tests can control a uniform cluster or deploy a composed stack directly, without using the scenario runner.
ManualCluster for one uniform cluster · DeployContext for an AppDeployment tree
A BDD harness, debugging tool, or ordinary Rust test can control a uniform cluster through ManualCluster. It can also deploy an existing composed stack through DeployContext. In both cases the test code decides when to call, stop, or restart each component.
Runner-driven and manually driven tests use the same cluster and process deployment code:
flowchart TB
R["runner sequences the test"]:::driver --> SETUP["system setup"]:::setup
U["your Rust code / BDD steps<br/>sequence the test"]:::driver --> SETUP
SETUP --> C["one uniform cluster"]:::shape
SETUP --> S["composed stack"]:::shape
S --> CC["uniform child clusters"]:::shape
S --> P["standalone processes"]:::process
C --> N["node bring-up<br/><small>topology → ports + peers → config → binary → start → readiness</small>"]:::engine
CC --> N
P --> SP["process bring-up<br/><small>launch settings → binary → start → readiness</small>"]:::engine
N --> H["running resources<br/><small>clients · lifecycle control · reverse cleanup</small>"]:::runtime
SP --> H
classDef driver stroke:#9b6dd6,stroke-width:2.5px;
classDef setup stroke:#777,stroke-width:2px,stroke-dasharray:4 3;
classDef shape stroke:#4a90d9,stroke-width:2.5px;
classDef process stroke:#e08a3c,stroke-width:2.5px;
classDef engine stroke:#777,stroke-width:2px;
classDef runtime stroke:#4caf7d,stroke-width:2.5px;
A uniform cluster can be the whole system or one child of a composed stack. Both use the same node startup path. A composed stack can also contain standalone processes. The runner, ManualCluster, and DeployContext call these shared deployment APIs in different ways.
In API terms, ManualCluster<QueueEnv> reuses the Application definition and local cluster implementation for QueueEnv. It does not execute AppDeployment::deploy or create a DeployContext; direct composed-stack deployment is the separate path shown later in this section.
One Uniform Cluster: ManualCluster
This test uses the queue from the same job-stack example. It is a normal async test: TF starts and owns the processes, while the test owns the sequence and assertions.
#[tokio::test]
async fn drives_queue_cluster_without_a_scenario() -> Result<(), DynError> {
let cluster = ManualCluster::<QueueEnv>::from_topology(QueueTopology::new(2));
let node0 = cluster.start_node("node-0").await?.client;
let node1 = cluster.start_node("node-1").await?.client;
cluster.wait_network_ready().await?;
enqueue(&node0, "manual-job").await?;
wait_for_queue_len(&[node0, node1], 1).await?;
cluster.restart_node("node-1").await?;
cluster.wait_node_ready("node-1").await?;
let restarted = cluster
.node_client("node-1")
.ok_or("node-1 client missing after restart")?;
wait_for_queue_len(&[restarted], 1).await?;
Ok(())
}
Dropping the cluster stops every child process, including on an early ? or panic. StartNodeOptions adds peer selection, config overrides and patches, persistent or snapshot directories, extra arguments, and per-start timeouts.
A Composed Stack: Direct AppDeployment
The same JobStackApp recipe used by .with_app(...) can be deployed directly. The returned aggregate exposes every component handle, so ordinary Rust can use and control the queue cluster, result-store cluster, and worker process:
let mut deployment =
DeployContext::<AppHostEnv>::new(AppHostTopology, NodeClients::default());
let stack = deployment.deploy(JobStackApp::new()).await?;
assert_eq!(stack.queue().node_count(), 2);
assert_eq!(stack.results().node_count(), 2);
let queue = stack.queue().first_client().ok_or("queue has no clients")?;
let results = stack.results().clients();
let worker = stack.worker().clone();
worker.restart().await?;
enqueue(&queue, "imperative-job").await?;
wait_for_completed_result(&results, "imperative-job").await?;
drop(deployment); // reverse cleanup for the whole stack
assert!(!worker.is_running().await);
In this form, DeployContext keeps the child AppDeployments, their typed handles, and the cleanup callbacks. Dropping it runs cleanup in reverse order, just as scenario teardown does.
| Declarative scenario | ManualCluster |
Direct AppDeployment |
|
|---|---|---|---|
| Who sequences behavior? | TF's runner | Your Rust code or external harness | Your Rust code or external harness |
| System shape | Uniform cluster or composed stack | One uniform cluster | One component or composed stack |
| Reusable definition | Application, optionally AppDeployment |
Application |
AppDeployment and its child apps |
| Test behavior | Workloads and expectations | Client calls, helpers, assertions | Handle calls, helpers, assertions |
| Cleanup owner | Scenario runtime | ManualCluster |
DeployContext |
Manual control is also available without abandoning a scenario. A scenario can opt into node control with with_node_control(), and app deployments return ClusterHandle / LocalAppCluster and LocalProcessHandle values with direct lifecycle methods.
Next: how TF assigns ports and how applications produce node configuration.
Scenario Capabilities · Chaos and Controlled Failure · ManualCluster: Imperative Node Control
6 · Configuration and Deployment Policy
ports, peer addresses, node config, readiness, and retry.
TF assigns ports and peers. Application code builds the node config. The selected backend writes the files and starts the process.
deployment policy controls readiness, retry, and retained artifacts without changing the application config
TF allocates collision-free ports and prepares each node's peer list. Application code converts those values into the config and command expected by its binary. The local, Compose, or Kubernetes backend delivers the files, starts the binary, and applies the requested readiness and retry policy.
flowchart TB
T["1 · topology<br/><small>the test asks for three queue nodes</small>"]:::input
V["2 · framework prepares node 1<br/><small>identity · reserved port · peer addresses</small>"]:::framework
C["3 · application builds queue configuration<br/><small>node id · HTTP port · peers · sync interval</small>"]:::app
B["4 · backend launches the node<br/><small>write config file · resolve binary · pass args and environment · start</small>"]:::backend
H["5 · ready running resource<br/><small>typed client · lifecycle control · registered cleanup</small>"]:::runtime
T --> V --> C --> B --> H
POL["deployment policy<br/><small>readiness · retry · retained artifacts</small>"]:::policy -. "governs launch" .-> B
POL -. "gates access" .-> H
classDef input stroke:#777,stroke-width:2px;
classDef framework stroke:#4a90d9,stroke-width:2.5px;
classDef app stroke:#9b6dd6,stroke-width:2.5px;
classDef backend stroke:#777,stroke-width:2px,stroke-dasharray:4 3;
classDef runtime stroke:#4caf7d,stroke-width:2.5px;
classDef policy stroke:#c89b3c,stroke-width:2.5px;
Application-Owned Configuration
The queue's real config builder receives one framework-generated node view plus all peer views and returns the value understood by the queue binary:
fn build_cluster_node_config(
node: &ClusterNodeView,
peers: &[ClusterPeerView],
) -> Result<QueueNodeConfig, Error> {
Ok(QueueNodeConfig {
node_id: node.index() as u64,
http_port: node.network_port(),
peers: peers
.iter()
.map(|peer| QueuePeerInfo {
node_id: peer.index() as u64,
http_address: peer.authority(),
})
.collect(),
sync_interval_ms: 500,
})
}
The local adapter then says where the binary comes from, how to serialize that typed config, and which port is its API:
fn local_process_spec() -> LocalProcessSpec {
LocalProcessSpec::new("QUEUE_NODE_BIN")
.with_binary_provider(queue_binary_provider())
.with_rust_log("queue_node=info")
}
fn render_local_config(config: &QueueNodeConfig) -> Result<Vec<u8>, DynError> {
yaml_node_config(config)
}
fn http_api_port(config: &QueueNodeConfig) -> u16 {
config.http_port
}
Scenario deployment, ManualCluster, and uniform child clusters all call these same application functions.
Delivering Configuration: Local Files and cfgsync
TF renders the same per-node artifacts for each backend. The local backend writes them directly, while container backends deliver them through cfgsync:
flowchart TB
C["typed per-node configuration"]:::app --> A["rendered per-node artifacts<br/><small>config file + any additional files</small>"]:::artifact
A --> L["local backend<br/><small>write directly into the node working directory</small>"]:::local
A --> S["container backends<br/><small>serve artifacts through cfgsync</small>"]:::container
S --> F["cfgsync client in each container<br/><small>register · fetch · write files</small>"]:::container
L --> N["start node binary"]:::process
F --> N
classDef app stroke:#9b6dd6,stroke-width:2.5px;
classDef artifact stroke:#777,stroke-width:2px,stroke-dasharray:4 3;
classDef local stroke:#4a90d9,stroke-width:2.5px;
classDef container stroke:#c89b3c,stroke-width:2.5px;
classDef process stroke:#e08a3c,stroke-width:2.5px;
Locally, TF writes files into the process working directory. Compose and Kubernetes nodes cannot see that host directory. For those backends, a cfgsync server holds each node's artifacts, and a client inside the container fetches and writes them before executing the node. cfgsync only transports generated configuration. It does not preserve application state or create snapshots.
Test-Side Changes for One Start
Tests normally keep the generated ports and peers and patch only the behavior they care about. The Section 5 manual-cluster test really starts its second node with a faster synchronization interval:
let node1 = cluster
.start_node_with(
"node-1",
StartNodeOptions::<QueueEnv>::default().create_patch(|mut config| {
config.sync_interval_ms = 50;
Ok(config)
}),
)
.await?
.client;
Use config_override only when the test intends to replace the complete generated config. config_patch preserves framework-assigned values unless the callback deliberately changes them.
Deployment Policy
The node config is passed to the application binary. DeploymentPolicy separately controls TF's readiness checks, retry behavior, cleanup, and artifact retention:
let policy = DeploymentPolicy {
readiness_enabled: true,
readiness_requirement: HttpReadinessRequirement::AtLeast(2),
retry_policy: Some(RetryPolicy::new(
5,
Duration::from_millis(500),
Duration::from_secs(5),
)),
cleanup_policy: CleanupPolicy::new(true),
..DeploymentPolicy::default()
};
For the primary scenario cluster, set this through .with_deployment_policy(policy). A child cluster created by an AppDeployment carries policy on its ClusterRequest. deploy_local_cluster(...) uses the default policy.
SLOW_TEST_ENV doubles timeouts
Retrythe local backend respawns a failed cluster attempt with backoff; Compose and Kubernetes currently do not repeat deployment
Artifactslocal files live in node working directories; container backends receive rendered config through cfgsync
Retentionpreserve_artifacts, TF_KEEP_LOGS, or a panic keep local working directories for post-mortems
Next: what happens to node state during restart and restore.
Ports, Peers, Node Config, and Readiness · Static Artifacts and cfgsync · Readiness, Retry, and Artifact Preservation · Diagnostics and Retained Artifacts
7 · State and Reproducibility
working directories, snapshot input, config changes, and deterministic deployment seeds.
A restart keeps the node's working directory. A snapshot starts a new node from copied state.
persist_dirchoose a stable working-directory locationsnapshot_dirsaved state copied in at spawnordinary restart: same working directory · restore: new working directory seeded from a snapshot
config_override replaces the generated per-node config; config_patch transforms it
Seedswith_deployment_seed feeds deterministic deployment providers
Next: connecting the same test to clusters that TF did not start.
Persistence, Snapshots, and Recovery Testing · Seeds and Reproducibility
8 · Cluster Sources and Ownership
managed, attached, and external clusters use one request API but provide different levels of control.
You request every cluster through the same API. Available control depends on whether TF started it or connected to it.
deploy_cluster(ClusterRequest::…) returns clients in all three modes; only managed clusters always provide full lifecycle control
The job-stack example asks TF to start both clusters. A test can instead connect to an existing deployment. deploy_cluster handles all three cases and returns node clients for each one. Full start, stop, and restart control is guaranteed only when TF manages the nodes.
let cluster = ctx.deploy_cluster(ClusterRequest::managed(deployment)).await?;
let attached = ctx.deploy_cluster(ClusterRequest::attached(existing)).await?;
let external = ctx.deploy_cluster(ClusterRequest::external(endpoints)).await?;
| Managed | Attached | External | |
|---|---|---|---|
| Clients | ✓ | ✓ | ✓ |
| Node control | ✓ | per backend | — |
| Readiness waits | ✓ | ✓ | — |
| Torn down by the framework | ✓ | — | — |
The scenario builder exposes the same modes through with_existing_cluster, with_external_nodes, and with_external_only_nodes. Workloads and expectations use node clients, so they do not need to change when a test moves from a locally managed cluster to an existing deployment.
Next: how TF finds the binaries it has been asked to start.
Shared Cluster Provisioning · Existing and External Clusters
9 · Binary Resolution
the worker_binary_provider() call inside line ②.
A binary provider returns an executable path. Providers can try an override, a local build, or a download in order.
the worker_binary_provider() call inside line ② — FallbackBinaryProvider tries sources in order, with a cache and cross-process locking
Every process TF starts needs an executable path. A binary provider can return an explicit path, read one from an environment variable, build the binary locally, or download an artifact. A fallback provider tries several providers in order.
The job worker's real provider chain tries an env var override and falls back to a local build:
FallbackBinaryProvider::new([
Arc::new(EnvBinaryProvider::new("MULTI_APP_JOB_WORKER_BIN")),
Arc::new(BuildBinaryProvider {
command: BuildCommand::new("cargo")
.with_args(["build", "-p", "multi-app-job-worker", "--bin", "multi-app-job-worker"]),
output_path: "target/debug/multi-app-job-worker".into(),
working_dir: Some(workspace),
lock_dir: None,
}),
])
The available providers are explicit path, environment variable, local build, and checksummed download with post-processing. FallbackBinaryProvider chains them, with a resolution cache and cross-process locking.
Next: selecting the local, Compose, or Kubernetes backend.
10 · Deployment Backends
line ⑥: local, Compose, and Kubernetes deployment.
Uniform scenarios can run locally, with Compose, or on Kubernetes. Backend capabilities currently differ.
line ⑥ picks the backend; app composition is local-only today
Line ⑥ selects the local backend. Uniform scenarios can also use the Compose and Kubernetes deployers. The table lists the deployment and control features currently implemented by each backend.
| Local | Compose | Kubernetes | |
|---|---|---|---|
| Node startup | processes + temp dirs | generated compose file | Helm chart + values |
| Config delivery | filesystem | cfgsync artifacts | cfgsync artifacts |
| Node control | full | restart | manual mode only |
| App composition | ✓ | — | — |
| Attach / external | external nodes | ✓ | ✓ |
App composition currently runs only on the local backend. Uniform scenarios run on all three. Local working directories are temporary and removed after a successful run unless TF_KEEP_LOGS or preserve_artifacts is set. They are also retained after a panic.
Next: reading changing application state during a test.
Capability Matrix · Local · Compose · Kubernetes · Diagnostics
11 · Observability
continuous state capture for tests, plus external metrics, logs, and traces.
Tests read application state through observation. Metrics, logs, and traces are exported through telemetry.
Observer polls on a cadence — snapshots · history · subscriptionsobservation is a runtime extension; telemetry is a backend capability
Continuous observation
An Observer polls application state on a cadence; tests read latest_snapshot(), history(), or subscribe() from an ObservationHandle. Sources can be dynamic, re-queried as nodes come and go.
Telemetry
Metrics, logs, tracing, and Grafana/OTLP endpoints are configured through the observability capability and environment variables. They serve external monitoring, not test logic.
Continuous observation is implemented as a runtime extension (section 4); telemetry is a backend capability configured on the scenario, not an extension.
Next: matching common test cases to the APIs covered above.
Continuous Observation · Telemetry and External Observability · Runtime Extensions
12 · Choosing What to Test
common test cases and the APIs normally used for them.
The table below shows which APIs are normally used for each kind of test.
| Test kind | Framework tools | Read |
|---|---|---|
| Convergence / consistency | traffic workload + expectation polling every node client | Workloads, Expectations |
| Recovery across a restart | restart_node or process restart(); working directories survive restarts |
Imperative Control, Persistence |
| Restore from saved state | snapshot_dir seeding + an expectation on the restored data |
Persistence |
| Role failover | find the role through observation, restart it via node control, expect a new holder | Chaos, Observation |
| Chaos under load | traffic workload + RandomRestartWorkload / the chaos builder in one scenario |
Chaos |
| Load / soak | bounded traffic workloads paced across the run window | Workloads |
| Deployment and config validation | the same uniform scenario per backend, plus readiness policy | Backends, Config |
| Behavior of a third-party binary | LocalProcessApp + LaunchSpec around the unmodified executable |
Section 3 |
| Against a live network | attached or external sources with unchanged workloads and expectations | Sources |