Kubernetes Deployer

K8sDeployer installs each scenario as a Helm release in a throwaway namespace and reaches nodes through NodePorts or port-forwards.

The k8s deployer lives in the testing-framework-runner-k8s crate. It talks to whatever cluster your current kubeconfig context points at (kube::Client::try_default()), installs a Helm release, waits for the workloads, and builds node clients against externally reachable ports.

use kvstore_runtime_ext::KvK8sDeployer; // = K8sDeployer<KvEnv>
use testing_framework_core::scenario::Deployer;
use testing_framework_runner_k8s::K8sRunnerError;

let deployer = KvK8sDeployer::new();
let runner = match deployer.deploy(&scenario).await {
    Ok(runner) => runner,
    Err(K8sRunnerError::ClientInit { .. }) => return Ok(()), // no cluster available
    Err(error) => return Err(error.into()),
};
runner.run(&mut scenario).await?;

Run the demonstration binary with cargo run -p kvstore-examples --bin kvstore_k8s_convergence.

Charts and Values

The environment trait K8sDeployEnv produces installable assets via prepare_assets, returning a PreparedK8sStack. Two asset shapes exist:

Generated single-template charts. Apps implementing K8sBinaryApp get the standard shape for free: render_binary_config_node_manifest renders one ConfigMap (the serialized node config), one Deployment (single replica, --config arg, config mounted from the ConfigMap), and one NodePort Service per node, then render_manifest_chart_assets wraps them in a minimal chart (RenderedHelmChartAssets).
Real chart directories. NodeRuntimeSpec builds RunnerChartValues (node image, pull policy, fullname override, asset mount layout, node group, optional shared bootstrap service with cfgsync configs and start scripts) and a HelmReleaseBundle with --set values and --set-file entries for start scripts and bootstrap configs. RunnerAssetLayout fixes where bootstrap configs and runner scripts land inside the chart's mount path.

Node images are resolved from env vars: for a conventional BinaryConfigK8sSpec the primary override is <APP>_K8S_IMAGE, the fallback <APP>_IMAGE, and the default <binary-name>:local with imagePullPolicy: IfNotPresent.

Each run installs into fresh identifiers: namespace tf-testnet-<timestamp>-<pid>, release tf-runner (override via K8sDeployEnv::cluster_identifiers).

Lifecycle Waits

After helm install, the deployer waits in stages:

Deployment readiness: each node Deployment must report ready replicas (timeout K8S_RUNNER_DEPLOYMENT_TIMEOUT_SECS, default 180 s).
Port discovery: each node Service must have allocated NodePorts for the API and auxiliary ports declared by collect_port_specs.
HTTP readiness: nodes are probed over their NodePorts at node_readiness_path(). The probe host is K8S_RUNNER_NODE_HOST if set, else KUBERNETES_SERVICE_HOST, else 127.0.0.1. If NodePort probing fails (common when the cluster's node IPs are not routable from the runner), the deployer transparently falls back to kubectl port-forward per service and probes over 127.0.0.1.
Policy-gated cluster readiness: a final probe pass controlled by DeploymentPolicy.readiness_enabled / readiness_requirement and the deployer's with_readiness(bool) switch. See Readiness, Retry, and Artifact Preservation.

HTTP wait tuning: K8S_RUNNER_HTTP_TIMEOUT_SECS (default 240), K8S_RUNNER_HTTP_PROBE_TIMEOUT_SECS (default 30), K8S_RUNNER_HTTP_POLL_INTERVAL_SECS (default 1).

Node Control

The Kubernetes deployer does not wire a node-control handle into managed scenario deployments. A scenario built with with_node_control() compiles against this backend, but runtime restart calls fail. For node lifecycle control on Kubernetes, use the Kubernetes ManualCluster below.

Manual Mode

K8sDeployer::manual_cluster_from_descriptors(descriptors) (or ManualCluster::from_topology) installs the same Helm release, discovers every node's ports, then scales all node Deployments to zero so your code decides when each node starts:

let deployer = OpenRaftKvK8sDeployer::new();
let cluster = deployer
    .manual_cluster_from_descriptors(OpenRaftKvTopology::new(3))
    .await?;

cluster.start_node("node-0").await?;
cluster.start_node("node-1").await?;
cluster.wait_network_ready().await?;
cluster.restart_node("node-0").await?;
cluster.stop_all();

Start, stop, and restart are implemented by patching Deployment replicas between 0 and 1 and waiting for the rollout. start_node_with accepts StartNodeOptions, with two k8s-specific limits: persist_dir / snapshot_dir are rejected, and peer selection or config overrides require the environment to implement cfgsync override artifacts (cfgsync_service + build_cfgsync_override_artifacts); the override is pushed to the in-cluster cfgsync service through a temporary port-forward before the node starts. The failover demonstration uses this path end to end: cargo run -p openraft-kv-examples --bin openraft_kv_k8s_failover. Contrast with the declarative local variant in ManualCluster: Imperative Node Control.

Attaching to an Existing Cluster

Existing-cluster mode is supported with a k8s descriptor: with_existing_cluster(ExistingCluster::for_k8s_selector("app.kubernetes.io/instance=tf-runner")) (optionally namespaced with for_k8s_selector_in_namespace). Services matching the selector are listed, each service's single TCP NodePort (or the port named http/api) becomes the node endpoint, and clients are built via Application::external_node_client. deploy_with_metadata returns K8sDeploymentMetadata (namespace + label selector) so a later scenario can attach to the stack this one installed. See Existing and External Clusters.

Observability and Cleanup

Observability inputs resolve exactly as in compose (LOGOS_BLOCKCHAIN_* env vars merged with the scenario capability), and TESTNET_PRINT_ENDPOINTS prints Prometheus/Grafana and per-node pprof endpoints. Cleanup uninstalls the Helm release and deletes the namespace (Kubernetes API first, kubectl delete namespace fallback), after killing any port-forward processes. Set K8S_RUNNER_PRESERVE to keep the release and namespace for inspection.

Requirements recap:

Requirement	Why
Reachable cluster in current kubeconfig context	`Client::try_default()` at deploy time
`helm` on PATH	Release install/uninstall
`kubectl` on PATH	Port-forward fallback, namespace-delete fallback
Node images loadable by the cluster	`<APP>_K8S_IMAGE` / `<APP>_IMAGE` / `<binary>:local`

6.6 KiB Raw Permalink Blame History