Continuous Observation

The observation runtime polls application state in the background and stores snapshots, histories, and event streams for workloads and expectations. It provides typed state inside the test process rather than external telemetry.

Shared Polling Runtime

Chaos and convergence tests repeatedly query state such as the current leader, whether every node has seen a key, or what changed after a restart. The observation runtime (testing-framework/core/src/observation/) runs one background polling task with shared error and staleness tracking. Workloads and expectations read the stored state.

Telemetry exports metrics, logs, and traces to external endpoints. Observation instead keeps typed application state inside the test and makes it synchronously queryable during the run.

The Observer Trait

An application defines how to poll and interpret its state; the runtime schedules the polling:

#[async_trait]
pub trait Observer: Send + Sync + 'static {
    type Source: Clone + Send + Sync + 'static;   // app-owned source handle
    type State: Send + Sync + 'static;            // retained materialized state
    type Snapshot: Clone + Send + Sync + 'static; // current view
    type Event: Clone + Send + Sync + 'static;    // delta emitted per cycle

    async fn init(&self, sources: &[ObservedSource<Self::Source>]) -> Result<Self::State, DynError>;

    async fn poll(
        &self,
        sources: &[ObservedSource<Self::Source>],
        state: &mut Self::State,
    ) -> Result<Vec<Self::Event>, DynError>;

    fn snapshot(&self, state: &Self::State) -> Self::Snapshot;
}

Each cycle the runtime refreshes the source set, calls poll to advance State and collect delta Events, then derives a Snapshot from the state. ObservedSource<S> is just a name plus the app-owned source value (ObservedSource::new(name, source)), typically a node client.

Sources are re-queried every cycle through SourceProvider<S>:

#[async_trait]
pub trait SourceProvider<S>: Send + Sync + 'static {
    async fn sources(&self) -> Result<Vec<ObservedSource<S>>, DynError>;
}

StaticSourceProvider::new(sources) covers the common fixed-cluster case. A custom provider makes sources dynamic: it can return a different set each cycle, which lets observation stay correct across node restarts. SourceProviderFactory<E, S> builds the provider once node clients exist; any closure Fn(&E::Deployment, NodeClients<E>) -> Result<BoxedSourceProvider<S>, DynError> qualifies.

Plugging Into a Scenario

ObservationExtensionFactory<E, O> is a runtime extension factory: at prepare time it builds the source provider, starts the runtime, and stores the read handle in the RunContext (background task registered for abort-on-teardown via PreparedRuntimeExtension::from_task). The builder has convenience methods for it (CoreBuilderExt):

// Clonable observer:
builder.with_observer(MyObserver, my_source_provider_fn, ObservationConfig::default())
// Observer built lazily per run:
builder.with_observer_factory(|| MyObserver::new(), my_source_provider_fn, config)

ObservationConfig has two fields: interval (time between cycles, default 1 s, must be non-zero) and history_limit (retained non-empty event batches, default 64).

Outside scenarios, for example around a ManualCluster, start it directly: ObservationRuntime::start(provider, observer, config), then handle(), into_parts() (handle + JoinHandle), or abort(). Dropping the runtime aborts the task.

Reading: the ObservationHandle

Workloads and expectations retrieve the handle by type and read four things:

Method	Returns
`latest_snapshot()`	`Option<ObservationSnapshot<O::Snapshot>>` — most recent successful view
`history()`	Retained non-empty `ObservationBatch<O::Event>`s, oldest first, bounded by `history_limit`
`last_error()`	`Option<ObservationFailure>` — the most recent failed cycle
`subscribe()`	`broadcast::Receiver` of future non-empty batches

Snapshots vs batches vs events: a snapshot is the whole current view (cycle, observed_at, source_count, value); an event is one delta discovered during a cycle; a batch groups the events of one cycle. Cycles that produce no events produce no batch. history() and subscribe() only ever see non-empty batches, while latest_snapshot() is refreshed on every successful cycle.

Freshness and failures. On a failed cycle, the runtime records an ObservationFailure (with stage: SourceRefresh if source discovery failed, stage: Poll if the observer failed) and retains the last successful snapshot. The next successful cycle clears last_error. To check staleness, compare snapshot.cycle or observed_at across reads, and inspect last_error() when a wait times out; it usually names the source that stopped answering.

Worked Example: the OpenRaft Cluster Observer

examples/openraft_kv/testing/integration/src/observation.rs observes a Raft cluster. State and snapshot are the same type (the latest per-node states plus any per-source failures), and no delta events are emitted (Event = ()):

#[derive(Clone, Debug, Default)]
pub struct OpenRaftClusterObserver;

#[async_trait]
impl Observer for OpenRaftClusterObserver {
    type Source = OpenRaftKvClient;
    type State = OpenRaftClusterSnapshot;
    type Snapshot = OpenRaftClusterSnapshot;
    type Event = ();

    async fn init(&self, sources: &[ObservedSource<Self::Source>]) -> Result<Self::State, DynError> {
        Ok(capture_cluster_snapshot(sources).await)
    }

    async fn poll(
        &self,
        sources: &[ObservedSource<Self::Source>],
        state: &mut Self::State,
    ) -> Result<Vec<Self::Event>, DynError> {
        *state = capture_cluster_snapshot(sources).await;
        Ok(Vec::new())
    }

    fn snapshot(&self, state: &Self::State) -> Self::Snapshot {
        state.clone()
    }
}

capture_cluster_snapshot queries each source's /state endpoint and records per-node errors as OpenRaftSourceFailure values instead of failing the cycle. A node restarting therefore appears as a named failure inside the snapshot. The snapshot type provides agreed_leader(different_from), all_voters_match(...), all_kv_match(...), and summary() for timeout messages.

Two source providers accompany it:

// Fixed: scenario runs, sources from the run's node clients.
pub fn openraft_cluster_source_provider(
    _deployment: &<OpenRaftKvEnv as Application>::Deployment,
    node_clients: NodeClients<OpenRaftKvEnv>,
) -> Result<BoxedSourceProvider<OpenRaftKvClient>, DynError> {
    Ok(Box::new(StaticSourceProvider::new(named_sources(node_clients.snapshot()))))
}

and OpenRaftManualClusterSourceProvider, a dynamic provider that re-resolves clients from a ManualCluster on every cycle so observation follows manual restarts. The scenario builder wires the fixed one in via with_observer(OpenRaftClusterObserver, openraft_cluster_source_provider, OpenRaftClusterObserver::config()).

The failover scenario waits on this observed state:

let observer = ctx.require_extension::<ObservationHandle<OpenRaftClusterObserver>>()?;
let leader = wait_for_observed_leader(&observer, timeout, None).await?;

External example: logos-blockchain's BlockFeed is an adopter-side analog of this pattern: an observer in its own repository materializes block records, per-node head snapshots, and transaction statistics, using the same Observer/ObservationHandle mechanism.

7.9 KiB Raw Blame History