diff --git a/README.md b/README.md index a3aea03..7ab6e57 100644 --- a/README.md +++ b/README.md @@ -1,155 +1,184 @@ -# Logos Blockchain Testing Framework +# Logos Testing Framework -A comprehensive testing framework for the Logos blockchain implementation, providing declarative scenario definitions, multiple deployment backends, and production-grade observability. +A Rust framework for system-level tests of networked applications. It can start +local processes and node clusters, deploy uniform clusters with Docker Compose +or Kubernetes, connect to existing deployments, run test activity, evaluate +outcomes, and clean up the resources it started. -## Overview +The framework is application-agnostic. Application repositories provide their +node configuration, clients, readiness checks, and backend-specific launch +settings. -This framework enables you to define, deploy, and execute integration tests for Logos blockchain scenarios across different environments—from local processes to containerized Kubernetes deployments—using a unified API. +## Start Here -**Key capabilities:** -- **Declarative scenario model** — Define topology, workloads, and success criteria using a fluent builder API -- **Multiple deployment backends** — Local processes, Docker Compose, or Kubernetes -- **Built-in workloads** — Transaction injection, DA (Data Availability) traffic, and restart-based chaos (requires node control; compose runner supported) -- **Observability-first** — Integrated Prometheus metrics, structured logging, and OpenTelemetry support -- **Production-ready** — Used in CI/CD pipelines with reproducible containerized environments - -## Quick Start - -### Prerequisites - -- Rust toolchain (nightly) -- `versions.env` file at repository root (included) -- For Docker Compose: Docker daemon -- For Kubernetes: Cluster access, `kubectl`, and `helm` - -### Run Your First Test +The workspace pins its Rust toolchain in `rust-toolchain.toml`. The local +kvstore example needs no additional setup; its node binary is built on first +use: ```bash -# Host mode (local processes) - fastest iteration -scripts/run/run-examples.sh -t 60 -v 1 -e 1 host - -# Compose mode (Docker containers) - reproducible environment -scripts/run/run-examples.sh -t 60 -v 1 -e 1 compose - -# K8s mode (Kubernetes cluster) - production-like fidelity -scripts/run/run-examples.sh -t 60 -v 1 -e 1 k8s +cargo run -p kvstore-examples --bin kvstore_app_host_convergence ``` -The script handles circuit setup, binary building, image preparation, and scenario execution automatically. +This starts three local node processes, writes data, restarts a node, checks +the result, and removes the processes and temporary directories. + +The composed-application example runs as integration tests: + +```bash +cargo test -p multi-app-e2e +``` + +It covers a queue cluster, worker process, and result-store cluster through a +runner-driven scenario and direct imperative tests. + +For Compose examples, run a Docker daemon and prepare the image named by the +example; the Compose deployer checks that it exists locally but does not build +or pull it. Kubernetes examples require a reachable cluster, `kubectl`, Helm, +and a node image available to that cluster. + +See [Quickstart](book/src/quickstart.md) and +[Running the Examples](book/src/running-examples.md) for the complete commands +and requirements. + +## Ways to Write Tests + +### Scenarios + +A scenario records the system to deploy, workloads to run, expectations to +evaluate, runtime limits, and enabled capabilities. The runner performs +deployment, readiness checks, concurrent workloads, cooldown, expectation +evaluation, and cleanup. + +```rust +let mut scenario = AppHost::scenario() + .with_app(KvLocalApp::nodes(3)) + .with_workload(KvAppHostConvergence::new(3)) + .with_run_duration(Duration::from_secs(5)) + .build()?; + +let runner = AppHostLocalDeployer::default() + .deploy(&scenario) + .await?; + +runner.run(&mut scenario).await?; +``` + +### Imperative Tests + +`ManualCluster` gives ordinary Rust or a BDD harness direct control of one +uniform cluster. Tests can start, stop, restart, and wait for nodes without +using workloads or expectations. + +A composed application can also be deployed directly through `DeployContext` +when test code needs to control the complete stack step by step. + +### Composed Applications + +`AppDeployment` describes how application components are started and connected. +`AppHost` runs one root deployment as part of a scenario. Child deployments can +start uniform clusters through `LocalAppCluster` and standalone binaries through +`LocalProcessApp`, then expose typed handles to workloads and expectations. + +App composition currently runs only with the local process deployer. Compose +and Kubernetes support uniform application clusters, not an `AppDeployment` +tree containing several application types. + +### Existing Deployments + +Scenarios can use managed nodes, attach to an existing Compose project or +Kubernetes deployment, or construct clients for external endpoints. Available +node control depends on the selected source and backend. + +## Deployment Backends + +Uniform scenarios use the same scenario runtime on all three backends. Each +application supplies a thin backend adapter containing details such as the +binary or image, config location, and service ports. + +| Capability | Local | Compose | Kubernetes | +|---|---|---|---| +| Uniform managed scenarios | Yes | Yes | Yes | +| Managed node control | Start, stop, restart | Restart | Use Kubernetes `ManualCluster` | +| Existing clusters | No | Compose project or services | Label selector and namespace | +| External endpoints | Yes | Yes | Yes | +| `AppHost` composition | Yes | No | No | +| Config delivery | Files in node working directories | cfgsync | cfgsync | + +The local deployer resolves executable paths through path, environment, build, +or download providers. Compose generates a project and services. Kubernetes +installs a Helm release in a per-run namespace. Container backends deliver +generated per-node configuration and other static files through cfgsync. + +See the [Capability Matrix](book/src/capability-matrix.md), +[Local Deployer](book/src/deployer-local.md), +[Compose Deployer](book/src/deployer-compose.md), and +[Kubernetes Deployer](book/src/deployer-k8s.md). + +## Repository Layout + +```text +testing-framework/ +├── core/ scenario runtime, topology, provisioning, control +├── app/ AppHost, AppDeployment, typed handles, composition +└── deployers/ + ├── local/ local processes and binary providers + ├── compose/ generated Docker Compose projects + └── k8s/ Helm and Kubernetes deployment + +cfgsync/ +├── artifacts/ backend-neutral per-node files +├── core/ protocol, server, client, and rendering +├── adapter/ application config materialization +└── runtime/ cfgsync server and client binaries + +examples/ self-contained example applications and tests +book/ mdBook source and presentation theme +scripts/ checks, cleanup, and observability helpers +``` + +The example applications include uniform clusters, composed stacks, consensus +failover, queues, WebSocket pub/sub, metrics, and unmodified NATS and Redis +servers. See [examples/README.md](examples/README.md) for the recommended entry +points. ## Documentation -**Complete documentation available at:** https://logos-blockchain.github.io/logos-blockchain-testing/ +- [The Framework in Brief](book/src/framework-in-brief.md) +- [Quickstart](book/src/quickstart.md) +- [Application and Environment Model](book/src/application-model.md) +- [Composing Applications](book/src/part-ii.md) +- [Scenario Runtime](book/src/part-iii.md) +- [Uniform Clusters and Configuration](book/src/part-iv.md) +- [Deployers and Sources](book/src/part-v.md) +- [Environment Variables](book/src/environment-variables.md) +- [Troubleshooting](book/src/troubleshooting.md) -### Essential Guides +Published book: -| Topic | Link | -|-------|------| -| **Getting Started** | [Quickstart Guide](https://logos-blockchain.github.io/logos-blockchain-testing/quickstart.html) | -| **Core Concepts** | [Testing Philosophy](https://logos-blockchain.github.io/logos-blockchain-testing/testing-philosophy.html) | -| **Examples** | [Basic](https://logos-blockchain.github.io/logos-blockchain-testing/examples.html) \| [Advanced](https://logos-blockchain.github.io/logos-blockchain-testing/examples-advanced.html) | -| **Deployment Options** | [Runners Overview](https://logos-blockchain.github.io/logos-blockchain-testing/runners.html) | -| **API Reference** | [Builder API](https://logos-blockchain.github.io/logos-blockchain-testing/dsl-cheat-sheet.html) | -| **Operations** | [Setup & Configuration](https://logos-blockchain.github.io/logos-blockchain-testing/operations.html) | -| **Troubleshooting** | [Common Issues](https://logos-blockchain.github.io/logos-blockchain-testing/troubleshooting.html) | +Build and test it locally with: -## Repository Structure - -``` -logos-blockchain-testing/ -├── testing-framework/ # Core library crates -│ ├── core/ # Scenario model, runtime orchestration -│ ├── workflows/ # Workloads (tx, DA, chaos) and expectations -│ ├── configs/ # Node configuration builders -│ ├── runners/ # Deployment backends (local, compose, k8s) -│ └── assets/stack/ # Docker/K8s deployment assets -├── examples/ # Runnable demo binaries -│ └── src/bin/ # local_runner, compose_runner, k8s_runner -├── scripts/ # Helper utilities (run-examples.sh, build-bundle.sh) -└── book/ # Documentation sources (mdBook) +```bash +mdbook build book +mdbook test book ``` -## Architecture - -The framework follows a clear separation of concerns: - -**Scenario Definition** → **Topology Builder** → **Deployer** → **Runner** → **Workloads** → **Expectations** - -- **Scenario**: Declarative description of test intent (topology + workloads + success criteria) -- **Deployer**: Provisions infrastructure on chosen backend (host/compose/k8s) -- **Runner**: Orchestrates execution, manages lifecycle, collects observability -- **Workloads**: Generate traffic and conditions (transactions, DA blobs, chaos) -- **Expectations**: Evaluate success/failure based on observed behavior +Install `mdbook` first if it is not already available. ## Development -### Building the Documentation +Useful focused checks from the workspace root: ```bash -# Install mdBook -cargo install mdbook mdbook-mermaid - -# Build and serve locally -cd book && mdbook serve -# Open http://localhost:3000 +cargo fmt --all -- --check +cargo test -p testing-framework-core +cargo test -p testing-framework-app +cargo test -p multi-app-e2e +cargo clippy --all --all-targets --all-features -- -D warnings ``` -### Running Tests - -```bash -# Run framework unit tests -cargo test - -# Run integration examples -scripts/run/run-examples.sh -t 60 -v 2 -e 1 host -``` - -### Creating Prebuilt Bundles - -For compose/k8s deployments, you can create prebuilt bundles to speed up image builds: - -```bash -# Build Linux bundle (required for compose/k8s) -scripts/build/build-bundle.sh --platform linux - -# Use the bundle when building images -export LOGOS_BLOCKCHAIN_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz -scripts/build/build_test_image.sh -``` - -## Environment Variables - -Key environment variables for customization: - -| Variable | Purpose | Default | -|----------|---------|---------| -| `LOGOS_BLOCKCHAIN_TESTNET_IMAGE` | Docker image tag for compose/k8s | `logos-blockchain-testing:local` | -| `LOGOS_BLOCKCHAIN_DEMO_NODES` | Number of nodes | Varies by example | -| `LOGOS_BLOCKCHAIN_LOG_DIR` | Directory for persistent log files | (temporary) | -| `LOGOS_BLOCKCHAIN_LOG_LEVEL` | Logging verbosity | `info` | - -See [Operations Guide](https://logos-blockchain.github.io/logos-blockchain-testing/operations.html) for complete configuration reference. - -## CI/CD Integration - -The framework is designed for CI/CD pipelines: - -- **Host runner**: Fast smoke tests with minimal overhead -- **Compose runner**: Reproducible containerized environment with Prometheus -- **K8s runner**: Production-like cluster validation - -Example CI workflow: `.github/workflows/lint.yml` (see `compose_smoke` job) +The lint workflow also checks dependency policy with `cargo-deny`, unused +dependencies with `cargo-machete`, and TOML formatting with Taplo. ## License -This project is part of the Logos blockchain implementation. - -## Links - -- **Documentation**: https://logos-blockchain.github.io/logos-blockchain-testing/ -- **Logos Project**: https://github.com/logos-co -- **Logos Node (repo: logos-blockchain-node)**: https://github.com/logos-co/logos-blockchain-node - -## Support - -For issues, questions, or contributions, please refer to the [Troubleshooting Guide](https://logos-blockchain.github.io/logos-blockchain-testing/troubleshooting.html) or file an issue in this repository. +MIT OR Apache-2.0. diff --git a/book/COMPREHENSIVE_REPO_SYNC_REVIEW.md b/book/COMPREHENSIVE_REPO_SYNC_REVIEW.md deleted file mode 100644 index e90b979..0000000 --- a/book/COMPREHENSIVE_REPO_SYNC_REVIEW.md +++ /dev/null @@ -1,70 +0,0 @@ -# Book → Repo Sync Review (2025-12-20) - -Reviewed against `git rev-parse HEAD` at the time of writing, plus local working tree changes. - -## Checks Run - -- `mdbook build book` -- `mdbook test book` -- `cargo build -p doc-snippets` -- Verified `book/src/SUMMARY.md` covers all pages in `book/src/` (no orphaned pages) -- Verified all `scripts/...` paths referenced from the book exist -- Compared `NOMOS_*` environment variables used in `scripts/`, `testing-framework/`, and `examples/` vs. `book/src/environment-variables.md` - -## Findings / Fixes Applied - -- `book/src/environment-variables.md` was not a complete reference: it missed multiple `NOMOS_*` variables used by the repo (scripts + framework). Added the missing variables and corrected a misleading note about `RUST_LOG` vs node logging. -- `book/src/running-examples.md` “Quick Smoke Matrix” section didn’t reflect current `scripts/run/run-test-matrix.sh` flags. Added the commonly used options and clarified the relationship to `LOGOS_BLOCKCHAIN_SKIP_IMAGE_BUILD`. -- `book/src/part-iv.md` existed but was not in `book/src/SUMMARY.md`. Removed it so the rendered book doesn’t silently diverge from the filesystem. -- `mdbook test book` was failing because: - - Many Rust examples were written as ` ```rust` (doctested by default) but depend on workspace crates; they aren’t standalone doctest snippets. - - Several unlabeled code blocks (e.g. tree/log output) were treated as Rust by rustdoc. - - Updated code fences to ` ```rust,ignore` for non-standalone Rust examples and to ` ```text` for non-Rust output blocks so `mdbook test book` succeeds. - -## Pages Reviewed (No Skips) - -All pages under `book/src/` currently included by `book/src/SUMMARY.md`: - -- `annotated-tree.md` -- `api-levels.md` -- `architecture-overview.md` -- `authoring-scenarios.md` -- `best-practices.md` -- `chaos.md` -- `ci-integration.md` -- `custom-workload-example.md` -- `design-rationale.md` -- `dsl-cheat-sheet.md` -- `environment-variables.md` -- `examples-advanced.md` -- `examples.md` -- `extending.md` -- `faq.md` -- `glossary.md` -- `internal-crate-reference.md` -- `introduction.md` -- `logging-observability.md` -- `node-control.md` -- `operations-overview.md` -- `part-i.md` -- `part-ii.md` -- `part-iii.md` -- `part-v.md` -- `part-vi.md` -- `prerequisites.md` -- `project-context-primer.md` -- `quickstart.md` -- `runners.md` -- `running-examples.md` -- `running-scenarios.md` -- `scenario-builder-ext-patterns.md` -- `scenario-lifecycle.md` -- `scenario-model.md` -- `testing-philosophy.md` -- `topology-chaos.md` -- `troubleshooting.md` -- `usage-patterns.md` -- `what-you-will-learn.md` -- `workloads.md` -- `workspace-layout.md` - diff --git a/book/README.md b/book/README.md index ca17ccc..33c872b 100644 --- a/book/README.md +++ b/book/README.md @@ -1,533 +1,18 @@ -# Documentation Maintenance Guide +# Testing Framework Book -This guide helps maintainers keep the book synchronized with code changes. Use these checklists when modifying the framework. +The book is an mdBook rooted at `book/src/SUMMARY.md`. Its current editing conventions and verification checklist live in [`../docs/book-maintenance.md`](../docs/book-maintenance.md). -**Key Tool:** The `examples/doc-snippets/` crate contains compilable versions of code examples from the book. Always run `cargo build -p doc-snippets` after API changes to catch broken examples early. - -## Quick Reference: What to Update When - -| Change Type | Pages to Check | Estimated Time | -|-------------|----------------|----------------| -| API method renamed/changed | [API Changes](#api-changes) | 1-2 hours | -| New workload/expectation added | [New Features](#new-features) | 30 minutes | -| Environment variable added/changed | [Environment Variables](#environment-variables) | 15 minutes | -| Script path/interface changed | [Scripts & Tools](#scripts--tools) | 30 minutes | -| New runner/deployer added | [New Runner](#new-runner) | 2-3 hours | -| Trait signature changed | [Trait Changes](#trait-changes) | 1-2 hours | - ---- - -## Detailed Checklists - -### API Changes - -**When:** Builder API methods, trait methods, or core types change - -**Examples:** -- Rename: `.transactions_with()` → `.with_transactions()` -- New method: `.with_timeout()` -- Parameter change: `.validators(3)` → `.validators(count, config)` - -**Update these pages:** +Build and test it from the repository root: ```bash -# 1. Search for affected API usage -rg "old_method_name" book/src/ - -# 2. Update these files: -- [ ] src/dsl-cheat-sheet.md # Builder API reference (highest priority) -- [ ] src/quickstart.md # First example users see -- [ ] src/examples.md # 4 complete scenarios -- [ ] src/examples-advanced.md # 3 advanced scenarios -- [ ] src/introduction.md # "A Scenario in 20 Lines" example -- [ ] src/project-context-primer.md # Quick example section -- [ ] src/authoring-scenarios.md # Scenario patterns -- [ ] src/best-practices.md # Code organization examples -- [ ] src/custom-workload-example.md # Complete implementation -- [ ] src/extending.md # Trait implementation examples +mdbook build book +mdbook test book ``` -**Verification:** -```bash -# Compile doc-snippets to catch API breakage -cargo build -p doc-snippets - -# Check if book links are valid -mdbook test -``` - ---- - -### New Features - -#### New Workload or Expectation - -**When:** Adding a new traffic generator or success criterion - -**Examples:** -- New workload: `MemoryPressureWorkload` -- New expectation: `ExpectZeroDroppedTransactions` - -**Update these pages:** +Preview it locally with: ```bash -- [ ] src/workloads.md # Add to built-in workloads section -- [ ] src/dsl-cheat-sheet.md # Add DSL helper if provided -- [ ] src/examples-advanced.md # Consider adding example usage -- [ ] src/glossary.md # Add term definition -- [ ] src/internal-crate-reference.md # Document crate location +mdbook serve book ``` -**Optional (if significant feature):** -```bash -- [ ] src/what-you-will-learn.md # Add to learning outcomes -- [ ] src/best-practices.md # Add usage guidance -``` - -#### New Runner/Deployer - -**When:** Adding support for a new deployment target (e.g., AWS ECS) - -**Update these pages:** - -```bash -# Core documentation -- [ ] src/runners.md # Add to comparison table and decision guide -- [ ] src/operations-overview.md # Update runner-agnostic matrix -- [ ] src/architecture-overview.md # Update deployer list and diagram -- [ ] src/running-examples.md # Add runner-specific section - -# Reference pages -- [ ] src/dsl-cheat-sheet.md # Add deployer import/usage -- [ ] src/internal-crate-reference.md # Document new crate -- [ ] src/glossary.md # Add runner type definition - -# Potentially affected -- [ ] src/ci-integration.md # Add CI example if applicable -- [ ] src/troubleshooting.md # Add common issues -- [ ] src/faq.md # Add FAQ entries -``` - -#### New Topology Helper - -**When:** Adding topology generation helpers (e.g., `.network_mesh()`) - -**Update these pages:** - -```bash -- [ ] src/dsl-cheat-sheet.md # Add to topology section -- [ ] src/authoring-scenarios.md # Add usage pattern -- [ ] src/topology-chaos.md # Add topology description -- [ ] src/examples.md # Consider adding example -``` - ---- - -### Trait Changes - -**When:** Core trait signatures change (breaking changes) - -**Examples:** -- `Workload::init()` adds new parameter -- `Expectation::evaluate()` changes return type -- `Deployer::deploy()` signature update - -**Update these pages:** - -```bash -# Critical - these show full trait definitions -- [ ] src/extending.md # Complete trait outlines (6+ examples) -- [ ] src/custom-workload-example.md # Full implementation example -- [ ] src/scenario-model.md # Core model documentation - -# Important - these reference traits -- [ ] src/api-levels.md # Trait usage patterns -- [ ] src/architecture-overview.md # Extension points diagram -- [ ] src/internal-crate-reference.md # Trait locations -``` - -**Verification:** -```bash -# Ensure trait examples would compile -cargo doc --no-deps --document-private-items -``` - ---- - -### Environment Variables - -**When:** New environment variable added, changed, or removed - -**Examples:** -- New: `LOGOS_BLOCKCHAIN_NEW_FEATURE_ENABLED` -- Changed: `LOGOS_BLOCKCHAIN_LOG_LEVEL` accepts new values -- Deprecated: `OLD_FEATURE_FLAG` - -**Update these pages:** - -```bash -# Primary location (single source of truth) -- [ ] src/environment-variables.md # Add to appropriate category table - -# Secondary mentions -- [ ] src/prerequisites.md # If affects setup -- [ ] src/running-examples.md # If affects runner usage -- [ ] src/troubleshooting.md # If commonly misconfigured -- [ ] src/glossary.md # If significant/commonly referenced -``` - -**Environment Variables Table Location:** -``` -src/environment-variables.md - ├─ Runner Configuration - ├─ Node Binary & Paths - ├─ Circuit Assets - ├─ Logging & Tracing - ├─ Observability & Metrics - ├─ Proof System - ├─ Docker & Images - ├─ Testing Behavior - └─ CI/CD -``` - ---- - -### Scripts & Tools - -**When:** Helper scripts move, rename, or change interface - -**Examples:** -- Script moved: `scripts/run-examples.sh` → `scripts/run/run-examples.sh` -- New script: `scripts/clean-all.sh` -- Interface change: `run-examples.sh` adds new required flag - -**Update these pages:** - -```bash -# High impact -- [ ] src/quickstart.md # Uses run-examples.sh prominently -- [ ] src/running-examples.md # Documents all scripts -- [ ] src/prerequisites.md # References setup scripts -- [ ] src/examples.md # Script recommendations -- [ ] src/examples-advanced.md # Script recommendations - -# Moderate impact -- [ ] src/ci-integration.md # May reference scripts in workflows -- [ ] src/troubleshooting.md # Cleanup scripts -- [ ] src/architecture-overview.md # Asset preparation scripts -``` - -**Find all script references:** -```bash -rg "scripts/" book/src/ --no-heading -``` - ---- - -### Operational Changes - -#### Docker Image Changes - -**When:** Image build process, tag names, or embedded assets change - -**Update these pages:** - -```bash -- [ ] src/prerequisites.md # Image build instructions -- [ ] src/runners.md # Compose/K8s prerequisites -- [ ] src/environment-variables.md # LOGOS_BLOCKCHAIN_TESTNET_IMAGE, LOGOS_BLOCKCHAIN_BINARIES_TAR -- [ ] src/architecture-overview.md # Assets and Images section -``` - -#### Observability Stack Changes - -**When:** Prometheus, Grafana, OTLP, or metrics configuration changes - -**Update these pages:** - -```bash -- [ ] src/logging-observability.md # Primary documentation -- [ ] src/environment-variables.md # LOGOS_BLOCKCHAIN_METRICS_*, LOGOS_BLOCKCHAIN_OTLP_* -- [ ] src/architecture-overview.md # Observability section -- [ ] src/runners.md # Runner observability support -``` - -#### CI/CD Changes - -**When:** CI workflow changes, new actions, or integration patterns - -**Update these pages:** - -```bash -- [ ] src/ci-integration.md # Complete workflow examples -- [ ] src/best-practices.md # CI recommendations -- [ ] src/operations-overview.md # CI mentioned in runner matrix -``` - ---- - -### Node Protocol Changes - -**When:** Changes to Logos blockchain protocol or node behavior - -**Examples:** -- New consensus parameter -- DA protocol change -- Network layer update - -**Update these pages:** - -```bash -# Context pages (high-level only) -- [ ] src/project-context-primer.md # Protocol overview -- [ ] src/glossary.md # Protocol terms -- [ ] src/faq.md # May need protocol updates - -# Usually NOT affected (framework is protocol-agnostic) -- Testing framework abstracts protocol details -- Only update if change affects testing methodology -``` - ---- - -### Crate Structure Changes - -**When:** Crate reorganization, renames, or new crates added - -**Examples:** -- New crate: `testing-framework-metrics` -- Crate rename: `runner-examples` → `examples` -- Module moved: `core::scenario` → `core::model` - -**Update these pages:** - -```bash -# Critical -- [ ] src/internal-crate-reference.md # Complete crate listing -- [ ] src/architecture-overview.md # Crate dependency diagram -- [ ] src/workspace-layout.md # Directory structure -- [ ] src/annotated-tree.md # File tree with annotations - -# Code examples (update imports) -- [ ] src/dsl-cheat-sheet.md # Import statements -- [ ] src/extending.md # use statements in examples -- [ ] src/custom-workload-example.md # Full imports -``` - -**Find all import statements:** -```bash -rg "^use testing_framework" book/src/ -``` - ---- - -## Testing Documentation Changes - -### Build the Book - -```bash -cd book -mdbook build - -# Output: ../target/book/ -``` - -### Test Documentation - -```bash -# Check for broken links -mdbook test - -# Preview locally -mdbook serve -# Open http://localhost:3000 -``` - -### Test Code Examples (Doc Snippets) - -**The `examples/doc-snippets/` crate contains compilable versions of code examples from the book.** - -This ensures examples stay synchronized with the actual API and don't break when code changes. - -**Why doc-snippets exist:** -- Code examples in the book (73 blocks across 18 files) can drift from reality -- Compilation failures catch API breakage immediately -- Single source of truth for code examples - -**Current coverage:** 40+ snippet files corresponding to examples in: -- `quickstart.md` (7 snippets) -- `examples.md` (4 scenarios) -- `examples-advanced.md` (3 scenarios) -- `dsl-cheat-sheet.md` (11 DSL examples) -- `custom-workload-example.md` (2 trait implementations) -- `internal-crate-reference.md` (6 extension examples) -- And more... - -**Testing snippets:** - -```bash -# Compile all doc snippets -cargo build -p doc-snippets - -# Run with full warnings -cargo build -p doc-snippets --all-features - -# Check during CI -cargo check -p doc-snippets -``` - -**When to update snippets:** - -1. **API method changed** → Update corresponding snippet file - ```bash - # Example: If .transactions_with() signature changes - # Update: examples/doc-snippets/src/examples_transaction_workload.rs - ``` - -2. **New code example added to book** → Create new snippet file - ```bash - # Example: Adding new topology pattern - # Create: examples/doc-snippets/src/topology_mesh_example.rs - ``` - -3. **Trait signature changed** → Update trait implementation snippets - ```bash - # Update: custom_workload_example_*.rs - # Update: internal_crate_reference_add_*.rs - ``` - -**Snippet naming convention:** -``` -book/src/examples.md → examples_*.rs -book/src/quickstart.md → quickstart_*.rs -book/src/dsl-cheat-sheet.md → dsl_cheat_sheet_*.rs -``` - -**Best practice:** -When updating code examples in markdown, update the corresponding snippet file first, verify it compiles, then copy to the book. This ensures examples are always valid. - -### Check for Common Issues - -```bash -# Find outdated API references -rg "old_deprecated_api" src/ - -# Find broken GitHub links -rg "github.com.*404" src/ - -# Find TODO/FIXME markers -rg "(TODO|FIXME|XXX)" src/ - -# Check for inconsistent terminology -rg "(Nomos node|nomos blockchain)" src/ # Should be "Logos node|Logos blockchain" -``` - ---- - -## Maintenance Schedule - -### On Every PR - -- [ ] Check if changes affect documented APIs -- [ ] Update relevant pages per checklist above -- [ ] Update corresponding doc-snippets if code examples changed -- [ ] Run `cargo build -p doc-snippets` to verify examples compile -- [ ] Build book to verify no broken links -- [ ] Verify code examples still make sense - -### Monthly - -- [ ] Review recent PRs for documentation impact -- [ ] Update environment variables table -- [ ] Check script references are current -- [ ] Verify GitHub source links are not 404 - -### Quarterly - -- [ ] Full audit of code examples against latest API -- [ ] Verify all doc-snippets still compile with latest dependencies -- [ ] Check for code examples in book that don't have corresponding snippets -- [ ] Review troubleshooting for new patterns -- [ ] Update FAQ with common questions -- [ ] Check all Mermaid diagrams render correctly - -### Major Release - -- [ ] Complete review of all technical content -- [ ] Verify all version-specific references -- [ ] Update "What You Will Learn" outcomes -- [ ] Add release notes for documentation changes - ---- - -## Content Organization - -### Stability Tiers (Change Frequency) - -**Stable (Rarely Change)** -- Part I — Foundations (philosophy, architecture, design rationale) -- Part VI — Appendix (glossary, FAQ, troubleshooting symptoms) -- Front matter (project context, introduction) - -**Semi-Stable (Occasional Changes)** -- Part II — User Guide (usage patterns, best practices, examples) -- Part V — Operations (prerequisites, CI, logging) - -**High Volatility (Frequent Changes)** -- API references (dsl-cheat-sheet.md, extending.md) -- Code examples (73 blocks across 18 files) -- Environment variables (50+ documented) -- Runner comparisons (features evolve) - -### Page Dependency Map - -**Core pages** (many other pages reference these): -- `dsl-cheat-sheet.md` ← Referenced by examples, quickstart, authoring -- `environment-variables.md` ← Referenced by operations, troubleshooting, runners -- `runners.md` ← Referenced by operations, quickstart, examples -- `glossary.md` ← Referenced throughout the book - -**When updating core pages, check for broken cross-references.** - ---- - -## Common Patterns - -### Adding a Code Example - -```markdown -# 1. Add the code block -```rust -use testing_framework_core::scenario::ScenarioBuilder; -// ... example code -``` - -# 2. Add context -**When to use:** [explain use case] - -# 3. Link to complete source (if applicable) -[View in source](https://github.com/logos-blockchain/logos-blockchain-testing/blob/master/examples/src/bin/example.rs) -``` - -### Adding a Cross-Reference - -```markdown -See [Environment Variables](environment-variables.md) for complete configuration reference. -``` - -### Adding a "When to Read" Callout - -```markdown -> **When should I read this?** [guidance on when this content is relevant] -``` - ---- - -## Contact & Questions - -When in doubt: -1. Check this README for guidance -2. Review recent similar changes in git history -3. Ask the team in technical documentation discussions - -**Remember:** Documentation quality directly impacts framework adoption and user success. Taking time to update docs properly is an investment in the project's future. +Generated output goes to `target/book/`. diff --git a/book/book.toml b/book/book.toml index bb2fe50..5444cbd 100644 --- a/book/book.toml +++ b/book/book.toml @@ -2,12 +2,13 @@ authors = ["Logos Testing"] language = "en" src = "src" -title = "Logos Blockchain Testing Framework Book" +title = "Testing Framework Book" [build] # Keep book output in target/ to avoid polluting the workspace root. build-dir = "../target/book" [output.html] -additional-js = ["theme/mermaid-init.js"] -default-theme = "light" +additional-css = ["theme/tour-v2.css"] +additional-js = ["theme/mermaid-init.js", "theme/tour-v2.js"] +default-theme = "light" diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index cc63a44..c42681b 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -1,44 +1,55 @@ # Summary -- [Project Context Primer](project-context-primer.md) -- [What You Will Learn](what-you-will-learn.md) + +- [Introduction](introduction.md) - [Quickstart](quickstart.md) -- [Part I — Foundations](part-i.md) - - [Introduction](introduction.md) - - [Architecture Overview](architecture-overview.md) - - [Testing Philosophy](testing-philosophy.md) - - [Scenario Lifecycle](scenario-lifecycle.md) - - [Design Rationale](design-rationale.md) -- [Part II — User Guide](part-ii.md) - - [Workspace Layout](workspace-layout.md) - - [Annotated Tree](annotated-tree.md) - - [Authoring Scenarios](authoring-scenarios.md) - - [Core Content: Workloads & Expectations](workloads.md) - - [Core Content: ScenarioBuilderExt Patterns](scenario-builder-ext-patterns.md) - - [Best Practices](best-practices.md) - - [Usage Patterns](usage-patterns.md) - - [Examples](examples.md) - - [Advanced & Artificial Examples](examples-advanced.md) - - [Running Scenarios](running-scenarios.md) - - [Runners](runners.md) - - [RunContext: BlockFeed & Node Control](node-control.md) - - [Chaos Workloads](chaos.md) - - [Topology & Chaos Patterns](topology-chaos.md) - - [Manual Clusters: Imperative Control](manual-cluster.md) -- [Part III — Developer Reference](part-iii.md) - - [Scenario Model (Developer Level)](scenario-model.md) - - [API Levels: Builder DSL vs. Direct](api-levels.md) - - [Extending the Framework](extending.md) - - [Example: New Workload & Expectation (Rust)](custom-workload-example.md) - - [Internal Crate Reference](internal-crate-reference.md) -- [Part IV — Operations & Deployment](part-iv.md) - - [Overview](operations-overview.md) - - [Prerequisites & Setup](prerequisites.md) - - [Running Examples](running-examples.md) - - [CI Integration](ci-integration.md) +- [The Framework in Brief](framework-in-brief.md) +- [Part I — Mental Model](part-i.md) + - [Application, AppDeployment, and Environments](application-model.md) + - [Scenario Model and Lifecycle](scenario-model.md) + - [Choosing an Entry Pattern](entry-patterns.md) + - [Ownership and Design Boundaries](boundaries.md) +- [Part II — Composing Applications](part-ii.md) + - [AppHost and with_app](app-host.md) + - [AppDeployment and DeployContext](app-deployment.md) + - [Handle Ownership and Teardown](handles-teardown.md) + - [One Binary: LocalProcessApp](local-process-app.md) + - [Uniform Child Clusters: LocalAppCluster](local-app-cluster.md) + - [Composing Heterogeneous Stacks](composing-stacks.md) + - [Backend Scope](app-backend-scope.md) +- [Part III — Scenario Runtime](part-iii.md) + - [Workloads and Concurrency](workloads.md) + - [Expectations and Evaluation](expectations.md) + - [The Verb Layer](verb-layer.md) + - [Scenario Capabilities](capabilities.md) + - [Chaos and Controlled Failure](chaos.md) + - [Runtime Extensions](runtime-extensions.md) + - [Continuous Observation](observation.md) + - [Telemetry and External Observability](telemetry.md) +- [Part IV — Uniform Clusters and Configuration](part-iv.md) + - [Implementing Application](implementing-application.md) + - [Topology and Deployment Plans](topology.md) + - [Ports, Peers, Node Config, and Readiness](node-config.md) + - [Static Artifacts and cfgsync](cfgsync.md) + - [Seeds and Reproducibility](seeds.md) + - [ManualCluster: Imperative Node Control](manual-cluster.md) + - [Persistence, Snapshots, and Recovery Testing](persistence.md) +- [Part V — Deployers and Sources](part-v.md) + - [Capability Matrix](capability-matrix.md) + - [Local Deployer](deployer-local.md) + - [Compose Deployer](deployer-compose.md) + - [Kubernetes Deployer](deployer-k8s.md) + - [Shared Cluster Provisioning](cluster-provisioning.md) + - [Existing and External Clusters](external-clusters.md) + - [Binary Providers](binary-providers.md) + - [Readiness, Retry, and Artifact Preservation](deployment-policies.md) +- [Part VI — Extending and Reference](part-vi.md) + - [Public Extension Points](extension-points.md) + - [Crate and API Map](crate-map.md) + - [Framework vs Application Boundaries](tf-boundaries.md) +- [Part VII — Operations](part-vii.md) + - [Running the Examples](running-examples.md) + - [Continuous Integration](ci.md) + - [Diagnostics and Retained Artifacts](diagnostics.md) - [Environment Variables](environment-variables.md) - - [Logging & Observability](logging-observability.md) -- [Part V — Appendix](part-v.md) - - [Builder API Quick Reference](dsl-cheat-sheet.md) - - [Troubleshooting Scenarios](troubleshooting.md) - - [FAQ](faq.md) + - [Troubleshooting](troubleshooting.md) - [Glossary](glossary.md) diff --git a/book/src/annotated-tree.md b/book/src/annotated-tree.md deleted file mode 100644 index d4d05e9..0000000 --- a/book/src/annotated-tree.md +++ /dev/null @@ -1,101 +0,0 @@ -# Annotated Tree - -Directory structure with key paths annotated: - -```text -logos-blockchain-testing/ -├─ testing-framework/ # Core library crates -│ ├─ configs/ # Node config builders, topology generation, tracing/logging config -│ ├─ core/ # Scenario model (ScenarioBuilder), runtime (Runner, Deployer), topology, node spawning -│ ├─ workflows/ # Workloads (transactions, chaos), expectations (liveness), builder DSL extensions -│ ├─ deployers/ # Deployment backends -│ │ ├─ local/ # LocalDeployer (spawns local processes) -│ │ ├─ compose/ # ComposeDeployer (Docker Compose + Prometheus) -│ │ └─ k8s/ # K8sDeployer (Kubernetes Helm) -│ └─ assets/ # Docker/K8s stack assets -│ └─ stack/ -│ ├─ monitoring/ # Prometheus config -│ ├─ scripts/ # Container entrypoints -│ └─ cfgsync.yaml # Config sync server template -│ -├─ examples/ # PRIMARY ENTRY POINT: runnable binaries -│ └─ src/bin/ -│ ├─ local_runner.rs # Host processes demo (LocalDeployer) -│ ├─ compose_runner.rs # Docker Compose demo (ComposeDeployer) -│ └─ k8s_runner.rs # Kubernetes demo (K8sDeployer) -│ -├─ scripts/ # Helper utilities -│ ├─ run-examples.sh # Convenience script (handles setup + runs examples) -│ ├─ build-bundle.sh # Build prebuilt binaries+circuits bundle -│ └─ setup-logos-blockchain-circuits.sh # Fetch circuit assets (Linux + host) -│ -└─ book/ # This documentation (mdBook) -``` - -## Key Directories Explained - -### `testing-framework/` -Core library crates providing the testing API. - -| Crate | Purpose | Key Exports | -|-------|---------|-------------| -| `configs` | Node configuration builders | Topology generation, tracing config | -| `core` | Scenario model & runtime | `ScenarioBuilder`, `Deployer`, `Runner` | -| `workflows` | Workloads & expectations | `ScenarioBuilderExt`, `ChaosBuilderExt` | -| `deployers/local` | Local process deployer | `LocalDeployer` | -| `deployers/compose` | Docker Compose deployer | `ComposeDeployer` | -| `deployers/k8s` | Kubernetes deployer | `K8sDeployer` | - -### `testing-framework/assets/stack/` -Docker/K8s deployment assets: -- **`monitoring/`**: Prometheus config -- **`scripts/`**: Container entrypoints - -### `scripts/` -Convenience utilities: -- **`run-examples.sh`**: All-in-one script for host/compose/k8s modes (recommended) -- **`build-bundle.sh`**: Create prebuilt binaries+circuits bundle for compose/k8s -- **`build_test_image.sh`**: Build the compose/k8s Docker image (bakes in assets) -- **`setup-logos-blockchain-circuits.sh`**: Fetch circuit assets for both Linux and host -- **`cfgsync.yaml`**: Configuration sync server template - -### `examples/` (Start Here!) -**Runnable binaries** demonstrating framework usage: -- `local_runner.rs` — Local processes -- `compose_runner.rs` — Docker Compose (requires `LOGOS_BLOCKCHAIN_TESTNET_IMAGE` built) -- `k8s_runner.rs` — Kubernetes (requires cluster + image) - -**Run with:** `cargo run -p runner-examples --bin ` - -### `scripts/` -Helper utilities: -- **`setup-logos-blockchain-circuits.sh`**: Fetch circuit assets from releases - -## Observability - -**Compose runner** includes: -- **Prometheus** at `http://localhost:9090` (metrics scraping) -- Node metrics exposed per node -- Access in expectations: `ctx.telemetry().prometheus().map(|p| p.base_url())` - -**Logging** controlled by: -- `LOGOS_BLOCKCHAIN_LOG_DIR` — Write per-node log files -- `LOGOS_BLOCKCHAIN_LOG_LEVEL` — Global log level (error/warn/info/debug/trace) -- `LOGOS_BLOCKCHAIN_LOG_FILTER` — Target-specific filtering (e.g., `cryptarchia=trace`) -- `LOGOS_BLOCKCHAIN_TESTS_TRACING` — Enable file logging for local runner - -See [Logging & Observability](logging-observability.md) for details. - -## Navigation Guide - -| To Do This | Go Here | -|------------|---------| -| **Run an example** | `examples/src/bin/` → `cargo run -p runner-examples --bin ` | -| **Write a custom scenario** | `testing-framework/core/` → Implement using `ScenarioBuilder` | -| **Add a new workload** | `testing-framework/workflows/src/workloads/` → Implement `Workload` trait | -| **Add a new expectation** | `testing-framework/workflows/src/expectations/` → Implement `Expectation` trait | -| **Modify node configs** | `testing-framework/configs/src/topology/configs/` | -| **Extend builder DSL** | `testing-framework/workflows/src/builder/` → Add trait methods | -| **Add a new deployer** | `testing-framework/deployers/` → Implement `Deployer` trait | - -For detailed guidance, see [Internal Crate Reference](internal-crate-reference.md). diff --git a/book/src/api-levels.md b/book/src/api-levels.md deleted file mode 100644 index 6a14ef0..0000000 --- a/book/src/api-levels.md +++ /dev/null @@ -1,119 +0,0 @@ -# API Levels: Builder DSL vs. Direct Instantiation - -The framework supports two styles for constructing scenarios: - -1. **High-level Builder DSL** (recommended): fluent helper methods (e.g. `.transactions_with(...)`) -2. **Low-level direct instantiation**: construct workload/expectation types explicitly, then attach them - -Both styles produce the same runtime behavior because they ultimately call the same core builder APIs. - -## High-Level Builder DSL (Recommended) - -The DSL is implemented as extension traits (primarily `testing_framework_workflows::ScenarioBuilderExt`) on the core scenario builder. - -```rust,ignore -use std::time::Duration; - -use testing_framework_core::scenario::ScenarioBuilder; -use testing_framework_workflows::ScenarioBuilderExt; - -let plan = ScenarioBuilder::topology_with(|t| t.network_star().nodes(3)) - .wallets(5) - .transactions_with(|txs| txs.rate(5).users(3)) - .expect_consensus_liveness() - .with_run_duration(Duration::from_secs(60)) - .build(); -``` - -**When to use:** -- Most test code (smoke, regression, CI) -- When you want sensible defaults and minimal boilerplate - -## Low-Level Direct Instantiation - -Direct instantiation gives you explicit control over the concrete types you attach: - -```rust,ignore -use std::{ - num::NonZeroUsize, - time::Duration, -}; - -use testing_framework_core::scenario::ScenarioBuilder; -use testing_framework_workflows::{ - expectations::ConsensusLiveness, - workloads::transaction, -}; - -let tx_workload = transaction::Workload::with_rate(5) - .expect("transaction rate must be non-zero") - .with_user_limit(NonZeroUsize::new(3)); - -let plan = ScenarioBuilder::topology_with(|t| t.network_star().nodes(3)) - .wallets(5) - .with_workload(tx_workload) - .with_expectation(ConsensusLiveness::default()) - .with_run_duration(Duration::from_secs(60)) - .build(); -``` - -**When to use:** -- Custom workload/expectation implementations -- Reusing preconfigured workload instances across multiple scenarios -- Debugging / exploring the underlying workload types - -## Method Correspondence - -| High-Level DSL | Low-Level Direct | -|----------------|------------------| -| `.transactions_with(\|txs\| txs.rate(5).users(3))` | `.with_workload(transaction::Workload::with_rate(5).expect(...).with_user_limit(...))` | -| `.expect_consensus_liveness()` | `.with_expectation(ConsensusLiveness::default())` | - -## Bundled Expectations (Important) - -Workloads can bundle expectations by implementing `Workload::expectations()`. - -These bundled expectations are attached automatically whenever you call `.with_workload(...)` (including when you use the DSL), because the core builder expands workload expectations during attachment. - -## Mixing Both Styles - -Mixing is common: use the DSL for built-ins, and direct instantiation for custom pieces. - -```rust,ignore -use std::time::Duration; - -use testing_framework_core::scenario::ScenarioBuilder; -use testing_framework_workflows::{ScenarioBuilderExt, workloads::transaction}; - -let tx_workload = transaction::Workload::with_rate(5) - .expect("transaction rate must be non-zero"); - -let plan = ScenarioBuilder::topology_with(|t| t.network_star().nodes(3)) - .wallets(5) - .with_workload(tx_workload) // direct instantiation - .expect_consensus_liveness() // DSL - .with_run_duration(Duration::from_secs(60)) - .build(); -``` - -## Implementation Detail (How the DSL Works) - -The DSL methods are thin wrappers. For example: - -`builder.transactions_with(|txs| txs.rate(5).users(3))` - -is roughly equivalent to: - -`builder.transactions().rate(5).users(3).apply()` - -## Troubleshooting - -**DSL method not found** -- Ensure the extension traits are in scope, e.g. `use testing_framework_workflows::ScenarioBuilderExt;` -- Cross-check method names in [Builder API Quick Reference](dsl-cheat-sheet.md) - -## See Also - -- [Builder API Quick Reference](dsl-cheat-sheet.md) -- [Example: New Workload & Expectation (Rust)](custom-workload-example.md) -- [Extending the Framework](extending.md) diff --git a/book/src/app-backend-scope.md b/book/src/app-backend-scope.md new file mode 100644 index 0000000..665694e --- /dev/null +++ b/book/src/app-backend-scope.md @@ -0,0 +1,70 @@ +# Backend Scope + +The app layer currently deploys components only through the local backend. Compose and Kubernetes support uniform single-application scenarios. + +This chapter lists the supported combinations and the APIs missing from the container backends. + +--- + +## What Works Where + +| Scenario shape | Local | Compose | Kubernetes | +|----------------|-------|---------|------------| +| Uniform cluster (`ScenarioBuilder` over a topology) | yes | yes | yes | +| AppHost composed stack (`AppHost::scenario().with_app(...)`) | yes | no | no | +| `with_app` presets over an existing uniform scenario | yes | yes | yes | + +An app preset that deploys nothing, such as the "existing cluster" presets in [AppHost and with_app](app-host.md), works on every backend because it only wraps `ctx.deployment()` and `ctx.node_clients()` in a typed handle. Deploying new components through the app layer is local-only: `LocalProcessApp` and `LocalAppCluster` use the local deployer's process primitives (`ProcessNode`, `ManualCluster`, `ProcessDeployer`), and `AppHostLocalDeployer` is a local process deployer. + +Single-app Compose and Kubernetes deployers are unchanged by the app layer. The kvstore and OpenRaft examples keep dedicated bins for them (`kvstore_compose_convergence`, `kvstore_k8s_convergence`, `openraft_kv_compose_failover`, `openraft_kv_k8s_failover`); see [Compose Deployer](deployer-compose.md) and [Kubernetes Deployer](deployer-k8s.md). + +--- + +## Why the Gap Exists + +The app layer starts, addresses, and stops individual units and can run application code between those starts, for example to check readiness or pass an address to a dependent component. The local deployer provides per-unit APIs. Compose and Kubernetes currently render and deploy a complete uniform scenario as one planned unit. Supporting `AppDeployment` on those backends requires corresponding per-unit planning and deployment APIs. + +--- + +## Choosing a Shape Today + +```mermaid +flowchart TD + Q{System under test} -->|one uniform cluster| U[ScenarioBuilder over a topology] + Q -->|composed stack| A[AppHost + root AppDeployment] + U --> B{Backend} + B --> L1[Local] + B --> C1[Compose] + B --> K1[Kubernetes] + A --> L2[Local only] +``` + +- **Composed stacks: run locally.** The local backend provides direct process control and per-unit restarts for heterogeneous stacks (see [Composing Heterogeneous Stacks](composing-stacks.md)). +- **Uniform single-app scenarios: use any supported backend.** A topology containing one application can run locally, with Compose, or on Kubernetes, subject to the capability matrix. +- **Split suites by shape, not by app.** If one system needs both a composed integration stack and a large containerized soak test of its main cluster, express them as two scenarios: an AppHost stack running locally, and a uniform scenario of the main app running on Compose/K8s. The kvstore example uses one environment crate with separate binaries per shape and backend. + +--- + +## Keep Workloads Backend-Independent + +Write workloads against typed handles and clients, not against backend details. A workload that requires a `StoreHandle` does not care whether the store came from a `LocalAppCluster` today or a future containerized unit: + +```rust,ignore +async fn start(&self, ctx: &RunContext) -> Result<(), DynError> { + let store = ctx.require_app::()?; // no backend visible here + store.put("/kv/scope-check", "ok").await?; + Ok(()) +} +``` + +If another backend later supports app composition, backend-specific changes should remain in the root deployment and its child adapters. Workloads and expectations can continue using the same handles. The OpenRaft "existing cluster" preset already works with Compose and Kubernetes because it only reads `ctx.deployment()` and `ctx.node_clients()`. + +**Note:** node-control-style fault injection inside a composed stack (restart one child-cluster node) is a handle method on [`LocalAppCluster`](local-app-cluster.md), so it is local-only by construction. Fault injection on containerized uniform scenarios goes through the scenario-level node control capability instead; see the openraft `openraft_kv_k8s_failover` bin. + +--- + +## See Also + +- [Capability Matrix](capability-matrix.md): the full feature-by-backend table. +- [Local Deployer](deployer-local.md): the backend the app layer builds on. +- [AppHost and with_app](app-host.md): the entry point this scope applies to. diff --git a/book/src/app-deployment.md b/book/src/app-deployment.md new file mode 100644 index 0000000..b01d905 --- /dev/null +++ b/book/src/app-deployment.md @@ -0,0 +1,145 @@ +# AppDeployment and DeployContext + +Application repositories implement `AppDeployment` for deployable components. The implementation uses `DeployContext` to deploy children, expose handles, and register managed resources with scenario cleanup. + +The framework runs deployments, stores handles, and performs teardown without defining application binaries or clients. The application crate decides which components start and which typed handles workloads receive. + +--- + +## The Trait Contract + +```rust,ignore +#[async_trait] +pub trait AppDeployment: Send + 'static { + type Handle: AppHandle; + + async fn deploy(self, ctx: &mut DeployContext) -> Result; +} +``` + +The trait has these properties: + +- **`deploy` consumes `self`.** A deployment value describes one preparation attempt and returns its runtime access handle. +- **The handle is typed.** `Handle` can be any `Clone + Send + Sync + 'static` type. Managed lifetime is registered separately; see [Handle Ownership and Teardown](handles-teardown.md). +- **`Clone` is required by the factory.** `with_app` needs `A: AppDeployment + Clone + Sync` because `AppDeploymentFactory` clones the description on each `prepare`. Deployment structs should contain configuration such as node counts, ports, and paths rather than live resources. + +A minimal implementation, from the kvstore example: + +```rust,ignore +// examples/kvstore/testing/integration/src/app.rs +#[derive(Clone)] +pub struct KvLocalApp { + deployment: KvTopology, +} + +#[async_trait] +impl AppDeployment for KvLocalApp { + type Handle = LocalAppCluster; + + async fn deploy(self, ctx: &mut DeployContext) -> Result { + ctx.deploy_local_cluster::(self.deployment).await + } +} +``` + +--- + +## DeployContext API + +One context belongs to one scenario preparation. It carries the active cluster provisioner, outer deployment and clients, exposed handles, and a cleanup stack. Routing every managed child through this context registers cleanup as soon as the resource is acquired, including when deployment fails partway. + +| Method | Purpose | +|--------|---------| +| `deploy(app)` | Runs a child deployment, returns its handle. Does **not** expose it. | +| `deploy_and_expose(app)` | Runs a child deployment and exposes a clone of its handle. | +| `expose(handle)` | Registers the default (unnamed) handle for its concrete type. | +| `expose_named(name, handle)` | Registers a named handle; allows several instances of one type. | +| `get::()` / `get_named::(name)` | `Option` clone of an exposed handle. | +| `require::()` / `require_named::(name)` | `Result` — typed missing-handle error. | +| `contains::()` | Whether a default handle for `T` is exposed. | +| `handles()` | Borrows the registry of handles exposed so far. | +| `deployment()` | The outer scenario deployment descriptor (`E::Deployment`). | +| `node_clients()` | Clients for nodes owned by the outer scenario (`NodeClients`). | +| `deploy_cluster::(request)` | Provisions a managed, attached, or external cluster through the active provisioner. | +| `deploy_local_cluster::(deployment)` | Convenience for an eager managed cluster with the active provisioner. | + +`deploy` does not expose its returned handle. Use it when only the parent needs the child handle. Use `deploy_and_expose` when workloads should also be able to request the child directly. Both `expose` and `expose_named` return `AppDeployError::DuplicateHandle` if the type or type/name pair is already registered. + +--- + +## Nested Deployments + +A deployment composes children by calling `ctx.deploy(...)` or `ctx.deploy_and_expose(...)` on other `AppDeployment` values. The parent decides what is visible: + +```rust,ignore +#[async_trait] +impl AppDeployment for ParentApp { + type Handle = ParentHandle; + + async fn deploy(self, ctx: &mut DeployContext) -> Result { + let child = ctx.deploy(ChildApp).await?; // child handle NOT exposed + let parent = ParentHandle { child }; + + ctx.expose(parent.clone())?; // only the parent is visible + + Ok(parent) + } +} +``` + +Workloads can then require `ParentHandle` but not `ChildHandle`: the child stays an implementation detail. Its managed resources remain registered with scenario cleanup whether or not the returned handle is exposed or embedded. Expose the child too when workloads legitimately need it. + +```mermaid +flowchart TD + Root[Root AppDeployment] -->|deploy| C1[Child A] + Root -->|deploy_and_expose| C2[Child B] + Root -->|expose| RH[Root handle] + C2 --> BH[Child B handle] + RH --> W[Workloads] + BH --> W + RH:::hd + BH:::hd + W:::sc + classDef hd stroke:#4caf7d,stroke-width:2.5px; + classDef sc stroke:#9b6dd6,stroke-width:2.5px; +``` + +--- + +## The Outer Scenario: deployment() and node_clients() + +For `AppHost` scenarios, `deployment()` is the empty `AppHostTopology` and `node_clients()` is empty; everything lives in your handles. On a regular uniform-cluster scenario, they are how an app preset wraps the managed cluster itself: + +```rust,ignore +// examples/kvstore/testing/integration/src/app.rs +#[async_trait] +impl AppDeployment for KvExistingClusterApp { + type Handle = KvStoreCluster; + + async fn deploy(self, ctx: &mut DeployContext) -> Result { + Ok(KvStoreCluster::new( + ctx.deployment().clone(), + ctx.node_clients().clone(), + )) + } +} +``` + +This preset does not launch nodes. It returns typed access to the nodes already managed by the scenario. + +--- + +## Root-Handle Auto-Exposure + +After the root deployment returns, `AppDeploymentFactory` checks `ctx.contains::()`. If the root handle type is not already exposed, the factory exposes the returned handle as the default for its type. So: + +- A simple root app can just `return Ok(handle)` and workloads can `require_app::()` with no explicit `expose`. +- A root app that already exposed its own handle (like the stack apps in [Composing Heterogeneous Stacks](composing-stacks.md)) is left alone, so there is no duplicate error. + +--- + +## See Also + +- [AppHost and with_app](app-host.md): how a deployment gets registered and prepared. +- [Handle Ownership and Teardown](handles-teardown.md): what exposure means for resource lifetime. +- [One Binary: LocalProcessApp](local-process-app.md), [Uniform Child Clusters: LocalAppCluster](local-app-cluster.md): ready-made deployments to compose. diff --git a/book/src/app-host.md b/book/src/app-host.md new file mode 100644 index 0000000..eccbc39 --- /dev/null +++ b/book/src/app-host.md @@ -0,0 +1,125 @@ +# AppHost and with_app + +`AppHost` creates a scenario whose system under test is supplied by application deployments instead of an outer managed node topology. + +The core scenario engine models one `Application` and a uniform cluster of its nodes. For a composed stack containing a binary, an additional cluster, or several applications, start from `AppHost::scenario()` and register the stack with `.with_app(...)`. Workloads, expectations, run duration, and teardown follow the lifecycle in [Scenario Model and Lifecycle](scenario-model.md). + +--- + +## The Zero-Node Scenario + +`AppHost::scenario()` returns a `ScenarioBuilder` seeded with `AppHostTopology`: + +| Type | Role | +|------|------| +| `AppHostTopology` | Deployment descriptor with `node_count() == 0`. The outer scenario manages no nodes. | +| `AppHostEnv` | Null environment: `NodeClient = ()`, and `build_node_client` always errors. Clients come from app handles instead. | +| `AppHostScenarioBuilder` | Alias for `ScenarioBuilder`. | +| `AppHostLocalDeployer` | Alias for `ProcessDeployer` — the local deployer that executes the scenario. | + +Because the outer topology is empty, app deployments create the processes and clusters used by the run. + +```rust,ignore +use testing_framework_app::{AppHost, AppHostLocalDeployer, AppScenarioBuilderExt}; +use testing_framework_core::scenario::Deployer; + +let mut scenario = AppHost::scenario() + .with_app(KvLocalApp::nodes(3)) + .with_run_duration(Duration::from_secs(5)) + .with_workload(KvAppHostConvergence::new(3)) + .build()?; + +let deployer = AppHostLocalDeployer::default(); +let runner = deployer.deploy(&scenario).await?; +runner.run(&mut scenario).await?; +``` + +The runnable `kvstore_app_host_convergence` binary uses this structure: + +```bash +cargo run -p kvstore-examples --bin kvstore_app_host_convergence +``` + +--- + +## How with_app Runs + +`AppScenarioBuilderExt::with_app(app)` wraps your [`AppDeployment`](app-deployment.md) in an `AppDeploymentFactory` and registers it as a runtime extension factory, the same lifecycle hook covered in [Runtime Extensions](runtime-extensions.md). Going through the extension mechanism ties managed deployment cleanup to the scenario lifetime and makes exposed handles available during the run. + +```mermaid +flowchart LR + B["with_app(app)"] --> F[AppDeploymentFactory] + F -->|prepare| C[DeployContext] + C -->|"deploy(root app)"| H["handles + cleanup"] + H --> R[AppRuntime extension] + R -->|require_app| W[Workloads] + H:::hd + R:::hd + W:::sc + classDef hd stroke:#4caf7d,stroke-width:2.5px; + classDef sc stroke:#9b6dd6,stroke-width:2.5px; +``` + +During scenario preparation the factory: + +1. Clones your app (this is why the factory requires `Clone`) and builds a fresh `DeployContext`. +2. Runs the root deployment's `deploy`, which may deploy and expose child apps. +3. Auto-exposes the returned root handle if the deployment did not expose one of that type itself (`!ctx.contains::()`). +4. Transfers the handle registry and cleanup stack into an `AppRuntime` extension. + +If any step fails, the partially built context is dropped and every resource deployed so far is released (see [Handle Ownership and Teardown](handles-teardown.md)). + +A scenario accepts one `with_app` registration. Every `AppDeploymentFactory` produces the same extension type (`AppRuntime`), and the runtime rejects duplicate extension types. A second registration fails during preparation with `duplicate runtime extension type registered: AppRuntime`. Compose several applications inside one root `AppDeployment` and expose the child handles from there, as shown in [Composing Heterogeneous Stacks](composing-stacks.md). + +--- + +## with_app Outside AppHost + +`with_app` is defined for every scenario builder, not only `AppHostScenarioBuilder`. On a regular uniform-cluster scenario, an "existing cluster" preset can wrap the outer scenario's deployment and node clients in a typed handle without deploying another resource. The OpenRaft example uses this pattern: + +```rust,ignore +// examples/openraft_kv/testing/integration/src/scenario.rs +fn with_existing_openraft_kv_app(app: OpenRaftKvExistingClusterApp) -> Self { + OpenRaftKvScenarioBuilder::with_deployment(app.topology()) + .with_app(app) + .with_cluster_observer() +} +``` + +Here the scenario still manages a uniform OpenRaft cluster, and the app layer just gives workloads a typed `OpenRaftKvCluster` handle over it. + +--- + +## Retrieving Handles in Workloads + +Workloads never see the deploy context. They retrieve exposed handles through `AppRunContextExt`, implemented on `RunContext`: + +| Method | Returns | +|--------|---------| +| `app::()` | `Option` — default handle for `T`, if exposed | +| `app_named::(name)` | `Option` — named handle for `T` | +| `require_app::()` | `Result` — errors if missing | +| `require_app_named::(name)` | `Result` — errors if missing | + +```rust,ignore +use testing_framework_app::AppRunContextExt; + +async fn start(&self, ctx: &RunContext) -> Result<(), DynError> { + let cluster = ctx.require_app::>()?; + cluster.restart_node("node-0").await?; + cluster.wait_node_ready("node-0").await?; + Ok(()) +} +``` + +Workloads normally use the `require_*` variants so that a missing handle produces a typed error containing the requested handle type. + +Every retrieval clones the handle. Handles are normally small access values backed by `Arc`; scenario cleanup still determines managed resource lifetime. + +--- + +## Where to Go Next + +- [AppDeployment and DeployContext](app-deployment.md): implementing the deployment itself. +- [One Binary: LocalProcessApp](local-process-app.md) and [Uniform Child Clusters: LocalAppCluster](local-app-cluster.md): the two built-in building blocks. +- [Backend Scope](app-backend-scope.md): why AppHost scenarios run on the local deployer today. diff --git a/book/src/application-model.md b/book/src/application-model.md new file mode 100644 index 0000000..a8e5f59 --- /dev/null +++ b/book/src/application-model.md @@ -0,0 +1,139 @@ +# Application, AppDeployment, and Environments + +This chapter distinguishes the `Application` trait, the `AppDeployment` trait, and the concrete environment types that implement `Application`. + +--- + +## The Application Trait + +`Application` is the contract between the scenario engine and whatever system you are testing. It bundles the backend-specific types the engine needs, without the engine ever knowing what your application does: + +```rust,ignore +use testing_framework_core::scenario::Application; + +pub trait Application: Send + Sync + 'static { + type Deployment: DeploymentDescriptor + Clone + 'static; + type NodeClient: Clone + Send + Sync + 'static; + type NodeConfig: Clone + Send + Sync + 'static; + + fn external_node_client(source: &ExternalNodeSource) -> Result; + fn build_node_client(access: &NodeAccess) -> Result; + fn node_readiness_path() -> &'static str; // default: "/" +} +``` + +The associated types and methods are: + +- **`Deployment`**: the topology descriptor, i.e. how many nodes exist and how they relate. +- **`NodeClient`**: the typed client workloads use to talk to one node. +- **`NodeConfig`**: the per-node configuration your binary consumes. +- **Client constructors**: `build_node_client` turns deployer-provided `NodeAccess` into a client; `external_node_client` does the same for nodes the framework did not start. Both return an "unsupported" error by default. An environment must implement the operations it supports ([Ownership and Design Boundaries](boundaries.md)). +- **`node_readiness_path`**: the HTTP path deployers probe during readiness gating. + +An implementation of `Application` is called an **environment**. Everything generic in the framework (`ScenarioBuilder`, `Workload`, `Expectation`, `RunContext`) is parameterized over one. + +Source: `testing-framework/core/src/env.rs`. + +--- + +## The AppDeployment Trait + +`Application` describes a *uniform* node population. A system containing a cluster plus another process, or several different clusters, is represented through `AppDeployment` in `testing-framework-app`: + +```rust,ignore +use testing_framework_app::{AppDeployment, AppHandle, DeployContext}; + +pub trait AppDeployment: Send + 'static { + type Handle: AppHandle; + + async fn deploy(self, ctx: &mut DeployContext) -> Result; +} +``` + +An `AppDeployment` is a deployable unit: it consumes its description, prepares whatever it represents, and returns a typed runtime handle. `AppHandle` is a blanket implementation, so any `Clone + Send + Sync + 'static` type qualifies. The handle provides access and control; managed resources acquired through framework adapters are owned separately by scenario cleanup. + +Deployments compose: inside `deploy`, the `DeployContext` lets a parent deployment call `ctx.deploy(child)` or `ctx.deploy_and_expose(child)`, then `ctx.expose(handle)` to publish typed handles to workloads. See [AppDeployment and DeployContext](app-deployment.md) for the full context API. + +An `AppDeployment` registered with `.with_app(...)` runs during scenario preparation. It participates in the scenario lifecycle; it does not replace that lifecycle. + +Source: `testing-framework/app/src/deployment.rs`. + +--- + +## Concrete Environments + +### AppHostEnv: an environment without outer nodes + +`AppHostEnv` is an environment with no outer nodes at all. Its topology, `AppHostTopology`, reports a node count of zero; its `NodeClient` and `NodeConfig` are both `()`; asking it for a node client is an error. It exists so that a scenario can be composed *entirely* from application deployments: + +```rust,ignore +use testing_framework_app::{AppHost, AppScenarioBuilderExt}; + +let builder = AppHost::scenario() // ScenarioBuilder, zero nodes + .with_app(KvLocalApp::nodes(3)); // apps provide all processes +``` + +The system is supplied by `with_app` deployments, and workloads access it through typed handles instead of outer node clients. See [AppHost and with_app](app-host.md). + +Source: `testing-framework/app/src/host.rs`. + +### KvEnv: a uniform node environment + +The kvstore example shows a full environment for a real binary: + +```rust,ignore +pub struct KvEnv; + +impl Application for KvEnv { + type Deployment = KvTopology; // ClusterTopology + type NodeClient = KvHttpClient; + type NodeConfig = KvNodeConfig; + + fn build_node_client(access: &NodeAccess) -> Result { + Ok(KvHttpClient::new(access.api_base_url()?)) + } + + fn node_readiness_path() -> &'static str { + "/health/ready" + } +} +``` + +`KvEnv` additionally implements `LocalBinaryApp` (in `examples/kvstore/testing/integration/src/local_env.rs`) to tell the local deployer which binary to run, how to render per-node configs, and which port serves the HTTP API. The same environment type also backs `AppDeployment` presets like `KvLocalApp`, whose handle is a whole child cluster. [Implementing Application](implementing-application.md) walks through this in detail. + +Source: `examples/kvstore/testing/integration/src/app.rs`. + +--- + +## How the Three Relate + +```mermaid +graph TD + SB["ScenarioBuilder<E>"] -->|"E: Application"| APP["Application(env contract)"] + KV["KvEnv"] -.->|implements| APP + AH["AppHostEnv(zero-node env)"] -.->|implements| APP + AD["AppDeployment<E, P>"] -->|"deploys via"| DC["DeployContext<E, P>"] + AD -->|returns| H["typed Handle"] + SB -->|".with_app(...)"| AD + SB:::sc + H:::hd + classDef sc stroke:#9b6dd6,stroke-width:2.5px; + classDef hd stroke:#4caf7d,stroke-width:2.5px; +``` + +| Role | What it is | What it answers | Example | +|---|---|---|---| +| `Application` | Trait bundling `Deployment`, `NodeClient`, `NodeConfig` for the scenario engine | "What types does the engine plumb around?" | `KvEnv` | +| `AppDeployment` | Trait for one deployable unit returning a typed handle | "How is this piece prepared, and what can test code access?" | `KvLocalApp` | +| Concrete environment | A type implementing `Application` | "Which system am I testing, uniform or zero-node?" | `KvEnv`, `AppHostEnv` | + +`Application` and `AppDeployment` are not alternatives. Every scenario has one environment type `E`, and app deployments are registered inside that scenario. A uniform kvstore cluster uses `KvEnv` directly; a heterogeneous stack uses `AppHostEnv` and supplies its components as app deployments. + +--- + +## Where to Go Next + +- [Scenario Model and Lifecycle](scenario-model.md): what a scenario is and how it runs. +- [Choosing an Entry Pattern](entry-patterns.md): which combination of these pieces fits your system. +- [Part II — Composing Applications](part-ii.md): the app layer in depth. +- [Part IV — Uniform Clusters and Configuration](part-iv.md): implementing an environment for your own binary. diff --git a/book/src/architecture-overview.md b/book/src/architecture-overview.md deleted file mode 100644 index 73c8759..0000000 --- a/book/src/architecture-overview.md +++ /dev/null @@ -1,256 +0,0 @@ -# Architecture Overview - -The framework follows a clear flow: **Topology → Scenario → Deployer → Runner → Workloads → Expectations**. - -## Core Flow - -```mermaid -flowchart LR - A(Topologyshape cluster) --> B(Scenarioplan) - B --> C(Deployerprovision & readiness) - C --> D(Runnerorchestrate execution) - D --> E(Workloadsdrive traffic) - E --> F(Expectationsverify outcomes) -``` - -## Crate Architecture - -```mermaid -flowchart TB - subgraph Examples["Runner Examples"] - LocalBin[local_runner.rs] - ComposeBin[compose_runner.rs] - K8sBin[k8s_runner.rs] - end - - subgraph Workflows["Workflows (Batteries Included)"] - DSL[ScenarioBuilderExtFluent API] - TxWorkload[Transaction Workload] - ChaosWorkload[Chaos Workload] - Expectations[Built-in Expectations] - end - - subgraph Core["Core Framework"] - ScenarioModel[Scenario Model] - Traits[Deployer + Runner Traits] - BlockFeed[BlockFeed] - NodeClients[Node Clients] - Topology[Topology Generation] - end - - subgraph Deployers["Runner Implementations"] - LocalDeployer[LocalDeployer] - ComposeDeployer[ComposeDeployer] - K8sDeployer[K8sDeployer] - end - - subgraph Support["Supporting Crates"] - Configs[Configs & Topology] - Nodes[Node API Clients] - end - - Examples --> Workflows - Examples --> Deployers - Workflows --> Core - Deployers --> Core - Deployers --> Support - Core --> Support - Workflows --> Support - - style Examples fill:#e1f5ff - style Workflows fill:#e1ffe1 - style Core fill:#fff4e1 - style Deployers fill:#ffe1f5 - style Support fill:#f0f0f0 -``` - -### Layer Responsibilities - -**Runner Examples (Entry Points)** -- Executable binaries that demonstrate framework usage -- Wire together deployers, scenarios, and execution -- Provide CLI interfaces for different modes - -**Workflows (High-Level API)** -- `ScenarioBuilderExt` trait provides fluent DSL -- Built-in workloads (transactions, chaos) -- Common expectations (liveness, inclusion) -- Simplifies scenario authoring - -**Core Framework (Foundation)** -- `Scenario` model and lifecycle orchestration -- `Deployer` and `Runner` traits (extension points) -- `BlockFeed` for real-time block observation -- `RunContext` providing node clients and metrics -- Topology generation and validation - -**Runner Implementations** -- `LocalDeployer` - spawns processes on host -- `ComposeDeployer` - orchestrates Docker Compose -- `K8sDeployer` - deploys to Kubernetes cluster -- Each implements `Deployer` trait - -**Supporting Crates** -- `configs` - Topology configuration and generation -- `nodes` - HTTP/RPC client for node APIs - -### Extension Points - -```mermaid -flowchart LR - Custom[Your Code] -.implements.-> Workload[Workload Trait] - Custom -.implements.-> Expectation[Expectation Trait] - Custom -.implements.-> Deployer[Deployer Trait] - - Workload --> Core[Core Framework] - Expectation --> Core - Deployer --> Core - - style Custom fill:#ffe1f5 - style Core fill:#fff4e1 -``` - -**Extend by implementing:** -- `Workload` - Custom traffic generation patterns -- `Expectation` - Custom success criteria -- `Deployer` - Support for new deployment targets - -See [Extending the Framework](extending.md) for details. - -### Components - -- **Topology** describes the cluster: how many nodes and the high-level network parameters they should follow. -- **Scenario** combines that topology with the activities to run and the checks to perform, forming a single plan. -- **Deployer** provisions infrastructure on the chosen backend (local processes, Docker Compose, or Kubernetes), waits for readiness, and returns a Runner. -- **Runner** orchestrates scenario execution: starts workloads, observes signals, evaluates expectations, and triggers cleanup. -- **Workloads** generate traffic and conditions that exercise the system. -- **Expectations** observe the run and judge success or failure once activity completes. - -Each layer has a narrow responsibility so that cluster shape, deployment choice, -traffic generation, and health checks can evolve independently while fitting -together predictably. - -## Entry Points - -The framework is consumed via **runnable example binaries** in `examples/src/bin/`: - -- `local_runner.rs` — Spawns nodes as host processes -- `compose_runner.rs` — Deploys via Docker Compose (requires `LOGOS_BLOCKCHAIN_TESTNET_IMAGE` built) -- `k8s_runner.rs` — Deploys via Kubernetes Helm (requires cluster + image) - -**Recommended:** Use the convenience script: - -```bash -scripts/run/run-examples.sh -t -n -# mode: host, compose, or k8s -``` - -This handles circuit setup, binary building/bundling, image building, and execution. - -**Alternative:** Direct cargo run (requires manual setup): - -```bash -cargo run -p runner-examples --bin -``` - -These binaries use the framework API (`ScenarioBuilder`) to construct and execute scenarios. - -## Builder API - -Scenarios are defined using a fluent builder pattern: - -```rust,ignore -use std::time::Duration; - -use testing_framework_core::scenario::ScenarioBuilder; -use testing_framework_workflows::ScenarioBuilderExt; - -pub fn scenario_plan() -> testing_framework_core::scenario::Scenario<()> { - ScenarioBuilder::topology_with(|t| t.network_star().nodes(3)) - .wallets(50) - .transactions_with(|txs| txs.rate(5).users(20)) - .expect_consensus_liveness() - .with_run_duration(Duration::from_secs(90)) - .build() -} -``` - -**Key API Points:** -- Topology uses `.topology_with(|t| { t.nodes(N) })` closure pattern -- Workloads are configured via `_with` closures (`transactions_with`, `chaos_with`) -- Chaos workloads require `.enable_node_control()` and a compatible runner - -## Deployers - -Three deployer implementations: - -| Deployer | Backend | Prerequisites | Node Control | -|----------|---------|---------------|--------------| -| `LocalDeployer` | Host processes | Binaries (built on demand or via bundle) | No | -| `ComposeDeployer` | Docker Compose | Image with embedded assets/binaries | Yes | -| `K8sDeployer` | Kubernetes Helm | Cluster + image loaded | Not yet | - -**Compose-specific features:** -- Observability is external (set `LOGOS_BLOCKCHAIN_METRICS_QUERY_URL` / `LOGOS_BLOCKCHAIN_METRICS_OTLP_INGEST_URL` / `LOGOS_BLOCKCHAIN_GRAFANA_URL` as needed) -- Optional OTLP trace/metrics endpoints (`LOGOS_BLOCKCHAIN_OTLP_ENDPOINT`, `LOGOS_BLOCKCHAIN_OTLP_METRICS_ENDPOINT`) -- Node control for chaos testing (restart nodes) - -## Assets and Images - -### Docker Image -Built via `scripts/build/build_test_image.sh`: -- Embeds circuit assets and binaries -- Includes runner scripts: `run_nomos_node.sh` -- Tagged as `LOGOS_BLOCKCHAIN_TESTNET_IMAGE` (default: `logos-blockchain-testing:local`) -- **Recommended:** Use prebuilt bundle via `scripts/build/build-bundle.sh --platform linux` and set `LOGOS_BLOCKCHAIN_BINARIES_TAR` before building image - -### Circuit Assets -Circuit assets required by the node binary: -- **Host path:** `~/.logos-blockchain-circuits` (default) -- **Container path:** `/opt/circuits` (for compose/k8s) -- **Override:** `LOGOS_BLOCKCHAIN_CIRCUITS=/custom/path/to/dir` (must point to a directory) -- **Fetch via:** `scripts/setup/setup-logos-blockchain-circuits.sh v0.3.1 ~/.logos-blockchain-circuits` or use `scripts/run/run-examples.sh` - -### Compose Stack -Templates and configs in `testing-framework/runners/compose/assets/`: -- `docker-compose.yml.tera` — Stack template (nodes) -- Cfgsync config: `testing-framework/assets/stack/cfgsync.yaml` -- Monitoring assets (not deployed by the framework): `testing-framework/assets/stack/monitoring/` - -## Logging Architecture - -**Two separate logging pipelines:** - -| Component | Configuration | Output | -|-----------|--------------|--------| -| **Runner binaries** | `RUST_LOG` | Framework orchestration logs | -| **Node processes** | `LOGOS_BLOCKCHAIN_LOG_LEVEL`, `LOGOS_BLOCKCHAIN_LOG_FILTER` (+ `LOGOS_BLOCKCHAIN_LOG_DIR` on host runner) | Consensus, mempool, network logs | - -**Node logging:** -- **Local runner:** Writes to temporary directories by default (cleaned up). Set `LOGOS_BLOCKCHAIN_TESTS_TRACING=true` + `LOGOS_BLOCKCHAIN_LOG_DIR` for persistent files. -- **Compose runner:** Default logs to container stdout/stderr (`docker logs`). To write per-node files, set `tracing_settings.logger: !File` in `testing-framework/assets/stack/cfgsync.yaml` (and mount a writable directory). -- **K8s runner:** Logs to pod stdout/stderr (`kubectl logs`). To write per-node files, set `tracing_settings.logger: !File` in `testing-framework/assets/stack/cfgsync.yaml` (and mount a writable directory). - -**File naming:** Per-node files use prefix `logos-blockchain-node-{index}` (may include timestamps). - -## Observability - -**Prometheus-compatible metrics querying (optional):** -- The framework does **not** deploy Prometheus/Grafana. -- Provide a Prometheus-compatible base URL (PromQL API) via `LOGOS_BLOCKCHAIN_METRICS_QUERY_URL`. -- Accessible in expectations when configured: `ctx.telemetry().prometheus().map(|p| p.base_url())` - -**Grafana dashboards (optional):** -- Dashboards live in `testing-framework/assets/stack/monitoring/grafana/dashboards/` and can be imported into your Grafana. -- If you set `LOGOS_BLOCKCHAIN_GRAFANA_URL`, the deployer prints it in `TESTNET_ENDPOINTS`. - -**Node APIs:** -- HTTP endpoints per node for consensus info and network status -- Accessible in expectations: `ctx.node_clients().node_clients().get(0)` - -**OTLP (optional):** -- Trace endpoint: `LOGOS_BLOCKCHAIN_OTLP_ENDPOINT=http://localhost:4317` -- Metrics endpoint: `LOGOS_BLOCKCHAIN_OTLP_METRICS_ENDPOINT=http://localhost:4318` -- Disabled by default (no noise if unset) - -For detailed logging configuration, see [Logging & Observability](logging-observability.md). diff --git a/book/src/authoring-scenarios.md b/book/src/authoring-scenarios.md deleted file mode 100644 index 95e9ef9..0000000 --- a/book/src/authoring-scenarios.md +++ /dev/null @@ -1,383 +0,0 @@ -# Authoring Scenarios - -Creating a scenario is a declarative exercise. This page walks you through the core authoring loop with concrete examples, explains the units and timing model, and shows how to structure scenarios in Rust test suites. - ---- - -## The Core Authoring Loop - -Every scenario follows the same pattern: - -```mermaid -flowchart LR - A[1. Topology] --> B[2. Workloads] - B --> C[3. Expectations] - C --> D[4. Duration] - D --> E[5. Deploy & Run] -``` - -1. **Shape the topology** — How many nodes, what network shape -2. **Attach workloads** — What traffic to generate (transactions, chaos) -3. **Define expectations** — What success looks like (liveness, inclusion, recovery) -4. **Set duration** — How long to run the experiment -5. **Choose a runner** — Where to execute (local, compose, k8s) - ---- - -## Hello Scenario: Your First Test - -Let's build a minimal consensus liveness test step-by-step. - -### Step 1: Shape the Topology - -```rust,ignore -use testing_framework_core::scenario::ScenarioBuilder; -use testing_framework_workflows::ScenarioBuilderExt; - -let scenario = ScenarioBuilder::topology_with(|t| { - t.network_star() // Star network (one gateway + nodes) - .nodes(3) // 3 nodes -}) -``` - -**What goes in topology?** -- Node counts (nodes) -- Network shape (`network_star()` is currently the only built-in layout) - -**What does NOT go in topology?** -- Traffic rates (that's workloads) -- Success criteria (that's expectations) -- Runtime configuration (that's duration/runner) - -### Step 2: Attach Workloads - -```rust,ignore -.wallets(20) // Seed funded wallet accounts for transaction workloads -.transactions_with(|tx| { - tx.rate(10) // 10 transactions per block - .users(5) // distributed across 5 wallets -}) -``` - -**What goes in workloads?** -- Transaction traffic (rate, users) -- Chaos injection (restarts, delays) - -**Units explained:** -- `.rate(10)` = **10 transactions per block** (not per second!) -- `.users(5)` = use 5 distinct wallet accounts -- The framework adapts to block time automatically - -### Step 3: Define Expectations - -```rust,ignore -.expect_consensus_liveness() -``` - -**What goes in expectations?** -- Health checks that run after the scenario completes -- Liveness (blocks produced) -- Inclusion (workload activity landed on-chain) -- Recovery (system survived chaos) - -**When do expectations run?** -After the duration window ends, during the **evaluation phase** of the scenario lifecycle. - -### Step 4: Set Duration - -```rust,ignore -use std::time::Duration; - -.with_run_duration(Duration::from_secs(60)) -``` - -**How long is enough?** -- Minimum: 2× the expected block time × number of blocks you want -- For consensus liveness: 30-60 seconds -- For transaction inclusion: 60-120 seconds -- For chaos recovery: 2-5 minutes - -**What happens during this window?** -- Nodes are running -- Workloads generate traffic -- Metrics/logs are collected -- BlockFeed broadcasts observations in real-time - -### Step 5: Build and Deploy - -```rust,ignore -.build(); - -// Choose a runner -use testing_framework_core::scenario::Deployer; -use testing_framework_runner_local::LocalDeployer; - -let deployer = LocalDeployer::default(); -let runner = deployer.deploy(&scenario).await?; -let _result = runner.run(&mut scenario).await?; -``` - ---- - -## Complete "Hello Scenario" - -Putting it all together: - -```rust,ignore -use std::time::Duration; - -use anyhow::Result; -use testing_framework_core::scenario::{Deployer, ScenarioBuilder}; -use testing_framework_runner_local::LocalDeployer; -use testing_framework_workflows::ScenarioBuilderExt; - -#[tokio::test] -async fn hello_consensus_liveness() -> Result<()> { - let mut scenario = ScenarioBuilder::topology_with(|t| { - t.network_star() - .nodes(3) - }) - .wallets(20) - .transactions_with(|tx| tx.rate(10).users(5)) - .expect_consensus_liveness() - .with_run_duration(Duration::from_secs(60)) - .build(); - - let deployer = LocalDeployer::default(); - let runner = deployer.deploy(&scenario).await?; - runner.run(&mut scenario).await?; - - Ok(()) -} -``` - -**Run it:** -```bash -cargo test hello_consensus_liveness -``` - ---- - -## Understanding Units & Timing - -### Transaction Rate: Per-Block, Not Per-Second - -**Wrong mental model:** `.rate(10)` = 10 tx/second - -**Correct mental model:** `.rate(10)` = 10 tx/block - -**Why?** The blockchain produces blocks at variable rates depending on consensus timing. The framework submits the configured rate **per block** to ensure predictable load regardless of block time. - -**Example:** -- Block time = 2 seconds -- `.rate(10)` → 10 tx/block → 5 tx/second average -- Block time = 5 seconds -- `.rate(10)` → 10 tx/block → 2 tx/second average - -### Duration: Wall-Clock Time - -`.with_run_duration(Duration::from_secs(60))` means the scenario runs for **60 seconds of real time**, not 60 blocks. - -**How many blocks will be produced?** -Depends on consensus timing (slot time, active slot coefficient). Typical: 1-2 seconds per block. - -**Rule of thumb:** -- 60 seconds → ~30-60 blocks -- 120 seconds → ~60-120 blocks - ---- - -## Structuring Scenarios in a Test Suite - -### Pattern 1: Integration Test Module - -```rust,ignore -// tests/integration_test.rs -use std::time::Duration; - -use anyhow::Result; -use testing_framework_core::scenario::{Deployer, ScenarioBuilder}; -use testing_framework_runner_local::LocalDeployer; -use testing_framework_workflows::ScenarioBuilderExt; - -#[tokio::test] -async fn test_consensus_liveness() -> Result<()> { - let mut scenario = ScenarioBuilder::topology_with(|t| { - t.network_star().nodes(3) - }) - .expect_consensus_liveness() - .with_run_duration(Duration::from_secs(30)) - .build(); - - let deployer = LocalDeployer::default(); - let runner = deployer.deploy(&scenario).await?; - runner.run(&mut scenario).await?; - Ok(()) -} - -#[tokio::test] -async fn test_transaction_inclusion() -> Result<()> { - let mut scenario = ScenarioBuilder::topology_with(|t| { - t.network_star().nodes(2) - }) - .wallets(10) - .transactions_with(|tx| tx.rate(5).users(5)) - .expect_consensus_liveness() - .with_run_duration(Duration::from_secs(60)) - .build(); - - let deployer = LocalDeployer::default(); - let runner = deployer.deploy(&scenario).await?; - runner.run(&mut scenario).await?; - Ok(()) -} -``` - -### Pattern 2: Shared Scenario Builders - -Extract common topology patterns: - -```rust,ignore -// tests/helpers.rs -use testing_framework_core::scenario::ScenarioBuilder; -use testing_framework_workflows::ScenarioBuilderExt; - -pub fn minimal_topology() -> ScenarioBuilder { - ScenarioBuilder::topology_with(|t| { - t.network_star().nodes(2) - }) -} - -pub fn production_like_topology() -> ScenarioBuilder { - ScenarioBuilder::topology_with(|t| { - t.network_star().nodes(7) - }) -} - -// tests/consensus_tests.rs -use std::time::Duration; - -use helpers::*; - -#[tokio::test] -async fn small_cluster_liveness() -> anyhow::Result<()> { - let mut scenario = minimal_topology() - .expect_consensus_liveness() - .with_run_duration(Duration::from_secs(30)) - .build(); - // ... deploy and run - Ok(()) -} - -#[tokio::test] -async fn large_cluster_liveness() -> anyhow::Result<()> { - let mut scenario = production_like_topology() - .expect_consensus_liveness() - .with_run_duration(Duration::from_secs(60)) - .build(); - // ... deploy and run - Ok(()) -} -``` - -### Pattern 3: Parameterized Scenarios - -Test the same behavior across different scales: - -```rust,ignore -use std::time::Duration; - -use anyhow::Result; -use testing_framework_core::scenario::{Deployer, ScenarioBuilder}; -use testing_framework_runner_local::LocalDeployer; -use testing_framework_workflows::ScenarioBuilderExt; - -async fn test_liveness_with_topology(nodes: usize) -> Result<()> { - let mut scenario = ScenarioBuilder::topology_with(|t| { - t.network_star() - .nodes(nodes) - }) - .expect_consensus_liveness() - .with_run_duration(Duration::from_secs(60)) - .build(); - - let deployer = LocalDeployer::default(); - let runner = deployer.deploy(&scenario).await?; - runner.run(&mut scenario).await?; - Ok(()) -} - -#[tokio::test] -async fn liveness_small() -> Result<()> { - test_liveness_with_topology(2, 1).await -} - -#[tokio::test] -async fn liveness_medium() -> Result<()> { - test_liveness_with_topology(5, 2).await -} - -#[tokio::test] -async fn liveness_large() -> Result<()> { - test_liveness_with_topology(10, 3).await -} -``` - ---- - -## What Belongs Where? - -### Topology - -**Do include:** -- Node counts (`.nodes(3)`) -- Network shape (`.network_star()`) - -**Don't include:** -- Traffic rates (workload concern) -- Expected outcomes (expectation concern) -- Runtime behavior (runner/duration concern) - -### Workloads - -**Do include:** -- Transaction traffic (`.transactions_with(|tx| ...)`) -- Chaos traffic (`.chaos().restart()` or `RandomRestartWorkload`) -- Chaos injection (`.with_workload(RandomRestartWorkload::new(...))`) -- Rates, users, timing - -**Don't include:** -- Node configuration (topology concern) -- Success criteria (expectation concern) - -### Expectations - -**Do include:** -- Health checks (`.expect_consensus_liveness()`) -- Inclusion verification (built-in to workloads) -- Custom assertions (`.with_expectation(MyExpectation::new())`) - -**Don't include:** -- Traffic generation (workload concern) -- Cluster shape (topology concern) - ---- - -## Best Practices - -1. **Keep scenarios focused**: One scenario = one behavior under test -2. **Start small**: 2-3 nodes, 30-60 seconds -3. **Use descriptive names**: `test_consensus_survives_node_restart` not `test_1` -4. **Extract common patterns**: Shared topology builders, helper functions -5. **Document intent**: Add comments explaining what you're testing and why -6. **Mind the units**: `.rate(N)` is per-block, `.with_run_duration()` is wall-clock -7. **Set realistic durations**: Allow enough time for multiple blocks + workload effects - ---- - -## Next Steps - -- **[Core Content: Workloads & Expectations](workloads.md)** — Comprehensive reference for built-in workloads and expectations -- **[Examples](examples.md)** — More scenario patterns (chaos, advanced topologies) -- **[Running Scenarios](running-scenarios.md)** — How execution works, artifacts produced, per-runner details -- **[API Levels](api-levels.md)** — When to use builder DSL vs. direct instantiation diff --git a/book/src/best-practices.md b/book/src/best-practices.md deleted file mode 100644 index 6bc802a..0000000 --- a/book/src/best-practices.md +++ /dev/null @@ -1,237 +0,0 @@ -# Best Practices - -This page collects proven patterns for authoring, running, and maintaining test scenarios that are reliable, maintainable, and actionable. - -## Scenario Design - -**State your intent** -- Document the goal of each scenario (throughput, resilience) so expectation choices are obvious -- Use descriptive variable names that explain topology purpose (e.g., `star_topology_3val_2exec` vs `topology`) -- Add comments explaining why specific rates or durations were chosen - -**Keep runs meaningful** -- Choose durations that allow multiple blocks and make timing-based assertions trustworthy -- Use [FAQ: Run Duration Calculator](faq.md#how-long-should-a-scenario-run) to estimate minimum duration -- Avoid runs shorter than 30 seconds unless testing startup behavior specifically - -**Separate concerns** -- Start with deterministic workloads for functional checks -- Add chaos in dedicated resilience scenarios to avoid noisy failures -- Don't mix high transaction load with aggressive chaos in the same test (hard to debug) - -**Start small, scale up** -- Begin with minimal topology (1-2 nodes) to validate scenario logic -- Gradually increase topology size and workload rates -- Use Host runner for fast iteration, then validate on Compose before production - -## Code Organization - -**Reuse patterns** -- Standardize on shared topology and workload presets so results are comparable across environments and teams -- Extract common topology builders into helper functions -- Create workspace-level constants for standard rates and durations - -**Example: Topology preset** - -```rust,ignore -pub fn standard_topology() -> GeneratedTopology { - TopologyBuilder::new() - .network_star() - .nodes(3) - .generate() -} -``` - -**Example: Shared constants** - -```rust,ignore -pub const STANDARD_TX_RATE: f64 = 10.0; -pub const SHORT_RUN_DURATION: Duration = Duration::from_secs(60); -pub const LONG_RUN_DURATION: Duration = Duration::from_secs(300); -``` - -## Debugging & Observability - -**Observe first, tune second** -- Rely on liveness and inclusion signals to interpret outcomes before tweaking rates or topology -- Enable detailed logging (`RUST_LOG=debug`, `LOGOS_BLOCKCHAIN_LOG_LEVEL=debug`) only after initial failure -- Use `LOGOS_BLOCKCHAIN_TESTS_KEEP_LOGS=1` to persist logs when debugging failures - -**Use BlockFeed effectively** -- Subscribe to BlockFeed in expectations for real-time block monitoring -- Track block production rate to detect liveness issues early -- Use block statistics (`block_feed.stats().total_transactions()`) to verify inclusion - -**Collect metrics** -- Set up Prometheus/Grafana via `scripts/setup/setup-observability.sh compose up` for visualizing node behavior -- Use metrics to identify bottlenecks before adding more load -- Monitor mempool size, block size, and consensus timing - -## Environment & Runner Selection - -**Environment fit** -- Pick runners that match the feedback loop you need: - - **Host**: Fast iteration during development, quick CI smoke tests - - **Compose**: Reproducible environments (recommended for CI), chaos testing - - **K8s**: Production-like fidelity, large topologies (10+ nodes) - -**Runner-specific considerations** - -| Runner | When to Use | When to Avoid | -|--------|-------------|---------------| -| Host | Development iteration, fast feedback | Chaos testing, container-specific issues | -| Compose | CI pipelines, chaos tests, reproducibility | Very large topologies (>10 nodes) | -| K8s | Production-like testing, cluster behaviors | Local development, fast iteration | - -**Minimal surprises** -- Seed only necessary wallets and keep configuration deltas explicit when moving between CI and developer machines -- Use `versions.env` to pin node versions consistently across environments -- Document non-default environment variables in scenario comments or README - -## CI/CD Integration - -**Use matrix builds** - -```yaml -strategy: - matrix: - runner: [host, compose] - topology: [small, medium] -``` - -**Cache aggressively** -- Cache Rust build artifacts (`target/`) -- Cache circuit parameters (`~/.logos-blockchain-circuits/`) -- Cache Docker layers (use BuildKit cache) - -**Collect logs on failure** - -```yaml -- name: Collect logs on failure - if: failure() - run: | - mkdir -p test-logs - find /tmp -name "nomos-*.log" -exec cp {} test-logs/ \; -- uses: actions/upload-artifact@v3 - if: failure() - with: - name: test-logs-${{ matrix.runner }} - path: test-logs/ -``` - -**Time limits** -- Set job timeout to prevent hung runs: `timeout-minutes: 30` -- Use shorter durations in CI (60s) vs local testing (300s) -- Run expensive tests (k8s, large topologies) only on main branch or release tags - -**See also:** [CI Integration](ci-integration.md) for complete workflow examples - -## Anti-Patterns to Avoid - -```bash -# BAD: Will hang/timeout on proof generation -cargo run -p runner-examples --bin local_runner -``` - -**DON'T: Use tiny durations** -```rust,ignore -// BAD: Not enough time for blocks to propagate -.with_run_duration(Duration::from_secs(5)) - -// GOOD: Allow multiple consensus rounds -.with_run_duration(Duration::from_secs(60)) -``` - -**DON'T: Ignore cleanup failures** -```rust,ignore -// BAD: Next run inherits leaked state -runner.run(&mut scenario).await?; -// forgot to call cleanup or use CleanupGuard - -// GOOD: Cleanup via guard (automatic on panic) -let _cleanup = CleanupGuard::new(runner.clone()); -runner.run(&mut scenario).await?; -``` - -**DON'T: Mix concerns in one scenario** -```rust,ignore -// BAD: Hard to debug when it fails -.transactions_with(|tx| tx.rate(50).users(100)) // high load -.chaos_with(|c| c.restart().min_delay(...)) // AND chaos - -// GOOD: Separate tests for each concern -// Test 1: High transaction load only -// Test 2: Chaos resilience only -``` - -**DON'T: Hardcode paths or ports** -```rust,ignore -// BAD: Breaks on different machines -let path = PathBuf::from("/home/user/circuits"); -let port = 9000; // might conflict - -// GOOD: Use env vars and dynamic allocation -let path = std::env::var("LOGOS_BLOCKCHAIN_CIRCUITS") - .unwrap_or_else(|_| "~/.logos-blockchain-circuits".to_string()); -let port = get_available_tcp_port(); -``` - -**DON'T: Ignore resource limits** -```bash -# BAD: Large topology without checking resources -scripts/run/run-examples.sh -n 20 compose -# (might OOM or exhaust ulimits) - -# GOOD: Scale gradually and monitor resources -scripts/run/run-examples.sh -n 3 compose # start small -docker stats # monitor resource usage -# then increase if resources allow -``` - -## Scenario Design Heuristics - -**Minimal viable topology** -- Consensus: 3 nodes (minimum for Byzantine fault tolerance) -- Network: Star topology (simplest for debugging) - -**Workload rate selection** -- Start with 1-5 tx/s per user, then increase -- Chaos: 30s+ intervals between restarts (allow recovery) - -**Duration guidelines** - -| Test Type | Minimum Duration | Typical Duration | -|-----------|------------------|------------------| -| Smoke test | 30s | 60s | -| Integration test | 60s | 120s | -| Load test | 120s | 300s | -| Resilience test | 120s | 300s | -| Soak test | 600s (10m) | 3600s (1h) | - -**Expectation selection** - -| Test Goal | Expectations | -|-----------|--------------| -| Basic functionality | `expect_consensus_liveness()` | -| Transaction handling | `expect_consensus_liveness()` + custom inclusion check | -| Resilience | `expect_consensus_liveness()` + recovery time measurement | - -## Testing the Tests - -**Validate scenarios before committing** -1. Run on Host runner first (fast feedback) -2. Run on Compose runner (reproducibility check) -3. Check logs for warnings or errors -4. Verify cleanup (no leaked processes/containers) -5. Run 2-3 times to check for flakiness - -**Handling flaky tests** -- Increase run duration (timing-sensitive assertions need longer runs) -- Reduce workload rates (might be saturating nodes) -- Check resource limits (CPU/RAM/ulimits) -- Add debugging output to identify race conditions -- Consider if test is over-specified (too strict expectations) - -**See also:** -- [Troubleshooting](troubleshooting.md) for common failure patterns -- [FAQ](faq.md) for design decisions and gotchas diff --git a/book/src/binary-providers.md b/book/src/binary-providers.md new file mode 100644 index 0000000..11b7fbb --- /dev/null +++ b/book/src/binary-providers.md @@ -0,0 +1,123 @@ +# Binary Providers + +Binary providers resolve the executable a local node process runs. The source can be a path, an env var, a build command, a download, or an ordered fallback chain. + +Every local node launch has exactly one provider selected on its `LocalProcessSpec`. Providers live in `testing_framework_runner_local::binary` and are re-exported from the crate root. They apply to the [Local Deployer](deployer-local.md) only; compose and k8s nodes run container images instead (see the [Capability Matrix](capability-matrix.md)). + +--- + +## The Trait + +```rust,ignore +pub trait BinaryProvider: Send + Sync { + fn try_resolve(&self) -> Result, BinaryProviderError>; + fn display(&self) -> String; + fn cache_key(&self) -> String; + + // Provided: cache lookup, then resolve_uncached. + fn resolve(&self) -> Result { /* ... */ } + fn resolve_uncached(&self) -> Result { /* ... */ } +} + +pub type BinaryProviderRef = Arc; +``` + +`try_resolve` returns `Ok(None)` when the provider is valid but cannot produce a binary in the current environment, which is not an error. Standalone, `resolve` turns `None` into `BinaryProviderError::NotFound`; inside a `FallbackBinaryProvider`, `None` means "try the next provider". Other errors (a failed build, a checksum mismatch) abort resolution immediately. + +--- + +## The Providers + +| Provider | Resolves from | Unresolved (`None`) when | +|---|---|---| +| `PathBinaryProvider` | A fixed absolute path | Path is not a file (relative paths are an error) | +| `EnvBinaryProvider` | An env var containing a path | Var unset or not pointing at a file | +| `BuildBinaryProvider` | Running a build command | Never — build failure is an error | +| `DownloadBinaryProvider` | Fetching a URL into a cache | Never — download failure is an error | +| `FallbackBinaryProvider` | First chain member to resolve | Every member returned `None` | + +**`PathBinaryProvider::new(path)`**: a deterministic explicit path. No filesystem search, no `PATH` lookup. + +**`EnvBinaryProvider::new("MY_NODE_BIN")`**: the standard override hook. `LocalProcessSpec::new(env_var)` installs one of these by default. + +**`BuildBinaryProvider`** delegates to any command: + +```rust,ignore +BuildBinaryProvider { + command: BuildCommand::new("cargo").with_args(["build", "-p", "kvstore-node"]), + output_path: "target/debug/kvstore-node".into(), // relative to working_dir + working_dir: Some(workspace_root), // default: current dir + lock_dir: None, // default: /.tf-binaries +} +``` + +The command is not Cargo-specific; it can invoke a Make target, shell script, or cache fetch. After the command succeeds, the configured `output_path` must exist or resolution fails with `MissingBuildOutput`. + +**`DownloadBinaryProvider`** fetches into a cache directory (default `target/.tf-binaries` under the current directory): + +- `DownloadUrl::Fixed(url)` or `DownloadUrl::Env(var)` selects the source. +- `DownloadChecksum::Fixed(sha256)` or `DownloadChecksum::Env(var)` enables SHA-256 verification; mismatches fail with `ChecksumMismatch` before anything is written to the final path. +- A `DownloadProcessor` post-processes artifacts that are not directly executable (archives, bundles). It receives the verified download and must materialize the executable at the output path. `DownloadProcessorFn::new(cache_key, closure)` (or `.with_processor_fn(...)`) is the lightweight adapter; the `cache_key` is part of cache identity, so changing your extraction logic invalidates previously prepared binaries. +- On Unix, the result is marked executable (`0o755`). Downloads are staged through temporary `.download`/`.part` files and renamed into place. + +**`FallbackBinaryProvider::new([a, b, ...])`**: an ordered chain, tried first to last. From the launch spec's perspective it is still a single provider. + +--- + +## Caching and Cache Identity + +Successful resolutions are cached **per process** in a global map keyed by `cache_key()`, so repeated node starts with the same provider config do not rebuild, redownload, or re-scan. Cache identity encodes the full request: + +- `path:`, `env:` +- `build:::` +- `download::::` +- fallback: the members' keys joined with commas + +Change any component and you get a fresh resolution. The download provider also caches on disk: the cached file name hashes the URL, resolved checksum, and processor key, so an already-downloaded binary is reused across processes without refetching. + +--- + +## Concurrent Resolution Locking + +Builds and downloads may be triggered by several test processes at once (e.g. `cargo nextest` running integration tests in parallel). Providers that materialize files take a **cross-process file lock** before doing work: a lock file created with `create_new` under `.tf-binaries` (build) or the download cache dir, retried every 200 ms for up to 10 minutes, then `BinaryProviderError::LockTimeout`. The lock file is removed when the guard drops. + +A killed test process can leave a stale lock file behind. If resolution hangs and then times out, look for leftover `*.lock` files under `.tf-binaries` and delete them. + +--- + +## Worked Example: kvstore's Fallback Chain + +From `examples/kvstore/testing/integration/src/local_env.rs`, which prefers an explicit env override and otherwise builds from source: + +```rust,ignore +use std::{path::PathBuf, sync::Arc}; +use testing_framework_runner_local::{ + BinaryProviderRef, BuildBinaryProvider, BuildCommand, EnvBinaryProvider, + FallbackBinaryProvider, LocalProcessSpec, +}; + +fn kvstore_binary_provider() -> FallbackBinaryProvider { + let providers: [BinaryProviderRef; 2] = [ + Arc::new(EnvBinaryProvider::new("KVSTORE_NODE_BIN")), + Arc::new(BuildBinaryProvider { + command: BuildCommand::new("cargo") + .with_args(["build", "-p", "kvstore-node", "--bin", "kvstore-node"]), + output_path: PathBuf::from(format!( + "target/debug/kvstore-node{}", + std::env::consts::EXE_SUFFIX + )), + working_dir: Some(workspace_root()), + lock_dir: None, + }), + ]; + FallbackBinaryProvider::new(providers) +} + +fn local_process_spec() -> LocalProcessSpec { + LocalProcessSpec::new("KVSTORE_NODE_BIN") + .with_binary_provider(kvstore_binary_provider()) + .with_rust_log("kvstore_node=info") +} +``` + +First run: `KVSTORE_NODE_BIN` is unset, the env provider yields `None`, the build provider compiles the node under the workspace lock, and the path is cached for the rest of the process. Set `KVSTORE_NODE_BIN=/path/to/kvstore-node` to skip the build entirely, which is useful for prebuilt release binaries or mixed-version clusters (via `local_process_spec_for_node`, see [Local Deployer](deployer-local.md)). diff --git a/book/src/boundaries.md b/book/src/boundaries.md new file mode 100644 index 0000000..3b9554a --- /dev/null +++ b/book/src/boundaries.md @@ -0,0 +1,87 @@ +# Ownership and Design Boundaries + +This chapter lists the responsibilities of the framework and of an application repository. + +--- + +## The Boundary + +The scenario engine never names a concrete application. Its only coupling point is the `Application` trait: a bundle of associated types (`Deployment`, `NodeClient`, `NodeConfig`) that the engine plumbs around generically. Everything that knows what your system *is* (its binary, its config format, its client, its notion of "healthy") lives on your side of that trait. + +| Concern | Owner | +|---|---| +| Process lifetime (spawn, stop, restart, PIDs) | Framework | +| Working directories and temp dirs | Framework | +| Cleanup guards and teardown ordering | Framework | +| Topology mechanics (ports, peers, node names) | Framework | +| Readiness probing, retry, and gating | Framework | +| Handle storage and lookup (`HandleRegistry`, `AppRuntime`, `RunContext`) | Framework | +| Workload/expectation scheduling and aggregation | Framework | +| Node binaries and how to obtain them | Application repo | +| `NodeConfig` shape and rendering | Application repo | +| Typed node clients | Application repo | +| Readiness endpoints and app-specific checks | Application repo | +| Domain handles (`StoreHandle`, `WalletHandle`, ...) | Application repo | +| Meaningful workloads, expectations, scenarios | Application repo | + +--- + +## What the Framework Owns + +**Process lifetime and working directories.** Deployers spawn node processes into per-run working directories, track PIDs, and stop everything on teardown. Artifact retention is policy (`CleanupPolicy::preserve_artifacts`), not something scenarios hand-roll. + +**Cleanup.** Teardown is guard-based and automatic: cleanup guards chain and run in reverse registration order when the `RunHandle` drops, and the same guards run on the failure path. App-layer adapters register managed resources in a LIFO cleanup stack so dependants stop before dependencies, independently of exposed handle clones ([Handle Ownership and Teardown](handles-teardown.md)). + +**Topology mechanics.** Port allocation, peer wiring, node naming, and readiness gating with retry are all generic over `E: Application`. The engine asks your environment *what* to render and probe, never *why*. + +**Handle storage and lookup.** `DeployContext` collects typed handles during preparation; `AppRuntime` carries them into the run; `AppRunContextExt` returns clones to workloads. Duplicate exposure of a type/name pair is an error, never a silent replacement. + +--- + +## What the Application Repository Owns + +The kvstore example is the template. Its integration crate supplies, in its own repository: + +- **The binary and how to get it**: a `FallbackBinaryProvider` chain that uses `KVSTORE_NODE_BIN` if set and otherwise builds `kvstore-node` with cargo ([Binary Providers](binary-providers.md)). +- **Config**: `KvNodeConfig`, built per node from the framework's port/peer views and rendered to YAML. +- **Client**: `KvHttpClient`, constructed in `Application::build_node_client`. +- **Readiness**: `node_readiness_path()` returning `/health/ready`. +- **Domain handles and presets**: `KvStoreCluster`, `KvLocalApp`, `KvExistingClusterApp`. +- **Scenarios that mean something**: write workloads, convergence expectations, runnable bins. + +```rust,ignore +// Application side: domain knowledge, no orchestration. +fn node_readiness_path() -> &'static str { + "/health/ready" +} + +// Framework side: orchestration, no domain knowledge. +// It only ever sees E::NodeClient, E::NodeConfig, E::Deployment. +``` + +Sources: `examples/kvstore/testing/integration/src/{app,local_env}.rs`, `testing-framework/app/src/lib.rs`. + +--- + +## How the Boundary Is Enforced + +**Unsupported defaults.** `Application::build_node_client` and `external_node_client` return an "unsupported" error by default. Capabilities are available only when the environment implements them. + +**Generic application types.** There is no global list of known applications or framework config file naming their binaries. `testing-framework-core` compiles against `E: Application`, so it does not depend on adopter types or their dependencies. The same runtime can therefore be instantiated with kvstore, openraft_kv, nats, or an application from another repository. + +**Application-owned composition.** Application repositories implement `AppDeployment`, compose children through `DeployContext`, and expose typed handles. The framework supplies the context, registry, and lifecycle without defining the application stack. + +**CI boundary check.** `scripts/run/check-boundaries.sh` checks an application-side topology crate for framework-extension symbols (`cfgsync`, `ComposeDeployEnv`, `K8sDeployEnv`, `runner-compose`, `runner-k8s`). This detects backend dependencies in topology code. The compiler enforces the reverse direction because core crates do not reference concrete application types. + +> **External example:** the current boundary script targets logos-blockchain's `lb-topology` crate (in its own checkout), which keeps that adopter's topology code local/topology-focused. The pattern generalizes: point the same grep at your own integration crates. + +Application-specific config formats and startup rules belong in the environment implementation or an `AppDeployment`, not in framework crates. + +--- + +## Where to Go Next + +- [Application, AppDeployment, and Environments](application-model.md): the trait that defines the boundary. +- [Implementing Application](implementing-application.md): building your side of it. +- [Framework vs Application Boundaries](tf-boundaries.md): the reference treatment in Part VI. +- [Public Extension Points](extension-points.md): the sanctioned ways to extend the framework itself. diff --git a/book/src/capabilities.md b/book/src/capabilities.md new file mode 100644 index 0000000..d80fefe --- /dev/null +++ b/book/src/capabilities.md @@ -0,0 +1,117 @@ +# Scenario Capabilities + +Capabilities record, in the type system, which deployer services a scenario requests, such as node control or external telemetry. Unsupported combinations fail during construction, compilation, or deployment rather than during a workload. + +--- + +## The Capability Type Parameter + +The core builder is generic over a capability marker: `Builder` with `Caps = ()` by default. Building a scenario produces `Scenario`, and deployers are typed as `Deployer`, so a deployer that cannot provide a capability does not accept scenarios that demand it. + +The public wrappers (`testing-framework/core/src/scenario/definition/builder.rs`): + +| Builder type | Capability | Entered via | +|--------------|-----------|-------------| +| `ScenarioBuilder` | `()` | `ScenarioBuilder::with_deployment(...)` / `::new(provider)` | +| `NodeControlScenarioBuilder` | `NodeControlCapability` | `.with_node_control()` (alias `.enable_node_control()`) | +| `ObservabilityScenarioBuilder` | `ObservabilityCapability` | `.with_observability()` or any `ObservabilityBuilderExt` method | + +All three expose the same fluent surface (`with_workload`, `with_expectation`, `with_run_duration`, ...), so the capability switch can happen anywhere in the chain: + +```rust,ignore +let scenario = ScenarioBuilder::with_deployment(topology) + .with_node_control() // () -> NodeControlCapability + .with_workload(my_restart_workload) + .with_run_duration(Duration::from_secs(60)) + .build()?; +``` + +`RequiresNodeControl` (`testing-framework/core/src/scenario/capabilities.rs`) is how `build()` and deployers reason about the marker: + +```rust,ignore +pub trait RequiresNodeControl { + const REQUIRED: bool; +} +// (): false NodeControlCapability: true ObservabilityCapability: false +``` + +`build()` uses it to validate the source configuration: a scenario that requires node control but only has external, uncontrolled nodes fails with a `SourceConfiguration` error ("node control is not available for cluster mode 'external-only' ..."). See [Existing and External Clusters](external-clusters.md). + +--- + +## Node Control Without ManualCluster + +Restarting nodes from a declarative workload does **not** require `ManualCluster`. The node-control capability provides access instead: + +1. Call `.with_node_control()` on the builder. +2. Deploy with a deployer that supports the capability (local ships full node control, compose supports restart; the k8s deployer wires no node control handle into managed scenarios, so use its `ManualCluster` mode instead). +3. Inside a workload, take the handle from the context: + +```rust,ignore +let Some(control) = ctx.node_control() else { + return Err("this workload requires node control".into()); +}; + +control.restart_node("node-1").await?; +``` + +`ManualCluster` is the imperative API for tests that control the entire node lifecycle themselves; see [ManualCluster: Imperative Node Control](manual-cluster.md). The scenario form above runs workloads, expectations, and teardown through the scenario runtime. [Chaos and Controlled Failure](chaos.md) shows a full failover scenario built this way. + +### NodeControlHandle + +`NodeControlHandle` (`testing-framework/core/src/scenario/control.rs`) is the deployer-agnostic control surface. Every method has a default implementation returning a "not supported by this deployer" error, so partial support is explicit at run time: + +| Method | Effect | +|--------|--------| +| `restart_node(name)` | Stop and start a named node | +| `restart_node_with(name, options)` | Restart with `StartNodeOptions` overrides | +| `start_node(name)` | Start a node, returning `StartedNode` | +| `start_node_with(name, options)` | Start with overrides | +| `stop_node(name)` | Stop a named node | +| `wait_node_ready(name)` | Wait for one named node's readiness gate | +| `node_client(name)` | Current client for a node, if any | +| `node_pid(name)` | OS pid where applicable | + +`StartedNode` is a plain pair: the node `name` and a fresh `E::NodeClient`. + +`ClusterWaitHandle` is the matching wait surface: a single `wait_network_ready()` used for readiness gates. It is exposed publicly on the runner as `Runner::wait_network_ready()` (before `run` starts) and on `ManualCluster`; inside workloads, prefer waiting on observed application state instead. + +### StartNodeOptions + +`StartNodeOptions` customizes a dynamic start or restart. Overview (full treatment in Part IV: [Ports, Peers, Node Config, and Readiness](node-config.md) and [Persistence, Snapshots, and Recovery Testing](persistence.md)): + +| Field | Builder method | Purpose | +|-------|----------------|---------| +| `peers: Option` | `with_peers` | `DefaultLayout`, `None`, or `Named(vec)` | +| `config_override: Option` | `with_config_override` | Replace the generated config | +| `config_patch` | `create_patch(fn)` | Transform the generated config before spawn | +| `persist_dir: Option` | `with_persist_dir` | Place the working directory at a findable location ([Persistence](persistence.md)) | +| `snapshot_dir: Option` | `with_snapshot_dir` | Seed the working dir from a snapshot | +| `args: Vec` | `with_args` | Extra process arguments | +| `runtime.start_timeout` | `with_runtime` / `with_start_timeout` | Readiness timeout override | + +--- + +## The Observability Capability + +`ObservabilityCapability` carries optional telemetry endpoints (Prometheus query URL, OTLP ingest URL, Grafana URL). It does not require node control and is populated through `ObservabilityBuilderExt` (`testing-framework/core/src/scenario/builder_ext.rs`): + +```rust,ignore +use testing_framework_core::scenario::ObservabilityBuilderExt; + +let builder = ScenarioBuilder::with_deployment(topology) + .with_metrics_query_url_str("http://127.0.0.1:9090"); +``` + +Each method transitions `ScenarioBuilder` into `ObservabilityScenarioBuilder` (and is a plain setter if you are already there). `Url`-typed, `_str` (panicking), and `try_..._str` (fallible) variants exist for all three endpoints. Deployers merge these values with environment variables; the details, including what telemetry is and is not, are in [Telemetry and External Observability](telemetry.md). + +Capabilities use one marker per scenario, not a set. Choosing `with_node_control()` gives the scenario node control; choosing an observability method supplies telemetry endpoints. Each deployer declares which `Caps` it supports; the [Capability Matrix](capability-matrix.md) lists the available combinations. + +--- + +## See Also + +- [Chaos and Controlled Failure](chaos.md) — node control from workloads +- [ManualCluster: Imperative Node Control](manual-cluster.md) — the imperative alternative +- [Telemetry and External Observability](telemetry.md) — the observability capability in use +- [Capability Matrix](capability-matrix.md) — deployer support by capability diff --git a/book/src/capability-matrix.md b/book/src/capability-matrix.md new file mode 100644 index 0000000..358bd52 --- /dev/null +++ b/book/src/capability-matrix.md @@ -0,0 +1,46 @@ +# Capability Matrix + +This page records what each deployer backend currently supports, based on the deployer implementations. + +The framework ships three deployers: `ProcessDeployer` (local processes), `ComposeDeployer` (Docker Compose), and `K8sDeployer` (Kubernetes/Helm). All three drive the same scenario runtime; they differ in where nodes run and which capabilities they wire into it. + +| Feature | Local | Compose | K8s | +|---|---|---|---| +| Uniform managed scenarios | Yes | Yes | Yes | +| Node control (`with_node_control`) | Yes — start, stop, restart | Restart only (managed); restart + stop (attached) | No — use `ManualCluster` | +| Observability / telemetry inputs | No — telemetry is empty | Yes | Yes | +| Attach / existing clusters | No — rejected | Yes — compose project/services | Yes — label selector | +| External nodes | Yes | Yes | Yes | +| App layer / AppHost composition | Yes (only backend) | No | No | +| Binary providers | Yes | No — container images | No — container images | +| cfgsync artifacts | No — direct config files | Yes | Yes | + +--- + +## Row-by-Row + +**Uniform managed scenarios.** All three deployers implement the `Deployer` trait for scenarios built with `ScenarioBuilder` over a topology: `deployer.deploy(&scenario).await` returns a `Runner`. This is the common path shown in the [Local](deployer-local.md), [Compose](deployer-compose.md), and [Kubernetes](deployer-k8s.md) chapters. + +**Node control.** The local deployer implements `Deployer` and backs it with a `NodeManager` that can start, stop, and restart node processes, including `StartNodeOptions` (peer selection, config overrides, persist/snapshot dirs). The compose deployer wires a `ComposeNodeControl` handle that supports `restart_node` via `docker compose restart`; in attached (existing-cluster) mode it also supports `stop_node` via `docker container stop`. The k8s deployer does not wire a node control handle into managed scenario deployments at all; node lifecycle control on Kubernetes goes through the k8s `ManualCluster` (see [Kubernetes Deployer](deployer-k8s.md) and [ManualCluster](manual-cluster.md)). + +**Observability / telemetry inputs.** Compose and k8s resolve `ObservabilityInputs` from `LOGOS_BLOCKCHAIN_METRICS_QUERY_URL` / `LOGOS_BLOCKCHAIN_METRICS_OTLP_INGEST_URL` / `LOGOS_BLOCKCHAIN_GRAFANA_URL` env vars merged with the scenario's `ObservabilityCapability`, pass the OTLP ingest URL into workspace preparation, and build the run's `Metrics` telemetry handle from the query URL. The local orchestrator constructs its runtime with `Metrics::empty()` and never resolves observability inputs. See [Telemetry and External Observability](telemetry.md). + +**Attach / existing clusters.** `with_existing_cluster(...)` switches the scenario to `ClusterMode::ExistingCluster`. Compose accepts descriptors built with `ExistingCluster::for_compose_project` / `for_compose_services`; k8s accepts `for_k8s_selector` / `for_k8s_selector_in_namespace`. The local deployer explicitly rejects existing-cluster mode with a source-orchestration error. Details in [Existing and External Clusters](external-clusters.md). + +**External nodes.** All three deployers resolve `with_external_node(s)` sources into node clients through `Application::external_node_client`. The local deployer additionally falls back to a generic endpoint parser (`build_external_client`) when the application does not override that hook. + +**App layer / AppHost composition.** The app layer is local-only today: `AppHostLocalDeployer` is a type alias for `ProcessDeployer`. There is no compose or k8s AppHost deployer. See [Backend Scope](app-backend-scope.md). + +**Binary providers.** Binary resolution (`PathBinaryProvider`, `EnvBinaryProvider`, `BuildBinaryProvider`, `DownloadBinaryProvider`, `FallbackBinaryProvider`) lives in the local deployer crate and feeds `LocalProcessSpec`. Compose and k8s nodes run container images instead, so image selection happens through descriptor specs and env-var overrides, not binary providers. See [Binary Providers](binary-providers.md). + +**cfgsync artifacts.** The compose deployer writes a `cfgsync.yaml` into its workspace and can launch a Docker-backed cfgsync config server sidecar (`ComposeConfigServerMode::Docker`); the k8s deployer supports cfgsync-backed config overrides in manual-cluster flows and cfgsync-rendered bootstrap assets in chart values. The local deployer materializes rendered config files directly into each node's working directory with no cfgsync involvement. See [Static Artifacts and cfgsync](cfgsync.md). + +--- + +## Backend Selection + +The local backend runs node processes directly and provides full node control. It requires no infrastructure beyond the node binary, which a [binary provider](binary-providers.md) can build. + +Use Compose for container images, container networking, or telemetry endpoints. Use Kubernetes to exercise charts, NodePort or port-forward access paths, and cluster infrastructure. Attach to an already-running stack when the cluster outlives the test (see [Existing and External Clusters](external-clusters.md)). + +Readiness gating, deploy retries, and artifact preservation are controlled uniformly through `DeploymentPolicy`; see [Readiness, Retry, and Artifact Preservation](deployment-policies.md). diff --git a/book/src/cfgsync.md b/book/src/cfgsync.md new file mode 100644 index 0000000..00b7462 --- /dev/null +++ b/book/src/cfgsync.md @@ -0,0 +1,92 @@ +# Static Artifacts and cfgsync + +This chapter covers how typed app configs become per-node file artifacts and how containerized backends deliver them to nodes that cannot see your filesystem. + +--- + +## Why cfgsync Exists + +The local deployer writes each node's rendered config into the node's working directory. Compose and Kubernetes nodes run in containers without access to the host directory where the framework generated those configs. cfgsync transfers the generated per-node files at startup and when a node is restarted with overridden options. + +cfgsync consists of a typed artifact model and an HTTP service. A node container starts, registers with the cfgsync server, fetches its artifact set, writes the files locally, and then starts the application. The same artifact types support runtime config overrides through `replace_node_artifacts` for dynamic node starts on Kubernetes. + +```mermaid +sequenceDiagram + participant R as Runner + participant S as cfgsync server + participant C as node container + R->>S: render config + artifacts, start server + C->>S: POST /register (identifier, ip, metadata) + C->>S: POST /node + S-->>C: NodeArtifactsPayload (files) + C->>C: write files, exec node binary +``` + +--- + +## The Crates + +| Crate | Role | +|---|---| +| `cfgsync-artifacts` | Pure data model: `ArtifactFile { path, content }`, `ArtifactSet` (with `ensure_unique_paths`) | +| `cfgsync-core` | Protocol types, HTTP server/router, protocol client, config sources, render helpers | +| `cfgsync-adapter` | App-facing materialization: registrations in, artifacts out | +| `cfgsync-runtime` | Runnable server/client: `cfgsync-server` and `cfgsync-client` binaries, env-driven client | + +**Protocol (`cfgsync-core`).** `NodeRegistration` carries a stable `identifier`, an IPv4 address, and an opaque `RegistrationPayload`, adapter-owned JSON metadata the framework never interprets (`with_metadata(&T)` / `from_json_str`). The server answers `/node` with a `NodeArtifactsPayload` (schema version + files) or a structured error: `MissingConfig` (unknown node), `NotReady` (registered, artifacts pending), `Internal`. `Client` wraps the endpoints: `register_node`, `fetch_node_config`, `fetch_node_config_status` (→ `ConfigFetchStatus::{Ready, NotReady, Missing}`), and the administrative `ReplaceNodeArtifactsRequest` for swapping one node's served files. + +**Sources.** A server serves whatever its `NodeConfigSource` resolves: + +- `StaticConfigSource`: an in-memory map of identifier → payload, built from payloads or from a `NodeArtifactsBundle` (per-node entries plus `shared_files` served to everyone). Registration succeeds only for known identifiers. Supports `replace_node_artifacts`. +- `RegistrationConfigSource` (`cfgsync-adapter`) is registration-aware: it records registrations, snapshots them, and asks a materializer for artifacts on every resolve. Per-node overrides installed via `replace_node_artifacts` win over materialized files (shared files are still appended). + +**Materialization (`cfgsync-adapter`).** The adapter contract is one trait: + +```rust,ignore +pub trait RegistrationSnapshotMaterializer: Send + Sync { + fn materialize_snapshot( + &self, + registrations: &RegistrationSnapshot, + ) -> Result; +} +``` + +`RegistrationSnapshot` is the current registration set, sorted by identifier for determinism. The result is `NotReady` (keep polling) or `Ready(MaterializedArtifacts)`: per-node `ArtifactSet`s keyed by identifier plus a shared set appended to every node (`resolve(identifier)` merges them). Wrappers: `CachedSnapshotMaterializer` caches results per snapshot, `PersistingSnapshotMaterializer` additionally pushes ready artifacts into a `MaterializedArtifactsSink`. A prebuilt `MaterializedArtifacts` value is itself a materializer that is always ready. + +**Runtime (`cfgsync-runtime`).** `ServerConfig { port, source }` loads from YAML; `ServerSource` is `static` (serve precomputed artifacts, no registration required) or `registration` (require registration first). The runtime `Client` adds local materialization: `OutputMap` routes artifact paths to disk (`OutputMap::under(root)`, `config_and_shared(config_path, shared_dir)`, or explicit `route(...)`), and `run_client_from_env` drives the whole register-fetch-write loop from `CFG_SERVER_ADDR`, `CFG_HOST_IDENTIFIER`, `CFG_HOST_IP`, `CFG_REGISTRATION_METADATA_JSON`, and output paths (`CFG_FILE_PATH`, `CFG_DEPLOYMENT_PATH`). This is what runs inside node containers before the app binary starts. + +--- + +## From Typed Config to Artifacts + +The boundary between your typed `NodeConfig` and cfgsync lives in `testing-framework/core/src/cfgsync/mod.rs`. + +Apps that implement `ClusterNodeConfigApplication` (see [Implementing Application](implementing-application.md)) get `StaticNodeConfigProvider` for free: build a config for node `i`, rewrite it for backend hostnames (`node-0.svc` instead of `127.0.0.1`), and serialize it. On top of that: + +- `build_static_artifacts::(deployment, hostnames)` produces a `MaterializedArtifacts` with one `/config.yaml` per `node-` identifier. Hostname count must match the node count. +- `render_and_write_registration_server::(...)` renders both the cfgsync server config YAML and the precomputed artifacts YAML to disk, with an `enrich_artifacts` hook for app-specific extras (shared files, additional per-node files). +- `build_node_artifact_override::(deployment, index, hostnames, options)` builds the replacement artifact set for a node started with non-default `StartNodeOptions`; `PeerSelection`, `config_override`, and `config_patch` are interpreted for container backends here (see [node-config.md](node-config.md)). + +The backends consume these directly: the Compose deployer calls `write_registration_server_compose_configs` to render the server config and artifacts into the generated stack directory before `docker compose up` ([Compose Deployer](deployer-compose.md)); the K8s deployer exposes `cfgsync_service`, `cfgsync_hostnames`, and `build_cfgsync_override_artifacts` hooks on its environment trait and pushes override artifacts through `replace_node_artifacts` when its manual cluster starts nodes with options ([Kubernetes Deployer](deployer-k8s.md)). + +```mermaid +graph LR + A["ClusterNodeConfigApplication(typed NodeConfig)"] --> B["build_static_artifacts(MaterializedArtifacts)"] + B --> C[cfgsync server] + C --> D["cfgsync client in container(writes files)"] + D --> E[node process] + C:::pr + D:::pr + E:::pr + classDef pr stroke:#e08a3c,stroke-width:2.5px; +``` + +--- + +## Choosing a Shape + +**Precomputed (used by the framework deployers):** all registrations are known up front, so the deployer materializes every artifact before the stack starts and serves them through a `registration`-kind source. The rendered `cfgsync.artifacts.yaml` remains in the stack directory for inspection. + +**Registration-driven:** when artifacts depend on runtime facts (e.g. which IPs registered), implement `RegistrationSnapshotMaterializer` yourself and return `NotReady` until the snapshot is complete. The runnable examples in `cfgsync/runtime/examples/` (`minimal_cfgsync.rs`, `precomputed_registration_cfgsync.rs`, `wait_for_registrations_cfgsync.rs`) show both shapes end to end. + +The framework deployers currently use the precomputed path. The registration-driven materializer is public API for integrations whose per-node configs cannot be finalized before nodes start. diff --git a/book/src/chaos.md b/book/src/chaos.md index 5a63917..9427c5b 100644 --- a/book/src/chaos.md +++ b/book/src/chaos.md @@ -1,57 +1,138 @@ -# Chaos Workloads +# Chaos and Controlled Failure -> **When should I read this?** You don't need chaos testing to be productive with the framework. Focus on basic scenarios first—chaos is for resilience validation and operational readiness drills once your core tests are stable. +Chaos scenarios deliberately stop or restart nodes and then check recovery. Ordinary workloads perform these operations through the node-control capability. -Chaos in the framework uses node control to introduce failures and validate -recovery. The built-in restart workload lives in -`testing_framework_workflows::workloads::chaos::RandomRestartWorkload`. +--- -## How it works -- Requires `NodeControlCapability` (`enable_node_control()` in the scenario - builder) and a runner that provides a `NodeControlHandle`. -- Randomly selects nodes to restart based on your - include/exclude flags. -- Respects min/max delay between restarts and a target cooldown to avoid - flapping the same node too frequently. -- Runs alongside other workloads; expectations should account for the added - disruption. -- Support varies by runner: node control is not provided by the local runner - and is not yet implemented for the k8s runner. Use a runner that advertises - `NodeControlHandle` support (e.g., compose) for chaos workloads. +## The Shape of a Chaos Scenario + +A chaos test is three ordinary pieces wired together: + +1. A scenario built with `.with_node_control()` (see [Scenario Capabilities](capabilities.md)). +2. A workload that drives traffic, disrupts a node via `ctx.node_control()`, waits for recovery, and drives traffic again. +3. An expectation that verifies the end state converged despite the disruption. + +```mermaid +flowchart LR + T[Drive traffic]:::sc --> D[Disruptrestart node]:::sc + D --> W[Wait for recoveryobserved state]:::sc + W --> T2[Drive traffic again]:::sc + T2 --> V[Expectation:state converged]:::sc + + classDef sc stroke:#9b6dd6,stroke-width:2.5px; +``` + +No `ManualCluster` is involved: the deployer provides a `NodeControlHandle` because the scenario declared the capability. + +--- + +## Worked Example: OpenRaft Leader Failover + +The openraft_kv failover test bootstraps a three-node Raft cluster, expands it to three voters, writes a batch, restarts the *leader*, then writes a second batch through the node elected next. Its workload is in `examples/openraft_kv/testing/workloads/src/failover.rs`: -## Usage ```rust,ignore -use std::time::Duration; +#[async_trait] +impl Workload for OpenRaftKvFailoverWorkload { + fn name(&self) -> &str { + "openraft_kv_failover_workload" + } -use testing_framework_core::scenario::ScenarioBuilder; -use testing_framework_workflows::{ScenarioBuilderExt, workloads::chaos::RandomRestartWorkload}; + async fn start(&self, ctx: &RunContext) -> Result<(), DynError> { + let clients = ctx.node_clients().snapshot(); + let observer = ctx.require_extension::>()?; -pub fn random_restart_plan() -> testing_framework_core::scenario::Scenario< - testing_framework_core::scenario::NodeControlCapability, -> { - ScenarioBuilder::topology_with(|t| t.network_star().nodes(2)) - .enable_node_control() - .with_workload(RandomRestartWorkload::new( - Duration::from_secs(45), // min delay - Duration::from_secs(75), // max delay - Duration::from_secs(120), // target cooldown - true, // include nodes - )) - .expect_consensus_liveness() - .with_run_duration(Duration::from_secs(150)) - .build() + ensure_cluster_size(&clients, 3)?; + self.bootstrap_cluster(&clients).await?; + + let initial_leader = wait_for_observed_leader(&observer, self.timeout, None).await?; + let membership = OpenRaftMembership::discover(&clients).await?; + + self.promote_cluster(&observer, &clients, initial_leader, &membership).await?; + self.write_initial_batch(&clients, initial_leader).await?; + + let new_leader = self + .restart_leader_and_wait_for_failover(ctx, &observer, initial_leader) + .await?; + self.write_second_batch(&clients, new_leader).await?; + + Ok(()) + } } ``` -## Expectations to pair -- **Consensus liveness**: ensure blocks keep progressing despite restarts. -- **Height convergence**: optionally check all nodes converge after the chaos - window. -- Any workload-specific inclusion checks if you’re also driving transactions. +The disruption itself is a few lines: -## Best practices -- Keep delays/cooldowns realistic; avoid back-to-back restarts that would never - happen in production. -- Limit chaos scope: toggle nodes based on what you want to - test. -- Combine with observability: monitor metrics/logs to explain failures. +```rust,ignore +let Some(control) = ctx.node_control() else { + return Err("openraft failover workload requires node control".into()); +}; + +control.restart_node(&format!("node-{leader_id}")).await?; + +let new_leader = wait_for_observed_leader(observer, self.timeout, Some(leader_id)).await?; +``` + +The guard returns a clear error if the capability is missing. + +### Assembling the Scenario + +`build_failover_scenario` (`examples/openraft_kv/examples/src/lib.rs`) puts workload, expectation, and capability together: + +```rust,ignore +pub fn build_failover_scenario( + run_duration: Duration, + workload_timeout: Duration, +) -> anyhow::Result> { + Ok(OpenRaftKvScenarioBuilder::with_existing_openraft_kv_app( + OpenRaftKvExistingClusterApp::nodes(3), + ) + .enable_node_control() + .with_run_duration(run_duration) + .with_workload(OpenRaftKvClusterAccessible::new(3)) + .with_workload( + OpenRaftKvFailoverWorkload::new() + .first_batch(INITIAL_WRITE_BATCH) + .second_batch(SECOND_WRITE_BATCH) + .timeout(workload_timeout) + .key_prefix(RAFT_KEY_PREFIX), + ) + .with_expectation( + OpenRaftKvConverges::new(TOTAL_WRITES) + .timeout(run_duration) + .key_prefix(RAFT_KEY_PREFIX), + ) + .build()?) +} +``` + +The return type is `Scenario`, so only deployers that provide node control will accept it. Run it locally or on compose: + +```bash +cargo run -p openraft-kv-examples --bin openraft_kv_basic_failover +cargo run -p openraft-kv-examples --bin openraft_kv_compose_failover +``` + +The `openraft_kv_k8s_failover` bin executes the same failover flow on Kubernetes, but uses `ManualCluster` imperatively (`start_node` per node, `restart_node`, `wait_network_ready`). See [ManualCluster: Imperative Node Control](manual-cluster.md). + +--- + +## Patterns + +**Restart-and-verify.** The minimal chaos loop: write known data, `restart_node`, wait for readiness, verify the data survived. Restarts reuse the node's existing working directory, so on-disk state survives them by default; use `with_snapshot_dir` to seed a restore from saved state; details are in [Persistence, Snapshots, and Recovery Testing](persistence.md). + +**Leader failover.** Restart the node that currently holds a distinguished role. The failover workload discovers the leader from observed cluster state instead of assuming a node index. Passing the old identity to the wait (`different_from: Some(leader_id)` above) verifies that leadership changed rather than accepting the old leader after it restarts. + +**Readiness waits after disruption.** Wait before sending traffic after a restart. The example waits on *observed application state* (an agreed leader across all nodes) via an [observation handle](observation.md), which checks more than an HTTP readiness probe. In imperative flows, `ManualCluster::wait_network_ready()` covers transport-level readiness (the k8s bin calls it right after `restart_node`); inside a declarative workload, wait on observed state. + +**Pair chaos with continuous observation.** A background observer polls every node through the disruption, so waits read stored snapshots and can report the last observation (`timed out waiting for observed leader agreement ...; last observation: node=0 leader=None ...`). A workload can also poll clients directly, but must then track the polling state itself. + +Managed clusters get a minimum 30-second cooldown window after the workload phase before expectations run, allowing post-chaos state to settle; see [Expectations and Evaluation](expectations.md). + +--- + +## See Also + +- [Scenario Capabilities](capabilities.md) — `with_node_control` and `StartNodeOptions` +- [Continuous Observation](observation.md) — the observer used for recovery waits +- [Persistence, Snapshots, and Recovery Testing](persistence.md) — restart with retained state +- [ManualCluster: Imperative Node Control](manual-cluster.md) — the imperative variant diff --git a/book/src/ci-integration.md b/book/src/ci-integration.md deleted file mode 100644 index a258d2c..0000000 --- a/book/src/ci-integration.md +++ /dev/null @@ -1,360 +0,0 @@ -# CI Integration - -Both **LocalDeployer** and **ComposeDeployer** work well in CI environments. Choose based on your tradeoffs. - -## Runner Comparison for CI - -**LocalDeployer (Host Runner):** -- Faster startup (no Docker overhead) -- Good for quick smoke tests -- **Trade-off:** Less isolation (processes share host resources) - -**ComposeDeployer (Recommended for CI):** -- Better isolation (containerized) -- Reproducible environment -- Can integrate with external Prometheus/Grafana (optional) -- **Trade-offs:** Slower startup (Docker image build), requires Docker daemon - -**K8sDeployer:** -- Production-like environment -- Full resource isolation -- **Trade-offs:** Slowest (cluster setup + image loading), requires cluster access -- Best for nightly/weekly runs or production validation - -**Existing Examples:** - -See `.github/workflows/lint.yml` (jobs: `host_smoke`, `compose_smoke`) for CI examples running the demo scenarios in this repository. - -## Complete CI Workflow Example - -Here's a comprehensive GitHub Actions workflow demonstrating host and compose runners with caching, matrix testing, and log collection: - -```yaml -name: Testing Framework CI - -on: - push: - branches: [main, develop] - pull_request: - branches: [main] - -env: - CARGO_TERM_COLOR: always - RUST_BACKTRACE: 1 - -jobs: - # Quick smoke test with host runner (no Docker) - host_smoke: - name: Host Runner Smoke Test - runs-on: ubuntu-latest - timeout-minutes: 15 - - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Set up Rust toolchain - uses: actions-rs/toolchain@v1 - with: - profile: minimal - toolchain: nightly - override: true - - - name: Cache Rust dependencies - uses: actions/cache@v3 - with: - path: | - ~/.cargo/bin/ - ~/.cargo/registry/index/ - ~/.cargo/registry/cache/ - ~/.cargo/git/db/ - target/ - key: ${{ runner.os }}-cargo-host-${{ hashFiles('**/Cargo.lock') }} - restore-keys: | - ${{ runner.os }}-cargo-host- - - - name: Cache logos-blockchain-node build - uses: actions/cache@v3 - with: - path: | - ../logos-blockchain-node/target/release/logos-blockchain-node - key: ${{ runner.os }}-nomos-${{ hashFiles('../logos-blockchain-node/**/Cargo.lock') }} - restore-keys: | - ${{ runner.os }}-nomos- - - - name: Run host smoke test - run: | - # Use run-examples.sh which handles setup automatically - scripts/run/run-examples.sh -t 120 -n 3 host - - - name: Upload logs on failure - if: failure() - uses: actions/upload-artifact@v3 - with: - name: host-runner-logs - path: | - .tmp/ - *.log - retention-days: 7 - - # Compose runner matrix (with Docker) - compose_matrix: - name: Compose Runner (${{ matrix.topology }}) - runs-on: ubuntu-latest - timeout-minutes: 25 - - strategy: - fail-fast: false - matrix: - topology: - - "3v1e" - - "5v1e" - - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Set up Rust toolchain - uses: actions-rs/toolchain@v1 - with: - profile: minimal - toolchain: nightly - override: true - - - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v2 - - - name: Cache Rust dependencies - uses: actions/cache@v3 - with: - path: | - ~/.cargo/bin/ - ~/.cargo/registry/index/ - ~/.cargo/registry/cache/ - ~/.cargo/git/db/ - target/ - key: ${{ runner.os }}-cargo-compose-${{ hashFiles('**/Cargo.lock') }} - restore-keys: | - ${{ runner.os }}-cargo-compose- - - - name: Cache Docker layers - uses: actions/cache@v3 - with: - path: /tmp/.buildx-cache - key: ${{ runner.os }}-buildx-${{ hashFiles('Dockerfile', 'scripts/build/build_test_image.sh') }} - restore-keys: | - ${{ runner.os }}-buildx- - - - name: Run compose test - env: - TOPOLOGY: ${{ matrix.topology }} - run: | - # Build and run with the specified topology - scripts/run/run-examples.sh -t 120 -n ${TOPOLOGY:0:1} compose - - - name: Collect Docker logs on failure - if: failure() - run: | - mkdir -p logs - for container in $(docker ps -a --filter "name=nomos-compose-" -q); do - docker logs $container > logs/$(docker inspect --format='{{.Name}}' $container).log 2>&1 - done - - - name: Upload logs and artifacts - if: failure() - uses: actions/upload-artifact@v3 - with: - name: compose-${{ matrix.topology }}-logs - path: | - logs/ - .tmp/ - retention-days: 7 - - - name: Clean up Docker resources - if: always() - run: | - docker compose down -v 2>/dev/null || true - docker ps -a --filter "name=nomos-compose-" -q | xargs -r docker rm -f - - # Summary job (requires all tests to pass) - ci_success: - name: CI Success - needs: [host_smoke, compose_matrix] - runs-on: ubuntu-latest - if: always() - - steps: - - name: Check all jobs - run: | - if [[ "${{ needs.host_smoke.result }}" != "success" ]] || \ - [[ "${{ needs.compose_matrix.result }}" != "success" ]]; then - echo "One or more CI jobs failed" - exit 1 - fi - echo "All CI jobs passed!" -``` - -## Workflow Features - -1. **Matrix Testing:** Runs compose tests with different topologies (`3v1e`, `5v1e`) -2. **Caching:** Caches Rust dependencies, Docker layers, and logos-blockchain-node builds for faster runs -3. **Log Collection:** Automatically uploads logs and artifacts when tests fail -4. **Timeout Protection:** Reasonable timeouts prevent jobs from hanging indefinitely -6. **Clean Teardown:** Ensures Docker resources are cleaned up even on failure - -## Customization Points - -**Topology Matrix:** - -Add more topologies for comprehensive testing: - -```yaml -matrix: - topology: - - "3v1e" - - "5v1e" - - "10v2e" # Larger scale -``` - -**Timeout Adjustments:** - -Increase `timeout-minutes` for longer-running scenarios or slower environments: - -```yaml -timeout-minutes: 30 # Instead of 15 -``` - -**Artifact Retention:** - -Change `retention-days` based on your storage needs: - -```yaml -retention-days: 14 # Keep logs for 2 weeks -``` - -**Conditional Execution:** - -Run expensive tests only on merge to main: - -```yaml -if: github.event_name == 'push' && github.ref == 'refs/heads/main' -``` - -## Best Practices - -### Use Helper Scripts - -Prefer `scripts/run/run-examples.sh` which handles all setup automatically: - -```bash -scripts/run/run-examples.sh -t 120 -n 3 host -``` - -This is more reliable than manual `cargo run` commands. - -### Cache Aggressively - -Cache Rust dependencies, logos-blockchain-node builds, and Docker layers to speed up CI: - -```yaml -- name: Cache Rust dependencies - uses: actions/cache@v3 - with: - path: | - ~/.cargo/bin/ - ~/.cargo/registry/index/ - ~/.cargo/registry/cache/ - ~/.cargo/git/db/ - target/ - key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }} -``` - -### Collect Logs on Failure - -Always upload logs when tests fail for easier debugging: - -```yaml -- name: Upload logs on failure - if: failure() - uses: actions/upload-artifact@v3 - with: - name: test-logs - path: | - .tmp/ - *.log - retention-days: 7 -``` - -### Split Workflows for Faster Iteration - -For large projects, split host/compose/k8s into separate workflow files: - -- `.github/workflows/test-host.yml` — Fast smoke tests -- `.github/workflows/test-compose.yml` — Reproducible integration tests -- `.github/workflows/test-k8s.yml` — Production-like validation (nightly) - -### Run K8s Tests Less Frequently - -K8s tests are slower. Consider running them only on main branch or scheduled: - -```yaml -on: - push: - branches: [main] - schedule: - - cron: '0 2 * * *' # Daily at 2 AM -``` - -## Platform-Specific Notes - -### Ubuntu Runners - -- Docker pre-installed and running -- Best for compose/k8s runners -- Most common choice - -### macOS Runners - -- Docker Desktop not installed by default -- Slower and more expensive -- Use only if testing macOS-specific issues - -### Self-Hosted Runners - -- Cache Docker images locally for faster builds -- Set resource limits (`SLOW_TEST_ENV=true` if needed) -- Ensure cleanup scripts run (`docker system prune`) - -## Debugging CI Failures - -### Enable Debug Logging - -Add debug environment variables temporarily: - -```yaml -env: - RUST_LOG: debug - LOGOS_BLOCKCHAIN_LOG_LEVEL: debug -``` - -### Preserve Containers (Compose) - -Set `COMPOSE_RUNNER_PRESERVE=1` to keep containers running for inspection: - -```yaml -- name: Run compose test (preserve on failure) - env: - COMPOSE_RUNNER_PRESERVE: 1 - run: scripts/run/run-examples.sh -t 120 -n 3 compose -``` - -### Access Artifacts - -Download uploaded artifacts from the GitHub Actions UI to inspect logs locally. - -## Next Steps - -- [Running Examples](running-examples.md) — Manual execution for local development -- [Environment Variables](environment-variables.md) — Full variable reference -- [Troubleshooting](troubleshooting.md) — Common CI-specific issues - diff --git a/book/src/ci.md b/book/src/ci.md new file mode 100644 index 0000000..341bc1a --- /dev/null +++ b/book/src/ci.md @@ -0,0 +1,77 @@ +# Continuous Integration + +This chapter covers how this repository checks itself, and patterns for running framework-based tests in your own CI. + +--- + +## Workflows in This Repository + +Two GitHub Actions workflows live in `.github/workflows/`. + +### `lint.yml` + +Runs on every push and pull request, with per-ref concurrency cancellation. All jobs pin the `nightly-2025-09-14` toolchain and cache `~/.cargo/registry`, `~/.cargo/git`, and `target/` keyed on `Cargo.lock`. + +| Job | Command | Checks | +|---|---|---| +| `fmt` | `cargo +nightly-2025-09-14 fmt --all -- --check` | formatting | +| `clippy` | `cargo clippy --all --all-targets --all-features -- -D warnings` | lints, warnings as errors | +| `deny` | `cargo deny check -c .cargo-deny.toml --show-stats -D warnings` | licenses, advisories, bans | +| `taplo` | `taplo fmt --check` and `taplo lint` | TOML formatting and lints | +| `machete` | `cargo machete` | unused dependencies | + +### `deploy-pages.yml` + +Builds this book with `mdbook build book` and publishes `target/book` to GitHub Pages. It triggers on pushes to `master` that touch `book/**`, or manually via `workflow_dispatch`. + +**Note:** CI currently covers linting and the book. The unit tests and example scenarios run via `cargo test` / `cargo run` on developer machines; there is no test workflow yet. + +--- + +## Helper Scripts + +- `scripts/run/checks.sh` is an informational, best-effort environment sanity check. It reports workspace and disk state, the Rust toolchain, Docker and Docker Compose availability, the Kubernetes context (including whether a `:local` image tag will be visible to `kind`, `minikube`, or `docker-desktop` clusters), and the current values of runner debug flags such as `COMPOSE_RUNNER_PRESERVE` and `K8S_RUNNER_PRESERVE`. Run it first when a backend misbehaves. +- `scripts/run/check-boundaries.sh` is a boundary guard for an adopter checkout living next to this repository. It fails if the adopter's topology crate references extension-specific symbols (`cfgsync`, compose/k8s deployer types), keeping the [framework/application boundary](tf-boundaries.md) enforceable by grep. + +--- + +## Patterns for Consumers + +If your repository builds tests on this framework, the following translate directly into CI configuration. + +### Cache the build for `BuildBinaryProvider` + +Local scenarios that resolve node binaries through a `BuildBinaryProvider` invoke `cargo build` at deploy time. On a cold runner this can dominate the job. Cache `~/.cargo/registry`, `~/.cargo/git`, and `target/` keyed on `Cargo.lock`, as `lint.yml` does, so the deploy-time build is incremental. Resolution is also cached in-process and serialized across concurrent test processes with a file lock, so parallel test binaries do not race the same build; see [Binary Providers](binary-providers.md). + +### Ensure Docker for compose tests + +Compose scenarios need a working Docker daemon and the node images already present: the runner verifies images with `docker image inspect` and fails with `MissingImage` rather than building or pulling. Add an image build/pull step before the test step. Decide your skip policy explicitly: the in-repo example binaries treat `ComposeRunnerError::DockerUnavailable` as a graceful skip, which is convenient locally but silently masks coverage loss in CI. In a pipeline, prefer failing (or gating the job on a Docker-capable runner). + +### Slow runners + +Set `SLOW_TEST_ENV=true` on constrained runners; the framework doubles its internal timeouts (`testing_framework_core::adjust_timeout`). The k8s deployer's wait timeouts can also be tuned individually; see [Environment Variables](environment-variables.md). + +### Preserve artifacts on failure + +By default every backend tears down and deletes its working state. To retain evidence from failed CI runs, preserve and upload the artifacts: + +```yaml +- name: Run scenarios + run: cargo test -p my-scenarios + env: + TF_KEEP_LOGS: "1" # keep local per-node working directories + COMPOSE_RUNNER_PRESERVE: "1" # keep the compose workspace and containers +- name: Upload artifacts + if: failure() + uses: actions/upload-artifact@v4 + with: + name: scenario-artifacts + path: | + **/.tmp* +``` + +Local node directories are created under the test process's working directory; note that panicking tests preserve their node directories automatically. The equivalent in code is `CleanupPolicy { preserve_artifacts: true }` via `with_deployment_policy`. What lands in those directories and how to read them is covered in [Diagnostics and Retained Artifacts](diagnostics.md). + +### Reproduce failures + +Log or fix the deployment seed (`with_deployment_seed`) so a failing CI run can be replayed locally with the same generated deployment; see [Seeds and Reproducibility](seeds.md). diff --git a/book/src/cluster-provisioning.md b/book/src/cluster-provisioning.md new file mode 100644 index 0000000..b60cff8 --- /dev/null +++ b/book/src/cluster-provisioning.md @@ -0,0 +1,121 @@ +# Shared Cluster Provisioning + +App composition and uniform scenarios share a cluster-provisioning model. A request describes the cluster source and required behavior. A provisioner returns a backend-independent `ClusterHandle` and registers any managed lifetime with the app cleanup stack. + +--- + +## One Request, Three Sources + +`ClusterRequest` separates what the test needs from how a backend supplies it: + +```rust,ignore +let managed = ClusterRequest::::managed(QueueTopology::new(3)); +let attached = ClusterRequest::::attached(existing_cluster); +let external = ClusterRequest::::external(node_sources); +``` + +| Source | Nodes started by the framework | Clients | Node control | Framework teardown | +|---|---:|---:|---:|---:| +| `Managed` | Yes, unless start mode is on demand | Yes | When requested and supported | Yes | +| `Attached` | No | Yes | According to the attached cluster's control profile | Only resources the framework itself acquires | +| `External` | No | Yes | No | No | + +Managed and attached sources can also include external nodes with `with_external_nodes(...)`. This is useful when one logical cluster combines framework-visible nodes from more than one source. + +--- + +## Requesting Behavior + +The request carries requirements that are meaningful across backends: + +| Method | Meaning | +|---|---| +| `with_policy(policy)` | Apply readiness, retry, cleanup, and network-control policy. | +| `with_start_mode(Eager)` | Start managed nodes while provisioning. This is the default. | +| `with_start_mode(OnDemand)` | Prepare a managed cluster but let test code start nodes explicitly. | +| `with_control(Full)` | Require the node-control surface on the returned handle. | +| `with_network_control()` | Require backend network control. | +| `with_network_recovery(recovery)` | Register application recovery after a network effect is released; also requests network control. | + +Backends may support different combinations of these requirements. The [Capability Matrix](capability-matrix.md) records current coverage. + +--- + +## Provisioning Inside an AppDeployment + +`DeployContext` is parameterized by a `ClusterProvisioner`. Its `deploy_cluster` method is the app-layer entry point: + +```rust,ignore +#[async_trait] +impl AppDeployment for QueueApp { + type Handle = ClusterHandle; + + async fn deploy( + self, + ctx: &mut DeployContext, + ) -> Result { + ctx.deploy_cluster(ClusterRequest::::managed(self.topology)) + .await + } +} +``` + +`DeployContext::deploy_cluster` requests the full common node-control surface because app workloads receive the returned cluster handle directly. The local convenience `deploy_local_cluster` expresses the common managed, eager case. Use `deploy_cluster` when ownership mode, start mode, or policy must be visible in the app definition. + +`with_app(app)` selects the default local provisioner. `with_app_using(app, provisioner)` supplies another provisioner. The root deployment must implement `AppDeployment` for that provisioner type; code written only as `AppDeployment` uses the default local type. + +--- + +## The Returned Handle + +`ClusterHandle` presents the common runtime surface: + +- clients: `node_clients`, `clients`, `first_client`, `node_client`; +- cluster description: `deployment`, `node_count`, `control_profile`; +- node operations when present: `start_node`, `stop_node`, `restart_node`, `wait_node_ready`; +- cluster readiness: `wait_network_ready`; +- network effects when present: `network_control`. + +Unavailable operations return an error or `None`; callers can inspect `control_profile()` when behavior depends on ownership mode. + +The handle does not own managed lifetime. The provisioner returns a private cleanup guard alongside the runtime surfaces. `DeployContext` moves that guard into the scenario cleanup stack, which runs in reverse acquisition order on normal completion and partial deployment failure. + +--- + +## Backend Boundary + +`ClusterProvisioner` has one operation: + +```rust,ignore +#[async_trait] +pub trait ClusterProvisioner: Clone + Send + Sync + 'static { + async fn provision_cluster( + &self, + request: ClusterRequest, + ) -> Result, DynError>; +} +``` + +A backend implementation translates the request into concrete resources, clients, control adapters, readiness, and cleanup. `ClusterUnit` carries these values from the provisioner; applications normally use the resulting `ClusterHandle`. + +The local provisioner currently supports managed and external sources. Attached support and equivalent Compose/Kubernetes app provisioners require backend implementations, but not another application-composition model. + +--- + +## Relation to Other Entry Patterns + +- A uniform scenario asks its deployer to provision the scenario's primary cluster. +- A composed stack asks its `DeployContext` to provision one or more child clusters. +- An attached or external test changes `ClusterSource`, while workloads keep using clients and available controls. +- `ManualCluster` uses local provisioning machinery directly and gives imperative code responsibility for sequencing. + +The entry patterns differ in who describes and drives the test. They do not need separate definitions of what a cluster is, which controls it exposes, or who tears it down. + +--- + +## See Also + +- [Existing and External Clusters](external-clusters.md): declaring non-managed sources in ordinary scenarios. +- [AppDeployment and DeployContext](app-deployment.md): composing cluster and process children. +- [Handle Ownership and Teardown](handles-teardown.md): the lifetime boundary in detail. +- [Readiness, Retry, and Cleanup](deployment-policies.md): the policies carried by a request. diff --git a/book/src/composing-stacks.md b/book/src/composing-stacks.md new file mode 100644 index 0000000..42248b4 --- /dev/null +++ b/book/src/composing-stacks.md @@ -0,0 +1,154 @@ +# Composing Heterogeneous Stacks + +A root `AppDeployment` deploys the components, passes dependency addresses between them, and exposes typed handles. + +A scenario accepts one `with_app` registration (see [AppHost and with_app](app-host.md)), so the root deployment composes its children. It exposes component handles needed by workloads and may also expose an aggregate stack handle. The `examples/multi_app` fixture contains a queue cluster, job-worker process, and kv result-store cluster in one job-processing pipeline. + +--- + +## The Root App + +```rust,ignore +// examples/multi_app/fixture/src/lib.rs +#[derive(Clone)] +struct JobStackApp { + queue_nodes: usize, + result_nodes: usize, +} + +impl JobStackApp { + fn new() -> Self { + Self { + queue_nodes: 2, + result_nodes: 2, + } + } +} + +#[async_trait] +impl AppDeployment for JobStackApp { + type Handle = JobStackHandle; + + async fn deploy(self, ctx: &mut DeployContext) -> Result { + let queue = ctx + .deploy_and_expose(QueueLocalApp::nodes(self.queue_nodes)) + .await?; + let results = ctx + .deploy_and_expose(KvLocalApp::nodes(self.result_nodes)) + .await?; + + let queue_url = queue + .first_client() + .ok_or("queue cluster has no clients")? + .base_url() + .clone(); + let results_url = results + .first_client() + .ok_or("result store has no clients")? + .base_url() + .clone(); + let worker = ctx + .deploy_and_expose(JobWorkerApp::new(queue_url, results_url)) + .await?; + + let stack = JobStackHandle { queue, results, worker }; + ctx.expose(stack.clone())?; + + Ok(stack) + } +} +``` + +The example establishes these relationships: + +- **Children are deployed through the context** (`deploy_and_expose`), so each cluster and the worker process are owned by the runtime for the whole run. +- **Dependencies are constructor arguments.** The worker receives the queue and result-store URLs from the already-deployed clusters, so the root deployment shows the dependency graph. +- **Both levels are exposed**: each component handle and the aggregate `JobStackHandle`, allowing workloads to request the smallest handle they need. + +```mermaid +flowchart TD + Root[JobStackApp] --> Q["queue cluster x2LocalAppCluster<QueueEnv>"] + Root --> R["result store x2LocalAppCluster<KvEnv>"] + Q --> W["job workerLocalProcessApp"] + R --> W + Q --> St[JobStackHandle] + R --> St + W --> St + Q:::cl + R:::cl + W:::pr + St:::hd + classDef cl stroke:#4a90d9,stroke-width:2.5px; + classDef pr stroke:#e08a3c,stroke-width:2.5px; + classDef hd stroke:#4caf7d,stroke-width:2.5px; +``` + +--- + +## Workloads Require What They Need + +Each workload asks for exactly the handles it uses, the whole stack or one component: + +```rust,ignore +async fn start(&self, ctx: &RunContext) -> Result<(), DynError> { + let stack = ctx.require_app::()?; + let queue = stack.queue.first_client().ok_or("queue cluster has no clients")?; + + for index in 0..self.count { + let response: EnqueueResponse = queue + .post("/queue/enqueue", &EnqueueRequest { payload: job_key(index) }) + .await?; + if !response.accepted { + return Err(format!("queue rejected job {index}").into()); + } + } + Ok(()) +} +``` + +Assembling and running the scenario is unchanged from any other AppHost run: + +```rust,ignore +let mut scenario = AppHost::scenario() + .with_app(JobStackApp::new()) + .with_run_duration(Duration::from_secs(10)) + .with_workload(EnqueueJobs::new(10)) + .with_expectation(AllJobsCompleted::new(10)) + .build()?; + +let deployer = AppHostLocalDeployer::default(); +let runner = deployer.deploy(&scenario).await?; +runner.run(&mut scenario).await?; +``` + +```bash +cargo test -p multi-app-e2e +``` + +--- + +## Wiring Dependencies Between Components + +**Pass dependencies through constructors.** Deploy the dependency first and pass its handle or address into the dependent's constructor, as `JobWorkerApp::new(queue_url, results_url)` does above. This records the dependency graph and acquisition order in the root deployment. A child can call `ctx.require::()`, but then it depends on another deployment having exposed `T` earlier. If that did not happen, deployment fails at run time with `HandleMissing`. + +The same rule applies to process-level wiring. The job worker is a [`LocalProcessApp`](local-process-app.md) whose `LaunchSpec` receives the queue and store URLs as command-line arguments: deploy the dependency, read its client's `base_url()`, and feed the address into the process. Do not have the process guess. + +**Use named handles for two instances of one type.** The registry allows one default handle per concrete type; a second `expose` of the same type is a duplicate error (see [Handle Ownership and Teardown](handles-teardown.md)). Two kv clusters in one stack therefore need names: + +```rust,ignore +ctx.expose_named("primary", primary)?; +ctx.expose_named("replica", replica)?; + +// in the workload: +let primary = ctx.require_app_named::>("primary")?; +``` + +**Expose components as well as the stack when both are used.** A workload that touches one component can request its handle directly, while stack-level workloads can request the aggregate handle. + +--- + +## See Also + +- [AppDeployment and DeployContext](app-deployment.md): the composition API in detail. +- [Uniform Child Clusters: LocalAppCluster](local-app-cluster.md): the child clusters used here. +- [Backend Scope](app-backend-scope.md): where composed stacks can run today. diff --git a/book/src/crate-map.md b/book/src/crate-map.md new file mode 100644 index 0000000..5473b50 --- /dev/null +++ b/book/src/crate-map.md @@ -0,0 +1,107 @@ +# Crate and API Map + +This chapter maps which crate owns which concept, what each one exports, and how they depend on each other. + +The workspace splits into three layers: the app-agnostic core, the deployment backends, and the cfgsync configuration pipeline. Example applications live in their own workspace layout under `examples/` and depend on the framework, never the other way around. + +```mermaid +graph BT + art[cfgsync-artifacts] + cc[cfgsync-core] --> art + ca[cfgsync-adapter] --> cc + ca --> art + cr[cfgsync-runtime] --> ca + core[testing-framework-core] --> ca + local[testing-framework-runner-local] --> core + compose[testing-framework-runner-compose] --> core + k8s[testing-framework-runner-k8s] --> core + k8s --> cc + k8s --> art + app[testing-framework-app] --> core + app --> local +``` + +--- + +## testing-framework-core + +Path: `testing-framework/core`. The scenario engine and everything app-agnostic: builder, runtime, topology, observation, sources, capabilities. Every other framework crate depends on it. + +| Module | Contents | +|---|---| +| `env` | The `Application` trait (re-exported from `scenario`) | +| `scenario` | `ScenarioBuilder`, `Scenario`, `Workload`, `Expectation`, `RunContext`, `RunHandle`, `Runner`, `Deployer`, `RuntimeExtensionFactory`, `DeploymentPolicy`, cluster provisioning (`ClusterRequest`, `ClusterSource`, `ClusterHandle`, `ClusterProvisioner`), control traits, capability markers, sources, observability inputs | +| `topology` | `DeploymentDescriptor`, `DeploymentProvider`, `FixedDeploymentProvider`, `DeploymentSeed`, `DeploymentPlan`, `TopologyShapeBuilder`, `ClusterTopology`, `NodeCountTopology` | +| `observation` | `Observer`, `SourceProvider`, `StaticSourceProvider`, `SourceProviderFactory`, `ObservationExtensionFactory`, `ObservationRuntime`, `ObservationHandle`, `ObservationConfig` | +| `workloads` | Generic reusable workloads and verbs: `ChaosBuilderExt`, `RestartChaosBuilderExt`, `RandomRestartWorkload`, `NetworkPartitionWorkload` | +| `runtime` | `manual` (the `ManualClusterHandle` interface), `process`, `retry` | +| `cfgsync` | Bridges deployments to the cfgsync pipeline (re-exports `cfgsync-adapter`, rendering output types) | + +Key builder entry points: `ScenarioBuilder::with_deployment`, `::new(provider)`, and the capability-gated variants `with_node_control()` and `with_observability()`. `ObservabilityBuilderExt` and `CoreBuilderExt` live here too. + +--- + +## testing-framework-app + +Path: `testing-framework/app`. The app layer for heterogeneous stacks: singleton processes, extra clusters, or several applications composed into one system. Depends on core plus the local deployer; the app layer is local-only today (see [Backend Scope](app-backend-scope.md)). + +| Export | Role | +|---|---| +| `AppHost`, `AppHostEnv`, `AppHostTopology`, `AppHostScenarioBuilder`, `AppHostLocalDeployer` | Zero-node scenario entry point: `AppHost::scenario().with_app(...)` | +| `AppDeployment`, `AppHandle` | The composition trait and its blanket handle bound | +| `DeployContext` | Deploy children, expose typed/named handles, provision clusters through `deploy_cluster` | +| `AppDeploymentFactory`, `AppScenarioBuilderExt`, `AppRunContextExt` | Builder registration (`with_app`) and workload-side handle lookup (`app`, `require_app`, ...) | +| `LocalProcessApp`, `LocalProcessHandle` | One supervised local process as an app | +| `LocalAppCluster` | Alias for the common `ClusterHandle` used by local child clusters | +| `AppRuntime`, `HandleRegistry`, `AppDeployError` | Runtime handle storage and errors; managed cleanup is kept separately | + +--- + +## Deployment Backends + +Each backend implements `Deployer` for its environment trait and returns the same core `Runner`. + +**`testing-framework-runner-local`** (`testing-framework/deployers/local`) spawns nodes as local processes. Exports `ProcessDeployer`, `ManualCluster`, `NodeManager`, the `LocalDeployerEnv` / `LocalBinaryApp` environment traits with config/port helpers (`LocalProcessSpec`, `LocalNodePorts`, `build_local_cluster_node_config`, ...), process primitives (`LaunchSpec`, `NodeEndpoints`, `ProcessNode`), and the whole `binary` module (`BinaryProvider` and its implementations). Honors `TF_KEEP_LOGS` for tempdir retention. + +**`testing-framework-runner-compose`** (`.../compose`) renders a Docker Compose stack. Exports `ComposeDeployer`, `ComposeDeployEnv`, descriptor builders (`ComposeDescriptor`, `NodeDescriptor`), compose lifecycle commands (`compose_up`, `compose_down`, `dump_compose_logs`), and the Docker config-server support used to serve cfgsync artifacts to containers. + +**`testing-framework-runner-k8s`** (`.../k8s`) installs a Helm release. Exports `K8sDeployer`, `K8sDeployEnv`, `ManualCluster` (K8s variant), Helm/chart-value infrastructure (`HelmInstallSpec`, `RunnerChartValues`, `render_binary_config_node_chart_assets`, ...), and wait/cleanup helpers. Depends directly on `cfgsync-core` and `cfgsync-artifacts` for artifact delivery. + +--- + +## cfgsync + +cfgsync is the typed pipeline that turns app config into per-node files: app config → registration snapshot → per-node artifact sets → backend rendering. Consumed by the compose and k8s deployers (locally, configs are written straight to disk). See [Static Artifacts and cfgsync](cfgsync.md). + +| Crate | Responsibility | Key exports | +|---|---|---| +| `cfgsync-artifacts` | App-agnostic artifact model | `ArtifactFile`, `ArtifactSet` | +| `cfgsync-core` | Protocol, client/server, template rendering, bundles | `Client`, `serve_cfgsync`, `NodeRegistration`, `NodeArtifactsPayload`, `RenderedCfgsync`, `NodeArtifactsBundle`, config sources | +| `cfgsync-adapter` | Materializing registration snapshots into artifacts | `RegistrationSnapshotMaterializer`, `CachedSnapshotMaterializer`, `PersistingSnapshotMaterializer`, `MaterializedArtifacts`, `RegistrationConfigSource` | +| `cfgsync-runtime` | Standalone server/client binaries-facing runtime | `serve_from_config`, `run_client_from_env`, `ServerConfig` | + +--- + +## Examples Workspace Layout + +Every example app follows the same four-part shape under `examples//`: + +```text +examples/kvstore/ +├── kvstore-node/ # the application binary under test +├── testing/ +│ ├── integration/ # crate kvstore-runtime-ext: Application impl, +│ │ # local/compose/k8s env impls, observation +│ └── workloads/ # crate kvstore-runtime-workloads: Workloads + Expectations +└── examples/ # crate kvstore-examples: runnable bins +``` + +The naming is uniform: `-runtime-ext`, `-runtime-workloads`, `-examples`. `nats` and `redis_streams` have no node crate because they run upstream binaries or images. `multi_app` uses an acceptance-suite layout instead: a `job-worker/` binary crate, a `fixture/` crate (`multi-app-fixture`: the stack deployment, handles, workload, and expectation), and an `e2e/` crate (`multi-app-e2e`) whose integration tests drive the fixture. It demonstrates application composition. + +Run any example bin with: + +```bash +cargo run -p kvstore-examples --bin kvstore_basic_convergence +``` + +**Note:** the dependency arrows only ever point from examples toward the framework and from backends toward core. If you find yourself wanting an arrow in the other direction, read [Framework vs Application Boundaries](tf-boundaries.md). The trait-level view of the same surface is in [Public Extension Points](extension-points.md). diff --git a/book/src/custom-workload-example.md b/book/src/custom-workload-example.md deleted file mode 100644 index c6d9439..0000000 --- a/book/src/custom-workload-example.md +++ /dev/null @@ -1,134 +0,0 @@ -# Example: New Workload & Expectation (Rust) - -A minimal, end-to-end illustration of adding a custom workload and matching -expectation. This shows the shape of the traits and where to plug into the -framework; expand the logic to fit your real test. - -## Workload: simple reachability probe - -Key ideas: -- **name**: identifies the workload in logs. -- **expectations**: workloads can bundle defaults so callers don’t forget checks. -- **init**: derive inputs from the generated topology (e.g., pick a target node). -- **start**: drive async activity using the shared `RunContext`. - -```rust,ignore -use async_trait::async_trait; -use testing_framework_core::{ - scenario::{DynError, Expectation, RunContext, RunMetrics, Workload}, - topology::generation::GeneratedTopology, -}; - -pub struct ReachabilityWorkload { - target_idx: usize, -} - -impl ReachabilityWorkload { - pub fn new(target_idx: usize) -> Self { - Self { target_idx } - } -} - -#[async_trait] -impl Workload for ReachabilityWorkload { - fn name(&self) -> &str { - "reachability_workload" - } - - fn expectations(&self) -> Vec> { - vec![Box::new( - crate::custom_workload_example_expectation::ReachabilityExpectation::new( - self.target_idx, - ), - )] - } - - fn init( - &mut self, - topology: &GeneratedTopology, - _run_metrics: &RunMetrics, - ) -> Result<(), DynError> { - if topology.nodes().get(self.target_idx).is_none() { - return Err(Box::new(std::io::Error::new( - std::io::ErrorKind::Other, - "no node at requested index", - ))); - } - Ok(()) - } - - async fn start(&self, ctx: &RunContext) -> Result<(), DynError> { - let client = ctx - .node_clients() - .node_clients() - .get(self.target_idx) - .ok_or_else(|| { - Box::new(std::io::Error::new( - std::io::ErrorKind::Other, - "missing target client", - )) as DynError - })?; - - // Lightweight API call to prove reachability. - client - .consensus_info() - .await - .map(|_| ()) - .map_err(|e| e.into()) - } -} -``` - -## Expectation: confirm the target stayed reachable - -Key ideas: -- **start_capture**: snapshot baseline if needed (not used here). -- **evaluate**: assert the condition after workloads finish. - -```rust,ignore -use async_trait::async_trait; -use testing_framework_core::scenario::{DynError, Expectation, RunContext}; - -pub struct ReachabilityExpectation { - target_idx: usize, -} - -impl ReachabilityExpectation { - pub fn new(target_idx: usize) -> Self { - Self { target_idx } - } -} - -#[async_trait] -impl Expectation for ReachabilityExpectation { - fn name(&self) -> &str { - "target_reachable" - } - - async fn evaluate(&mut self, ctx: &RunContext) -> Result<(), DynError> { - let client = ctx - .node_clients() - .node_clients() - .get(self.target_idx) - .ok_or_else(|| { - Box::new(std::io::Error::new( - std::io::ErrorKind::Other, - "missing target client", - )) as DynError - })?; - - client - .consensus_info() - .await - .map(|_| ()) - .map_err(|e| e.into()) - } -} -``` - -## How to wire it -- Build your scenario as usual and call `.with_workload(ReachabilityWorkload::new(0))`. -- The bundled expectation is attached automatically; you can add more with - `.with_expectation(...)` if needed. -- Keep the logic minimal and fast for smoke tests; grow it into richer probes - for deeper scenarios. diff --git a/book/src/deployer-compose.md b/book/src/deployer-compose.md new file mode 100644 index 0000000..16ea134 --- /dev/null +++ b/book/src/deployer-compose.md @@ -0,0 +1,79 @@ +# Compose Deployer + +`ComposeDeployer` runs each node as a Docker Compose service generated from your deployment descriptor. + +The compose deployer lives in the `testing-framework-runner-compose` crate. It generates a compose file per run, brings the stack up, discovers the host ports Docker assigned, probes readiness, and hands control to the scenario runner. It requires a running Docker daemon; otherwise deployment returns `ComposeRunnerError::DockerUnavailable`. + +```rust,ignore +use kvstore_runtime_ext::KvComposeDeployer; // = ComposeDeployer +use testing_framework_core::scenario::Deployer; +use testing_framework_runner_compose::ComposeRunnerError; + +let deployer = KvComposeDeployer::new(); +let runner = match deployer.deploy(&scenario).await { + Ok(runner) => runner, + Err(ComposeRunnerError::DockerUnavailable) => return Ok(()), // skip without Docker + Err(error) => return Err(error.into()), +}; +runner.run(&mut scenario).await?; +``` + +Run the demonstration binary with `cargo run -p kvstore-examples --bin kvstore_compose_convergence`. + +--- + +## Deployment Pipeline + +```mermaid +flowchart LR + A[Workspacetempdir] --> B[Write configs+ cfgsync.yaml] + B --> C[Rendercompose.generated.yml] + C --> D[docker composecreate + up] + D --> E[Port discoverydocker compose port] + E --> F[Readinessprobes] + F --> G[Node clients+ Runner] +``` + +1. **Workspace.** A temporary `ComposeWorkspace` is created; the app's `ComposeDeployEnv::prepare_compose_configs` writes per-node config files (for `ComposeBinaryApp` environments, one static config per node under `stack/configs/`, rewritten for service hostnames `node-0`, `node-1`, ...). +2. **cfgsync.** If the environment enables `ComposeConfigServerMode::Docker`, a cfgsync config server container is started on an ephemeral port and the deployer waits for it to accept TCP connections before proceeding. The default mode is `Disabled`. See [Static Artifacts and cfgsync](cfgsync.md). +3. **Compose file.** The env's `compose_descriptor` (image, entrypoint, volumes, ports, environment, optional platform per service) is rendered through the Tera template at `testing-framework/deployers/compose/assets/docker-compose.yml.tera` into `compose.generated.yml`. The template is resolved relative to the repository root (`CARGO_WORKSPACE_DIR` override respected). Required images are checked with `docker image inspect` up front. The deployer never builds or pulls them; a missing image fails the deploy with `MissingImage`. +4. **Bring-up.** `docker compose create` and `docker compose up` run under a unique project name (`compose-stack-`). On failure, container logs are dumped before cleanup. +5. **Ports.** Container ports map to ephemeral host ports; the deployer resolves each with `docker compose port` and records them as `NodeHostPorts { api, testing }`. The host defaults to `127.0.0.1` and can be overridden with `COMPOSE_RUNNER_HOST`. +6. **Readiness.** Per the env's `ComposeReadinessProbe`: HTTP GET against `Application::node_readiness_path()` on each mapped API port, or raw TCP reachability. Gated by `DeploymentPolicy.readiness_enabled` and the deployer's own `with_readiness(bool)` switch; when disabled, the stack gets a short fixed grace period instead. See [Readiness, Retry, and Artifact Preservation](deployment-policies.md). +7. **Clients.** `build_node_client` runs against the discovered host/port pairs, producing the scenario's typed node clients. + +--- + +## Node Control + +With `with_node_control()` on the builder, the deployer installs a `ComposeNodeControl` handle bound to the generated compose file and project. It supports **restart only**: `restart_node(name)` shells out to `docker compose restart `. Start and stop of individual services are not wired for managed compose scenarios. The openraft_kv failover scenario runs on this backend: `cargo run -p openraft-kv-examples --bin openraft_kv_compose_failover`. + +--- + +## Attaching to an Existing Stack + +The compose deployer fully supports existing-cluster mode. A scenario built with `with_existing_cluster(ExistingCluster::for_compose_project("my-project"))` skips workspace generation entirely: services are discovered from the running project (or taken from `for_compose_services`), each container's labeled API port is inspected, and clients are built through `Application::external_node_client`. In this mode node control gains `stop_node` in addition to `restart_node`, implemented with `docker container stop` / `docker container restart` against discovered container IDs. + +`deploy_with_metadata` returns `ComposeDeploymentMetadata` alongside the runner; its `existing_cluster()` / `IntoExistingCluster` impl lets a later scenario attach to the stack this one deployed. See [Existing and External Clusters](external-clusters.md). + +--- + +## Observability + +Compose resolves `ObservabilityInputs` by merging `LOGOS_BLOCKCHAIN_METRICS_QUERY_URL`, `LOGOS_BLOCKCHAIN_METRICS_OTLP_INGEST_URL`, and `LOGOS_BLOCKCHAIN_GRAFANA_URL` env vars with the scenario's observability capability (capability values win). The OTLP ingest URL is passed into config preparation so node configs can point at your collector; the metrics query URL becomes the run's Prometheus-backed `Metrics` handle. Setting `TESTNET_PRINT_ENDPOINTS` prints Prometheus/Grafana endpoints and per-node pprof profile URLs to stdout. See [Telemetry and External Observability](telemetry.md). + +--- + +## Cleanup + +The runner's cleanup guard runs `docker compose down`, shuts down the cfgsync container if one was started, and removes the workspace. Setting `COMPOSE_RUNNER_PRESERVE` (or `TESTNET_RUNNER_PRESERVE`) keeps the stack running and persists the workspace directory for post-mortem inspection; the preserved path is logged. + +--- + +**Requirements recap:** + +| Requirement | Why | +|---|---| +| Docker daemon running | `ensure_docker_available` gates every deploy | +| Node container images | Must exist locally before deploy; missing images fail with `MissingImage` | +| Repository checkout | The compose Tera template is read from the repo tree | diff --git a/book/src/deployer-k8s.md b/book/src/deployer-k8s.md new file mode 100644 index 0000000..2255377 --- /dev/null +++ b/book/src/deployer-k8s.md @@ -0,0 +1,97 @@ +# Kubernetes Deployer + +`K8sDeployer` installs each scenario as a Helm release in a throwaway namespace and reaches nodes through NodePorts or port-forwards. + +The k8s deployer lives in the `testing-framework-runner-k8s` crate. It talks to whatever cluster your current kubeconfig context points at (`kube::Client::try_default()`), installs a Helm release, waits for the workloads, and builds node clients against externally reachable ports. + +```rust,ignore +use kvstore_runtime_ext::KvK8sDeployer; // = K8sDeployer +use testing_framework_core::scenario::Deployer; +use testing_framework_runner_k8s::K8sRunnerError; + +let deployer = KvK8sDeployer::new(); +let runner = match deployer.deploy(&scenario).await { + Ok(runner) => runner, + Err(K8sRunnerError::ClientInit { .. }) => return Ok(()), // no cluster available + Err(error) => return Err(error.into()), +}; +runner.run(&mut scenario).await?; +``` + +Run the demonstration binary with `cargo run -p kvstore-examples --bin kvstore_k8s_convergence`. + +--- + +## Charts and Values + +The environment trait `K8sDeployEnv` produces installable assets via `prepare_assets`, returning a `PreparedK8sStack`. Two asset shapes exist: + +- **Generated single-template charts.** Apps implementing `K8sBinaryApp` get the standard shape for free: `render_binary_config_node_manifest` renders one ConfigMap (the serialized node config), one Deployment (single replica, `--config` arg, config mounted from the ConfigMap), and one NodePort Service per node, then `render_manifest_chart_assets` wraps them in a minimal chart (`RenderedHelmChartAssets`). +- **Real chart directories.** `NodeRuntimeSpec` builds `RunnerChartValues` (node image, pull policy, fullname override, asset mount layout, node group, optional shared bootstrap service with cfgsync configs and start scripts) and a `HelmReleaseBundle` with `--set` values and `--set-file` entries for start scripts and bootstrap configs. `RunnerAssetLayout` fixes where bootstrap configs and runner scripts land inside the chart's mount path. + +Node images are resolved from env vars: for a conventional `BinaryConfigK8sSpec` the primary override is `_K8S_IMAGE`, the fallback `_IMAGE`, and the default `:local` with `imagePullPolicy: IfNotPresent`. + +Each run installs into fresh identifiers: namespace `tf-testnet--`, release `tf-runner` (override via `K8sDeployEnv::cluster_identifiers`). + +--- + +## Lifecycle Waits + +After `helm install`, the deployer waits in stages: + +1. **Deployment readiness**: each node Deployment must report ready replicas (timeout `K8S_RUNNER_DEPLOYMENT_TIMEOUT_SECS`, default 180 s). +2. **Port discovery**: each node Service must have allocated NodePorts for the API and auxiliary ports declared by `collect_port_specs`. +3. **HTTP readiness**: nodes are probed over their NodePorts at `node_readiness_path()`. The probe host is `K8S_RUNNER_NODE_HOST` if set, else `KUBERNETES_SERVICE_HOST`, else `127.0.0.1`. If NodePort probing fails (common when the cluster's node IPs are not routable from the runner), the deployer transparently falls back to `kubectl port-forward` per service and probes over `127.0.0.1`. +4. **Policy-gated cluster readiness**: a final probe pass controlled by `DeploymentPolicy.readiness_enabled` / `readiness_requirement` and the deployer's `with_readiness(bool)` switch. See [Readiness, Retry, and Artifact Preservation](deployment-policies.md). + +HTTP wait tuning: `K8S_RUNNER_HTTP_TIMEOUT_SECS` (default 240), `K8S_RUNNER_HTTP_PROBE_TIMEOUT_SECS` (default 30), `K8S_RUNNER_HTTP_POLL_INTERVAL_SECS` (default 1). + +--- + +## Node Control + +The Kubernetes deployer does not wire a node-control handle into managed scenario deployments. A scenario built with `with_node_control()` compiles against this backend, but runtime restart calls fail. For node lifecycle control on Kubernetes, use the Kubernetes `ManualCluster` below. + +--- + +## Manual Mode + +`K8sDeployer::manual_cluster_from_descriptors(descriptors)` (or `ManualCluster::from_topology`) installs the same Helm release, discovers every node's ports, then **scales all node Deployments to zero** so your code decides when each node starts: + +```rust,ignore +let deployer = OpenRaftKvK8sDeployer::new(); +let cluster = deployer + .manual_cluster_from_descriptors(OpenRaftKvTopology::new(3)) + .await?; + +cluster.start_node("node-0").await?; +cluster.start_node("node-1").await?; +cluster.wait_network_ready().await?; +cluster.restart_node("node-0").await?; +cluster.stop_all(); +``` + +Start, stop, and restart are implemented by patching Deployment replicas between 0 and 1 and waiting for the rollout. `start_node_with` accepts `StartNodeOptions`, with two k8s-specific limits: `persist_dir` / `snapshot_dir` are rejected, and peer selection or config overrides require the environment to implement cfgsync override artifacts (`cfgsync_service` + `build_cfgsync_override_artifacts`); the override is pushed to the in-cluster cfgsync service through a temporary port-forward before the node starts. The failover demonstration uses this path end to end: `cargo run -p openraft-kv-examples --bin openraft_kv_k8s_failover`. Contrast with the declarative local variant in [ManualCluster: Imperative Node Control](manual-cluster.md). + +--- + +## Attaching to an Existing Cluster + +Existing-cluster mode is supported with a k8s descriptor: `with_existing_cluster(ExistingCluster::for_k8s_selector("app.kubernetes.io/instance=tf-runner"))` (optionally namespaced with `for_k8s_selector_in_namespace`). Services matching the selector are listed, each service's single TCP NodePort (or the port named `http`/`api`) becomes the node endpoint, and clients are built via `Application::external_node_client`. `deploy_with_metadata` returns `K8sDeploymentMetadata` (namespace + label selector) so a later scenario can attach to the stack this one installed. See [Existing and External Clusters](external-clusters.md). + +--- + +## Observability and Cleanup + +Observability inputs resolve exactly as in compose (`LOGOS_BLOCKCHAIN_*` env vars merged with the scenario capability), and `TESTNET_PRINT_ENDPOINTS` prints Prometheus/Grafana and per-node pprof endpoints. Cleanup uninstalls the Helm release and deletes the namespace (Kubernetes API first, `kubectl delete namespace` fallback), after killing any port-forward processes. Set `K8S_RUNNER_PRESERVE` to keep the release and namespace for inspection. + +--- + +**Requirements recap:** + +| Requirement | Why | +|---|---| +| Reachable cluster in current kubeconfig context | `Client::try_default()` at deploy time | +| `helm` on PATH | Release install/uninstall | +| `kubectl` on PATH | Port-forward fallback, namespace-delete fallback | +| Node images loadable by the cluster | `_K8S_IMAGE` / `_IMAGE` / `:local` | diff --git a/book/src/deployer-local.md b/book/src/deployer-local.md new file mode 100644 index 0000000..7d84241 --- /dev/null +++ b/book/src/deployer-local.md @@ -0,0 +1,109 @@ +# Local Deployer + +`ProcessDeployer` runs every node as a local OS process. It is the default backend. + +The local deployer lives in the `testing-framework-runner-local` crate. It requires no Docker daemon and no cluster: it resolves a node binary, writes each node's config into a private working directory, spawns the processes, probes readiness, and hands the running cluster to the scenario runner. + +```rust,ignore +use kvstore_runtime_ext::KvLocalDeployer; // = ProcessDeployer +use testing_framework_core::scenario::Deployer; + +let deployer = KvLocalDeployer::default(); +let runner = deployer.deploy(&scenario).await?; +runner.run(&mut scenario).await?; +``` + +Run the demonstration binary with `cargo run -p kvstore-examples --bin kvstore_basic_convergence`. No manual binary setup is needed, because kvstore's fallback provider chain builds the node binary on first use (see [Binary Providers](binary-providers.md)). + +--- + +## What deploy Does + +For a managed scenario, `ProcessDeployer::deploy`: + +1. Validates the cluster mode: the local deployer rejects `ClusterMode::ExistingCluster` (attach is a compose/k8s feature, see [Existing and External Clusters](external-clusters.md)). +2. Builds the source orchestration plan and spawns one `ProcessNode` per topology entry. +3. Probes readiness and retries the whole spawn on failure (see below). +4. Merges external node clients into the managed set. +5. Assembles the runtime and returns a `Runner` whose cleanup guard owns the node processes. + +The main runtime types are: + +- **`ProcessDeployer`**: the deployer. `E` implements `LocalDeployerEnv` (full-control hooks) or the compact `LocalBinaryApp` trait (one binary + one config file + one HTTP port per node). +- **`LaunchSpec`**: the launch plan for one process: binary path, files to materialize, CLI args, env vars. +- **`ProcessNode`**: a spawned child process plus its tempdir, endpoints, and typed client. + +--- + +## Working Directories and Logs + +Each node gets its own temporary working directory, created under the current directory (or under a caller-supplied persist path). Config files and any other `LaunchFile` entries are written there before spawn, and the process starts with that directory as its cwd. + +Node stdout and stderr are inherited from the test process, so node logs interleave with your test output; control verbosity with the `RUST_LOG` value configured on the app's `LocalProcessSpec` (for example `.with_rust_log("kvstore_node=info")`). + +On drop, each `ProcessNode` kills its child and removes the tempdir. Two things preserve working directories instead of deleting them: + +- `DeploymentPolicy` with `cleanup_policy.preserve_artifacts = true` (see [Readiness, Retry, and Artifact Preservation](deployment-policies.md)), or the `TF_KEEP_LOGS=1` env var. +- A panicking test thread, in which case directories are kept automatically for inspection. + +Nodes started with a `persist_dir` or seeded from a `snapshot_dir` (via `StartNodeOptions`) copy or place state accordingly before spawn (see [Persistence, Snapshots, and Recovery Testing](persistence.md)). + +--- + +## Ports + +The deployer reserves real OS ports up front: `allocate_available_port()` binds an ephemeral listener and releases it, and `reserve_local_node_ports` reserves the network port plus any app-named extra ports for each node. Endpoints are surfaced as `NodeEndpoints` (an API socket address plus named extra ports), from which the app builds its typed `NodeClient`. + +--- + +## Readiness and Retry + +Readiness is governed by the scenario's `DeploymentPolicy` combined with the deployer's own switch: + +| Control | Effect | +|---|---| +| `ProcessDeployer::with_membership_check(false)` | Disables local readiness probing entirely | +| `DeploymentPolicy.readiness_enabled` | Must also be true for probes to run | +| `DeploymentPolicy.readiness_requirement` | `AllNodesReady`, `AnyNodeReady`, or `AtLeast(n)` | +| `DeploymentPolicy.retry_policy` | Attempts and backoff; defaults to 3 attempts, 250 ms base, 2 s max | + +The probe shape comes from the environment: `LocalReadinessProbe::HttpGet { path }` (default, using `Application::node_readiness_path()`) or `LocalReadinessProbe::Tcp`. If spawn or readiness fails, all nodes from that attempt are dropped and the entire cluster is respawned with exponential backoff and jitter, up to the retry budget. + +--- + +## Node Control + +The local deployer supports the complete node-control surface. Building the scenario with `with_node_control()` deploys through `Deployer`, which wraps the spawned nodes in a `NodeManager`. Workloads can then start, stop, and restart nodes by name, with full `StartNodeOptions` support (peer selection, config overrides and patches, persist and snapshot directories, extra args, start timeouts). The openraft_kv failover scenario uses this path: + +```bash +cargo run -p openraft-kv-examples --bin openraft_kv_basic_failover +``` + +See [Scenario Capabilities](capabilities.md) for the capability-gated builder. + +--- + +## Manual Clusters + +For orchestration outside the scenario runner, such as Cucumber steps or another test harness, the deployer provides an imperative cluster: + +```rust,ignore +let deployer = ProcessDeployer::::new(); +let cluster = deployer.manual_cluster_from_descriptors(descriptors); + +cluster.start_node("node-0").await?; +cluster.wait_network_ready().await?; +cluster.stop_all(); +``` + +`ManualCluster` exposes `start_node(_with)`, `stop_node`, `restart_node(_with)`, `wait_node_ready`, `wait_network_ready`, `node_client`, `node_pid`, `node_clients`, and `add_external_sources` / `add_external_clients`. It is covered in depth in [ManualCluster: Imperative Node Control](manual-cluster.md). + +--- + +## Binary Resolution + +Every local node needs an executable. `LocalProcessSpec::new("MY_NODE_BIN")` defaults to an env-var provider; `with_binary_provider` swaps in any `BinaryProvider`, including fallback chains that try an env override first and build with Cargo otherwise. Resolution is cached per process and locked across processes. Full detail in [Binary Providers](binary-providers.md). + +--- + +The local deployer supports external node sources (`with_external_node`) but not attached existing clusters. If `Application::external_node_client` is not implemented, it falls back to parsing the endpoint (`http://host:port`) and building a client from the resolved socket address. diff --git a/book/src/deployment-policies.md b/book/src/deployment-policies.md new file mode 100644 index 0000000..6cf66d8 --- /dev/null +++ b/book/src/deployment-policies.md @@ -0,0 +1,90 @@ +# Readiness, Retry, and Artifact Preservation + +`DeploymentPolicy` is the single policy struct that controls readiness gating, deploy retries, and artifact retention across all deployers. + +--- + +## The Policy + +From `testing-framework-core` (`core/src/scenario/deployment_policy.rs`): + +```rust,ignore +pub struct DeploymentPolicy { + pub readiness_enabled: bool, + pub readiness_requirement: HttpReadinessRequirement, + pub retry_policy: Option, + pub cleanup_policy: CleanupPolicy, +} + +pub struct RetryPolicy { + pub max_attempts: usize, + pub base_delay: Duration, + pub max_delay: Duration, +} + +pub struct CleanupPolicy { + pub preserve_artifacts: bool, +} +``` + +Defaults: `readiness_enabled: true`, `readiness_requirement: HttpReadinessRequirement::AllNodesReady`, `retry_policy: None`, `preserve_artifacts: false`. `HttpReadinessRequirement` is `AllNodesReady`, `AnyNodeReady`, or `AtLeast(usize)`. + +Set it on the builder: + +```rust,ignore +use std::time::Duration; +use testing_framework_core::scenario::{ + CleanupPolicy, DeploymentPolicy, HttpReadinessRequirement, RetryPolicy, +}; + +let scenario = KvScenarioBuilder::deployment_with(|_| KvTopology::new(3)) + .with_deployment_policy(DeploymentPolicy { + readiness_enabled: true, + readiness_requirement: HttpReadinessRequirement::AtLeast(2), + retry_policy: Some(RetryPolicy::new( + 5, + Duration::from_millis(500), + Duration::from_secs(5), + )), + cleanup_policy: CleanupPolicy::new(true), + }) + .build()?; +``` + +To adjust only the requirement, `with_http_readiness_requirement(...)` is the shortcut. + +--- + +## Readiness + +`readiness_enabled` and `readiness_requirement` gate the post-spawn probe pass in every deployer. Each backend also has its own deployer-level switch that must agree (`ProcessDeployer::with_membership_check(bool)`, `ComposeDeployer::with_readiness(bool)`, `K8sDeployer::with_readiness(bool)`), so effective readiness is `deployer switch && policy.readiness_enabled`. The probe shape (HTTP path vs TCP) comes from the application environment; see the per-deployer chapters ([Local](deployer-local.md), [Compose](deployer-compose.md), [K8s](deployer-k8s.md)). + +--- + +## Retry + +`retry_policy` drives the local deployer's spawn-and-readiness loop: on failure, all nodes from the attempt are dropped and the cluster is respawned with exponential backoff (from `base_delay`, capped at `max_delay`, with jitter) up to `max_attempts`. When `retry_policy` is `None`, the local deployer falls back to its built-in default of 3 attempts, 250 ms base delay, 2 s max delay. + +The Compose and Kubernetes deployers currently honor the readiness fields of the policy but do not repeat deployment on failure; `retry_policy` has no effect on those backends today. + +--- + +## Artifact Preservation + +`cleanup_policy.preserve_artifacts` controls **artifact and tempdir retention, not teardown ordering**. Teardown itself always follows the runner's cleanup-guard chain (see [Handle Ownership and Teardown](handles-teardown.md)); this flag only decides whether per-node working directories survive it. + +The local orchestrator computes retention as: + +```rust,ignore +policy.cleanup_policy.preserve_artifacts || keep_tempdir_from_env() // TF_KEEP_LOGS +``` + +so either the policy flag or `TF_KEEP_LOGS=1` (also `true`/`yes`) keeps every node's working directory (configs, on-disk state, anything the process wrote) after the run. Panicking tests preserve working directories regardless. + +The container deployers preserve through env vars rather than the policy: `COMPOSE_RUNNER_PRESERVE` / `TESTNET_RUNNER_PRESERVE` keep the compose stack and workspace, `K8S_RUNNER_PRESERVE` keeps the Helm release and namespace. See [Diagnostics and Retained Artifacts](diagnostics.md). + +| Backend | Policy `preserve_artifacts` | Env var | +|---|---|---| +| Local | Yes — keeps node tempdirs | `TF_KEEP_LOGS` | +| Compose | No effect | `COMPOSE_RUNNER_PRESERVE` / `TESTNET_RUNNER_PRESERVE` | +| K8s | No effect | `K8S_RUNNER_PRESERVE` | diff --git a/book/src/design-rationale.md b/book/src/design-rationale.md deleted file mode 100644 index 94961b6..0000000 --- a/book/src/design-rationale.md +++ /dev/null @@ -1,7 +0,0 @@ -# Design Rationale - -- **Modular crates** keep configuration, orchestration, workloads, and runners decoupled so each can evolve without breaking the others. -- **Pluggable runners** let the same scenario run on a laptop, a Docker host, or a Kubernetes cluster, making validation portable across environments. -- **Separated workloads and expectations** clarify intent: what traffic to generate versus how to judge success. This simplifies review and reuse. -- **Declarative topology** makes cluster shape explicit and repeatable, reducing surprise when moving between CI and developer machines. -- **Maintainability through predictability**: a clear flow from plan to deployment to verification lowers the cost of extending the framework and interpreting failures. diff --git a/book/src/diagnostics.md b/book/src/diagnostics.md new file mode 100644 index 0000000..a429f46 --- /dev/null +++ b/book/src/diagnostics.md @@ -0,0 +1,76 @@ +# Diagnostics and Retained Artifacts + +This chapter explains where node output and generated files go, how to keep them after a run, and how to turn a failed run into a diagnosis. + +--- + +## Where Output Goes + +**Process output.** Local node processes are spawned with inherited stdout/stderr (`testing-framework/deployers/local/src/process.rs`). Node logs interleave with your test's own output on the terminal; they are not redirected to files by the framework. Control node verbosity with the process env, e.g. `LocalProcessSpec::with_rust_log("my_node=debug")` or `with_env("RUST_LOG", ...)`. + +**Files.** Every local node runs inside its own temporary working directory, created per node per run **under the current working directory of the test process** (a `TempDir` with a random `.tmp*` name). The directory contains: + +- the materialized launch files: for the standard spec, the rendered config (`config.yaml` by default, `LocalProcessSpec::with_config_file` to change it) plus any extra `LaunchFile` entries; +- anything the node itself writes, since the process is spawned with the directory as its `current_dir` (databases, application logs, snapshots); +- state seeded before start: `with_snapshot_dir(path)` copies a snapshot into the directory before spawn. + +A typical kvstore node directory looks like: + +```text +.tmpAbC123/ +├── config.yaml # rendered by the framework before spawn +└── data/ # whatever the node itself created +``` + +If a persistent location was requested (`with_persist_dir(path)` on `LocalProcessApp`, or `persist_dir` in `StartNodeOptions`), the directory is instead created next to `path` with a `_` prefix, so restarts and recovery tests can find it; see [Persistence, Snapshots, and Recovery Testing](persistence.md). + +Compose runs render their whole stack (compose file, per-node configs under `stack/configs/`, cfgsync artifacts) into a `compose-stack-*` workspace in the system temp directory. Kubernetes runs render Helm charts into temporary chart directories and install them into a per-run namespace. + +--- + +## Keeping Artifacts + +By default all of the above is deleted at teardown. Three mechanisms retain it: + +| Mechanism | Scope | How | +|---|---|---| +| `CleanupPolicy` | one scenario | `with_deployment_policy(DeploymentPolicy { cleanup_policy: CleanupPolicy::new(true), .. })` | +| `keep_tempdir` | one process | `LocalProcessApp::keep_tempdir(true)` at build time, or `handle.keep_tempdir().await` at run time | +| `TF_KEEP_LOGS` | whole process | env var, no code change | + +The local orchestrator preserves node directories when **either** the policy or the env var asks for it: `policy.cleanup_policy.preserve_artifacts || keep_tempdir_from_env()`. `TF_KEEP_LOGS` accepts `1`, `true`, or `yes` (case-insensitive) and is also honored by `ManualCluster` node starts. See [Readiness, Retry, and Artifact Preservation](deployment-policies.md) for the full policy type. + +A panicking test thread preserves its node working directories automatically (`thread::panicking()` is checked in the process drop path), so a failed assertion usually leaves the directory behind without an additional flag. + +The container backends have their own preserve switches: `COMPOSE_RUNNER_PRESERVE` (or `TESTNET_RUNNER_PRESERVE`) keeps the compose workspace and skips `docker compose down`; `K8S_RUNNER_PRESERVE` skips Helm uninstall and namespace deletion; `K8S_RUNNER_DEBUG` additionally logs Helm install output. All are listed in [Environment Variables](environment-variables.md). + +--- + +## Teardown Ordering (What Preservation Does Not Change) + +Preservation only controls file deletion; it does not change stop order. At the end of a run the runner executes its cleanup guards. App-layer managed resources form one LIFO guard stack within that chain, so dependants acquired later stop before their dependencies. Handle-registry release is separate and does not own process or cluster lifetime. Details in [Handle Ownership and Teardown](handles-teardown.md). + +```mermaid +flowchart LR + A[run ends] --> B[cleanup guard chain] + B --> C[app cleanup stackreverse acquisition order] + C --> D{preserve?} + D -- no --> E[tempdirs deleted] + D -- yes --> F[tempdirs kept on disk] +``` + +--- + +## Post-Mortem Workflow + +1. **Reproduce with preservation.** Re-run the failing binary or test with `TF_KEEP_LOGS=1` (plus `COMPOSE_RUNNER_PRESERVE=1` for compose). On panic, artifacts are often already there from the first failure. + +2. **Locate the directories.** Local node dirs are the `.tmp*` entries under the directory you launched from (named `_*` when a persist dir was set). The `working_dir()` accessor on `LocalProcessHandle` and the spawn-time log lines give exact paths. + +3. **Inspect configs first.** Many deploy-time failures come from configuration. Check the rendered `config.yaml` for the ports, peer lists, and paths the framework generated. For Compose, diff the rendered files under the preserved workspace's `stack/` directory; for Kubernetes, re-run with `K8S_RUNNER_DEBUG=1` to see Helm output. + +4. **Read the node's own output.** Scroll the interleaved terminal output for the failing node's log lines, or raise its `RUST_LOG` and re-run. Anything the node writes to files is in its working directory. + +5. **Re-run deterministically.** If the deployment was generated from a seed, replay it with the same one via `with_deployment_seed` so the topology and generated identities match the failing run exactly; see [Seeds and Reproducibility](seeds.md). Combined with preserved state and `with_snapshot_dir`, you can restart a node from the exact bytes it crashed with. + +6. **Use imperative control when needed.** To inspect the cluster interactively, start one node at a time, or restart with modified options, rebuild the situation with [ManualCluster](manual-cluster.md). It uses the same working-directory and `TF_KEEP_LOGS` behavior. diff --git a/book/src/dsl-cheat-sheet.md b/book/src/dsl-cheat-sheet.md deleted file mode 100644 index 85262b3..0000000 --- a/book/src/dsl-cheat-sheet.md +++ /dev/null @@ -1,188 +0,0 @@ -# Builder API Quick Reference - -Quick reference for the scenario builder DSL. All methods are chainable. - -## Imports - -```rust,ignore -use std::time::Duration; - -use testing_framework_core::scenario::{Deployer, ScenarioBuilder}; -use testing_framework_runner_compose::ComposeDeployer; -use testing_framework_runner_k8s::K8sDeployer; -use testing_framework_runner_local::LocalDeployer; -use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt}; -``` - -## Topology - -```rust,ignore -use testing_framework_core::scenario::{Builder, ScenarioBuilder}; - -pub fn topology() -> Builder<()> { - ScenarioBuilder::topology_with(|t| { - t.network_star() // Star topology (all connect to seed node) - .nodes(3) // Number of nodes - }) -} -``` - -## Wallets - -```rust,ignore -use testing_framework_core::scenario::ScenarioBuilder; -use testing_framework_workflows::ScenarioBuilderExt; - -pub fn wallets_plan() -> testing_framework_core::scenario::Scenario<()> { - ScenarioBuilder::topology_with(|t| t.network_star().nodes(1)) - .wallets(50) // Seed 50 funded wallet accounts - .build() -} -``` - -## Transaction Workload - -```rust,ignore -use testing_framework_core::scenario::ScenarioBuilder; -use testing_framework_workflows::ScenarioBuilderExt; - -pub fn transactions_plan() -> testing_framework_core::scenario::Scenario<()> { - ScenarioBuilder::topology_with(|t| t.network_star().nodes(1)) - .wallets(50) - .transactions_with(|txs| { - txs.rate(5) // 5 transactions per block - .users(20) // Use 20 of the seeded wallets - }) // Finish transaction workload config - .build() -} -``` - -## Chaos Workload (Requires `enable_node_control()`) - -```rust,ignore -use std::time::Duration; - -use testing_framework_core::scenario::{NodeControlCapability, ScenarioBuilder}; -use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt}; - -pub fn chaos_plan() -> testing_framework_core::scenario::Scenario { - ScenarioBuilder::topology_with(|t| t.network_star().nodes(3)) - .enable_node_control() // Enable node control capability - .chaos_with(|c| { - c.restart() // Random restart chaos - .min_delay(Duration::from_secs(30)) // Min time between restarts - .max_delay(Duration::from_secs(60)) // Max time between restarts - .target_cooldown(Duration::from_secs(45)) // Cooldown after restart - .apply() // Required for chaos configuration - }) - .build() -} -``` - -## Expectations - -```rust,ignore -use testing_framework_core::scenario::ScenarioBuilder; -use testing_framework_workflows::ScenarioBuilderExt; - -pub fn expectations_plan() -> testing_framework_core::scenario::Scenario<()> { - ScenarioBuilder::topology_with(|t| t.network_star().nodes(1)) - .expect_consensus_liveness() // Assert blocks are produced continuously - .build() -} -``` - -## Run Duration - -```rust,ignore -use std::time::Duration; - -use testing_framework_core::scenario::ScenarioBuilder; -use testing_framework_workflows::ScenarioBuilderExt; - -pub fn run_duration_plan() -> testing_framework_core::scenario::Scenario<()> { - ScenarioBuilder::topology_with(|t| t.network_star().nodes(1)) - .with_run_duration(Duration::from_secs(120)) // Run for 120 seconds - .build() -} -``` - -## Build - -```rust,ignore -use testing_framework_core::scenario::ScenarioBuilder; -use testing_framework_workflows::ScenarioBuilderExt; - -pub fn build_plan() -> testing_framework_core::scenario::Scenario<()> { - ScenarioBuilder::topology_with(|t| t.network_star().nodes(1)).build() // Construct the final Scenario -} -``` - -## Deployers - -```rust,ignore -use testing_framework_runner_compose::ComposeDeployer; -use testing_framework_runner_k8s::K8sDeployer; -use testing_framework_runner_local::LocalDeployer; - -pub fn deployers() { - // Local processes - let _deployer = LocalDeployer::default(); - - // Docker Compose - let _deployer = ComposeDeployer::default(); - - // Kubernetes - let _deployer = K8sDeployer::default(); -} -``` - -## Execution - -```rust,ignore -use anyhow::Result; -use testing_framework_core::scenario::{Deployer, ScenarioBuilder}; -use testing_framework_runner_local::LocalDeployer; -use testing_framework_workflows::ScenarioBuilderExt; - -pub async fn execution() -> Result<()> { - let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().nodes(1)) - .expect_consensus_liveness() - .build(); - - let deployer = LocalDeployer::default(); - let runner = deployer.deploy(&plan).await?; - let _handle = runner.run(&mut plan).await?; - - Ok(()) -} -``` - -## Complete Example - -```rust,ignore -use std::time::Duration; - -use anyhow::Result; -use testing_framework_core::scenario::{Deployer, ScenarioBuilder}; -use testing_framework_runner_local::LocalDeployer; -use testing_framework_workflows::ScenarioBuilderExt; - -pub async fn run_test() -> Result<()> { - let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().nodes(3)) - .wallets(50) - .transactions_with(|txs| { - txs.rate(5) // 5 transactions per block - .users(20) - }) - .expect_consensus_liveness() - .with_run_duration(Duration::from_secs(90)) - .build(); - - let deployer = LocalDeployer::default(); - let runner = deployer.deploy(&plan).await?; - let _handle = runner.run(&mut plan).await?; - - Ok(()) -} -``` diff --git a/book/src/entry-patterns.md b/book/src/entry-patterns.md new file mode 100644 index 0000000..fe15288 --- /dev/null +++ b/book/src/entry-patterns.md @@ -0,0 +1,95 @@ +# Choosing an Entry Pattern + +The framework supports three scenario-based entry patterns and one imperative entry pattern. This chapter compares them. + +--- + +## The Four Patterns + +**1. Uniform managed cluster.** Your system is N identical nodes of one application. Implement `Application` (and the deployer-specific traits), describe a topology, and build a `ScenarioBuilder` over it. The framework spawns, gates, and tears down every node. + +```rust,ignore +let mut scenario = KvScenarioBuilder::deployment_with(|t| t) // 3-node default topology + .with_run_duration(Duration::from_secs(30)) + .with_workload(KvWriteWorkload::new().operations(300)) + .with_expectation(KvConverges::new("demo", 30)) + .build()?; + +let runner = KvLocalDeployer::default().deploy(&scenario).await?; +runner.run(&mut scenario).await?; +``` + +(The shipped `kvstore_basic_convergence` binary wraps the same topology in the `with_existing_kvstore_app` convenience preset; that hybrid is covered in [AppHost and with_app](app-host.md).) + +**2. Composed application stack.** Your system is heterogeneous: several clusters, singleton processes, or both. Start from `AppHost::scenario()` (a zero-node `ScenarioBuilder`) and register one root `AppDeployment` with `.with_app(...)`. The deployment composes children through `DeployContext` and exposes typed handles that workloads retrieve with `AppRunContextExt`. + +```rust,ignore +let mut scenario = AppHost::scenario() + .with_app(JobStackApp::new()) // queue cluster + worker + result store + .with_run_duration(Duration::from_secs(10)) + .with_workload(EnqueueJobs::new(10)) + .with_expectation(AllJobsCompleted::new(10)) + .build()?; + +let runner = AppHostLocalDeployer::default().deploy(&scenario).await?; +runner.run(&mut scenario).await?; +``` + +A scenario accepts one `with_app` registration. A second registration fails at prepare time with a duplicate-runtime-extension error; compose multiple apps inside one root deployment instead ([Composing Heterogeneous Stacks](composing-stacks.md)). + +**3. Attached and external nodes.** The system already runs somewhere else: a staging network, a long-lived cluster, another team's deployment. You plug it in as a source instead of deploying it: `with_existing_cluster(...)` / `with_existing_cluster_from(...)` attach a cluster description, `with_external_node(...)` / `with_external_nodes(...)` add endpoint-only nodes, and `with_external_only_nodes(...)` declares a scenario with no framework-managed nodes at all. `Application::external_node_client` turns each `ExternalNodeSource` into a typed client. Workloads and expectations are unchanged. See [Existing and External Clusters](external-clusters.md). + +**4. ManualCluster: imperative control.** Your code decides when nodes start, stop, and restart, step by step. `ManualCluster::from_topology(descriptors)` (or `ProcessDeployer::manual_cluster_from_descriptors`) gives you `start_node`, `start_node_with(StartNodeOptions)`, `stop_node`, `restart_node`, `wait_network_ready`, `wait_node_ready`, and `node_client`, but no workloads, no expectations, no runner. See [ManualCluster: Imperative Node Control](manual-cluster.md). + +**Note:** needing to restart nodes does *not* push you to ManualCluster. Declarative scenarios gain restart-capable workloads via `with_node_control()` on the builder ([Scenario Capabilities](capabilities.md)), and app-layer child clusters expose `restart_node` on their handles. + +--- + +## One Runtime, Three Declarative Patterns + +```mermaid +flowchart TD + U["Uniform clusterScenarioBuilder::with_deployment"] --> S["Scenario"] + A["Composed stackAppHost::scenario().with_app(...)"] --> S + X["Attached / externalwith_existing_cluster,with_external_nodes"] --> S + S --> R["Deployer::deploy → Runner::run(one lifecycle, see Scenario Model)"] + M["ManualClustermanaged nodes, you drive"] -.->|"bypasses the runner"| C["imperative node control"] + S:::sc + R:::sc + classDef sc stroke:#9b6dd6,stroke-width:2.5px; +``` + +All three declarative patterns produce a `Scenario` and use the same [lifecycle](scenario-model.md), so workloads and expectations can be reused across them when their required clients and capabilities are available. `ManualCluster` uses the node-startup implementation without the scenario runtime. Its nodes remain framework-managed while test code controls the sequence. + +--- + +## Decision Table + +| Shape of the system under test | Pattern | Read next | +|---|---|---| +| N identical nodes of one binary, framework-managed | Uniform managed cluster | [Part IV](part-iv.md) | +| Several apps or clusters composed into one stack | `AppHost` + `with_app` | [Part II](part-ii.md) | +| Already-running nodes you must not deploy | Attached / external sources | [Part V](part-v.md) | +| An external driver dictates every step | `ManualCluster` | [ManualCluster](manual-cluster.md) | + +--- + +## Choosing by Example + +**"Three kvstore nodes, write traffic, convergence check."** Uniform managed cluster. `KvEnv` already models the node; the framework owns the whole population. This is `cargo run -p kvstore-examples --bin kvstore_basic_convergence`. + +**"A queue cluster, a worker process, and a result-store cluster forming one pipeline."** Composed stack. One root `AppDeployment` deploys both clusters, wires the worker to them by URL, and exposes each handle plus a stack handle; a workload enqueues jobs and an expectation verifies the results. The `multi-app-e2e` acceptance test covers this shape; run it with `cargo test -p multi-app-e2e`. + +**"Run our smoke workload against the live staging network."** Attach. There is nothing to deploy: declare the endpoints with `with_external_only_nodes`, let `external_node_client` build clients, and keep the exact same workloads and expectations you use locally. + +**"A Gherkin suite where each step starts or kills a node."** ManualCluster. The BDD runner owns sequencing, and its steps call `start_node_with`, `stop_node`, and `wait_node_ready` directly. + +> **External example:** logos-blockchain's cucumber suite is a real example of the fourth pattern: Gherkin steps drive `ManualCluster` for dependency-ordered starts, restarts, and snapshot/restore flows, all in its own repository. + +--- + +## Where to Go Next + +- [Scenario Model and Lifecycle](scenario-model.md): the runtime every declarative pattern converges on. +- [Application, AppDeployment, and Environments](application-model.md): the types behind patterns 1 and 2. +- [Ownership and Design Boundaries](boundaries.md): what stays yours regardless of pattern. diff --git a/book/src/environment-variables.md b/book/src/environment-variables.md index 6896261..236f609 100644 --- a/book/src/environment-variables.md +++ b/book/src/environment-variables.md @@ -1,375 +1,104 @@ -# Environment Variables Reference +# Environment Variables -Complete reference of environment variables used by the testing framework, organized by category. +This chapter is the complete, audited list of environment variables the framework reads, and where each read happens. + +This chapter was produced by auditing the source (`grep -rn "env::var" testing-framework/ cfgsync/ --include="*.rs"`), not by convention. If a variable is not listed here, the framework does not read it. Re-run the grep after upgrading. --- -## Runner Selection & Topology +## Core (`testing-framework-core`) -Control which runner to use and the test topology: +| Variable | Purpose | Read in | When unset | +|---|---|---|---| +| `SLOW_TEST_ENV` | When exactly `true`, `adjust_timeout` doubles framework timeouts (slow CI runners) | `core/src/lib.rs` | normal timeouts | +| `LOGOS_BLOCKCHAIN_METRICS_QUERY_URL` | Prometheus-compatible query endpoint for `ObservabilityInputs::from_env` | `core/src/scenario/observability.rs` | metrics queries disabled (`Metrics::empty()`) | +| `LOGOS_BLOCKCHAIN_METRICS_OTLP_INGEST_URL` | OTLP metrics ingest endpoint | `core/src/scenario/observability.rs` | none | +| `LOGOS_BLOCKCHAIN_GRAFANA_URL` | Grafana base URL surfaced alongside run output | `core/src/scenario/observability.rs` | none | -| Variable | Default | Effect | -|----------|---------|--------| -| `LOGOS_BLOCKCHAIN_DEMO_NODES` | 1 | Number of nodes (all runners) | -| `LOGOS_BLOCKCHAIN_DEMO_RUN_SECS` | 60 | Run duration in seconds (all runners) | -| `LOCAL_DEMO_NODES` | — | Legacy: Number of nodes (host runner only) | -| `LOCAL_DEMO_RUN_SECS` | — | Legacy: Run duration (host runner only) | -| `COMPOSE_NODE_PAIRS` | — | Compose-specific topology format: "nodes" (e.g., `3`) | - -**Example:** - -```bash -# Run with 5 nodes, for 120 seconds -LOGOS_BLOCKCHAIN_DEMO_NODES=5 \ -LOGOS_BLOCKCHAIN_DEMO_RUN_SECS=120 \ -scripts/run/run-examples.sh -t 120 -n 5 host -``` +The three `LOGOS_BLOCKCHAIN_*` names are historical; they are only consulted when telemetry inputs come from the environment rather than from an `ObservabilityCapability`; see [Telemetry and External Observability](telemetry.md). --- -## Node Binaries (Host Runner) +## Local Deployer (`testing-framework-runner-local`) -Required for host runner when not using helper scripts: +| Variable | Purpose | Read in | When unset | +|---|---|---|---| +| `TF_KEEP_LOGS` | Preserve per-node working directories (`1`/`true`/`yes`) | `deployers/local/src/lib.rs`, honored by the orchestrator and `ManualCluster` | directories deleted at teardown (unless the deployment policy preserves them) | -| Variable | Required | Default | Effect | -|----------|----------|---------|--------| -| `LOGOS_BLOCKCHAIN_NODE_BIN` | Yes (host) | — | Path to `logos-blockchain-node` binary | -| `LOGOS_BLOCKCHAIN_NODE_PATH` | No | — | Path to logos-blockchain-node git checkout (dev workflow) | +Two provider types read **caller-named** variables, where the framework defines the mechanism and the application names the variable: -**Example:** +- `EnvBinaryProvider::new("MY_NODE_BIN")` reads that variable as an explicit executable path. Unset or not-a-file counts as unresolved, letting a `FallbackBinaryProvider` continue to the next provider. +- `DownloadUrl::Env(var)` / `DownloadChecksum::Env(var)` on `DownloadBinaryProvider` read the download URL and expected SHA-256 from the named variables. A missing URL variable is a hard error (`MissingDownloadUrl`); a missing checksum variable disables verification. -```bash -export LOGOS_BLOCKCHAIN_NODE_BIN=/path/to/logos-blockchain-node/target/release/logos-blockchain-node -``` +See [Binary Providers](binary-providers.md). --- -## Docker Images (Compose / K8s) +## Compose Deployer (`testing-framework-runner-compose`) -Required for compose and k8s runners: +| Variable | Purpose | Read in | When unset | +|---|---|---|---| +| `COMPOSE_RUNNER_PRESERVE` | Skip `docker compose down`, keep the workspace | `lifecycle/cleanup.rs` | full teardown | +| `TESTNET_RUNNER_PRESERVE` | Alias for the above | `lifecycle/cleanup.rs` | full teardown | +| `COMPOSE_RUNNER_HOST` | Host used to reach published container ports | `infrastructure/ports.rs` | `127.0.0.1` | +| `COMPOSE_RUNNER_HOST_GATEWAY` | Explicit `extra_hosts` gateway entry; `disable` or empty removes it | `docker/platform.rs` | falls through to `DOCKER_HOST_GATEWAY` | +| `DOCKER_HOST_GATEWAY` | Gateway IP mapped as `host.docker.internal:` | `docker/platform.rs` | `host.docker.internal:host-gateway` | +| `TESTNET_PRINT_ENDPOINTS` | If set (any value), print discovered endpoints after deploy | `deployer/orchestrator.rs` | silent | +| `REPO_ROOT_OVERRIDE_DIR` | Override repository-root detection for stack assets | `docker/workspace.rs` | falls through to `CARGO_WORKSPACE_DIR`, then manifest-relative detection | +| `CARGO_WORKSPACE_DIR` | Workspace root override (also used by template rendering) | `docker/workspace.rs`, `infrastructure/template.rs` | manifest-relative detection | +| `REL_ASSETS_STACK_DIR` | Alternative stack-assets directory (absolute, or relative to repo root) | `docker/workspace.rs` | bundled default assets | -| Variable | Required | Default | Effect | -|----------|----------|---------|--------| -| `LOGOS_BLOCKCHAIN_TESTNET_IMAGE` | Yes (compose/k8s) | `logos-blockchain-testing:local` | Docker image tag for node containers | -| `LOGOS_BLOCKCHAIN_TESTNET_IMAGE_PULL_POLICY` | No | `IfNotPresent` (local) / `Always` (ECR) | K8s `imagePullPolicy` used by the runner | -| `LOGOS_BLOCKCHAIN_BINARIES_TAR` | No | — | Path to prebuilt bundle (`.tar.gz`) for image build | -| `LOGOS_BLOCKCHAIN_SKIP_IMAGE_BUILD` | No | 0 | Skip image rebuild (compose/k8s); assumes image already exists | -| `LOGOS_BLOCKCHAIN_FORCE_IMAGE_BUILD` | No | 0 | Force rebuilding the image even when the script would normally skip it (e.g. non-local k8s) | - -**Example:** - -```bash -# Using prebuilt bundle -export LOGOS_BLOCKCHAIN_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz -export LOGOS_BLOCKCHAIN_TESTNET_IMAGE=logos-blockchain-testing:local -scripts/build/build_test_image.sh - -# Using pre-existing image (skip build) -export LOGOS_BLOCKCHAIN_SKIP_IMAGE_BUILD=1 -scripts/run/run-examples.sh -t 60 -n 3 compose -``` +Per-application image selection is again a mechanism with caller-derived names: `BinaryConfigNodeSpec::conventional("/usr/local/bin/kvstore-node", ...)` derives the prefix `KVSTORE` and reads `KVSTORE_IMAGE` (default `kvstore-node:local`) and `KVSTORE_PLATFORM` (`descriptor/node.rs`). --- -## Circuit Assets +## Kubernetes Deployer (`testing-framework-runner-k8s`) -Circuit asset configuration used by local runs and image builds: +| Variable | Purpose | Read in | When unset | +|---|---|---|---| +| `K8S_RUNNER_NODE_HOST` | Host used to reach NodePort services | `host.rs` | `KUBERNETES_SERVICE_HOST`, then `127.0.0.1` | +| `KUBERNETES_SERVICE_HOST` | Standard fallback for the above (e.g. Docker Desktop) | `host.rs` | `127.0.0.1` | +| `K8S_RUNNER_PRESERVE` | Skip Helm uninstall and namespace deletion | `env.rs` | full teardown | +| `K8S_RUNNER_DEBUG` | Log Helm install stdout/stderr | `infrastructure/helm.rs` | Helm output suppressed | +| `K8S_RUNNER_DEPLOYMENT_TIMEOUT_SECS` | Deployment readiness timeout (integer seconds) | `lifecycle/wait/mod.rs` | built-in default | +| `K8S_RUNNER_HTTP_TIMEOUT_SECS` | Node HTTP readiness timeout | `lifecycle/wait/mod.rs` | built-in default | +| `K8S_RUNNER_HTTP_PROBE_TIMEOUT_SECS` | Per-probe HTTP timeout | `lifecycle/wait/mod.rs` | built-in default | +| `K8S_RUNNER_HTTP_POLL_INTERVAL_SECS` | Readiness poll interval | `lifecycle/wait/mod.rs` | built-in default | +| `TESTNET_PRINT_ENDPOINTS` | If set, print Prometheus/Grafana/pprof endpoints after deploy | `deployer/orchestrator.rs` | silent | -| Variable | Default | Effect | -|----------|---------|--------| -| `LOGOS_BLOCKCHAIN_CIRCUITS` | `~/.logos-blockchain-circuits` | Directory containing circuit assets | -| `VERSION` | From `versions.env` | Circuit release tag (used by helper scripts) | -| `LOGOS_BLOCKCHAIN_CIRCUITS_VERSION` | — | Legacy alias for `VERSION` (supported by some build scripts) | -| `LOGOS_BLOCKCHAIN_CIRCUITS_PLATFORM` | Auto-detected | Override circuits platform (e.g. `linux-x86_64`, `macos-aarch64`) | -| `LOGOS_BLOCKCHAIN_CIRCUITS_HOST_DIR_REL` | `.tmp/logos-blockchain-circuits-host` | Output dir for host circuit bundle (relative to repo root) | -| `LOGOS_BLOCKCHAIN_CIRCUITS_LINUX_DIR_REL` | `.tmp/logos-blockchain-circuits-linux` | Output dir for linux circuit bundle (relative to repo root) | -| `LOGOS_BLOCKCHAIN_CIRCUITS_NONINTERACTIVE` | 0 | Set to `1` to overwrite outputs without prompting in setup scripts | -| `LOGOS_BLOCKCHAIN_CIRCUITS_REBUILD_RAPIDSNARK` | 0 | Set to `1` to force rebuilding rapidsnark (host bundle only) | - -**Example:** - -```bash -# Use custom circuit assets -LOGOS_BLOCKCHAIN_CIRCUITS=/custom/path/to/circuits \ -cargo run -p runner-examples --bin local_runner -``` +Image selection mirrors compose with a k8s-specific override first: `BinaryConfigK8sSpec::conventional` reads `_K8S_IMAGE`, then `_IMAGE`, then the `:local` default (`env.rs`). `workspace.rs` additionally exposes `resolve_workspace_root` / `resolve_optional_relative_dir` helpers that read a variable **named by the caller**. --- -## Node Logging +## cfgsync Runtime (`cfgsync-runtime`) -Control node log output (not framework runner logs): +These are read by the cfgsync **client inside node containers** at startup, not by your test process; the deployers set them when rendering the stack. See [Static Artifacts and cfgsync](cfgsync.md). -| Variable | Default | Effect | -|----------|---------|--------| -| `LOGOS_BLOCKCHAIN_LOG_LEVEL` | `info` | Global log level: `error`, `warn`, `info`, `debug`, `trace` | -| `LOGOS_BLOCKCHAIN_LOG_FILTER` | — | Fine-grained module filtering (e.g., `cryptarchia=trace`) | -| `LOGOS_BLOCKCHAIN_LOG_DIR` | — | Host runner: directory for per-node log files (persistent). Compose/k8s: use `cfgsync.yaml` for file logging. | -| `LOGOS_BLOCKCHAIN_TESTS_KEEP_LOGS` | 0 | Keep per-run temporary directories (useful for debugging/CI artifacts) | -| `LOGOS_BLOCKCHAIN_TESTS_TRACING` | false | Enable debug tracing preset (combine with `LOGOS_BLOCKCHAIN_LOG_DIR` unless external tracing backends configured) | - -**Important:** Node logging ignores `RUST_LOG`; use `LOGOS_BLOCKCHAIN_LOG_LEVEL` and `LOGOS_BLOCKCHAIN_LOG_FILTER` for node logs. - -**Example:** - -```bash -# Debug logging to files -LOGOS_BLOCKCHAIN_LOG_DIR=/tmp/test-logs \ -LOGOS_BLOCKCHAIN_LOG_LEVEL=debug \ -LOGOS_BLOCKCHAIN_LOG_FILTER="cryptarchia=trace" \ -cargo run -p runner-examples --bin local_runner - -# Inspect logs -ls /tmp/test-logs/ -# logos-blockchain-node-0.2024-12-18T14-30-00.log -# logos-blockchain-node-1.2024-12-18T14-30-00.log -``` - -**Common filter targets:** - -| Target Prefix | Subsystem | -|---------------------------|-----------| -| `lb_cryptarchia` | Consensus (Cryptarchia) | -| `lb_blend` | Mix network/privacy layer | -| `lb_chain_service` | Chain service (node APIs/state) | -| `lb_chain_network` | P2P networking | -| `lb_chain_leader_service` | Leader election | +| Variable | Purpose | When unset | +|---|---|---| +| `CFG_SERVER_ADDR` | cfgsync server URL | `http://127.0.0.1:` | +| `CFG_HOST_IP` | This node's IPv4 address for registration | `127.0.0.1` | +| `CFG_HOST_IDENTIFIER` | Node identifier for registration | `unidentified-node` | +| `CFG_REGISTRATION_METADATA_JSON` | Extra registration payload (JSON) | empty payload | +| `CFG_FILE_PATH` | Where to write the fetched `config.yaml` | config output not routed | +| `CFG_DEPLOYMENT_PATH` | Where to write the fetched deployment settings | deployment output not routed | +| `LOGOS_BLOCKCHAIN_CFGSYNC_PORT` | Default server port for the `cfgsync-client` binary | `4400` | --- -## Observability & Metrics +## Example-App Variables (Not Framework Variables) -Optional observability integration: +The example applications define their own variables through the mechanisms above. **These belong to the examples**: `KVSTORE_NODE_BIN` is defined by the kvstore example's environment implementation, not by the framework; your application will define its own equivalents. Found by auditing `examples/`: -| Variable | Default | Effect | -|----------|---------|--------| -| `LOGOS_BLOCKCHAIN_METRICS_QUERY_URL` | — | Prometheus-compatible base URL for runner to query (e.g., `http://localhost:9090`) | -| `LOGOS_BLOCKCHAIN_METRICS_OTLP_INGEST_URL` | — | Full OTLP HTTP ingest URL for node metrics export (e.g., `http://localhost:9090/api/v1/otlp/v1/metrics`) | -| `LOGOS_BLOCKCHAIN_GRAFANA_URL` | — | Grafana base URL for printing/logging (e.g., `http://localhost:3000`) | -| `LOGOS_BLOCKCHAIN_OTLP_ENDPOINT` | — | OTLP trace endpoint (optional) | -| `LOGOS_BLOCKCHAIN_OTLP_METRICS_ENDPOINT` | — | OTLP metrics endpoint (optional) | +| Variable | Example | Purpose | +|---|---|---| +| `KVSTORE_NODE_BIN`, `OPENRAFT_KV_NODE_BIN` | kvstore, openraft_kv | optional binary override (fallback builds with Cargo) | +| `QUEUE_NODE_BIN`, `PUBSUB_NODE_BIN`, `METRICS_COUNTER_NODE_BIN` | queue, pubsub, metrics_counter | required node binary path for local runs | +| `NATS_SERVER_BIN` | nats | path to an upstream `nats-server` executable | +| `NATS_IMAGE` / `NATS_PLATFORM` | nats | compose image override (default `nats:2.10`) | +| `REDIS_STREAMS_IMAGE` / `REDIS_STREAMS_PLATFORM` | redis_streams | compose image override (default `redis:7`) | +| `KVSTORE_IMAGE`, `QUEUE_IMAGE`, … (`_IMAGE`/`_PLATFORM`/`_K8S_IMAGE`) | all node apps | derived image overrides via the conventional specs | +| `METRICS_COUNTER_K8S_PROMETHEUS_NODE_PORT` | metrics_counter | fixed NodePort for the Prometheus service | +| `LOGOS_BLOCKCHAIN_METRICS_QUERY_URL` | metrics_counter | also consulted by the example to locate Prometheus | -**Example:** - -```bash -# Enable Prometheus querying -export LOGOS_BLOCKCHAIN_METRICS_QUERY_URL=http://localhost:9090 -export LOGOS_BLOCKCHAIN_METRICS_OTLP_INGEST_URL=http://localhost:9090/api/v1/otlp/v1/metrics -export LOGOS_BLOCKCHAIN_GRAFANA_URL=http://localhost:3000 - -scripts/run/run-examples.sh -t 60 -n 3 compose -``` - ---- - -## Compose Runner Specific - -Variables specific to Docker Compose deployment: - -| Variable | Default | Effect | -|----------|---------|--------| -| `COMPOSE_RUNNER_HOST` | `127.0.0.1` | Host address for port mappings | -| `COMPOSE_RUNNER_PRESERVE` | 0 | Keep containers running after test (for debugging) | -| `COMPOSE_RUNNER_HTTP_TIMEOUT_SECS` | — | Override HTTP readiness timeout (seconds) | -| `COMPOSE_RUNNER_HOST_GATEWAY` | `host.docker.internal:host-gateway` | Controls `extra_hosts` entry injected into compose (set to `disable` to omit) | -| `TESTNET_RUNNER_PRESERVE` | — | Alias for `COMPOSE_RUNNER_PRESERVE` | - -**Example:** - -```bash -# Keep containers after test for debugging -COMPOSE_RUNNER_PRESERVE=1 \ -scripts/run/run-examples.sh -t 60 -n 3 compose - -# Containers remain running -docker ps --filter "name=nomos-compose-" -docker logs -``` - ---- - -## K8s Runner Specific - -Variables specific to Kubernetes deployment: - -| Variable | Default | Effect | -|----------|---------|--------| -| `K8S_RUNNER_NAMESPACE` | Random UUID | Kubernetes namespace (pin for debugging) | -| `K8S_RUNNER_RELEASE` | Random UUID | Helm release name (pin for debugging) | -| `K8S_RUNNER_NODE_HOST` | — | NodePort host resolution for non-local clusters | -| `K8S_RUNNER_DEBUG` | 0 | Log Helm stdout/stderr for install commands | -| `K8S_RUNNER_PRESERVE` | 0 | Keep namespace/release after run (for debugging) | -| `K8S_RUNNER_DEPLOYMENT_TIMEOUT_SECS` | — | Override deployment readiness timeout | -| `K8S_RUNNER_HTTP_TIMEOUT_SECS` | — | Override HTTP readiness timeout (port-forwards) | -| `K8S_RUNNER_HTTP_PROBE_TIMEOUT_SECS` | — | Override HTTP readiness timeout (NodePort probes) | -| `K8S_RUNNER_PROMETHEUS_HTTP_TIMEOUT_SECS` | — | Override Prometheus readiness timeout | -| `K8S_RUNNER_PROMETHEUS_HTTP_PROBE_TIMEOUT_SECS` | — | Override Prometheus NodePort probe timeout | - -**Example:** - -```bash -# Pin namespace for debugging -K8S_RUNNER_NAMESPACE=nomos-test-debug \ -K8S_RUNNER_PRESERVE=1 \ -K8S_RUNNER_DEBUG=1 \ -scripts/run/run-examples.sh -t 60 -n 3 k8s - -# Inspect resources -kubectl get pods -n nomos-test-debug -kubectl logs -n nomos-test-debug -l nomos/logical-role=node -``` - ---- - -## Platform & Build Configuration - -Platform-specific build configuration: - -| Variable | Default | Effect | -|----------|---------|--------| -| `LOGOS_BLOCKCHAIN_BUNDLE_DOCKER_PLATFORM` | Host arch | Docker platform for bundle builds: `linux/arm64` or `linux/amd64` (macOS/Windows hosts) | -| `LOGOS_BLOCKCHAIN_BIN_PLATFORM` | — | Legacy alias for `LOGOS_BLOCKCHAIN_BUNDLE_DOCKER_PLATFORM` | -| `COMPOSE_CIRCUITS_PLATFORM` | Host arch | Circuits platform for image builds: `linux-aarch64` or `linux-x86_64` | -| `LOGOS_BLOCKCHAIN_EXTRA_FEATURES` | — | Extra cargo features to enable when building bundles (used by `scripts/build/build-bundle.sh`) | - -**macOS / Apple Silicon:** - -```bash -# Native performance (recommended for local testing) -export LOGOS_BLOCKCHAIN_BUNDLE_DOCKER_PLATFORM=linux/arm64 - -# Or target amd64 (slower via emulation) -export LOGOS_BLOCKCHAIN_BUNDLE_DOCKER_PLATFORM=linux/amd64 -``` - ---- - -## Timeouts & Performance - -Timeout and performance tuning: - -| Variable | Default | Effect | -|----------|---------|--------| -| `SLOW_TEST_ENV` | false | Doubles built-in readiness timeouts (useful in CI / constrained laptops) | -| `TESTNET_PRINT_ENDPOINTS` | 0 | Print `TESTNET_ENDPOINTS` / `TESTNET_PPROF` lines during deploy (set automatically by `scripts/run/run-examples.sh`) | - -**Example:** - -```bash -# Increase timeouts for slow environments -SLOW_TEST_ENV=true \ -scripts/run/run-examples.sh -t 120 -n 5 compose -``` - ---- - -## Node Configuration (Advanced) - -Node-level configuration passed through to logos-blockchain-node: - -| Variable | Default | Effect | -|----------|---------|--------| -| `CONSENSUS_SLOT_TIME` | — | Consensus slot time (seconds) | -| `CONSENSUS_ACTIVE_SLOT_COEFF` | — | Active slot coefficient (0.0-1.0) | -| `LOGOS_BLOCKCHAIN_USE_AUTONAT` | Unset | If set, use AutoNAT instead of a static loopback address for libp2p NAT settings | -| `LOGOS_BLOCKCHAIN_CFGSYNC_PORT` | 4400 | Port used for cfgsync service inside the stack | -| `LOGOS_BLOCKCHAIN_TIME_BACKEND` | `monotonic` | Select time backend (used by compose/k8s stack scripts and deployers) | - -**Example:** - -```bash -# Faster block production -CONSENSUS_SLOT_TIME=5 \ -CONSENSUS_ACTIVE_SLOT_COEFF=0.9 \ -cargo run -p runner-examples --bin local_runner -``` - ---- - -## Framework Runner Logging (Not Node Logs) - -Control framework runner process logs (uses `RUST_LOG`, not `NOMOS_*`): - -| Variable | Default | Effect | -|----------|---------|--------| -| `RUST_LOG` | — | Framework runner log level (e.g., `debug`, `info`) | -| `RUST_BACKTRACE` | — | Enable Rust backtraces on panic (`1` or `full`) | -| `CARGO_TERM_COLOR` | — | Cargo output color (`always`, `never`, `auto`) | - -**Example:** - -```bash -# Debug framework runner (not nodes) -RUST_LOG=debug \ -RUST_BACKTRACE=1 \ -cargo run -p runner-examples --bin local_runner -``` - ---- - -## Helper Script Variables - -Variables used by helper scripts (`scripts/run/run-examples.sh`, etc.): - -| Variable | Default | Effect | -|----------|---------|--------| -| `LOGOS_BLOCKCHAIN_NODE_REV` | From `versions.env` | logos-blockchain-node git revision to build/fetch | -| `LOGOS_BLOCKCHAIN_BUNDLE_VERSION` | From `versions.env` | Bundle schema version | -| `LOGOS_BLOCKCHAIN_IMAGE_SELECTION` | — | Internal: image selection mode set by `run-examples.sh` (`local`/`ecr`/`auto`) | -| `LOGOS_BLOCKCHAIN_NODE_APPLY_PATCHES` | 1 | Set to `0` to disable applying local patches when building bundles | -| `LOGOS_BLOCKCHAIN_NODE_PATCH_DIR` | `patches/logos-blockchain-node` | Patch directory applied to logos-blockchain-node checkout during bundle builds | -| `LOGOS_BLOCKCHAIN_NODE_PATCH_LEVEL` | — | Patch application level (`all` or an integer) for bundle builds | - ---- - -## Quick Reference Examples - -### Minimal Host Run - -```bash -scripts/run/run-examples.sh -t 60 -n 3 host -``` - -### Debug Logging (Host) - -```bash -LOGOS_BLOCKCHAIN_LOG_DIR=/tmp/logs \ -LOGOS_BLOCKCHAIN_LOG_LEVEL=debug \ -LOGOS_BLOCKCHAIN_LOG_FILTER="cryptarchia=trace" \ -scripts/run/run-examples.sh -t 60 -n 3 host -``` - -### Compose with Observability - -```bash -LOGOS_BLOCKCHAIN_METRICS_QUERY_URL=http://localhost:9090 \ -LOGOS_BLOCKCHAIN_GRAFANA_URL=http://localhost:3000 \ -scripts/run/run-examples.sh -t 60 -n 3 compose -``` - -### K8s with Debug - -```bash -K8S_RUNNER_NAMESPACE=nomos-debug \ -K8S_RUNNER_DEBUG=1 \ -K8S_RUNNER_PRESERVE=1 \ -scripts/run/run-examples.sh -t 60 -n 3 k8s -``` - -### CI Environment - -```yaml -env: - RUST_BACKTRACE: 1 - LOGOS_BLOCKCHAIN_TESTS_KEEP_LOGS: 1 -``` - ---- - -## See Also - -- [Prerequisites & Setup](prerequisites.md) — Required files and setup -- [Running Examples](running-examples.md) — How to run scenarios -- [Logging & Observability](logging-observability.md) — Log collection details -- [CI Integration](ci-integration.md) — CI-specific variables -- [Troubleshooting](troubleshooting.md) — Common issues with variables +See [Running the Examples](running-examples.md) for how these fit each binary. diff --git a/book/src/examples-advanced.md b/book/src/examples-advanced.md deleted file mode 100644 index f737ca4..0000000 --- a/book/src/examples-advanced.md +++ /dev/null @@ -1,328 +0,0 @@ -# Advanced Examples - -> **When should I read this?** Skim now to see what's possible, revisit later when you need load testing, chaos scenarios, or custom extensions. Start with [basic examples](examples.md) first. - -Realistic advanced scenarios demonstrating framework capabilities for production testing. - -**Adapt from Complete Source:** -- [compose_runner.rs](https://github.com/logos-blockchain/logos-blockchain-testing/blob/master/examples/src/bin/compose_runner.rs) — Compose examples with workloads -- [k8s_runner.rs](https://github.com/logos-blockchain/logos-blockchain-testing/blob/master/examples/src/bin/k8s_runner.rs) — K8s production patterns -- [Chaos testing patterns](https://github.com/logos-blockchain/logos-blockchain-testing/blob/master/testing-framework/workflows/src/workloads/chaos.rs) — Node control implementation - -## Summary - -| Example | Topology | Workloads | Deployer | Key Feature | -|---------|----------|-----------|----------|-------------| -| Load Progression | 3 nodes | Increasing tx rate | Compose | Dynamic load testing | -| Sustained Load | 4 nodes | High tx rate | Compose | Stress testing | -| Aggressive Chaos | 4 nodes | Frequent restarts + traffic | Compose | Resilience validation | - -## Load Progression Test - -Test consensus under progressively increasing transaction load: - -```rust,ignore -use std::time::Duration; - -use anyhow::Result; -use testing_framework_core::scenario::{Deployer, ScenarioBuilder}; -use testing_framework_runner_compose::ComposeDeployer; -use testing_framework_workflows::ScenarioBuilderExt; - -pub async fn load_progression_test() -> Result<()> { - for rate in [5, 10, 20, 30] { - println!("Testing with rate: {}", rate); - - let mut plan = - ScenarioBuilder::topology_with(|t| t.network_star().nodes(3)) - .wallets(50) - .transactions_with(|txs| txs.rate(rate).users(20)) - .expect_consensus_liveness() - .with_run_duration(Duration::from_secs(60)) - .build(); - - let deployer = ComposeDeployer::default(); - let runner = deployer.deploy(&plan).await?; - let _handle = runner.run(&mut plan).await?; - } - - Ok(()) -} -``` - -**When to use:** Finding the maximum sustainable transaction rate for a given topology. - -## Sustained Load Test - -Run high transaction load for extended duration: - -```rust,ignore -use std::time::Duration; - -use anyhow::Result; -use testing_framework_core::scenario::{Deployer, ScenarioBuilder}; -use testing_framework_runner_compose::ComposeDeployer; -use testing_framework_workflows::ScenarioBuilderExt; - -pub async fn sustained_load_test() -> Result<()> { - let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().nodes(4)) - .wallets(100) - .transactions_with(|txs| txs.rate(15).users(50)) - .expect_consensus_liveness() - .with_run_duration(Duration::from_secs(300)) - .build(); - - let deployer = ComposeDeployer::default(); - let runner = deployer.deploy(&plan).await?; - let _handle = runner.run(&mut plan).await?; - - Ok(()) -} -``` - -**When to use:** Validating stability under continuous high load over extended periods. - -## Aggressive Chaos Test - -Frequent node restarts with active traffic: - -```rust,ignore -use std::time::Duration; - -use anyhow::Result; -use testing_framework_core::scenario::{Deployer, ScenarioBuilder}; -use testing_framework_runner_compose::ComposeDeployer; -use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt}; - -pub async fn aggressive_chaos_test() -> Result<()> { - let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().nodes(4)) - .enable_node_control() - .wallets(50) - .transactions_with(|txs| txs.rate(10).users(20)) - .chaos_with(|c| { - c.restart() - .min_delay(Duration::from_secs(10)) - .max_delay(Duration::from_secs(20)) - .target_cooldown(Duration::from_secs(15)) - .apply() - }) - .expect_consensus_liveness() - .with_run_duration(Duration::from_secs(180)) - .build(); - - let deployer = ComposeDeployer::default(); - let runner = deployer.deploy(&plan).await?; - let _handle = runner.run(&mut plan).await?; - - Ok(()) -} -``` - -**When to use:** Validating recovery and liveness under aggressive failure conditions. - -**Note:** Requires `ComposeDeployer` for node control support. - -## Extension Ideas - -These scenarios require custom implementations but demonstrate framework extensibility: - -### Mempool & Transaction Handling - -#### Transaction Propagation & Inclusion Test - -**Concept:** Submit the same batch of independent transactions to different nodes in randomized order/offsets, then verify all transactions are included and final state matches across nodes. - -**Requirements:** -- **Custom workload:** Generates a fixed batch of transactions and submits the same set to different nodes via `ctx.node_clients()`, with randomized submission order and timing offsets per node -- **Custom expectation:** Verifies all transactions appear in blocks (order may vary), final state matches across all nodes (compare balances or state roots), and no transactions are dropped - -**Why useful:** Exercises mempool propagation, proposer fairness, and transaction inclusion guarantees under realistic race conditions. Tests that the protocol maintains consistency regardless of which node receives transactions first. - -**Implementation notes:** Requires both a custom `Workload` implementation (to submit same transactions to multiple nodes with jitter) and a custom `Expectation` implementation (to verify inclusion and state consistency). - -#### Cross-Validator Mempool Divergence & Convergence - -**Concept:** Drive different transaction subsets into different nodes (or differing arrival orders) to create temporary mempool divergence, then verify mempools/blocks converge to contain the union (no permanent divergence). - -**Requirements:** -- **Custom workload:** Targets specific nodes via `ctx.node_clients()` with disjoint or jittered transaction batches -- **Custom expectation:** After a convergence window, verifies that all transactions appear in blocks (order may vary) or that mempool contents converge across nodes -- Run normal workloads during convergence period - -**Expectations:** -- Temporary mempool divergence is acceptable (different nodes see different transactions initially) -- After convergence window, all transactions appear in blocks or mempools converge -- No transactions are permanently dropped despite initial divergence -- Mempool gossip/reconciliation mechanisms work correctly - -**Why useful:** Exercises mempool gossip and reconciliation under uneven input or latency. Ensures no node "drops" transactions seen elsewhere, validating that mempool synchronization mechanisms correctly propagate transactions across the network even when they arrive at different nodes in different orders. - -**Implementation notes:** Requires both a custom `Workload` implementation (to inject disjoint/jittered batches per node) and a custom `Expectation` implementation (to verify mempool convergence or block inclusion). Uses existing `ctx.node_clients()` capability—no new infrastructure needed. - -#### Adaptive Mempool Pressure Test - -**Concept:** Ramp transaction load over time to observe mempool growth, fee prioritization/eviction, and block saturation behavior, detecting performance regressions and ensuring backpressure/eviction work under increasing load. - -**Requirements:** -- **Custom workload:** Steadily increases transaction rate over time (optional: use fee tiers) -- **Custom expectation:** Monitors mempool size, evictions, and throughput (blocks/txs per slot), flagging runaway growth or stalls -- Run for extended duration to observe pressure buildup - -**Expectations:** -- Mempool size grows predictably with load (not runaway growth) -- Fee prioritization/eviction mechanisms activate under pressure -- Block saturation behavior is acceptable (blocks fill appropriately) -- Throughput (blocks/txs per slot) remains stable or degrades gracefully -- No stalls or unbounded mempool growth - -**Why useful:** Detects performance regressions in mempool management. Ensures backpressure and eviction mechanisms work correctly under increasing load, preventing memory exhaustion or unbounded growth. Validates that fee prioritization correctly selects high-value transactions when mempool is full. - -**Implementation notes:** Can be built with current workload model (ramping rate). Requires custom `Expectation` implementation that reads mempool metrics (via node HTTP APIs or Prometheus) and monitors throughput to judge behavior. No new infrastructure needed—uses existing observability capabilities. - -#### Invalid Transaction Fuzzing - -**Concept:** Submit malformed transactions and verify they're rejected properly. - -**Implementation approach:** -- Custom workload that generates invalid transactions (bad signatures, insufficient funds, malformed structure) -- Expectation verifies mempool rejects them and they never appear in blocks -- Test mempool resilience and filtering - -**Why useful:** Ensures mempool doesn't crash or include invalid transactions under fuzzing. - -### Network & Gossip - -#### Gossip Latency Gradient Scenario - -**Concept:** Test consensus robustness under skewed gossip delays by partitioning nodes into latency tiers (tier A ≈10ms, tier B ≈100ms, tier C ≈300ms) and observing propagation lag, fork rate, and eventual convergence. - -**Requirements:** -- Partition nodes into three groups (tiers) -- Apply per-group network delay via chaos: `netem`/`iptables` in compose; NetworkPolicy + `netem` sidecar in k8s -- Run standard workload (transactions/block production) -- Optional: Remove delays at end to check recovery - -**Expectations:** -- **Propagation:** Messages reach all tiers within acceptable bounds -- **Safety:** No divergent finalized heads; fork rate stays within tolerance -- **Liveness:** Chain keeps advancing; convergence after delays relaxed (if healed) - -**Why useful:** Real networks have heterogeneous latency. This stress-tests proposer selection and fork resolution when some peers are "far" (high latency), validating that consensus remains safe and live under realistic network conditions. - -**Current blocker:** Runner support for per-group delay injection (network delay via `netem`/`iptables`) is not present today. Would require new chaos plumbing in compose/k8s deployers to inject network delays per node group. - -#### Byzantine Gossip Flooding (libp2p Peer) - -**Concept:** Spin up a custom workload/sidecar that runs a libp2p host, joins the cluster's gossip mesh, and publishes a high rate of syntactically valid but useless/stale messages to selected topics, testing gossip backpressure, scoring, and queue handling under a "malicious" peer. - -**Requirements:** -- Custom workload/sidecar that implements a libp2p host -- Join the cluster's gossip mesh as a peer -- Publish high-rate syntactically valid but useless/stale messages to selected gossip topics -- Run alongside normal workloads (transactions/block production) - -**Expectations:** -- Gossip backpressure mechanisms prevent message flooding from overwhelming nodes -- Peer scoring correctly identifies and penalizes the malicious peer -- Queue handling remains stable under flood conditions -- Normal consensus operation continues despite malicious peer - -**Why useful:** Tests Byzantine behavior (malicious peer) which is critical for consensus protocol robustness. More realistic than RPC spam since it uses the actual gossip protocol. Validates that gossip backpressure, peer scoring, and queue management correctly handle adversarial peers without disrupting consensus. - -**Current blocker:** Requires adding gossip-capable helper (libp2p integration) to the framework. Would need a custom workload/sidecar implementation that can join the gossip mesh and inject messages. The rest of the scenario can use existing runners/workloads. - -#### Network Partition Recovery - -**Concept:** Test consensus recovery after network partitions. - -**Requirements:** -- Needs `block_peer()` / `unblock_peer()` methods in `NodeControlHandle` -- Partition subsets of nodes, wait, then restore connectivity -- Verify chain convergence after partition heals - -**Why useful:** Tests the most realistic failure mode in distributed systems. - -**Current blocker:** Node control doesn't yet support network-level actions (only process restarts). - -### Time & Timing - -#### Time-Shifted Blocks (Clock Skew Test) - -**Concept:** Test consensus and timestamp handling when nodes run with skewed clocks (e.g., +1s, −1s, +200ms jitter) to surface timestamp validation issues, reorg sensitivity, and clock drift handling. - -**Requirements:** -- Assign per-node time offsets (e.g., +1s, −1s, +200ms jitter) -- Run normal workload (transactions/block production) -- Observe whether blocks are accepted/propagated and the chain stays consistent - -**Expectations:** -- Blocks with skewed timestamps are handled correctly (accepted or rejected per protocol rules) -- Chain remains consistent across nodes despite clock differences -- No unexpected reorgs or chain splits due to timestamp validation issues - -**Why useful:** Clock skew is a common real-world issue in distributed systems. This validates that consensus correctly handles timestamp validation and maintains safety/liveness when nodes have different clock offsets, preventing timestamp-based attacks or failures. - -**Current blocker:** Runner ability to skew per-node clocks (e.g., privileged containers with `libfaketime`/`chrony` or time-offset netns) is not available today. Would require a new chaos/time-skew hook in deployers to inject clock offsets per node. - -#### Block Timing Consistency - -**Concept:** Verify block production intervals stay within expected bounds. - -**Implementation approach:** -- Custom expectation that consumes `BlockFeed` -- Collect block timestamps during run -- Assert intervals are within `(slot_duration * active_slot_coeff) ± tolerance` - -**Why useful:** Validates consensus timing under various loads. - -### Topology & Membership - -#### Dynamic Topology (Churn) Scenario - -**Concept:** Nodes join and leave mid-run (new identities/addresses added; some nodes permanently removed) to exercise peer discovery, bootstrapping, reputation, and load balancing under churn. - -**Requirements:** -- Runner must be able to spin up new nodes with fresh keys/addresses at runtime -- Update peer lists and bootstraps dynamically as nodes join/leave -- Optionally tear down nodes permanently (not just restart) -- Run normal workloads (transactions/block production) during churn - -**Expectations:** -- New nodes successfully discover and join the network -- Peer discovery mechanisms correctly handle dynamic topology changes -- Reputation systems adapt to new/removed peers -- Load balancing adjusts to changing node set -- Consensus remains safe and live despite topology churn - -**Why useful:** Real networks experience churn (nodes joining/leaving). Unlike restarts (which preserve topology), churn changes the actual topology size and peer set, testing how the protocol handles dynamic membership. This exercises peer discovery, bootstrapping, reputation systems, and load balancing under realistic conditions. - -**Current blocker:** Runner support for dynamic node addition/removal at runtime is not available today. Chaos today only restarts existing nodes; churn would require the ability to spin up new nodes with fresh identities/addresses, update peer lists/bootstraps dynamically, and permanently remove nodes. Would need new topology management capabilities in deployers. - -### API & External Interfaces - -#### API DoS/Stress Test - -**Concept:** Adversarial workload floods node HTTP/WS APIs with high QPS and malformed/bursty requests; expectation checks nodes remain responsive or rate-limit without harming consensus. - -**Requirements:** -- **Custom workload:** Targets node HTTP/WS API endpoints with mixed valid/invalid requests at high rate -- **Custom expectation:** Monitors error rates, latency, and confirms block production/liveness unaffected -- Run alongside normal workloads (transactions/block production) - -**Expectations:** -- Nodes remain responsive or correctly rate-limit under API flood -- Error rates/latency are acceptable (rate limiting works) -- Block production/liveness unaffected by API abuse -- Consensus continues normally despite API stress - -**Why useful:** Validates API hardening under abuse and ensures control/telemetry endpoints don't destabilize the node. Tests that API abuse is properly isolated from consensus operations, preventing DoS attacks on API endpoints from affecting blockchain functionality. - -**Implementation notes:** Requires custom `Workload` implementation that directs high-QPS traffic to node APIs (via `ctx.node_clients()` or direct HTTP clients) and custom `Expectation` implementation that monitors API responsiveness metrics and consensus liveness. Uses existing node API access—no new infrastructure needed. - -### State & Correctness - -#### Wallet Balance Verification - -**Concept:** Track wallet balances and verify state consistency. - -**Description:** After transaction workload completes, query all wallet balances via node API and verify total supply is conserved. Requires tracking initial state, submitted transactions, and final balances. Validates that the ledger maintains correctness under load (no funds lost or created). This is a **state assertion** expectation that checks correctness, not just liveness. diff --git a/book/src/examples.md b/book/src/examples.md deleted file mode 100644 index 6f696a2..0000000 --- a/book/src/examples.md +++ /dev/null @@ -1,118 +0,0 @@ -# Examples - -Concrete scenario shapes that illustrate how to combine topologies, workloads, -and expectations. - -**View Complete Source Code:** -- [local_runner.rs](https://github.com/logos-blockchain/logos-blockchain-testing/blob/master/examples/src/bin/local_runner.rs) — Host processes (local) -- [compose_runner.rs](https://github.com/logos-blockchain/logos-blockchain-testing/blob/master/examples/src/bin/compose_runner.rs) — Docker Compose -- [k8s_runner.rs](https://github.com/logos-blockchain/logos-blockchain-testing/blob/master/examples/src/bin/k8s_runner.rs) — Kubernetes - -**Runnable examples:** The repo includes complete binaries in `examples/src/bin/`: -- `local_runner.rs` — Host processes (local) -- `compose_runner.rs` — Docker Compose (requires image built) -- `k8s_runner.rs` — Kubernetes (requires cluster access and image loaded) - -**Recommended:** Use `scripts/run/run-examples.sh -t -n ` where mode is `host`, `compose`, or `k8s`. - -**Alternative:** Direct cargo run: `cargo run -p runner-examples --bin ` - -**Code patterns** below show how to build scenarios. Wrap these in `#[tokio::test]` functions for integration tests, or `#[tokio::main]` for binaries. - -## Simple consensus liveness - -Minimal test that validates basic block production: - -```rust,ignore -use std::time::Duration; - -use anyhow::Result; -use testing_framework_core::scenario::{Deployer, ScenarioBuilder}; -use testing_framework_runner_local::LocalDeployer; -use testing_framework_workflows::ScenarioBuilderExt; - -pub async fn simple_consensus() -> Result<()> { - let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().nodes(3)) - .expect_consensus_liveness() - .with_run_duration(Duration::from_secs(30)) - .build(); - - let deployer = LocalDeployer::default(); - let runner = deployer.deploy(&plan).await?; - let _handle = runner.run(&mut plan).await?; - - Ok(()) -} -``` - -**When to use**: smoke tests for consensus on minimal hardware. - -## Transaction workload - -Test consensus under transaction load: - -```rust,ignore -use std::time::Duration; - -use anyhow::Result; -use testing_framework_core::scenario::{Deployer, ScenarioBuilder}; -use testing_framework_runner_local::LocalDeployer; -use testing_framework_workflows::ScenarioBuilderExt; - -pub async fn transaction_workload() -> Result<()> { - let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().nodes(2)) - .wallets(20) - .transactions_with(|txs| txs.rate(5).users(10)) - .expect_consensus_liveness() - .with_run_duration(Duration::from_secs(60)) - .build(); - - let deployer = LocalDeployer::default(); - let runner = deployer.deploy(&plan).await?; - let _handle = runner.run(&mut plan).await?; - - Ok(()) -} -``` - -**When to use**: validate transaction submission and inclusion. - -## Chaos resilience - -Test system resilience under node restarts: - -```rust,ignore -use std::time::Duration; - -use anyhow::Result; -use testing_framework_core::scenario::{Deployer, ScenarioBuilder}; -use testing_framework_runner_compose::ComposeDeployer; -use testing_framework_workflows::{ChaosBuilderExt, ScenarioBuilderExt}; - -pub async fn chaos_resilience() -> Result<()> { - let mut plan = ScenarioBuilder::topology_with(|t| t.network_star().nodes(4)) - .enable_node_control() - .wallets(20) - .transactions_with(|txs| txs.rate(3).users(10)) - .chaos_with(|c| { - c.restart() - .min_delay(Duration::from_secs(20)) - .max_delay(Duration::from_secs(40)) - .target_cooldown(Duration::from_secs(30)) - .apply() - }) - .expect_consensus_liveness() - .with_run_duration(Duration::from_secs(120)) - .build(); - - let deployer = ComposeDeployer::default(); - let runner = deployer.deploy(&plan).await?; - let _handle = runner.run(&mut plan).await?; - - Ok(()) -} -``` - -**When to use**: resilience validation and operational readiness drills. - -**Note**: Chaos tests require `ComposeDeployer` or another runner with node control support. diff --git a/book/src/expectations.md b/book/src/expectations.md new file mode 100644 index 0000000..7af7951 --- /dev/null +++ b/book/src/expectations.md @@ -0,0 +1,171 @@ +# Expectations and Evaluation + +Expectations define success conditions. They can capture state before workloads start, check invariants while traffic runs, and evaluate the final state after the run settles. + +--- + +## The Expectation Trait + +`Expectation` lives in `testing-framework/core/src/scenario/expectation.rs`: + +```rust,ignore +use async_trait::async_trait; +use testing_framework_core::scenario::{DynError, Expectation, RunContext}; + +#[async_trait] +pub trait Expectation: Send + Sync { + fn name(&self) -> &str; + + fn init( + &mut self, + _descriptors: &E::Deployment, + _run_metrics: &RunMetrics, + ) -> Result<(), DynError> { + Ok(()) + } + + async fn start_capture(&mut self, _ctx: &RunContext) -> Result<(), DynError> { + Ok(()) + } + + /// Optional periodic check used by fail-fast expectation mode. + async fn check_during_capture(&mut self, _ctx: &RunContext) -> Result<(), DynError> { + Ok(()) + } + + async fn evaluate(&mut self, ctx: &RunContext) -> Result<(), DynError>; +} +``` + +The trait methods are: +- **`init`** runs at `build()` time with the resolved deployment and run metrics; a failure aborts the build. +- **`start_capture`** runs once per expectation *before any workload starts*. Use it to record a baseline (initial counters, starting state). A failure here is `ScenarioError::ExpectationCapture` and stops the run before traffic begins. +- **`check_during_capture`** is a fail-fast hook. The runner calls it on every expectation roughly once per second for the whole workload window (and the cooldown window). The default is a no-op, so existing end-of-run expectations are unaffected. The first check that returns `Err` aborts the run immediately with `ScenarioError::ExpectationFailedDuringCapture`. Use it for invariants that must hold throughout the run. +- **`evaluate`** checks the final condition after the run settles. It takes `&mut self`, so it can consume state accumulated during capture. + +--- + +## Registration + +Two paths feed the scenario's expectation list: + +1. **Explicit**: `.with_expectation(exp)` or `.with_expectation_boxed(boxed)` on any builder. +2. **Workload-attached**: when you call `.with_workload(w)`, the builder also collects `w.expectations()` (see [Workloads and Concurrency](workloads.md)). The default implementation returns none. + +Both end up in the same list and are treated identically at run time. + +Workload-attached expectations let a workload register the checks associated with its own traffic. Adding the workload also adds those checks. + +--- + +## Evaluation and Failure Aggregation + +```mermaid +flowchart LR + SC[start_capture]:::sc --> W[Workload window+ periodic checks]:::sc + W --> CD[Cooldown + settle]:::sc + CD --> EV[evaluate all]:::sc + EV --> R{failures?} + R -->|no| OK[run passes] + R -->|yes| AGG[aggregated report] + + classDef sc stroke:#9b6dd6,stroke-width:2.5px; +``` + +At the end of the run the runner evaluates **every** registered expectation, even after failures. Each failure is recorded as `name: error`, and the results are joined into a single `ScenarioError::Expectations` report: + +```text +expectations failed: +kv_converges: kv convergence not reached within 20s for 20 keys +openraft_kv_converges: timed out waiting for observed replicated state convergence ... +``` + +This is different from workload failures and capture-check failures, which abort immediately. + +--- + +## Cooldown: with_expectation_cooldown + +Workload traffic may need time to settle before evaluation: replication lags, queues drain, and restarted nodes rejoin. The builder exposes: + +```rust,ignore +.with_expectation_cooldown(Duration::from_secs(20)) +``` + +Verified behavior (`runner.rs` and `definition/validation.rs`): + +- If you never call it, the cooldown defaults to **10 seconds**. (`build()` also enforces a minimum run duration of 10 seconds.) +- After the workload window, the runner keeps the run alive for the cooldown window, still joining unfinished workloads and still running `check_during_capture` ticks. +- When the framework owns the node lifecycle (managed clusters), the cooldown window is raised to a **minimum of 30 seconds** so restarted or freshly deployed nodes stabilize. +- Before calling `evaluate`, the runner additionally sleeps a short settle wait derived from the same setting (at least 2 seconds when a cooldown is configured or node control is active) so runtime extensions such as [observers](observation.md) catch up. + +Set the cooldown to zero only for scenarios without managed nodes where staleness cannot matter. + +--- + +## Worked Example: Convergence Checks + +The kvstore example's `KvConverges` (`examples/kvstore/testing/workloads/src/expectations.rs`) is a plain polling expectation. It only implements `evaluate` and does its own retry loop against the node clients: + +```rust,ignore +use async_trait::async_trait; +use kvstore_runtime_ext::KvEnv; +use testing_framework_core::scenario::{DynError, Expectation, RunContext}; + +#[async_trait] +impl Expectation for KvConverges { + fn name(&self) -> &str { + "kv_converges" + } + + async fn evaluate(&mut self, ctx: &RunContext) -> Result<(), DynError> { + let clients = ctx.node_clients().snapshot(); + if clients.is_empty() { + return Err("no kv node clients available".into()); + } + + let deadline = tokio::time::Instant::now() + self.timeout; + while tokio::time::Instant::now() < deadline { + if self.is_converged(&clients).await? { + return Ok(()); + } + tokio::time::sleep(self.poll_interval).await; + } + + Err(format!( + "kv convergence not reached within {:?} for {} keys", + self.timeout, self.key_count + ) + .into()) + } +} +``` + +The example follows two conventions: + +- **Poll with a deadline inside `evaluate`.** Eventual consistency is the common case; a one-shot read makes flaky tests. +- **Make the error message carry the diagnosis.** State what was expected, how long you waited, and (where available) what was last observed. + +The openraft_kv variant, `OpenRaftKvConverges` (`examples/openraft_kv/testing/workloads/src/convergence.rs`), reads the cluster observer registered as a runtime extension instead of querying nodes directly: + +```rust,ignore +async fn evaluate(&mut self, ctx: &RunContext) -> Result<(), DynError> { + let expected = expected_kv(&self.key_prefix, self.total_writes); + let observer = ctx.require_extension::>()?; + + wait_for_observed_replication(&observer, &expected, self.timeout).await?; + + Ok(()) +} +``` + +The observer polls every node in the background, and the expectation waits for a matching snapshot without maintaining its own client polling state. See [Continuous Observation](observation.md) for the mechanism. + +--- + +## See Also + +- [Workloads and Concurrency](workloads.md) — the traffic these checks judge +- [Continuous Observation](observation.md) — snapshot-based state for expectations +- [Runtime Extensions](runtime-extensions.md) — how extension handles reach `evaluate` +- [Telemetry and External Observability](telemetry.md) — asserting on Prometheus metrics diff --git a/book/src/extending.md b/book/src/extending.md deleted file mode 100644 index 547e584..0000000 --- a/book/src/extending.md +++ /dev/null @@ -1,360 +0,0 @@ -# Extending the Framework - -This guide shows how to extend the framework with custom workloads, expectations, runners, and topology helpers. Each section includes the trait outline and a minimal code example. - -## Adding a Workload - -**Steps:** -1. Implement `testing_framework_core::scenario::Workload` -2. Provide a name and any bundled expectations -3. Use `init` to derive inputs from topology/metrics; fail fast if prerequisites missing -4. Use `start` to drive async traffic using `RunContext` clients -5. Expose from `testing-framework/workflows` and optionally add a DSL helper - -**Trait outline:** - -```rust,ignore -use async_trait::async_trait; -use testing_framework_core::scenario::{ - DynError, Expectation, RunContext, RunMetrics, Workload, -}; -use testing_framework_core::topology::generation::GeneratedTopology; - -struct MyExpectation; - -#[async_trait] -impl Expectation for MyExpectation { - fn name(&self) -> &str { - "my_expectation" - } - - async fn evaluate(&mut self, _ctx: &RunContext) -> Result<(), DynError> { - Ok(()) - } -} - -pub struct MyWorkload { - // Configuration fields - target_rate: u64, -} - -impl MyWorkload { - pub fn new(target_rate: u64) -> Self { - Self { target_rate } - } -} - -#[async_trait] -impl Workload for MyWorkload { - fn name(&self) -> &str { - "my_workload" - } - - fn expectations(&self) -> Vec> { - // Return bundled expectations that should run with this workload - vec![Box::new(MyExpectation)] - } - - fn init( - &mut self, - topology: &GeneratedTopology, - _run_metrics: &RunMetrics, - ) -> Result<(), DynError> { - // Validate prerequisites (e.g., enough nodes, wallet data present) - if topology.nodes().is_empty() { - return Err("no nodes available".into()); - } - Ok(()) - } - - async fn start(&self, ctx: &RunContext) -> Result<(), DynError> { - // Drive async activity: submit transactions, query nodes, etc. - let clients = ctx.node_clients().node_clients(); - - for client in clients { - let info = client.consensus_info().await?; - tracing::info!(height = info.height, "workload queried node"); - } - - Ok(()) - } -} -``` - -**Key points:** -- `name()` identifies the workload in logs -- `expectations()` bundles default checks (can be empty) -- `init()` validates topology before run starts -- `start()` executes concurrently with other workloads; it should complete before run duration expires - -See [Example: New Workload & Expectation](custom-workload-example.md) for a complete, runnable example. - -## Adding an Expectation - -**Steps:** -1. Implement `testing_framework_core::scenario::Expectation` -2. Use `start_capture` to snapshot baseline metrics (optional) -3. Use `evaluate` to assert outcomes after workloads finish -4. Return descriptive errors; the runner aggregates them -5. Export from `testing-framework/workflows` if reusable - -**Trait outline:** - -```rust,ignore -use async_trait::async_trait; -use testing_framework_core::scenario::{DynError, Expectation, RunContext}; - -pub struct MyExpectation { - expected_value: u64, - captured_baseline: Option, -} - -impl MyExpectation { - pub fn new(expected_value: u64) -> Self { - Self { - expected_value, - captured_baseline: None, - } - } -} - -#[async_trait] -impl Expectation for MyExpectation { - fn name(&self) -> &str { - "my_expectation" - } - - async fn start_capture(&mut self, ctx: &RunContext) -> Result<(), DynError> { - // Optional: capture baseline state before workloads start - let client = ctx.node_clients().node_clients().first() - .ok_or("no nodes")?; - - let info = client.consensus_info().await?; - self.captured_baseline = Some(info.height); - - tracing::info!(baseline = self.captured_baseline, "captured baseline"); - Ok(()) - } - - async fn evaluate(&mut self, ctx: &RunContext) -> Result<(), DynError> { - // Assert the expected condition holds after workloads finish - let client = ctx.node_clients().node_clients().first() - .ok_or("no nodes")?; - - let info = client.consensus_info().await?; - let final_height = info.height; - - let baseline = self.captured_baseline.unwrap_or(0); - let delta = final_height.saturating_sub(baseline); - - if delta < self.expected_value { - return Err(format!( - "expected at least {} blocks, got {}", - self.expected_value, delta - ).into()); - } - - tracing::info!(delta, "expectation passed"); - Ok(()) - } -} -``` - -**Key points:** -- `name()` identifies the expectation in logs -- `start_capture()` runs before workloads start (optional) -- `evaluate()` runs after workloads finish; return descriptive errors -- Expectations run sequentially; keep them fast - -## Adding a Runner (Deployer) - -**Steps:** -1. Implement `testing_framework_core::scenario::Deployer` for your capability type -2. Deploy infrastructure and return a `Runner` -3. Construct `NodeClients` and spawn a `BlockFeed` -4. Build a `RunContext` and provide a `CleanupGuard` for teardown - -**Trait outline:** - -```rust,ignore -use async_trait::async_trait; -use testing_framework_core::scenario::{ - CleanupGuard, Deployer, DynError, Metrics, NodeClients, RunContext, Runner, Scenario, - spawn_block_feed, -}; -use testing_framework_core::topology::deployment::Topology; - -pub struct MyDeployer { - // Configuration: cluster connection details, etc. -} - -impl MyDeployer { - pub fn new() -> Self { - Self {} - } -} - -#[async_trait] -impl Deployer<()> for MyDeployer { - type Error = DynError; - - async fn deploy(&self, scenario: &Scenario<()>) -> Result { - // 1. Launch nodes using scenario.topology() - // 2. Wait for readiness (e.g., consensus info endpoint responds) - // 3. Build NodeClients for nodes - // 4. Spawn a block feed for expectations (optional but recommended) - // 5. Create NodeControlHandle if you support restarts (optional) - // 6. Return a Runner wrapping RunContext + CleanupGuard - - tracing::info!("deploying scenario with MyDeployer"); - - let topology: Option = None; // Some(topology) if you spawned one - let node_clients = NodeClients::default(); // Or NodeClients::from_topology(...) - - let client = node_clients - .any_client() - .ok_or("no api clients available")? - .clone(); - let (block_feed, block_feed_guard) = spawn_block_feed(client).await?; - - let telemetry = Metrics::empty(); // or Metrics::from_prometheus(...) - let node_control = None; // or Some(Arc) - - let context = RunContext::new( - scenario.topology().clone(), - topology, - node_clients, - scenario.duration(), - telemetry, - block_feed, - node_control, - ); - - // If you also have other resources to clean up (containers/pods/etc), - // wrap them in your own CleanupGuard implementation and call - // CleanupGuard::cleanup(Box::new(block_feed_guard)) inside it. - Ok(Runner::new(context, Some(Box::new(block_feed_guard)))) - } -} -``` - -**Key points:** -- `deploy()` must return a fully prepared `Runner` -- Block until nodes are ready before returning (avoid false negatives) -- Use a `CleanupGuard` to tear down resources on failure (and on `RunHandle` drop) -- If you want chaos workloads, also provide a `NodeControlHandle` via `RunContext` - -## Adding Topology Helpers - -**Steps:** -1. Extend `testing_framework_core::topology::config::TopologyBuilder` with new layouts -2. Keep defaults safe: ensure at least one participant, clamp dispersal factors -3. Consider adding configuration presets for specialized parameters - -**Example:** - -```rust,ignore -use testing_framework_core::topology::{ - config::TopologyBuilder, - configs::network::Libp2pNetworkLayout, -}; - -pub trait TopologyBuilderExt { - fn network_full(self) -> Self; -} - -impl TopologyBuilderExt for TopologyBuilder { - fn network_full(self) -> Self { - self.with_network_layout(Libp2pNetworkLayout::Full) - } -} -``` - -**Key points:** -- Maintain method chaining (return `&mut Self`) -- Validate inputs: clamp factors, enforce minimums -- Document assumptions (e.g., "requires at least 4 nodes") - -## Adding a DSL Helper - -To expose your custom workload through the high-level DSL, add a trait extension: - -```rust,ignore -use async_trait::async_trait; -use testing_framework_core::scenario::{DynError, RunContext, ScenarioBuilder, Workload}; - -#[derive(Default)] -pub struct MyWorkloadBuilder { - target_rate: u64, - some_option: bool, -} - -impl MyWorkloadBuilder { - pub const fn target_rate(mut self, target_rate: u64) -> Self { - self.target_rate = target_rate; - self - } - - pub const fn some_option(mut self, some_option: bool) -> Self { - self.some_option = some_option; - self - } - - pub const fn build(self) -> MyWorkload { - MyWorkload { - target_rate: self.target_rate, - some_option: self.some_option, - } - } -} - -pub struct MyWorkload { - target_rate: u64, - some_option: bool, -} - -#[async_trait] -impl Workload for MyWorkload { - fn name(&self) -> &str { - "my_workload" - } - - async fn start(&self, _ctx: &RunContext) -> Result<(), DynError> { - Ok(()) - } -} - -pub trait MyWorkloadDsl { - fn my_workload_with( - self, - f: impl FnOnce(MyWorkloadBuilder) -> MyWorkloadBuilder, - ) -> Self; -} - -impl MyWorkloadDsl for ScenarioBuilder { - fn my_workload_with( - self, - f: impl FnOnce(MyWorkloadBuilder) -> MyWorkloadBuilder, - ) -> Self { - let builder = f(MyWorkloadBuilder::default()); - self.with_workload(builder.build()) - } -} -``` - -Users can then call: - -```rust,ignore -ScenarioBuilder::topology_with(|t| t.network_star().nodes(1)) - .my_workload_with(|w| { - w.target_rate(10) - .some_option(true) - }) - .build() -``` - -## See Also - -- [API Levels: Builder DSL vs. Direct](api-levels.md) - Understanding the two API levels -- [Custom Workload Example](custom-workload-example.md) - Complete runnable example -- [Internal Crate Reference](internal-crate-reference.md) - Where to add new code diff --git a/book/src/extension-points.md b/book/src/extension-points.md new file mode 100644 index 0000000..f5a3daf --- /dev/null +++ b/book/src/extension-points.md @@ -0,0 +1,202 @@ +# Public Extension Points + +This chapter lists every trait you implement to plug your application into the framework. + +The framework never imports your application. It defines public traits that your integration crate implements and calls them at defined points in the run lifecycle. Each entry below links to the corresponding chapter. + +| Trait | Defined in | You implement it to... | Taught in | +|---|---|---|---| +| `Application` | `testing-framework-core` (`env.rs`) | Bundle your deployment, client, and config types | [Implementing Application](implementing-application.md) | +| `DeploymentProvider` | `testing-framework-core` (`topology`) | Build a deployment plan, optionally from a seed | [Topology and Deployment Plans](topology.md) | +| `Workload` | `testing-framework-core` (`scenario`) | Drive traffic against the running system | [Workloads and Concurrency](workloads.md) | +| `Expectation` | `testing-framework-core` (`scenario`) | Define what success means | [Expectations and Evaluation](expectations.md) | +| `RuntimeExtensionFactory` | `testing-framework-core` (`scenario`) | Prepare a shared runtime value before workloads start | [Runtime Extensions](runtime-extensions.md) | +| `Observer` | `testing-framework-core` (`observation`) | Continuously materialize app state | [Continuous Observation](observation.md) | +| `SourceProvider` | `testing-framework-core` (`observation`) | Supply the current observation source set | [Continuous Observation](observation.md) | +| `SourceProviderFactory` | `testing-framework-core` (`observation`) | Build a source provider once node clients exist | [Continuous Observation](observation.md) | +| `AppDeployment` | `testing-framework-app` | Prepare one composable application preset | [AppDeployment and DeployContext](app-deployment.md) | +| `Deployer` | `testing-framework-core` (`scenario::runtime`) | Provision a scenario into a target environment | [Part V](part-v.md) | +| `NodeControlHandle` | `testing-framework-core` (`scenario`) | Expose start/stop/restart of nodes at runtime | [Scenario Capabilities](capabilities.md) | +| `ClusterWaitHandle` | `testing-framework-core` (`scenario`) | Expose cluster readiness waits | [Scenario Capabilities](capabilities.md) | +| `ObservabilityCapabilityProvider` | `testing-framework-core` (`scenario`) | Surface telemetry endpoints from capability markers | [Telemetry and External Observability](telemetry.md) | +| `BinaryProvider` | `testing-framework-runner-local` (`binary`) | Resolve the node executable for local processes | [Binary Providers](binary-providers.md) | +| `DownloadProcessor` | `testing-framework-runner-local` (`binary`) | Turn a downloaded artifact into an executable | [Binary Providers](binary-providers.md) | +| `IntoExistingCluster` | `testing-framework-core` (`scenario::sources`) | Convert a value into an existing-cluster descriptor | [Existing and External Clusters](external-clusters.md) | + +--- + +## Environment and Topology + +**`Application`** is the root of every integration. It bundles the backend-specific types the scenario engine is generic over: a deployment descriptor, a node client, and a node config. The three methods have working defaults: override `external_node_client` to support external sources, `build_node_client` to support deployer-discovered nodes, and `node_readiness_path` when your health endpoint is not `/`. + +```rust,ignore +#[async_trait] +pub trait Application: Send + Sync + 'static { + type Deployment: DeploymentDescriptor + Clone + 'static; + type NodeClient: Clone + Send + Sync + 'static; + type NodeConfig: Clone + Send + Sync + 'static; + + fn external_node_client(source: &ExternalNodeSource) -> Result; + fn build_node_client(access: &NodeAccess) -> Result; + fn node_readiness_path() -> &'static str; // default "/" +} +``` + +Plugs in as the `E` type parameter of `ScenarioBuilder`, `Workload`, `Expectation`, and every deployer. + +**`DeploymentProvider`** builds the deployment descriptor a scenario runs against, optionally driven by a `DeploymentSeed` for reproducible generation. `ScenarioBuilder::new` accepts one; `ScenarioBuilder::with_deployment` wraps a fixed value in the built-in `FixedDeploymentProvider`. + +```rust,ignore +pub trait DeploymentProvider: Send + Sync { + fn build(&self, seed: Option<&DeploymentSeed>) -> Result; +} +``` + +--- + +## Scenario Behavior + +**`Workload`** describes an action sequence executed during the run. `start` receives the `RunContext` (node clients, extensions, run metrics) and runs concurrently with other workloads. A workload can bundle its own checks via `expectations()`. + +```rust,ignore +#[async_trait] +pub trait Workload: Send + Sync { + fn name(&self) -> &str; + fn expectations(&self) -> Vec>> { Vec::new() } + fn init(&mut self, descriptors: &E::Deployment, metrics: &RunMetrics) -> Result<(), DynError> { Ok(()) } + async fn start(&self, ctx: &RunContext) -> Result<(), DynError>; +} +``` + +Registered with `with_workload` / `with_workload_boxed` on the builder. + +**`Expectation`** defines a check evaluated during or after the run. `start_capture` records a baseline, `check_during_capture` is the optional fail-fast hook polled during the run, and `evaluate` delivers the verdict at the end. + +```rust,ignore +#[async_trait] +pub trait Expectation: Send + Sync { + fn name(&self) -> &str; + fn init(&mut self, descriptors: &E::Deployment, metrics: &RunMetrics) -> Result<(), DynError> { Ok(()) } + async fn start_capture(&mut self, ctx: &RunContext) -> Result<(), DynError> { Ok(()) } + async fn check_during_capture(&mut self, ctx: &RunContext) -> Result<(), DynError> { Ok(()) } + async fn evaluate(&mut self, ctx: &RunContext) -> Result<(), DynError>; +} +``` + +Registered with `with_expectation` / `with_expectation_boxed`. + +**`RuntimeExtensionFactory`** prepares one typed value after deployment (node clients are available) and before workloads start. The value is stored by `TypeId` and retrieved in workloads via `ctx.extension::()` / `ctx.require_extension::()`. Return `PreparedRuntimeExtension::new(value)`, `::with_cleanup(value, guard)`, or `::from_task(value, join_handle)` to tie a background task's lifetime to the run. + +```rust,ignore +#[async_trait] +pub trait RuntimeExtensionFactory: Send + Sync { + async fn prepare( + &self, + deployment: &E::Deployment, + node_clients: NodeClients, + ) -> Result; +} +``` + +Registered with `with_runtime_extension_factory`. Registering two factories that produce the same extension type fails at prepare time with `duplicate runtime extension type registered`. + +--- + +## Observation + +**`Observer`** owns the app-side logic of the continuous observation runtime: `init` builds retained state from the source set, `poll` advances it each cycle and emits delta events, `snapshot` renders the current view. The runtime handles scheduling, history, and error tracking. + +```rust,ignore +#[async_trait] +pub trait Observer: Send + Sync + 'static { + type Source: Clone + Send + Sync + 'static; + type State: Send + Sync + 'static; + type Snapshot: Clone + Send + Sync + 'static; + type Event: Clone + Send + Sync + 'static; + + async fn init(&self, sources: &[ObservedSource]) -> Result; + async fn poll(&self, sources: &[ObservedSource], state: &mut Self::State) + -> Result, DynError>; + fn snapshot(&self, state: &Self::State) -> Self::Snapshot; +} +``` + +**`SourceProvider`** returns the current source set before each cycle, which lets the observed population change mid-run. Use `StaticSourceProvider` for a fixed set. + +```rust,ignore +#[async_trait] +pub trait SourceProvider: Send + Sync + 'static { + async fn sources(&self) -> Result>, DynError>; +} +``` + +**`SourceProviderFactory`** builds the provider once node clients exist. Any `Fn(&E::Deployment, NodeClients) -> Result, DynError>` closure implements it. All three plug into a scenario through `ObservationExtensionFactory`, which is itself a `RuntimeExtensionFactory`; see `examples/openraft_kv/testing/integration/src/observation.rs` for a complete implementation. + +--- + +## Application Composition + +**`AppDeployment`** prepares one reusable application preset, such as a process, child cluster, or composed stack, and returns a typed access or control handle. Framework adapters register managed resource lifetime separately with scenario cleanup. `AppHandle` is blanket-implemented for any `Clone + Send + Sync + 'static` type. + +```rust,ignore +#[async_trait] +pub trait AppDeployment: Send + 'static +where + E: Application, +{ + type Handle: AppHandle; + async fn deploy(self, ctx: &mut DeployContext) -> Result; +} +``` + +Registered with `AppScenarioBuilderExt::with_app`, which wraps it in an `AppDeploymentFactory` (a `RuntimeExtensionFactory`). Compose children inside `deploy` via `ctx.deploy(...)` / `ctx.deploy_and_expose(...)`. See [Handle Ownership and Teardown](handles-teardown.md) for handle access and cleanup semantics. + +--- + +## Deployment Backends + +**`Deployer`** is the contract every backend implements: turn a built `Scenario` into a `Runner`. `ProcessDeployer` (local), `ComposeDeployer`, and `K8sDeployer` are the in-repo implementations; `Caps` carries capability markers such as `NodeControlCapability`. + +```rust,ignore +#[async_trait] +pub trait Deployer: Send + Sync { + type Error; + async fn deploy(&self, scenario: &Scenario) -> Result, Self::Error>; +} +``` + +**`NodeControlHandle`** is the deployer-agnostic control surface behind node-control scenarios: `start_node(_with)`, `stop_node`, `restart_node(_with)`, `wait_node_ready`, `node_client`, and `node_pid`. Every method has a default that returns a "not supported by this deployer" error, so backends implement only what they support. **`ClusterWaitHandle`** provides the cluster-wide `wait_network_ready` operation. Both are combined by `ManualClusterHandle` in `core::runtime::manual`, the interface behind [ManualCluster](manual-cluster.md). + +**`ObservabilityCapabilityProvider`** lets deployers read telemetry endpoints out of whatever capability marker a scenario was built with; it is implemented for `()`, `NodeControlCapability`, and `ObservabilityCapability`. You only implement it when defining a new capability marker type. + +--- + +## Local Binary Resolution + +**`BinaryProvider`** resolves the executable path for a locally spawned node process. Implementations return `Ok(None)` when valid but unable to resolve, which is how `FallbackBinaryProvider` chains providers. The default `resolve` caches per process by `cache_key`. + +```rust,ignore +pub trait BinaryProvider: Send + Sync { + fn try_resolve(&self) -> Result, BinaryProviderError>; + fn display(&self) -> String; + fn cache_key(&self) -> String; + // provided: resolve(), resolve_uncached() +} +``` + +Built-in implementations: `PathBinaryProvider`, `EnvBinaryProvider`, `BuildBinaryProvider`, `DownloadBinaryProvider`, `FallbackBinaryProvider`. **`DownloadProcessor`** post-processes a checksum-verified download (for example, unpacking an archive) into the executable; `DownloadProcessorFn` adapts a closure with a stable `cache_key` so changed preparation logic invalidates the cache. + +```rust,ignore +pub trait DownloadProcessor: Send + Sync { + fn process(&self, artifact: &Path, output: &Path) -> Result<(), DownloadProcessorError>; + fn cache_key(&self) -> &str; +} +``` + +--- + +## Attaching Sources + +**`IntoExistingCluster`** converts a value into the typed `ExistingCluster` descriptor accepted by `with_existing_cluster_from`. It is implemented for `ExistingCluster` and `&ExistingCluster`; implement it for your own environment-selection types to keep attach logic in one place. External endpoints use `ExternalNodeSource` values directly and pair with `Application::external_node_client`. + +The required extension points depend on the entry pattern: a uniform managed cluster needs `Application` and the scenario traits, an AppHost stack adds `AppDeployment`, and attached clusters add the source traits. See [Choosing an Entry Pattern](entry-patterns.md). For where each implementation should live, see [Framework vs Application Boundaries](tf-boundaries.md) and the crate-level view in [Crate and API Map](crate-map.md). diff --git a/book/src/external-clusters.md b/book/src/external-clusters.md new file mode 100644 index 0000000..550c146 --- /dev/null +++ b/book/src/external-clusters.md @@ -0,0 +1,122 @@ +# Existing and External Clusters + +Scenarios can run against nodes the framework did not deploy: an attached existing cluster, standalone external endpoints, or a mix. + +Every scenario draws its node clients from three source classes: **managed** nodes the deployer spawns, **attached** nodes discovered in an existing cluster, and **external** nodes named by static endpoints. The builder records which sources you want; the deployer resolves them into one `NodeClients` inventory at deploy time. + +--- + +## The Source Model + +The source model uses these types from `testing-framework-core`: + +| Type | Shape | +|---|---| +| `ExistingCluster` | Typed descriptor of a cluster to attach to — a k8s label selector (optionally namespaced) or a compose project (optionally with explicit services) | +| `IntoExistingCluster` | Conversion trait; implemented by `ExistingCluster` itself and by deployer metadata types | +| `ExternalNodeSource` | A label plus an endpoint string, e.g. `http://10.0.0.5:8080` | +| `ClusterMode` | `Managed`, `ExistingCluster`, or `ExternalOnly` | +| `ClusterControlProfile` | `FrameworkManaged`, `ExistingClusterAttached`, `ExternalUncontrolled`, `ManualControlled` | + +`ExistingCluster` is constructed with `for_k8s_selector(selector)`, `for_k8s_selector_in_namespace(namespace, selector)`, `for_compose_project(project)`, or `for_compose_services(project, services)`. `ExternalNodeSource::new(label, endpoint)` wraps a plain endpoint string. + +The mode is derived, not set: a scenario with only a topology is `Managed`; adding an existing cluster makes it `ExistingCluster`; `with_external_only` makes it `ExternalOnly`. Invalid combinations (managed **and** attached at once) are unrepresentable. Each mode maps to a `ClusterControlProfile`, which workloads can consult to know whether the framework owns node lifecycles (`framework_owns_lifecycle()` is true only for `FrameworkManaged`). + +--- + +## Builder Methods + +```rust,ignore +use testing_framework_core::scenario::{ExistingCluster, ExternalNodeSource}; + +// Attach to a running compose project instead of deploying nodes. +let scenario = KvScenarioBuilder::deployment_with(|_| KvTopology::new(3)) + .with_existing_cluster(ExistingCluster::for_compose_project("compose-stack-1234".into())) + .with_workload(KvWriteWorkload::new().operations(100)) + .build()?; + +// Add a standalone external endpoint alongside managed nodes. +let scenario = KvScenarioBuilder::deployment_with(|_| KvTopology::new(2)) + .with_external_node(ExternalNodeSource::new( + "staging-gateway".into(), + "http://staging.example.net:8080".into(), + )) + .build()?; +``` + +| Method | Effect | +|---|---| +| `with_existing_cluster(cluster)` | Switch to existing-cluster mode with this descriptor | +| `with_existing_cluster_from(value)` | Same, converting through `IntoExistingCluster` (fallible) | +| `with_attach_source(attach)` | Alias for `with_existing_cluster` | +| `with_external_node(node)` | Add one external endpoint to the current mode | +| `with_external_nodes(nodes)` | Add several | +| `with_external_only()` | Drop the managed topology; keep only external nodes | +| `with_external_only_nodes(nodes)` | `with_external_nodes` + `with_external_only` in one call | + +External nodes compose with every mode: managed + external and attached + external are both valid hybrids. + +--- + +## From Source to Typed Client + +External and attached sources become typed clients through one hook on the `Application` trait: + +```rust,ignore +fn external_node_client(source: &ExternalNodeSource) -> Result; +``` + +The default implementation errors with "external node sources are not supported"; an application opts in by parsing `source.endpoint()` and constructing its client. The local deployer additionally falls back to a generic parser that resolves `http://host:port` endpoints and builds the client from the socket address when the app has not overridden the hook. + +--- + +## Resolution at Runtime + +At deploy time the scenario's sources become a `SourceOrchestrationPlan`, and each deployer supplies a `SourceProviders` set: a managed provider (the clients it just deployed), an attach provider, and an external provider. `orchestrate_sources_with_providers` resolves the plan: + +```mermaid +flowchart LR + P[SourceOrchestrationPlan] --> M[managed providerdeployer-spawned clients] + P --> A[attach providerdiscover existing cluster] + P --> X[external providerexternal_node_client] + M --> N[NodeClients] + A --> N + X --> N +``` + +The final inventory is ordered managed, then attached, then external. Managed mode with zero managed nodes is rejected; existing-cluster and external-only modes require at least one resolved client overall. + +**Per-deployer attach support:** + +- **Local**: no attach. `ProcessDeployer` rejects `ClusterMode::ExistingCluster` outright; external nodes are supported. +- **Compose**: requires a compose descriptor. Services are taken from the descriptor or discovered from the running project; each container's labeled API port is inspected and turned into an `ExternalNodeSource` fed to `external_node_client`. Attached mode also wires restart/stop node control. See [Compose Deployer](deployer-compose.md). +- **K8s**: requires a k8s descriptor. Services matching the label selector are listed in the namespace (default `default`); each service's single TCP NodePort (preferring ports named `http` or `api`) becomes the endpoint. See [Kubernetes Deployer](deployer-k8s.md). + +--- + +## Deploy-Then-Attach + +Both container deployers return metadata that converts back into an attach descriptor, so one process can deploy a stack and a second scenario can attach to it: + +```rust,ignore +let (runner, metadata) = ComposeDeployer::::new() + .deploy_with_metadata(&scenario) + .await?; + +// Later, or elsewhere: attach to the same project. +let attached = KvScenarioBuilder::deployment_with(|_| KvTopology::new(3)) + .with_existing_cluster_from(&metadata)? + .build()?; +``` + +`K8sDeployer::deploy_with_metadata` provides the equivalent `K8sDeploymentMetadata` (namespace + label selector). + +--- + +## Use Cases + +- **Staging and live networks.** Point `with_external_only_nodes` at long-lived endpoints and run workloads and expectations against them; the framework never touches their lifecycle (`ExternalUncontrolled`). +- **Shared test stacks.** Deploy a compose or k8s stack once, attach many fast scenarios to it, and preserve the stack between runs with the deployer preserve env vars (see [Readiness, Retry, and Artifact Preservation](deployment-policies.md)). +- **Hybrid scenarios.** Combine managed nodes with an external dependency, for example a locally deployed cluster that must interoperate with a fixed remote peer. + +Manual clusters have their own external hooks: `add_external_sources` and `add_external_clients` on `ManualCluster` merge external endpoints into an imperatively driven cluster (see [ManualCluster](manual-cluster.md)). diff --git a/book/src/faq.md b/book/src/faq.md deleted file mode 100644 index 7986074..0000000 --- a/book/src/faq.md +++ /dev/null @@ -1,32 +0,0 @@ -# FAQ - -**Why block-oriented timing?** -Slots advance at a fixed rate (NTP-synchronized, 2s by default), so reasoning -about blocks and consensus intervals keeps assertions aligned with protocol -behavior rather than arbitrary wall-clock durations. - -**Can I reuse the same scenario across runners?** -Yes. The plan stays the same; swap runners (local, compose, k8s) to target -different environments. - -**When should I enable chaos workloads?** -Only when testing resilience or operational recovery; keep functional smoke -tests deterministic. - -**How long should runs be?** -The framework enforces a minimum of **2× slot duration** (4 seconds with default 2s slots), but practical recommendations: - -- **Smoke tests**: 30s minimum (~14 blocks with default 2s slots, 0.9 coefficient) -- **Transaction workloads**: 60s+ (~27 blocks) to observe inclusion patterns -- **Chaos tests**: 120s+ (~54 blocks) to allow recovery after restarts - -Very short runs (< 30s) risk false confidence—one or two lucky blocks don't prove liveness. - -**Do I always need seeded wallets?** -Only for transaction scenarios. Pure chaos scenarios may not require them, but -liveness checks still need nodes producing blocks. - -**What if expectations fail but workloads “look fine”?** -Trust expectations first—they capture the intended success criteria. Use the -observability signals and runner logs to pinpoint why the system missed the -target. diff --git a/book/src/framework-in-brief.md b/book/src/framework-in-brief.md new file mode 100644 index 0000000..75c9b06 --- /dev/null +++ b/book/src/framework-in-brief.md @@ -0,0 +1,971 @@ + + +# The Framework in Brief + + +