logos-blockchain-testing/book/src/running-examples.md

# Running Examples

The framework provides three runner modes: **host** (local processes), **compose** (Docker Compose), and **k8s** (Kubernetes).

## Quick Start (Recommended)

Use `scripts/run/run-examples.sh` for all modes—it handles all setup automatically:

```bash
# Host mode (local processes)
scripts/run/run-examples.sh -t 60 -v 3 -e 1 host

# Compose mode (Docker Compose)
scripts/run/run-examples.sh -t 60 -v 3 -e 1 compose

# K8s mode (Kubernetes)
scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s
```

**Parameters:**
- `-t 60` — Run duration in seconds
- `-v 3` — Number of validators
- `-e 1` — Number of executors
- `host|compose|k8s` — Deployment mode

This script handles:
- Circuit asset setup
- Binary building/bundling
- Image building (compose/k8s)
- Image loading into cluster (k8s)
- Execution with proper environment

**Note:** For `k8s` runs against non-local clusters (e.g. EKS), the cluster pulls images from a registry. In that case, build + push your image separately (see `scripts/build/build_test_image.sh`) and set `NOMOS_TESTNET_IMAGE` to the pushed reference.

## Quick Smoke Matrix

For a small "does everything still run?" matrix across all runners:

```bash
scripts/run/run-test-matrix.sh -t 120 -v 1 -e 1
```

This runs host, compose, and k8s modes with various image-build configurations. Useful after making runner/image/script changes. Forwards `--metrics-*` options through to `scripts/run/run-examples.sh`.

**Environment overrides:**
- `VERSION=v0.3.1` — Circuit version
- `NOMOS_NODE_REV=<commit>` — nomos-node git revision
- `NOMOS_BINARIES_TAR=path/to/bundle.tar.gz` — Use prebuilt bundle
- `NOMOS_SKIP_IMAGE_BUILD=1` — Skip image rebuild (compose/k8s)
- `NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64|linux/amd64` — Docker platform for bundle builds (macOS/Windows)
- `COMPOSE_CIRCUITS_PLATFORM=linux-aarch64|linux-x86_64` — Circuits platform for image builds
- `SLOW_TEST_ENV=true` — Doubles built-in readiness timeouts (useful in CI / constrained laptops)
- `TESTNET_PRINT_ENDPOINTS=1` — Print `TESTNET_ENDPOINTS` / `TESTNET_PPROF` lines during deploy

## Dev Workflow: Updating nomos-node Revision

The repo pins a `nomos-node` revision in `versions.env` for reproducible builds. To update it or point to a local checkout:

```bash
# Pin to a new git revision (updates versions.env + Cargo.toml git revs)
scripts/ops/update-nomos-rev.sh --rev <git_sha>

# Use a local nomos-node checkout instead (for development)
scripts/ops/update-nomos-rev.sh --path /path/to/nomos-node

# If Cargo.toml was marked skip-worktree, clear it
scripts/ops/update-nomos-rev.sh --unskip-worktree
```

**Notes:**
- Don't commit absolute `NOMOS_NODE_PATH` values; prefer `--rev` for shared history/CI
- After changing rev/path, expect `Cargo.lock` to update on the next `cargo build`/`cargo test`

## Cleanup Helper

If you hit Docker build failures, I/O errors, or disk space issues:

```bash
scripts/ops/clean.sh
```

For extra Docker cache cleanup:

```bash
scripts/ops/clean.sh --docker
```

---

## Host Runner (Direct Cargo Run)

For manual control, run the `local_runner` binary directly:

```bash
POL_PROOF_DEV_MODE=true \
NOMOS_NODE_BIN=/path/to/nomos-node \
NOMOS_EXECUTOR_BIN=/path/to/nomos-executor \
cargo run -p runner-examples --bin local_runner
```

### Host Runner Environment Variables

| Variable | Default | Effect |
|----------|---------|--------|
| `NOMOS_DEMO_VALIDATORS` | 1 | Number of validators (legacy: `LOCAL_DEMO_VALIDATORS`) |
| `NOMOS_DEMO_EXECUTORS` | 1 | Number of executors (legacy: `LOCAL_DEMO_EXECUTORS`) |
| `NOMOS_DEMO_RUN_SECS` | 60 | Run duration in seconds (legacy: `LOCAL_DEMO_RUN_SECS`) |
| `NOMOS_NODE_BIN` | — | Path to nomos-node binary (required) |
| `NOMOS_EXECUTOR_BIN` | — | Path to nomos-executor binary (required) |
| `NOMOS_LOG_DIR` | None | Directory for per-node log files |
| `NOMOS_TESTS_KEEP_LOGS` | 0 | Keep per-run temporary directories (useful for debugging/CI) |
| `NOMOS_TESTS_TRACING` | false | Enable debug tracing preset |
| `NOMOS_LOG_LEVEL` | info | Global log level: error, warn, info, debug, trace |
| `NOMOS_LOG_FILTER` | None | Fine-grained module filtering (e.g., `cryptarchia=trace,nomos_da_sampling=debug`) |
| `POL_PROOF_DEV_MODE` | — | **REQUIRED**: Set to `true` for all runners |

**Note:** Requires circuit assets and host binaries. Use `scripts/run/run-examples.sh host` to handle setup automatically.

---

## Compose Runner (Direct Cargo Run)

For manual control, run the `compose_runner` binary directly. Compose requires a Docker image with embedded assets.

### Option 1: Prebuilt Bundle (Recommended)

```bash
# 1. Build a Linux bundle (includes binaries + circuits)
scripts/build/build-bundle.sh --platform linux
# Creates .tmp/nomos-binaries-linux-v0.3.1.tar.gz

# 2. Build image (embeds bundle assets)
export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
scripts/build/build_test_image.sh

# 3. Run
NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin compose_runner
```

### Option 2: Manual Circuit/Image Setup

```bash
# Fetch and copy circuits
scripts/setup/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits
cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/

# Build image
scripts/build/build_test_image.sh

# Run
NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin compose_runner
```

### Platform Note (macOS / Apple Silicon)

- Docker Desktop runs a `linux/arm64` engine by default
- For native performance: `NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64` (recommended for local testing)
- For amd64 targets: `NOMOS_BUNDLE_DOCKER_PLATFORM=linux/amd64` (slower via emulation)

### Compose Runner Environment Variables

| Variable | Default | Effect |
|----------|---------|--------|
| `NOMOS_TESTNET_IMAGE` | — | Image tag (required, must match built image) |
| `POL_PROOF_DEV_MODE` | — | **REQUIRED**: Set to `true` for all runners |
| `NOMOS_DEMO_VALIDATORS` | 1 | Number of validators |
| `NOMOS_DEMO_EXECUTORS` | 1 | Number of executors |
| `NOMOS_DEMO_RUN_SECS` | 60 | Run duration in seconds |
| `COMPOSE_NODE_PAIRS` | — | Alternative topology format: "validators×executors" (e.g., `3x2`) |
| `NOMOS_METRICS_QUERY_URL` | None | Prometheus-compatible base URL for runner to query |
| `NOMOS_METRICS_OTLP_INGEST_URL` | None | Full OTLP HTTP ingest URL for node metrics export |
| `NOMOS_GRAFANA_URL` | None | Grafana base URL for printing/logging |
| `COMPOSE_RUNNER_HOST` | 127.0.0.1 | Host address for port mappings |
| `COMPOSE_RUNNER_PRESERVE` | 0 | Keep containers running after test |
| `NOMOS_LOG_LEVEL` | info | Node log level (stdout/stderr) |
| `NOMOS_LOG_FILTER` | None | Fine-grained module filtering |

**Config file option:** `testing-framework/assets/stack/cfgsync.yaml` (`tracing_settings.logger`) — Switch node logs between stdout/stderr and file output

### Compose-Specific Features

- **Node control support**: Only runner that supports chaos testing (`.enable_node_control()` + chaos workloads)
- **External observability**: Set `NOMOS_METRICS_*` / `NOMOS_GRAFANA_URL` to enable telemetry links and querying
  - Quickstart: `scripts/setup/setup-observability.sh compose up` then `scripts/setup/setup-observability.sh compose env`

**Important:**
- Containers expect KZG parameters at `/kzgrs_test_params/kzgrs_test_params` (note the repeated filename)
- Use `scripts/run/run-examples.sh compose` to handle all setup automatically

---

## K8s Runner (Direct Cargo Run)

For manual control, run the `k8s_runner` binary directly. K8s requires the same image setup as Compose.

### Prerequisites

1. **Kubernetes cluster** with `kubectl` configured
2. **Test image built** (same as Compose, preferably with prebuilt bundle)
3. **Image available in cluster** (loaded or pushed to registry)

### Build and Load Image

```bash
# 1. Build image with bundle (recommended)
scripts/build/build-bundle.sh --platform linux
export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
scripts/build/build_test_image.sh

# 2. Load into cluster (choose one)
export NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local

# For kind:
kind load docker-image logos-blockchain-testing:local

# For minikube:
minikube image load logos-blockchain-testing:local

# For remote cluster (push to registry):
docker tag logos-blockchain-testing:local your-registry/logos-blockchain-testing:latest
docker push your-registry/logos-blockchain-testing:latest
export NOMOS_TESTNET_IMAGE=your-registry/logos-blockchain-testing:latest
```

### Run the Example

```bash
export NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local
export POL_PROOF_DEV_MODE=true
cargo run -p runner-examples --bin k8s_runner
```

### K8s Runner Environment Variables

| Variable | Default | Effect |
|----------|---------|--------|
| `NOMOS_TESTNET_IMAGE` | — | Image tag (required) |
| `POL_PROOF_DEV_MODE` | — | **REQUIRED**: Set to `true` for all runners |
| `NOMOS_DEMO_VALIDATORS` | 1 | Number of validators |
| `NOMOS_DEMO_EXECUTORS` | 1 | Number of executors |
| `NOMOS_DEMO_RUN_SECS` | 60 | Run duration in seconds |
| `NOMOS_METRICS_QUERY_URL` | None | Prometheus-compatible base URL for runner to query (PromQL) |
| `NOMOS_METRICS_OTLP_INGEST_URL` | None | Full OTLP HTTP ingest URL for node metrics export |
| `NOMOS_GRAFANA_URL` | None | Grafana base URL for printing/logging |
| `K8S_RUNNER_NAMESPACE` | Random | Kubernetes namespace (pin for debugging) |
| `K8S_RUNNER_RELEASE` | Random | Helm release name (pin for debugging) |
| `K8S_RUNNER_NODE_HOST` | — | NodePort host resolution for non-local clusters |
| `K8S_RUNNER_DEBUG` | 0 | Log Helm stdout/stderr for install commands |
| `K8S_RUNNER_PRESERVE` | 0 | Keep namespace/release after run (for debugging) |

### K8s + Observability (Optional)

```bash
export NOMOS_METRICS_QUERY_URL=http://your-prometheus:9090
# Prometheus OTLP receiver example:
export NOMOS_METRICS_OTLP_INGEST_URL=http://your-prometheus:9090/api/v1/otlp/v1/metrics
# Optional: print Grafana link in TESTNET_ENDPOINTS
export NOMOS_GRAFANA_URL=http://your-grafana:3000
cargo run -p runner-examples --bin k8s_runner
```

**Notes:**
- `NOMOS_METRICS_QUERY_URL` must be reachable from the runner process (often via `kubectl port-forward`)
- `NOMOS_METRICS_OTLP_INGEST_URL` must be reachable from nodes (pods/containers) and is backend-specific
  - Quickstart installer: `scripts/setup/setup-observability.sh k8s install` then `scripts/setup/setup-observability.sh k8s env`
  - Optional dashboards: `scripts/setup/setup-observability.sh k8s dashboards`

### Via `scripts/run/run-examples.sh` (Recommended)

```bash
scripts/run/run-examples.sh -t 60 -v 3 -e 1 k8s \
  --metrics-query-url http://your-prometheus:9090 \
  --metrics-otlp-ingest-url http://your-prometheus:9090/api/v1/otlp/v1/metrics
```

### In Code (Optional)

```rust
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ObservabilityBuilderExt as _;

let plan = ScenarioBuilder::with_node_counts(1, 1)
    .with_metrics_query_url_str("http://your-prometheus:9090")
    .with_metrics_otlp_ingest_url_str("http://your-prometheus:9090/api/v1/otlp/v1/metrics")
    .build();
```

### Important K8s Notes

- K8s runner mounts `testing-framework/assets/stack/kzgrs_test_params` as a hostPath volume
- File path inside pods: `/kzgrs_test_params/kzgrs_test_params`
- **No node control support yet**: Chaos workloads (`.enable_node_control()`) will fail
- Optimized for local clusters (Docker Desktop K8s / minikube / kind)
  - Remote clusters require additional setup (registry push, PV/CSI for assets, etc.)
- Use `scripts/run/run-examples.sh k8s` to handle all setup automatically

## Next Steps

- [CI Integration](ci-integration.md) — Automate tests in continuous integration
- [Environment Variables](environment-variables.md) — Full variable reference
- [Logging & Observability](logging-observability.md) — Log collection and metrics
- [Troubleshooting](troubleshooting.md) — Common issues and fixes