Operations

Operational readiness focuses on prerequisites, environment fit, and clear signals:

• Prerequisites:
  • versions.env file at repository root (required by helper scripts; defines VERSION, NOMOS_NODE_REV, NOMOS_BUNDLE_VERSION)
  • Keep a sibling nomos-node checkout available, or use scripts/run-examples.sh which clones/builds on demand
  • Ensure the chosen runner's platform needs are met (Docker for compose, cluster access for k8s)
  • CI uses prebuilt binary artifacts from the build-binaries workflow
• Artifacts: DA scenarios require KZG parameters (circuit assets) located at testing-framework/assets/stack/kzgrs_test_params. Fetch them via scripts/setup-nomos-circuits.sh or override the path with NOMOS_KZGRS_PARAMS_PATH.
• Environment flags: POL_PROOF_DEV_MODE=true is required for all runners (local, compose, k8s) unless you want expensive Groth16 proof generation that will cause tests to timeout. Configure logging via NOMOS_LOG_DIR, NOMOS_LOG_LEVEL, and NOMOS_LOG_FILTER (see Logging and Observability for details). Note that nodes ignore RUST_LOG and only respond to NOMOS_* variables.
• Readiness checks: verify runners report node readiness before starting workloads; this avoids false negatives from starting too early.
• Failure triage: map failures to missing prerequisites (wallet seeding, node control availability), runner platform issues, or unmet expectations. Start with liveness signals, then dive into workload-specific assertions.

Treat operational hygiene—assets present, prerequisites satisfied, observability reachable—as the first step to reliable scenario outcomes.

Environment Variable Reference

This section consolidates all environment variables used by the framework, helper scripts, and node processes.

Core Requirements

Binary and Asset Paths

Docker and Image Configuration

Node Logging Configuration

Important: Node processes ignore RUST_LOG and only respond to NOMOS_* variables.

Framework Logging Configuration

Note: Runner binaries (not node processes) use RUST_LOG for framework orchestration logs.

Observability and Telemetry

Scenario Configuration

Kubernetes-Specific

Quick Reference: Common Scenarios

Running local smoke test:

POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner

Running with custom log directory:

POL_PROOF_DEV_MODE=true \
NOMOS_LOG_DIR=/tmp/my-test-logs \
NOMOS_TESTS_KEEP_LOGS=1 \
cargo run -p runner-examples --bin local_runner

Running compose with observability:

POL_PROOF_DEV_MODE=true \
NOMOS_METRICS_QUERY_URL=http://localhost:9090 \
NOMOS_GRAFANA_URL=http://localhost:3000 \
scripts/run-examples.sh -t 60 -v 2 -e 1 compose

Running with debug logging:

POL_PROOF_DEV_MODE=true \
NOMOS_LOG_LEVEL=debug \
NOMOS_LOG_FILTER="consensus=trace,da=debug" \
RUST_LOG=testing_framework=debug \
cargo run -p runner-examples --bin local_runner

Building for Apple Silicon:

NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64 scripts/build_test_image.sh

CI Usage

Both LocalDeployer and ComposeDeployer work in CI environments:

LocalDeployer in CI:

• Faster (no Docker overhead)
• Good for quick smoke tests
• Trade-off: Less isolation (processes share host)

ComposeDeployer in CI (recommended):

• Better isolation (containerized)
• Reproducible environment
• Can integrate with external Prometheus/Grafana (optional)
• Trade-off: Slower startup (Docker image build)
• Trade-off: Requires Docker daemon

Complete CI Integration Example

Here's a production-ready GitHub Actions workflow demonstrating host and compose runners with proper artifact caching, log collection, and Cucumber test integration:

name: Integration Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  CARGO_TERM_COLOR: always
  RUST_BACKTRACE: 1

jobs:
  # Host runner: Fast smoke tests
  test-host:
    name: Host Runner Tests (Smoke)
    runs-on: ubuntu-latest
    timeout-minutes: 15
    
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      
      - name: Install Rust nightly
        uses: dtolnay/rust-toolchain@nightly
        with:
          components: rustfmt, clippy
      
      - name: Cache Rust dependencies
        uses: Swatinem/rust-cache@v2
        with:
          key: host-runner-${{ hashFiles('**/Cargo.lock') }}
      
      - name: Cache binary bundle
        id: cache-bundle
        uses: actions/cache@v4
        with:
          path: .tmp/nomos-binaries-host-*.tar.gz
          key: nomos-binaries-host-${{ hashFiles('versions.env') }}
      
      - name: Build binaries (if not cached)
        if: steps.cache-bundle.outputs.cache-hit != 'true'
        run: |
          scripts/build-bundle.sh --platform host --output .tmp/bundle.tar.gz
          tar -xzf .tmp/bundle.tar.gz
          mkdir -p testing-framework/assets/stack/bin
          cp artifacts/nomos-node artifacts/nomos-executor testing-framework/assets/stack/bin/
      
      - name: Extract cached binaries
        if: steps.cache-bundle.outputs.cache-hit == 'true'
        run: |
          tar -xzf .tmp/nomos-binaries-host-*.tar.gz
          mkdir -p testing-framework/assets/stack/bin
          cp artifacts/nomos-node artifacts/nomos-executor testing-framework/assets/stack/bin/
      
      - name: Run host smoke tests
        env:
          POL_PROOF_DEV_MODE: true
          NOMOS_TESTS_KEEP_LOGS: 1
          NOMOS_LOG_DIR: ${{ github.workspace }}/.tmp/host-logs
        run: |
          cargo test -p runner-examples --bin local_runner -- --nocapture
      
      - name: Run Cucumber host tests
        env:
          POL_PROOF_DEV_MODE: true
        run: |
          cargo test -p runner-examples --bin cucumber_host -- --nocapture
      
      - name: Upload logs on failure
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: host-runner-logs
          path: .tmp/host-logs/
          retention-days: 7

  # Compose runner: Full integration tests
  test-compose:
    name: Compose Runner Tests
    runs-on: ubuntu-latest
    timeout-minutes: 30
    
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Install Rust nightly
        uses: dtolnay/rust-toolchain@nightly
      
      - name: Cache Rust dependencies
        uses: Swatinem/rust-cache@v2
        with:
          key: compose-runner-${{ hashFiles('**/Cargo.lock') }}
      
      - name: Cache Linux binary bundle
        id: cache-linux-bundle
        uses: actions/cache@v4
        with:
          path: .tmp/nomos-binaries-linux-*.tar.gz
          key: nomos-binaries-linux-${{ hashFiles('versions.env') }}
      
      - name: Build Linux binaries (if not cached)
        if: steps.cache-linux-bundle.outputs.cache-hit != 'true'
        run: |
          scripts/build-bundle.sh --platform linux --output .tmp/linux-bundle.tar.gz
      
      - name: Cache Docker image layers
        uses: actions/cache@v4
        with:
          path: /tmp/.buildx-cache
          key: docker-buildx-${{ hashFiles('versions.env', 'testing-framework/assets/stack/**') }}
          restore-keys: |
            docker-buildx-
      
      - name: Build Docker image
        env:
          NOMOS_BINARIES_TAR: .tmp/nomos-binaries-linux-*.tar.gz
        run: |
          scripts/build_test_image.sh
      
      - name: Run compose smoke tests
        env:
          POL_PROOF_DEV_MODE: true
        run: |
          cargo test -p runner-examples --bin compose_runner -- --nocapture
      
      - name: Run Cucumber compose tests
        env:
          POL_PROOF_DEV_MODE: true
        run: |
          cargo test -p runner-examples --bin cucumber_compose -- --nocapture
      
      - name: Collect container logs on failure
        if: failure()
        run: |
          mkdir -p .tmp/compose-logs
          for container in $(docker ps -a --filter "name=nomos-compose" --format "{{.Names}}"); do
            docker logs "$container" > ".tmp/compose-logs/${container}.log" 2>&1 || true
          done
      
      - name: Upload container logs
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: compose-runner-logs
          path: .tmp/compose-logs/
          retention-days: 7
      
      - name: Cleanup containers
        if: always()
        run: |
          docker compose down --volumes --remove-orphans || true

  # Combined status check (required for branch protection)
  integration-tests:
    name: Integration Tests Status
    runs-on: ubuntu-latest
    needs: [test-host, test-compose]
    if: always()
    
    steps:
      - name: Check all jobs succeeded
        run: |
          if [ "${{ needs.test-host.result }}" != "success" ] || \
             [ "${{ needs.test-compose.result }}" != "success" ]; then
            echo "One or more integration test jobs failed"
            exit 1
          fi

Key Features:

1. Matrix Strategy: Separate jobs for host and compose runners
2. Artifact Caching:
   • Rust dependencies cached per Cargo.lock
   • Binary bundles cached per versions.env
   • Docker layers cached for faster image builds
3. Log Collection: Uploads logs only on failure, with 7-day retention
4. Cucumber Integration: Runs both programmatic and BDD tests
5. Proper Cleanup: Ensures containers are removed even on failure
6. Status Check: Combined job for branch protection rules

Customization:

# Add matrix for multiple topologies:
strategy:
  matrix:
    topology:
      - { validators: 1, executors: 1, duration: 60 }
      - { validators: 2, executors: 1, duration: 90 }
      - { validators: 3, executors: 2, duration: 120 }

# Or add scheduled nightly stress tests:
on:
  schedule:
    - cron: '0 2 * * *'  # 2 AM daily

See also: .github/workflows/lint.yml for current CI examples.

Running Examples

The framework provides three runner modes: host (local processes), compose (Docker Compose), and k8s (Kubernetes).

Recommended: Use scripts/run-examples.sh for all modes:

# Host mode (local processes)
scripts/run-examples.sh -t 60 -v 1 -e 1 host

# Compose mode (Docker Compose)
scripts/run-examples.sh -t 60 -v 1 -e 1 compose

# K8s mode (Kubernetes)
scripts/run-examples.sh -t 60 -v 1 -e 1 k8s

This script handles circuit setup, binary building/bundling, (local) image building, and execution.

Note: for k8s runs against non-local clusters (e.g. EKS), the cluster pulls images from a registry, so a local docker build is not used. In that case, build + push your image separately (see scripts/build_test_image.sh) and set NOMOS_TESTNET_IMAGE to the pushed reference.

Quick Smoke Matrix (Host/Compose/K8s)

For a small “does everything still run?” matrix (including --no-image-build variants where relevant), use:

scripts/run-test-matrix.sh -t 120 -v 1 -e 1

This is useful after making runner/image/script changes, and it forwards --metrics-* options through to scripts/run-examples.sh.

Environment overrides:

• VERSION=v0.3.1 — Circuit version
• NOMOS_NODE_REV=<commit> — nomos-node git revision
• NOMOS_BINARIES_TAR=path/to/bundle.tar.gz — Use prebuilt bundle
• NOMOS_SKIP_IMAGE_BUILD=1 — Skip image rebuild (compose/k8s)
• NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64|linux/amd64 — Docker platform used when building a Linux bundle on non-Linux hosts (macOS/Windows)
• COMPOSE_CIRCUITS_PLATFORM=linux-aarch64|linux-x86_64 — Circuits platform used when building the compose/k8s image (defaults based on host arch)
• SLOW_TEST_ENV=true — Doubles built-in readiness timeouts (useful in slower CI / constrained laptops)
• TESTNET_PRINT_ENDPOINTS=1 — Print TESTNET_ENDPOINTS / TESTNET_PPROF lines during deploy (set automatically by scripts/run-examples.sh)
• COMPOSE_RUNNER_HTTP_TIMEOUT_SECS=<secs> — Override compose node HTTP readiness timeout
• K8S_RUNNER_DEPLOYMENT_TIMEOUT_SECS=<secs> — Override k8s deployment readiness timeout
• K8S_RUNNER_HTTP_TIMEOUT_SECS=<secs> — Override k8s HTTP readiness timeout for port-forwards
• K8S_RUNNER_HTTP_PROBE_TIMEOUT_SECS=<secs> — Override k8s HTTP readiness timeout for NodePort probes
• K8S_RUNNER_PROMETHEUS_HTTP_TIMEOUT_SECS=<secs> — Override k8s Prometheus readiness timeout
• K8S_RUNNER_PROMETHEUS_HTTP_PROBE_TIMEOUT_SECS=<secs> — Override k8s Prometheus NodePort probe timeout

Updating nomos-node Revision (Dev Workflow)

The repo pins a nomos-node revision in versions.env for reproducible builds. To update it (or point to a local checkout), use the helper script:

# Pin to a new git revision (updates versions.env + Cargo.toml git revs)
scripts/update-nomos-rev.sh --rev <git_sha>

# Use a local nomos-node checkout instead (for development)
scripts/update-nomos-rev.sh --path /path/to/nomos-node

# If Cargo.toml was marked skip-worktree, clear it
scripts/update-nomos-rev.sh --unskip-worktree

Notes:

• Don’t commit absolute NOMOS_NODE_PATH values; prefer --rev for shared history/CI.
• After changing rev/path, expect Cargo.lock to update on the next cargo build/cargo test.

Cleanup Helper

If you hit Docker build failures, mysterious I/O errors, or are running out of disk space:

scripts/clean.sh

For extra Docker cache cleanup:

scripts/clean.sh --docker

Host Runner (Direct Cargo Run)

For manual control, you can run the local_runner binary directly:

POL_PROOF_DEV_MODE=true \
NOMOS_NODE_BIN=/path/to/nomos-node \
NOMOS_EXECUTOR_BIN=/path/to/nomos-executor \
cargo run -p runner-examples --bin local_runner

Environment variables:

• NOMOS_DEMO_VALIDATORS=3 — Number of validators (default: 1, or use legacy LOCAL_DEMO_VALIDATORS)
• NOMOS_DEMO_EXECUTORS=2 — Number of executors (default: 1, or use legacy LOCAL_DEMO_EXECUTORS)
• NOMOS_DEMO_RUN_SECS=120 — Run duration in seconds (default: 60, or use legacy LOCAL_DEMO_RUN_SECS)
• NOMOS_NODE_BIN / NOMOS_EXECUTOR_BIN — Paths to binaries (required for direct run)
• NOMOS_LOG_DIR=/tmp/logs — Directory for per-node log files (host runner). For compose/k8s, set tracing_settings.logger: !File in testing-framework/assets/stack/cfgsync.yaml.
• NOMOS_TESTS_KEEP_LOGS=1 — Keep per-run temporary directories (useful for debugging/CI artifacts)
• NOMOS_TESTS_TRACING=true — Enable the debug tracing preset (optional; combine with NOMOS_LOG_DIR unless you have external tracing backends configured)
• NOMOS_LOG_LEVEL=debug — Set log level (default: info)
• NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug" — Fine-grained module filtering

Note: Requires circuit assets and host binaries. Use scripts/run-examples.sh host to handle setup automatically.

Compose Runner (Direct Cargo Run)

For manual control, you can run the compose_runner binary directly. Compose requires a Docker image with embedded assets.

Recommended setup: Use a prebuilt bundle:

# Build a Linux bundle (includes binaries + circuits)
scripts/build-bundle.sh --platform linux
# Creates .tmp/nomos-binaries-linux-v0.3.1.tar.gz

# Build image (embeds bundle assets)
export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
scripts/build_test_image.sh

# Run
NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin compose_runner

Platform note (macOS / Apple silicon):

• Docker Desktop runs a linux/arm64 engine. If Linux bundle builds are slow/unstable when producing .tmp/nomos-binaries-linux-*.tar.gz, prefer NOMOS_BUNDLE_DOCKER_PLATFORM=linux/arm64 for local compose/k8s runs.
• If you need amd64 images/binaries specifically (e.g., deploying to amd64-only environments), set NOMOS_BUNDLE_DOCKER_PLATFORM=linux/amd64 and expect slower builds via emulation.

Alternative: Manual circuit/image setup (rebuilds during image build):

# Fetch and copy circuits
scripts/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits
cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/

# Build image
scripts/build_test_image.sh

# Run
NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin compose_runner

Environment variables:

• NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local — Image tag (required, must match built image)
• POL_PROOF_DEV_MODE=true — Required for all runners
• NOMOS_DEMO_VALIDATORS=3 / NOMOS_DEMO_EXECUTORS=2 / NOMOS_DEMO_RUN_SECS=120 — Topology overrides
• COMPOSE_NODE_PAIRS=1x1 — Alternative topology format: "validators×executors"
• NOMOS_METRICS_QUERY_URL — Prometheus-compatible base URL for the runner process to query (optional)
• NOMOS_METRICS_OTLP_INGEST_URL — Full OTLP HTTP ingest URL for node metrics export (optional)
• NOMOS_GRAFANA_URL — Grafana base URL for printing/logging (optional)
• COMPOSE_RUNNER_HOST=127.0.0.1 — Host address for port mappings
• COMPOSE_RUNNER_PRESERVE=1 — Keep containers running after test
• NOMOS_LOG_LEVEL=debug / NOMOS_LOG_FILTER=... — Control node log verbosity (stdout/stderr)
• testing-framework/assets/stack/cfgsync.yaml (tracing_settings.logger) — Switch node logs between stdout/stderr and file output

Compose-specific features:

• Node control support: Only runner that supports chaos testing (.enable_node_control() + chaos workloads)
• Observability is external: Set NOMOS_METRICS_* / NOMOS_GRAFANA_URL to enable telemetry links and querying
  • Quickstart: scripts/setup-observability.sh compose up then scripts/setup-observability.sh compose env

Important:

• Containers expect KZG parameters at /kzgrs_test_params/kzgrs_test_params (note the repeated filename)
• Use scripts/run-examples.sh compose to handle all setup automatically

K8s Runner (Direct Cargo Run)

For manual control, you can run the k8s_runner binary directly. K8s requires the same image setup as Compose.

Prerequisites:

1. Kubernetes cluster with kubectl configured
2. Test image built (same as Compose, preferably with prebuilt bundle)
3. Image available in cluster (loaded or pushed to registry)

Build and load image:

# Build image with bundle (recommended)
scripts/build-bundle.sh --platform linux
export NOMOS_BINARIES_TAR=.tmp/nomos-binaries-linux-v0.3.1.tar.gz
scripts/build_test_image.sh

# Load into cluster
export NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local
kind load docker-image logos-blockchain-testing:local  # For kind
# OR: minikube image load logos-blockchain-testing:local  # For minikube
# OR: docker push your-registry/logos-blockchain-testing:local  # For remote

Run the example:

export NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local
export POL_PROOF_DEV_MODE=true
cargo run -p runner-examples --bin k8s_runner

Environment variables:

• NOMOS_TESTNET_IMAGE — Image tag (required)
• POL_PROOF_DEV_MODE=true — Required for all runners
• NOMOS_DEMO_VALIDATORS / NOMOS_DEMO_EXECUTORS / NOMOS_DEMO_RUN_SECS — Topology overrides
• NOMOS_METRICS_QUERY_URL — Prometheus-compatible base URL for the runner process to query (PromQL)
• NOMOS_METRICS_OTLP_INGEST_URL — Full OTLP HTTP ingest URL for node metrics export (optional)
• NOMOS_GRAFANA_URL — Grafana base URL for printing/logging (optional)

Metrics + Grafana (optional):

export NOMOS_METRICS_QUERY_URL=http://your-prometheus:9090
# Prometheus OTLP receiver example:
export NOMOS_METRICS_OTLP_INGEST_URL=http://your-prometheus:9090/api/v1/otlp/v1/metrics
# Optional: print a Grafana link in TESTNET_ENDPOINTS
export NOMOS_GRAFANA_URL=http://your-grafana:3000
cargo run -p runner-examples --bin k8s_runner

Notes:

• NOMOS_METRICS_QUERY_URL must be reachable from the runner process (often via kubectl port-forward).
• NOMOS_METRICS_OTLP_INGEST_URL must be reachable from nodes (pods/containers) and is backend-specific (Prometheus vs VictoriaMetrics paths differ).
  • Quickstart installer: scripts/setup-observability.sh k8s install then scripts/setup-observability.sh k8s env (optional dashboards: scripts/setup-observability.sh k8s dashboards)

Via scripts/run-examples.sh (optional):

scripts/run-examples.sh -t 60 -v 1 -e 1 k8s \
  --metrics-query-url http://your-prometheus:9090 \
  --metrics-otlp-ingest-url http://your-prometheus:9090/api/v1/otlp/v1/metrics

In code (optional):

#![allow(unused)]
fn main() {
use testing_framework_core::scenario::ScenarioBuilder;
use testing_framework_workflows::ObservabilityBuilderExt as _;

let plan = ScenarioBuilder::with_node_counts(1, 1)
    .with_metrics_query_url_str("http://your-prometheus:9090")
    .with_metrics_otlp_ingest_url_str("http://your-prometheus:9090/api/v1/otlp/v1/metrics")
    .build();
}

Important:

• K8s runner mounts testing-framework/assets/stack/kzgrs_test_params as a hostPath volume with file /kzgrs_test_params/kzgrs_test_params inside pods
• No node control support yet: Chaos workloads (.enable_node_control()) will fail
• Use scripts/run-examples.sh k8s to handle all setup automatically

Circuit Assets (KZG Parameters)

DA workloads require KZG cryptographic parameters for polynomial commitment schemes.

Asset Location

Default path: testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params

Note the repeated filename: the directory kzgrs_test_params/ contains a file named kzgrs_test_params. This is the actual proving key file.

Container path (compose/k8s): /kzgrs_test_params/kzgrs_test_params

Override: Set NOMOS_KZGRS_PARAMS_PATH to use a custom location (must point to the file):

NOMOS_KZGRS_PARAMS_PATH=/path/to/custom/params cargo run -p runner-examples --bin local_runner

Directory vs File (KZG)

The system uses KZG assets in two distinct ways:

Getting Circuit Assets

Option 1: Use helper script (recommended):

# From the repository root
chmod +x scripts/setup-nomos-circuits.sh
scripts/setup-nomos-circuits.sh v0.3.1 /tmp/nomos-circuits

# Copy to default location
cp -r /tmp/nomos-circuits/* testing-framework/assets/stack/kzgrs_test_params/

Option 2: Build locally (advanced):

# This repository does not provide a `make kzgrs_test_params` target.
# If you need to regenerate KZG params from source, follow upstream tooling
# instructions (unspecified here) or use the helper scripts above to fetch a
# known-good bundle.

CI Workflow

The CI automatically fetches and places assets:

- name: Install circuits for host build
  run: |
    scripts/setup-nomos-circuits.sh v0.3.1 "$TMPDIR/nomos-circuits"
    cp -a "$TMPDIR/nomos-circuits"/. testing-framework/assets/stack/kzgrs_test_params/

When Are Assets Needed?

Error without assets:

Error: missing KZG parameters at testing-framework/assets/stack/kzgrs_test_params/kzgrs_test_params

If you see this error, the file kzgrs_test_params is missing from the directory. Use scripts/run-examples.sh or scripts/setup-nomos-circuits.sh to fetch it.

Logging and Observability

Node Logging vs Framework Logging

Critical distinction: Node logs and framework logs use different configuration mechanisms.

Common mistake: Setting RUST_LOG=debug only increases verbosity of the runner binary itself. Node logs remain at their default level unless you also set NOMOS_LOG_LEVEL=debug.

Example:

# This only makes the RUNNER verbose, not the nodes:
RUST_LOG=debug cargo run -p runner-examples --bin local_runner

# This makes the NODES verbose:
NOMOS_LOG_LEVEL=debug cargo run -p runner-examples --bin local_runner

# Both verbose (typically not needed):
RUST_LOG=debug NOMOS_LOG_LEVEL=debug cargo run -p runner-examples --bin local_runner

Logging Environment Variables

Example: Full debug logging to files:

NOMOS_TESTS_TRACING=true \
NOMOS_LOG_DIR=/tmp/test-logs \
NOMOS_LOG_LEVEL=debug \
NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug,nomos_da_dispersal=debug,nomos_da_verifier=debug,nomos_blend=debug,chain_service=info,chain_network=info,chain_leader=info" \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin local_runner

Per-Node Log Files

When NOMOS_LOG_DIR is set, each node writes logs to separate files:

File naming pattern:

• Validators: Prefix nomos-node-0, nomos-node-1, etc. (may include timestamp suffix)
• Executors: Prefix nomos-executor-0, nomos-executor-1, etc. (may include timestamp suffix)

Local runner note: The local runner uses per-run temporary directories under the current working directory and removes them after the run unless NOMOS_TESTS_KEEP_LOGS=1. Use NOMOS_LOG_DIR=/path/to/logs to write per-node log files to a stable location.

Filter Target Names

Common target prefixes for NOMOS_LOG_FILTER:

Example filter:

NOMOS_LOG_FILTER="cryptarchia=trace,nomos_da_sampling=debug,chain_service=info,chain_network=info,chain_leader=info"

Accessing Logs Per Runner

Local Runner

Default (temporary directories, auto-cleanup):

POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner
# Logs written to temporary directories in working directory
# Automatically cleaned up after test completes

Persistent file output:

NOMOS_LOG_DIR=/tmp/local-logs \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin local_runner

# After test completes:
ls /tmp/local-logs/
# Files with prefix: nomos-node-0*, nomos-node-1*, nomos-executor-0*
# May include timestamps in filename

Tip: Use NOMOS_LOG_DIR for persistent per-node log files, and NOMOS_TESTS_KEEP_LOGS=1 if you want to keep the per-run temporary directories (configs/state) for post-mortem inspection.

Compose Runner

Via Docker logs (default, recommended):

# List containers (note the UUID prefix in names)
docker ps --filter "name=nomos-compose-"

# Stream logs from specific container
docker logs -f <container-id-or-name>

# Or use name pattern matching:
docker logs -f $(docker ps --filter "name=nomos-compose-.*-validator-0" -q | head -1)

Via file collection (advanced):

To write per-node log files inside containers, set tracing_settings.logger: !File in testing-framework/assets/stack/cfgsync.yaml (and ensure the directory is writable). To access them, you must either:

1. Copy files out after the run:

# Ensure `testing-framework/assets/stack/cfgsync.yaml` is configured to log to `/logs`
# via `tracing_settings.logger: !File`.
NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
POL_PROOF_DEV_MODE=true \
cargo run -p runner-examples --bin compose_runner

# After test, copy files from containers:
docker ps --filter "name=nomos-compose-"
docker cp <container-id>:/logs/node* /tmp/

2. Mount a host volume (requires modifying compose template):

volumes:
  - /tmp/host-logs:/logs  # Add to docker-compose.yml.tera

Recommendation: Use docker logs by default. File collection inside containers is complex and rarely needed.

Keep containers for debugging:

COMPOSE_RUNNER_PRESERVE=1 \
NOMOS_TESTNET_IMAGE=logos-blockchain-testing:local \
cargo run -p runner-examples --bin compose_runner
# Containers remain running after test—inspect with docker logs or docker exec

Compose networking/debug knobs:

• COMPOSE_RUNNER_HOST=127.0.0.1 — host used for readiness probes (override for remote Docker daemons / VM networking)
• COMPOSE_RUNNER_HOST_GATEWAY=host.docker.internal:host-gateway — controls the extra_hosts entry injected into compose (set to disable to omit)
• TESTNET_RUNNER_PRESERVE=1 — alias for COMPOSE_RUNNER_PRESERVE=1
• COMPOSE_RUNNER_HTTP_TIMEOUT_SECS=<secs> — override compose node HTTP readiness timeout

Note: Container names follow pattern nomos-compose-{uuid}-validator-{index}-1 where {uuid} changes per run.

K8s Runner

Via kubectl logs (use label selectors):

# List pods
kubectl get pods

# Stream logs using label selectors (recommended)
# Helm chart labels:
# - nomos/logical-role=validator|executor
# - nomos/validator-index / nomos/executor-index
kubectl logs -l nomos/logical-role=validator -f
kubectl logs -l nomos/logical-role=executor -f

# Stream logs from specific pod
kubectl logs -f nomos-validator-0

# Previous logs from crashed pods
kubectl logs --previous -l nomos/logical-role=validator

Download logs for offline analysis:

# Using label selectors
kubectl logs -l nomos/logical-role=validator --tail=1000 > all-validators.log
kubectl logs -l nomos/logical-role=executor --tail=1000 > all-executors.log

# Specific pods
kubectl logs nomos-validator-0 > validator-0.log
kubectl logs nomos-executor-1 > executor-1.log

K8s environment notes:

• The k8s runner is optimized for local clusters (Docker Desktop Kubernetes / minikube / kind):
  • The default image logos-blockchain-testing:local must be available on the cluster’s nodes (Docker Desktop shares the local daemon; kind/minikube often requires an explicit image load step).
  • The Helm chart mounts KZG params via a hostPath to your workspace path; this typically won’t work on remote/managed clusters without replacing it with a PV/CSI volume or baking the params into an image.
• Debug helpers:
  • K8S_RUNNER_DEBUG=1 — logs Helm stdout/stderr for install commands.
  • K8S_RUNNER_PRESERVE=1 — keep the namespace/release after the run.
• K8S_RUNNER_NODE_HOST=<ip|hostname> — override NodePort host resolution for non-local clusters.
• K8S_RUNNER_NAMESPACE=<name> / K8S_RUNNER_RELEASE=<name> — pin namespace/release instead of random IDs (useful for debugging)

Specify namespace (if not using default):

kubectl logs -n my-namespace -l nomos/logical-role=validator -f

OTLP and Telemetry

OTLP exporters are optional. If you see errors about unreachable OTLP endpoints, it's safe to ignore them unless you're actively collecting traces/metrics.

To enable OTLP:

NOMOS_OTLP_ENDPOINT=http://localhost:4317 \
NOMOS_OTLP_METRICS_ENDPOINT=http://localhost:4318 \
cargo run -p runner-examples --bin local_runner

To silence OTLP errors: Simply leave these variables unset (the default).

Observability: Prometheus and Node APIs

Runners expose metrics and node HTTP endpoints for expectation code and debugging:

Prometheus-compatible metrics querying (optional):

• Runners do not provision Prometheus automatically.
• For a ready-to-run stack, use scripts/setup-observability.sh:
  • Compose: scripts/setup-observability.sh compose up then scripts/setup-observability.sh compose env
  • K8s: scripts/setup-observability.sh k8s install then scripts/setup-observability.sh k8s env
• Provide NOMOS_METRICS_QUERY_URL (PromQL base URL) to enable ctx.telemetry() queries.
• Access from expectations when configured: ctx.telemetry().prometheus().map(|p| p.base_url())

Grafana (optional):

• Runners do not provision Grafana automatically (but scripts/setup-observability.sh can).
• If you set NOMOS_GRAFANA_URL, the deployer prints it in TESTNET_ENDPOINTS.
• Dashboards live in testing-framework/assets/stack/monitoring/grafana/dashboards/ for import into your Grafana.

Node APIs:

• Access from expectations: ctx.node_clients().validator_clients().get(0)
• Endpoints: consensus info, network info, DA membership, etc.
• See testing-framework/core/src/nodes/api_client.rs for available methods

flowchart TD
    Expose[Runner exposes endpoints/ports] --> Collect[Runtime collects block/health signals]
    Collect --> Consume[Expectations consume signals<br/>decide pass/fail]
    Consume --> Inspect[Operators inspect logs/metrics<br/>when failures arise]