CI Integration

Both LocalDeployer and ComposeDeployer work well in CI environments. Choose based on your tradeoffs.

Runner Comparison for CI

LocalDeployer (Host Runner):

• Faster startup (no Docker overhead)
• Good for quick smoke tests
• Trade-off: Less isolation (processes share host resources)

ComposeDeployer (Recommended for CI):

• Better isolation (containerized)
• Reproducible environment
• Can integrate with external Prometheus/Grafana (optional)
• Trade-offs: Slower startup (Docker image build), requires Docker daemon

K8sDeployer:

• Production-like environment
• Full resource isolation
• Trade-offs: Slowest (cluster setup + image loading), requires cluster access
• Best for nightly/weekly runs or production validation

Existing Examples:

See .github/workflows/lint.yml (jobs: host_smoke, compose_smoke) for CI examples running the demo scenarios in this repository.

Complete CI Workflow Example

Here's a comprehensive GitHub Actions workflow demonstrating host and compose runners with caching, matrix testing, and log collection:

name: Testing Framework CI

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  POL_PROOF_DEV_MODE: true
  CARGO_TERM_COLOR: always
  RUST_BACKTRACE: 1

jobs:
  # Quick smoke test with host runner (no Docker)
  host_smoke:
    name: Host Runner Smoke Test
    runs-on: ubuntu-latest
    timeout-minutes: 15
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up Rust toolchain
        uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: nightly
          override: true
      
      - name: Cache Rust dependencies
        uses: actions/cache@v3
        with:
          path: |
            ~/.cargo/bin/
            ~/.cargo/registry/index/
            ~/.cargo/registry/cache/
            ~/.cargo/git/db/
            target/
          key: ${{ runner.os }}-cargo-host-${{ hashFiles('**/Cargo.lock') }}
          restore-keys: |
            ${{ runner.os }}-cargo-host-
      
      - name: Cache nomos-node build
        uses: actions/cache@v3
        with:
          path: |
            ../nomos-node/target/release/nomos-node
            ../nomos-node/target/release/nomos-executor
          key: ${{ runner.os }}-nomos-${{ hashFiles('../nomos-node/**/Cargo.lock') }}
          restore-keys: |
            ${{ runner.os }}-nomos-
      
      - name: Run host smoke test
        run: |
          # Use run-examples.sh which handles setup automatically
          scripts/run/run-examples.sh -t 120 -v 3 -e 1 host
      
      - name: Upload logs on failure
        if: failure()
        uses: actions/upload-artifact@v3
        with:
          name: host-runner-logs
          path: |
            .tmp/
            *.log
          retention-days: 7

  # Compose runner matrix (with Docker)
  compose_matrix:
    name: Compose Runner (${{ matrix.topology }})
    runs-on: ubuntu-latest
    timeout-minutes: 25
    
    strategy:
      fail-fast: false
      matrix:
        topology:
          - "3v1e"
          - "5v1e"
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up Rust toolchain
        uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: nightly
          override: true
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      
      - name: Cache Rust dependencies
        uses: actions/cache@v3
        with:
          path: |
            ~/.cargo/bin/
            ~/.cargo/registry/index/
            ~/.cargo/registry/cache/
            ~/.cargo/git/db/
            target/
          key: ${{ runner.os }}-cargo-compose-${{ hashFiles('**/Cargo.lock') }}
          restore-keys: |
            ${{ runner.os }}-cargo-compose-
      
      - name: Cache Docker layers
        uses: actions/cache@v3
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ hashFiles('Dockerfile', 'scripts/build/build_test_image.sh') }}
          restore-keys: |
            ${{ runner.os }}-buildx-
      
      - name: Run compose test
        env:
          TOPOLOGY: ${{ matrix.topology }}
        run: |
          # Build and run with the specified topology
          scripts/run/run-examples.sh -t 120 -v ${TOPOLOGY:0:1} -e ${TOPOLOGY:2:1} compose
      
      - name: Collect Docker logs on failure
        if: failure()
        run: |
          mkdir -p logs
          for container in $(docker ps -a --filter "name=nomos-compose-" -q); do
            docker logs $container > logs/$(docker inspect --format='{{.Name}}' $container).log 2>&1
          done
      
      - name: Upload logs and artifacts
        if: failure()
        uses: actions/upload-artifact@v3
        with:
          name: compose-${{ matrix.topology }}-logs
          path: |
            logs/
            .tmp/
          retention-days: 7
      
      - name: Clean up Docker resources
        if: always()
        run: |
          docker compose down -v 2>/dev/null || true
          docker ps -a --filter "name=nomos-compose-" -q | xargs -r docker rm -f

  # Cucumber/BDD integration tests (if enabled)
  cucumber_tests:
    name: Cucumber BDD Tests
    runs-on: ubuntu-latest
    timeout-minutes: 20
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up Rust toolchain
        uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: nightly
          override: true
      
      - name: Cache dependencies
        uses: actions/cache@v3
        with:
          path: |
            ~/.cargo/bin/
            ~/.cargo/registry/index/
            ~/.cargo/registry/cache/
            ~/.cargo/git/db/
            target/
          key: ${{ runner.os }}-cargo-cucumber-${{ hashFiles('**/Cargo.lock') }}
          restore-keys: |
            ${{ runner.os }}-cargo-cucumber-
      
      - name: Run Cucumber tests
        run: |
          # Build prerequisites
          scripts/build/build-bundle.sh --platform linux
          export NOMOS_BINARIES_TAR=$(ls -t .tmp/nomos-binaries-linux-*.tar.gz | head -1)
          
          # Run Cucumber tests (host runner)
          cargo test -p runner-examples --bin cucumber_host
      
      - name: Upload test report
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: cucumber-report
          path: |
            target/cucumber-reports/
          retention-days: 14

  # Summary job (requires all tests to pass)
  ci_success:
    name: CI Success
    needs: [host_smoke, compose_matrix, cucumber_tests]
    runs-on: ubuntu-latest
    if: always()
    
    steps:
      - name: Check all jobs
        run: |
          if [[ "${{ needs.host_smoke.result }}" != "success" ]] || \
             [[ "${{ needs.compose_matrix.result }}" != "success" ]] || \
             [[ "${{ needs.cucumber_tests.result }}" != "success" ]]; then
            echo "One or more CI jobs failed"
            exit 1
          fi
          echo "All CI jobs passed!"

Workflow Features

1. Matrix Testing: Runs compose tests with different topologies (3v1e, 5v1e)
2. Caching: Caches Rust dependencies, Docker layers, and nomos-node builds for faster runs
3. Log Collection: Automatically uploads logs and artifacts when tests fail
4. Timeout Protection: Reasonable timeouts prevent jobs from hanging indefinitely
5. Cucumber Integration: Shows how to integrate BDD tests into CI
6. Clean Teardown: Ensures Docker resources are cleaned up even on failure

Customization Points

Topology Matrix:

Add more topologies for comprehensive testing:

matrix:
  topology:
    - "3v1e"
    - "5v1e"
    - "10v2e"  # Larger scale

Timeout Adjustments:

Increase timeout-minutes for longer-running scenarios or slower environments:

timeout-minutes: 30  # Instead of 15

Artifact Retention:

Change retention-days based on your storage needs:

retention-days: 14  # Keep logs for 2 weeks

Conditional Execution:

Run expensive tests only on merge to main:

if: github.event_name == 'push' && github.ref == 'refs/heads/main'

Best Practices

Required: Set POL_PROOF_DEV_MODE

Always set POL_PROOF_DEV_MODE=true globally in your workflow env:

env:
  POL_PROOF_DEV_MODE: true  # REQUIRED!

Without this, tests will hang due to expensive proof generation.

Use Helper Scripts

Prefer scripts/run/run-examples.sh which handles all setup automatically:

scripts/run/run-examples.sh -t 120 -v 3 -e 1 host

This is more reliable than manual cargo run commands.

Cache Aggressively

Cache Rust dependencies, nomos-node builds, and Docker layers to speed up CI:

- name: Cache Rust dependencies
  uses: actions/cache@v3
  with:
    path: |
      ~/.cargo/bin/
      ~/.cargo/registry/index/
      ~/.cargo/registry/cache/
      ~/.cargo/git/db/
      target/
    key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}

Collect Logs on Failure

Always upload logs when tests fail for easier debugging:

- name: Upload logs on failure
  if: failure()
  uses: actions/upload-artifact@v3
  with:
    name: test-logs
    path: |
      .tmp/
      *.log
    retention-days: 7

Split Workflows for Faster Iteration

For large projects, split host/compose/k8s into separate workflow files:

• .github/workflows/test-host.yml — Fast smoke tests
• .github/workflows/test-compose.yml — Reproducible integration tests
• .github/workflows/test-k8s.yml — Production-like validation (nightly)

Run K8s Tests Less Frequently

K8s tests are slower. Consider running them only on main branch or scheduled:

on:
  push:
    branches: [main]
  schedule:
    - cron: '0 2 * * *'  # Daily at 2 AM

Platform-Specific Notes

Ubuntu Runners

• Docker pre-installed and running
• Best for compose/k8s runners
• Most common choice

macOS Runners

• Docker Desktop not installed by default
• Slower and more expensive
• Use only if testing macOS-specific issues

Self-Hosted Runners

• Cache Docker images locally for faster builds
• Set resource limits (SLOW_TEST_ENV=true if needed)
• Ensure cleanup scripts run (docker system prune)

Debugging CI Failures

Enable Debug Logging

Add debug environment variables temporarily:

env:
  RUST_LOG: debug
  NOMOS_LOG_LEVEL: debug

Preserve Containers (Compose)

Set COMPOSE_RUNNER_PRESERVE=1 to keep containers running for inspection:

- name: Run compose test (preserve on failure)
  env:
    COMPOSE_RUNNER_PRESERVE: 1
  run: scripts/run/run-examples.sh -t 120 -v 3 -e 1 compose

Access Artifacts

Download uploaded artifacts from the GitHub Actions UI to inspect logs locally.

Next Steps

• Running Examples — Manual execution for local development
• Environment Variables — Full variable reference
• Troubleshooting — Common CI-specific issues