logos-tutorial/docs/spec.md

Tutorial YAML Spec Format

This document describes the YAML format used by tools/tutorial_runner.py to define executable tutorials. Each .test.yaml file is the single source of truth — it drives both:

• Execution (run): steps are executed in a temp directory, commands run, outputs verified
• Markdown generation (generate): a .md tutorial is produced from the same YAML

Quick example

name: "My Tutorial"
output: my-tutorial.md
release: ""

intro: |
  One-paragraph description of what this tutorial covers.

what_you_build: "A short sentence describing the end result."

what_you_learn:
  - First learning objective
  - Second learning objective

prerequisites:
  - "**Nix** with flakes enabled."

sections:
  - title: "Set Up the Project"
    phase: scaffold
    text: |
      Intro paragraph for this section.
    steps:
      - title: "Create the directory"
        run: "mkdir -p my-project"

      - title: "Write the config"
        text: "Create `config.json`:"
        file:
          path: config.json
          language: json
          content: |
            { "name": "example" }

      - title: "Build"
        run: "nix build"
        expect_contains:
          - "Build successful"

Top-level fields

Field	Required	Type	Description
name	yes	string	Tutorial title. Used as the # heading in generated markdown and in runner output.
output	no	string	Default output filename for generate (relative to the tutorial directory, e.g., tutorial-wrapping-c-library.md). Can be overridden with -o.
project_name	no	string	Directory name for this tutorial's project (e.g., logos-calc-module). Used when chaining tutorials via requires: — each tutorial runs in a subdirectory of a shared parent. Ignored when running standalone.
requires	no	list of strings	Paths to prerequisite .test.yaml specs (relative to the current spec). The runner executes each prerequisite first in a sibling subdirectory (named by its project_name), then runs the current tutorial. Prerequisites are resolved transitively — if A requires B and B requires C, the run order is C, B, A. Shared prerequisites run once (deduped), and circular requires: are reported as an error. This enables cross-tutorial references like ../logos-calc-module. Requires project_name on both the current and prerequisite specs.
intro	no	string	Introductory paragraph(s). Rendered after the title in the markdown. Supports full markdown.
what_you_build	no	string	One-line summary prefixed with "What you'll build:" in the markdown.
what_you_learn	no	list of strings	Bullet list prefixed with "What you'll learn:".
comparison	no	string	Free-form markdown block rendered after the learning objectives (useful for comparison tables).
prerequisites	no	list of strings	Rendered as a bullet list under a prerequisites heading. Each item can contain markdown (code blocks, links, etc.).
release	no	string	Git tag applied to all {release} placeholders in GitHub URLs (e.g., tutorial-v2). See Release tags.
build_overrides	no	map	Nix --override-input flags for the runner. Keys are input names, values are relative paths to local repos. Only affects execution, not generation.
sections	yes	list	The tutorial content. See below.

Sections

Each section becomes a ## heading in the markdown. Sections with step: true get auto-numbered as "Step N: Title".

Field	Required	Type	Description
title	yes	string	Section heading.
step	no	boolean	If true, this section is numbered as "Step N: Title" in markdown and gets a --- separator after it (except the last section). Sections without step render as plain ## Title.
text	no	string	Introductory prose rendered before the steps. Supports full markdown (tables, blockquotes, code blocks, etc.).
steps	no	list	Ordered list of steps. See below.

Sections without steps are prose-only — the text is rendered as-is. This is useful for reference sections like "Troubleshooting" or "Common Patterns".

Steps

Steps are the core building blocks. Each step can combine multiple fields. The rendering order in the generated markdown is:

1. title → ### heading
2. text → prose paragraph
3. file → code block with file contents
4. run → bash code block
5. post_text → prose after the action
6. extra_run → additional command block (no heading)

A step without a title renders its content inline under the previous heading — useful for continuation content like "Then run:" followed by a code block.

Step fields

title (string, optional)

Rendered as a ### heading in the markdown. Steps without a title don't get a heading — their content flows under the previous step's heading.

text (string, optional)

Prose rendered before any action. Supports full markdown.

file (object, optional)

Writes a file to disk during execution and renders it as a code block in the markdown.

Subfield	Type	Description
path	string	Relative path within the project (e.g., src/main.cpp, lib/libcalc.h).
content	string	The file contents.
language	string	Syntax highlighting hint for the markdown code block. Auto-detected from extension if omitted (.c → c, .h/.cpp → cpp, .json → json, .nix → nix, etc.).
encoding	string	Set to base64 for binary files. Renders as *Binary file: \path`*` instead of a code block.

Runner behavior: Creates parent directories and writes the file to disk.

Generator behavior: Renders ```language code block with the content.

run (string, optional)

A shell command to execute and display.

Subfield	Type	Description
(value)	string	The command to execute. Supports {ext} (expands to so or dylib) and {shared_flags} (expands to -shared -fPIC or -dynamiclib).

Runner behavior: Expands platform placeholders, injects nix overrides if applicable, and runs the command. Checks exit code (0 = pass, non-zero = fail).

Generator behavior: Renders the command in a ```bash block. If code_block is present on the step, renders that instead (see below).

code_block (string, optional)

The exact content to show in the generated markdown, used instead of the run command. This is for cases where:

• The executed command differs from what readers should see (e.g., &&-chained commands displayed as separate lines)
• Platform-specific variants should be shown (Linux and macOS versions)
• The display should include comments, blank lines, or additional context

The runner ignores code_block — it only uses run for execution.

- title: "Build the shared library"
  run: "cd lib && gcc {shared_flags} -o libcalc.{ext} libcalc.c && cd .."
  code_block: |
    cd lib

    # Linux
    gcc -shared -fPIC -o libcalc.so libcalc.c

    # macOS
    # gcc -shared -fPIC -o libcalc.dylib libcalc.c

    cd ..

expect_contains (list of strings, optional)

Assertions checked by the runner against command output. Not rendered in the markdown.

run: "./lm/bin/lm metadata result/lib/calc_module_plugin.{ext}"
expect_contains:
  - "Name:         calc_module"
  - "Version:      1.0.0"

Runner behavior: Captures stdout+stderr, checks that every string appears in the output.

Generator behavior: Ignored — not rendered. Use post_text to show expected output to readers.

check_file (string, optional)

Verifies a file exists. Runner-only, not rendered in the markdown.

- check_file: "result/lib/calc_module_plugin.{ext}"

Runner behavior: Expands {ext}, globs for the file, passes if found.

Generator behavior: Ignored — not rendered.

ui_test (object, optional)

Runs headless UI tests against a Qt app using logos-qt-mcp. The app is launched with QT_QPA_PLATFORM=offscreen (no display needed) and tests connect to the QML inspector to verify elements, click buttons, and check results.

Two modes:

• Launch mode (preferred): launch runs the app as a background process, tests connect to its inspector, app is killed when done. The launch command is rendered in the generated markdown.
• Binary mode: build + binary let the test framework manage the app via --ci. Not rendered in markdown.

Subfield	Type	Description
launch	string	Command to launch the app (e.g., nix run .). Launched as a background process with offscreen Qt. Rendered in generated markdown as a bash code block.
build	string	Command to build the app binary (binary mode only, not rendered).
binary	string	Path to the app binary or nix-app to auto-resolve from flake (binary mode only).
qt_mcp	string	Path to the logos-qt-mcp package, relative to workdir (e.g., result-mcp). Falls back to --qt-mcp CLI flag or LOGOS_QT_MCP env var.
setup	list of strings	Commands to run before testing (e.g., nix build 'github:logos-co/logos-qt-mcp' -o result-mcp).
inspector_port	integer	TCP port for the QML inspector (default: 3768).
tests	list of objects	Test actions to execute. See below.

Test actions:

Action	Fields	What it does
click	target	Find element by text and click it
wait_for	texts, timeout (ms, default 10000), name	Poll until all texts are visible
expect_texts	texts	Assert all texts are visible now
set_text	find_by, find_value, value	Find element by property and set its text property
sleep	ms	Wait a fixed duration

Any action may also carry an optional screenshot field (a filename, e.g. screenshot: "result.png"). After that action runs, the runner captures the headless app via the qt-mcp app.screenshot() API and writes the PNG to an images/ directory next to the generated .md (<output-dir>/images/; temp runs fall back to the workdir). The generator then embeds it inline in that section as ![<name>](images/<file>.png), so screenshots appear in the published tutorial with no extra markup. The .png extension is added if omitted; the value is reduced to a basename.

The two-column HTML report (--report) inlines each captured screenshot as a base64 data: URI, so the report stays a single self-contained file that renders the screenshots even when served from GitHub Pages (where only the index.html is published). If a capture is missing (e.g. its step failed), the report falls back to the relative images/<file>.png link.

Runner behavior (launch mode): Runs setup commands, launches the app in the background with QT_QPA_PLATFORM=offscreen, waits for the QML inspector to be available, generates a .mjs test file, runs it, then kills the app. Reports pass/fail.

Runner behavior (binary mode): Runs setup + build, generates a .mjs test file, runs it via node test.mjs --ci <binary> --verbose. Reports pass/fail.

Generator behavior: In launch mode, launch is rendered as a ```bash code block. In binary mode, nothing is rendered. Use text: and post_text: for additional user-facing prose.

# Launch mode (preferred) — what's shown is what's executed
- ui_test:
    launch: "nix run ."
    setup:
      - "nix build 'github:logos-co/logos-qt-mcp' -o result-mcp"
    qt_mcp: "result-mcp"
    tests:
      - name: "Title visible"
        action: wait_for
        texts: ["Logos Calculator"]
        timeout: 15000
      - name: "Enter number"
        action: set_text
        find_by: "placeholderText"
        find_value: "a"
        value: "3"
      - name: "Click Add"
        action: click
        target: "Add"
      - name: "Result shows 3"
        action: wait_for
        texts: ["3"]
        timeout: 10000

post_text (string, optional)

Prose rendered after the step's action (file, run). Supports full markdown including code blocks, tables, blockquotes.

Use this for:

• Expected output blocks
• Explanations of what just happened
• Callout boxes and tips

- title: "View metadata"
  run: "./lm/bin/lm metadata result/lib/plugin.{ext}"
  post_text: |
    Output:

    ```
    Plugin Metadata:
    ================
    Name:         my_module
    Version:      1.0.0
    ```

extra_run (object, optional)

A continuation command rendered under the same step heading (no separate ###). Useful when a step has two related commands (e.g., build then verify).

Subfield	Type	Description
run	string	Command to execute.
code_block	string	Display override (same as step-level code_block).
post_text	string	Prose after the extra command.

- title: "Build the shared library"
  run: "cd lib && gcc {shared_flags} -o libcalc.{ext} libcalc.c && cd .."
  code_block: |
    cd lib
    gcc -shared -fPIC -o libcalc.so libcalc.c
    cd ..
  post_text: "Verify the symbols are exported:"
  extra_run:
    run: "nm -gU lib/libcalc.{ext} | grep calc"
    code_block: |
      # Linux
      nm -D lib/libcalc.so | grep calc

      # macOS
      # nm -gU lib/libcalc.dylib | grep calc
    post_text: |
      You should see symbols marked with `T`.

Platform placeholders

These placeholders are expanded at execution time by the runner and at generation time in rendered content:

Placeholder	Linux	macOS
{ext}	so	dylib
{shared_flags}	-shared -fPIC	-dynamiclib

Platform placeholders work in run, check_file, extra_run.run, and file.content fields. They are not expanded in code_block, text, or post_text — those are rendered verbatim.

When a command uses platform placeholders, provide a code_block showing both platform variants for the markdown.

Release tags

The release field lets you pin all GitHub URLs to a specific git tag. This avoids updating every URL individually when you want all nix build 'github:logos-co/...' commands to use the same release.

Use the {release} placeholder in run commands, code_block, and file.content:

release: "tutorial-v2"

sections:
  - title: "Set Up"
    steps:
      - run: "nix flake init -t github:logos-co/logos-module-builder{release}#with-external-lib"
      - run: "nix build 'github:logos-co/logos-module{release}#lm' --out-link ./lm"

When release is set to "tutorial-v2", {release} expands to /tutorial-v2:

nix build 'github:logos-co/logos-module/tutorial-v2#lm' --out-link ./lm

When release is empty or omitted, {release} expands to nothing:

nix build 'github:logos-co/logos-module#lm' --out-link ./lm

The --release CLI flag overrides the YAML field:

# Use a specific tag (overrides whatever is in the YAML)
python3 tools/tutorial_runner.py run spec.yaml --release tutorial-v3

# Generate markdown with a tag
python3 tools/tutorial_runner.py generate spec.yaml --release tutorial-v2

# Clear the tag even if the YAML sets one
python3 tools/tutorial_runner.py run spec.yaml --release ""

Runner behavior

• Creates a fresh temp directory (or uses --output-dir / --workdir) — all files, builds, and commands happen there
• Walks sections and steps in order
• Executes file, run, check_file, ui_test actions
• Tracks pass/fail/skip counts
• Stops on first failure by default (use --continue-on-fail to override)
• Prints a summary report at the end (and an HTML report with --report <path>)
• By default the temp directory is deleted when the run finishes (use --output-dir or --keep-workdir to keep it)

Working directory

The runner needs a directory to work in. There are four modes:

1. Default (temp dir, auto-deleted): A fresh /tmp/tutorial-test-XXXXX/ is created and removed after the run.
2. --keep-workdir: Same temp dir, but it's kept after the run so you can inspect the results (built artifacts, installed modules, etc.).
3. --output-dir <dir>: Run into <dir> and keep it — created if missing, never auto-deleted. This is the mode to use when you want the finished tutorial output (built modules, .lgx packages, result/ symlinks) in a known location. For a chained tutorial (requires:) <dir> becomes the chain root and each project lands in <dir>/<project_name>/; for a standalone spec the project is written directly into <dir>. The path is printed as output : <dir> at the end of the run.
4. --workdir <path>: Use an existing directory. It is never created for you, and it runs the spec standalone — requires: prerequisite chains are not run in this mode (the directory is used as the single project's workdir). Useful for re-running a spec against a previous build or for debugging. Note: unless --keep-workdir is also given, the directory is deleted on exit. Prefer --output-dir when you want a persistent, chain-aware output location.

The workdir path is printed at the top of every run:

  workdir  : /tmp/tutorial-test-abc123

Tutorial chaining (requires)

When a spec has requires:, the runner creates a shared parent directory and runs each prerequisite before the main tutorial. Each tutorial gets its own subdirectory named by project_name:

# Part 2 requires Part 1:
requires:
  - tutorial-wrapping-c-library.test.yaml
project_name: logos-calc-ui

# Running Part 2 automatically runs Part 1 first:
python3 tools/tutorial_runner.py run tests/tutorial-qml-ui-app.test.yaml

# Resulting directory structure:
/tmp/tutorial-chain-XXXXX/
├── logos-calc-module/    # Part 1 (prerequisite)
└── logos-calc-ui/        # Part 2 (main)

Commands like ../logos-calc-module in Part 2 resolve to Part 1's output. Results are cumulative — a failure in any tutorial stops the chain (unless --continue-on-fail).

requires: is resolved transitively. A spec only needs to declare its direct prerequisites; their prerequisites are pulled in automatically. For the three-part series, Part 3 can simply declare Part 2:

# Part 3 requires Part 2, which itself requires Part 1:
requires:
  - tutorial-qml-ui-app.test.yaml
project_name: logos-calc-ui-cpp

# Running Part 3 resolves the whole graph and runs in dependency order:
python3 tools/tutorial_runner.py run tests/tutorial-cpp-ui-app.test.yaml

# Resulting directory structure (Part 1 → Part 2 → Part 3):
/tmp/tutorial-chain-XXXXX/
├── logos-calc-module/    # Part 1 (transitive prerequisite, runs first)
├── logos-calc-ui/        # Part 2 (direct prerequisite, runs second)
└── logos-calc-ui-cpp/    # Part 3 (main, runs last)

The graph is walked depth-first in post-order, so a prerequisite always runs before the spec that needs it. A prerequisite shared by multiple specs runs only once, and a circular requires: chain is reported as an error rather than looping forever.

When running standalone (no requires:), project_name is ignored and the runner behaves as before.

HTML execution report (--report)

run --report <path> writes a self-contained HTML report next to (or instead of) the console output. The report has two columns per step:

• Left: the step's rendered tutorial markdown — identical to what generate produces, so the report shows exactly what the reader sees.
• Right: the command(s) actually executed for that step and their captured output, each with a pass/fail badge and exit code.

It records every executed action type:

Step type	Recorded on the right
file	the path written and a pass/fail marker (contents already shown on the left)
run / extra_run	the expanded command, exit code, and combined stdout/stderr (both the command and its (verify) continuation appear)
check_file	the glob checked and which file matched
ui_test	the launch command, the list of UI test actions, and the framework output (plus the app log tail on failure)

When a chain runs (requires:), every tutorial in the chain is included; a dropdown at the top switches between them. Markdown is rendered client-side via marked.js loaded from a CDN, so viewing the report needs network access on first open.

Pair --report with --continue-on-fail so the report captures the full run rather than stopping at the first failure. The CI workflow (.github/workflows/ci.yml) runs with --report and publishes the result to GitHub Pages, linked from a PR comment.

Live TUI (--tui / --iterative)

run --tui shows the same two-column view live in the terminal instead of (or in addition to) writing an HTML file. The left pane renders the current step's tutorial markdown; the right pane shows the command being executed and its output, updating as the run proceeds. Execution goes through the exact same handlers as a normal run, so behaviour is identical — only the presentation differs.

• Auto mode (default): steps execute back-to-back; failed steps pause briefly so they're readable.
• --iterative: the runner waits for a keypress before executing each step — down arrow, right arrow, or space to advance; q to quit. (Using --iterative without --tui is an error.)

--tui requires an interactive terminal (a TTY) and the rich package. rich is an optional dependency used only for the TUI; if it's not installed the runner prints an install hint and exits. All other functionality needs only pyyaml.

Generator behavior

• Walks the same YAML structure
• Produces markdown with proper headings, code blocks, and prose
• Sections with phase get numbered as "Step 1:", "Step 2:", etc.
• Sections without phase get plain ## Title headings
• expect_contains and check_file are not rendered (runner-only)
• code_block overrides the run value for display
• file content is rendered as a fenced code block with syntax highlighting
• Triple blank lines are collapsed to double

File layout

repos/logos-tutorial/
├── tests/
│   └── tutorial-wrapping-c-library.test.yaml   # The spec
├── tools/
│   ├── tutorial_runner.py                       # Runner + generator
│   └── run-tutorial                             # Nix wrapper script
├── tutorial-wrapping-c-library.md               # Generated output
└── docs/
    └── spec.md                                  # This file

Usage

# Run a tutorial (temp dir, auto-deleted)
python3 tools/tutorial_runner.py run tests/tutorial-wrapping-c-library.test.yaml --verbose

# Run into a persistent output directory (created if missing, never deleted).
# For a chained tutorial each part lands in <dir>/<project_name>/.
python3 tools/tutorial_runner.py run tests/tutorial-cpp-ui-app.test.yaml \
  --output-dir ./outputs --continue-on-fail

# Keep the auto-generated temp directory for debugging
python3 tools/tutorial_runner.py run tests/tutorial-wrapping-c-library.test.yaml --keep-workdir --verbose

# Re-run a spec standalone against an existing directory
python3 tools/tutorial_runner.py run tests/tutorial-wrapping-c-library.test.yaml --workdir /tmp/my-tutorial-test --verbose

# Write a two-column HTML report (rendered tutorial + commands run and their output)
python3 tools/tutorial_runner.py run tests/tutorial-cpp-ui-app.test.yaml \
  --report ./tutorial-report.html --continue-on-fail

# Generate markdown
python3 tools/tutorial_runner.py generate tests/tutorial-wrapping-c-library.test.yaml

# Generate to a specific file
python3 tools/tutorial_runner.py generate tests/tutorial-wrapping-c-library.test.yaml -o my-output.md

# Use nix wrapper (ensures python3 + pyyaml are available)
./tools/run-tutorial run tests/tutorial-wrapping-c-library.test.yaml --verbose