diff --git a/book/src/operations.md b/book/src/operations.md index 2033872..b323b7c 100644 --- a/book/src/operations.md +++ b/book/src/operations.md @@ -108,8 +108,9 @@ cargo run -p runner-examples --bin local_runner - `NOMOS_DEMO_EXECUTORS=2` — Number of executors (default: 1, or use legacy `LOCAL_DEMO_EXECUTORS`) - `NOMOS_DEMO_RUN_SECS=120` — Run duration in seconds (default: 60, or use legacy `LOCAL_DEMO_RUN_SECS`) - `NOMOS_NODE_BIN` / `NOMOS_EXECUTOR_BIN` — Paths to binaries (required for direct run) -- `NOMOS_TESTS_TRACING=true` — Enable persistent file logging -- `NOMOS_LOG_DIR=/tmp/logs` — Directory for per-node log files +- `NOMOS_LOG_DIR=/tmp/logs` — Directory for per-node log files (works across runners) +- `NOMOS_TESTS_KEEP_LOGS=1` — Keep per-run temporary directories (useful for debugging/CI artifacts) +- `NOMOS_TESTS_TRACING=true` — Enable the debug tracing preset (optional; combine with `NOMOS_LOG_DIR` unless you have external tracing backends configured) - `NOMOS_LOG_LEVEL=debug` — Set log level (default: info) - `NOMOS_LOG_FILTER=consensus=trace,da=debug` — Fine-grained module filtering @@ -317,7 +318,7 @@ RUST_LOG=debug NOMOS_LOG_LEVEL=debug cargo run -p runner-examples --bin local_ru | `NOMOS_LOG_DIR` | None (console only) | Directory for per-node log files. If unset, logs go to stdout/stderr. | | `NOMOS_LOG_LEVEL` | `info` | Global log level: `error`, `warn`, `info`, `debug`, `trace` | | `NOMOS_LOG_FILTER` | None | Fine-grained target filtering (e.g., `consensus=trace,da=debug`) | -| `NOMOS_TESTS_TRACING` | `false` | Enable tracing subscriber for local runner file logging | +| `NOMOS_TESTS_TRACING` | `false` | Enable the debug tracing preset (optional; combine with `NOMOS_LOG_DIR` unless you have external tracing backends configured) | | `NOMOS_OTLP_ENDPOINT` | None | OTLP trace endpoint (optional, disables OTLP noise if unset) | | `NOMOS_OTLP_METRICS_ENDPOINT` | None | OTLP metrics endpoint (optional) | @@ -339,7 +340,7 @@ When `NOMOS_LOG_DIR` is set, each node writes logs to separate files: - **Validators**: Prefix `nomos-node-0`, `nomos-node-1`, etc. (may include timestamp suffix) - **Executors**: Prefix `nomos-executor-0`, `nomos-executor-1`, etc. (may include timestamp suffix) -**Local runner caveat:** By default, the local runner writes logs to temporary directories in the working directory. These are automatically cleaned up after tests complete. To preserve logs, you MUST set both `NOMOS_TESTS_TRACING=true` AND `NOMOS_LOG_DIR=/path/to/logs`. +**Local runner note:** The local runner uses per-run temporary directories under the current working directory and removes them after the run unless `NOMOS_TESTS_KEEP_LOGS=1`. Use `NOMOS_LOG_DIR=/path/to/logs` to write per-node log files to a stable location. ### Filter Target Names @@ -374,7 +375,6 @@ POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner **Persistent file output:** ```bash -NOMOS_TESTS_TRACING=true \ NOMOS_LOG_DIR=/tmp/local-logs \ POL_PROOF_DEV_MODE=true \ cargo run -p runner-examples --bin local_runner @@ -385,7 +385,7 @@ ls /tmp/local-logs/ # May include timestamps in filename ``` -**Both flags required:** You MUST set both `NOMOS_TESTS_TRACING=true` (enables tracing file sink) AND `NOMOS_LOG_DIR` (specifies directory) to get persistent logs. +**Tip:** Use `NOMOS_LOG_DIR` for persistent per-node log files, and `NOMOS_TESTS_KEEP_LOGS=1` if you want to keep the per-run temporary directories (configs/state) for post-mortem inspection. #### Compose Runner diff --git a/book/src/quickstart.md b/book/src/quickstart.md index e4179ab..82be830 100644 --- a/book/src/quickstart.md +++ b/book/src/quickstart.md @@ -77,8 +77,8 @@ let _handle = runner.run(&mut plan).await?; - Nodes spawn as local processes - Consensus starts producing blocks - Scenario runs for the configured duration -- Node logs written to temporary directories in working directory (auto-cleaned up after test) -- To persist logs: set `NOMOS_TESTS_TRACING=true` and `NOMOS_LOG_DIR=/path/to/logs` (files will have prefix like `nomos-node-0*`, may include timestamps) +- Node state/logs written under a temporary per-run directory in the current working directory (removed after the run unless `NOMOS_TESTS_KEEP_LOGS=1`) +- To write per-node log files to a stable location: set `NOMOS_LOG_DIR=/path/to/logs` (files will have prefix like `nomos-node-0*`, may include timestamps) ## What Just Happened? diff --git a/book/src/troubleshooting.md b/book/src/troubleshooting.md index 2a37006..f1cc2ca 100644 --- a/book/src/troubleshooting.md +++ b/book/src/troubleshooting.md @@ -30,12 +30,12 @@ Common symptoms and likely causes: | Runner | Default Output | With `NOMOS_LOG_DIR` + Flags | Access Command | |--------|---------------|------------------------------|----------------| -| **Host** (local) | Temporary directories (cleaned up) | Per-node files with prefix `nomos-node-{index}` (requires `NOMOS_TESTS_TRACING=true`) | `cat $NOMOS_LOG_DIR/nomos-node-0*` | +| **Host** (local) | Per-run temporary directories under the current working directory (removed unless `NOMOS_TESTS_KEEP_LOGS=1`) | Per-node files with prefix `nomos-node-{index}` (set `NOMOS_LOG_DIR`) | `cat $NOMOS_LOG_DIR/nomos-node-0*` | | **Compose** | Docker container stdout/stderr | Per-node files inside containers (if path is mounted) | `docker ps` then `docker logs ` | -| **K8s** | Pod stdout/stderr | Per-node files inside pods (if path is mounted) | `kubectl logs -l app=nomos-validator` | +| **K8s** | Pod stdout/stderr | Per-node files inside pods (if path is mounted) | `kubectl logs -l nomos/logical-role=validator` | **Important Notes:** -- **Host runner** (local processes): Logs go to system temporary directories (NOT in working directory) by default and are automatically cleaned up after tests. To persist logs, you MUST set both `NOMOS_TESTS_TRACING=true` AND `NOMOS_LOG_DIR=/path/to/logs`. +- **Host runner** (local processes): Per-run temporary directories are created under the current working directory and removed after the run unless `NOMOS_TESTS_KEEP_LOGS=1`. To write per-node log files to a stable location, set `NOMOS_LOG_DIR=/path/to/logs`. - **Compose/K8s**: Per-node log files only exist inside containers/pods if `NOMOS_LOG_DIR` is set AND the path is writable inside the container/pod. By default, rely on `docker logs` or `kubectl logs`. - **File naming**: Log files use prefix `nomos-node-{index}*` or `nomos-executor-{index}*` with timestamps, e.g., `nomos-node-0.2024-12-01T10-30-45.log` (NOT just `.log` suffix). - **Container names**: Compose containers include project UUID, e.g., `nomos-compose--validator-0-1` where `` is randomly generated per run @@ -51,7 +51,6 @@ POL_PROOF_DEV_MODE=true cargo run -p runner-examples --bin local_runner 2>&1 | t **Persistent file output:** ```bash -NOMOS_TESTS_TRACING=true \ NOMOS_LOG_DIR=/tmp/debug-logs \ NOMOS_LOG_LEVEL=debug \ POL_PROOF_DEV_MODE=true \ @@ -109,28 +108,28 @@ docker logs > debug.log kubectl config view --minify | grep namespace # All validator pods (add -n if not using default) -kubectl logs -l app=nomos-validator -f +kubectl logs -l nomos/logical-role=validator -f # All executor pods -kubectl logs -l app=nomos-executor -f +kubectl logs -l nomos/logical-role=executor -f # Specific pod by name (find exact name first) -kubectl get pods -l app=nomos-validator # Find the exact pod name +kubectl get pods -l nomos/logical-role=validator # Find the exact pod name kubectl logs -f # Then use it # With explicit namespace -kubectl logs -n my-namespace -l app=nomos-validator -f +kubectl logs -n my-namespace -l nomos/logical-role=validator -f ``` **Download logs from crashed pods:** ```bash # Previous logs from crashed pod -kubectl get pods -l app=nomos-validator # Find crashed pod name first +kubectl get pods -l nomos/logical-role=validator # Find crashed pod name first kubectl logs --previous > crashed-validator.log # Or use label selector for all crashed validators -for pod in $(kubectl get pods -l app=nomos-validator -o name); do +for pod in $(kubectl get pods -l nomos/logical-role=validator -o name); do kubectl logs --previous $pod > $(basename $pod)-previous.log 2>&1 done ``` @@ -145,11 +144,11 @@ for pod in $(kubectl get pods -o name); do done > all-logs.txt # Or use label selectors (recommended) -kubectl logs -l app=nomos-validator --tail=500 > validators.log -kubectl logs -l app=nomos-executor --tail=500 > executors.log +kubectl logs -l nomos/logical-role=validator --tail=500 > validators.log +kubectl logs -l nomos/logical-role=executor --tail=500 > executors.log # With explicit namespace -kubectl logs -n my-namespace -l app=nomos-validator --tail=500 > validators.log +kubectl logs -n my-namespace -l nomos/logical-role=validator --tail=500 > validators.log ``` ## Debugging Workflow @@ -180,8 +179,8 @@ ps aux | grep nomos docker ps -a --filter "name=nomos-compose-" # K8s: check pod status (use label selectors, add -n if needed) -kubectl get pods -l app=nomos-validator -kubectl get pods -l app=nomos-executor +kubectl get pods -l nomos/logical-role=validator +kubectl get pods -l nomos/logical-role=executor kubectl describe pod # Get name from above first ``` @@ -272,11 +271,11 @@ Run a minimal baseline test (e.g., 2 validators, consensus liveness only). If it - **Cause**: Helper scripts (`run-examples.sh`, `build-bundle.sh`, `setup-circuits-stack.sh`) require `versions.env` file at repository root. - **Fix**: Ensure you're running from the repository root directory. The `versions.env` file should already exist and contains: ``` - VERSION=v0.3.1 - NOMOS_NODE_REV=d2dd5a5084e1daef4032562c77d41de5e4d495f8 - NOMOS_BUNDLE_VERSION=v4 + VERSION= + NOMOS_NODE_REV= + NOMOS_BUNDLE_VERSION= ``` - If the file is missing, restore it from version control or create it with the above content. + Use the checked-in `versions.env` at the repository root as the source of truth. ### "Port already in use"