* shadow: add k8s-based runner setup
Adds a containerized Shadow runner so the existing single-machine prototype
(shadow/{topogen.py,run.sh,*.awk}) can execute on a lab worker without SSH.
- shadow/Dockerfile: multi-stage build — nim test-node `main` (mirrors
nim-test-node/Dockerfile_amd64) + Ubuntu 22.04 runner with Shadow v3.3.0
built from source (rustup + cmake/glib deps).
- shadow/k8s-job.yaml: Job pinned to node-05 in zerotesting-shadow, with
seccomp Unconfined + SYS_PTRACE for Shadow's syscall interposition.
- shadow/kaniko-build-job.yaml: in-cluster image build via kaniko on node-04
(avoids 30+ min QEMU emulation on arm64 macs); pushes to
radiken/dst-shadow-nim.
- .dockerignore at repo root: keeps the buildx context lean for the new
root-context Shadow build; per-experiment builds under nim-test-node/ are
unaffected.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: fix kaniko git context branch case
GitHub branch refs are case-sensitive server-side. The branch landed as
Alan/shadow-k8s (capital A, matching team convention), but the manifest
referenced lowercase.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: add python3-yaml to the runner image
topogen.py imports yaml; without python3-yaml the sim aborts at
ModuleNotFoundError before generating shadow.yaml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: build nim test node dynamically for Shadow runner
The k8s test-node build links the nim main fully static (-static -mmusl,
-static-libgcc, -static-libstdc++) for container portability. Shadow won't
accept a static binary because its syscall interposer is LD_PRELOAD-based,
which only hooks dynamically linked ELFs. Smoke test confirmed:
Failed to verify plugin path '/tmp/t2/main'
Caused by: not a dynamically linked ELF
Drop the static linker flags in the Shadow stage of the Dockerfile so the
nim main ends up dynamically linked against the runner image's glibc /
libssl3 / libstdc++. The k8s build (nim-test-node/Dockerfile_amd64) is
unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: split runner image into base + per-experiment test-node
Until now we baked the nim test-node main binary into the Shadow runner
image. That couples the generic Shadow runtime to one specific test node
and means rebuilding the whole image (Rust + Shadow) any time the nim
source changes.
Split into two images:
- radiken/dst-shadow-base: Ubuntu 22.04 + Shadow v3.3.0 + python deps.
No nim binary. Built rarely. shadow/Dockerfile.
- radiken/dst-test-node-shadow: dynamic-linked nim main, two-stage build
with debian:bookworm-slim final stage. nim-test-node/Dockerfile_amd64_shadow.
Per-run Job specs use an init container to copy /node/main out of the
test-node image into an emptyDir, then run shadow from the base image with
the binary + shadow.yaml mounted in. Decouples the Shadow runtime from the
per-experiment binary.
Two kaniko Job manifests (shadow/kaniko-build-job.yaml and
shadow/kaniko-build-test-node-job.yaml), both pinned to node-04.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: bake python3-requests into base for the publisher host
The Shadow runner pod runs traffic_sync.py (mounted via ConfigMap) as a
publisher host inside shadow.yaml. traffic_sync.py uses the requests
library, which isn't in the upstream python3 package. Bake it in once
rather than apt-installing at every run start.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test-node: make storeMetrics scrape interval configurable
storeMetrics (used in Shadow mode to curl /metrics into metrics_pod-N.txt)
was hardcoded to a 5-minute interval. Shadow sims run for seconds-to-minutes
of simulated time, so a single scrape fires at t~=0 (before the mesh forms)
and the run ends before the next one. The captured libp2p_network_bytes
counter is therefore ~0 and useless for bandwidth comparison.
Add METRICS_INTERVAL_S env var (default 300 to preserve k8s behavior).
Shadow runs set it short (e.g. 15s) so the last scrape captures the
post-traffic cumulative byte counts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: drop superseded k8s-job.yaml, fix stale image comment
k8s-job.yaml was the old single-image by-hand runner manifest pointing at the
retired radiken/dst-shadow-nim image; the runner Job is now generated by
10ksim's builders. Also fix a kaniko comment that still named dst-shadow-nim.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix comments
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Zhuolun Li