Files
Zhuolun LiandClaude Opus 4.7 ec1d9a9b62 Shadow runner base image with dynamic test-node build and configurable metrics interval (#28)
* shadow: add k8s-based runner setup

Adds a containerized Shadow runner so the existing single-machine prototype
(shadow/{topogen.py,run.sh,*.awk}) can execute on a lab worker without SSH.

- shadow/Dockerfile: multi-stage build — nim test-node `main` (mirrors
  nim-test-node/Dockerfile_amd64) + Ubuntu 22.04 runner with Shadow v3.3.0
  built from source (rustup + cmake/glib deps).
- shadow/k8s-job.yaml: Job pinned to node-05 in zerotesting-shadow, with
  seccomp Unconfined + SYS_PTRACE for Shadow's syscall interposition.
- shadow/kaniko-build-job.yaml: in-cluster image build via kaniko on node-04
  (avoids 30+ min QEMU emulation on arm64 macs); pushes to
  radiken/dst-shadow-nim.
- .dockerignore at repo root: keeps the buildx context lean for the new
  root-context Shadow build; per-experiment builds under nim-test-node/ are
  unaffected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* shadow: fix kaniko git context branch case

GitHub branch refs are case-sensitive server-side. The branch landed as
Alan/shadow-k8s (capital A, matching team convention), but the manifest
referenced lowercase.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* shadow: add python3-yaml to the runner image

topogen.py imports yaml; without python3-yaml the sim aborts at
ModuleNotFoundError before generating shadow.yaml.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* shadow: build nim test node dynamically for Shadow runner

The k8s test-node build links the nim main fully static (-static -mmusl,
-static-libgcc, -static-libstdc++) for container portability. Shadow won't
accept a static binary because its syscall interposer is LD_PRELOAD-based,
which only hooks dynamically linked ELFs. Smoke test confirmed:

  Failed to verify plugin path '/tmp/t2/main'
  Caused by: not a dynamically linked ELF

Drop the static linker flags in the Shadow stage of the Dockerfile so the
nim main ends up dynamically linked against the runner image's glibc /
libssl3 / libstdc++. The k8s build (nim-test-node/Dockerfile_amd64) is
unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* shadow: split runner image into base + per-experiment test-node

Until now we baked the nim test-node main binary into the Shadow runner
image. That couples the generic Shadow runtime to one specific test node
and means rebuilding the whole image (Rust + Shadow) any time the nim
source changes.

Split into two images:

- radiken/dst-shadow-base: Ubuntu 22.04 + Shadow v3.3.0 + python deps.
  No nim binary. Built rarely. shadow/Dockerfile.
- radiken/dst-test-node-shadow: dynamic-linked nim main, two-stage build
  with debian:bookworm-slim final stage. nim-test-node/Dockerfile_amd64_shadow.

Per-run Job specs use an init container to copy /node/main out of the
test-node image into an emptyDir, then run shadow from the base image with
the binary + shadow.yaml mounted in. Decouples the Shadow runtime from the
per-experiment binary.

Two kaniko Job manifests (shadow/kaniko-build-job.yaml and
shadow/kaniko-build-test-node-job.yaml), both pinned to node-04.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* shadow: bake python3-requests into base for the publisher host

The Shadow runner pod runs traffic_sync.py (mounted via ConfigMap) as a
publisher host inside shadow.yaml. traffic_sync.py uses the requests
library, which isn't in the upstream python3 package. Bake it in once
rather than apt-installing at every run start.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test-node: make storeMetrics scrape interval configurable

storeMetrics (used in Shadow mode to curl /metrics into metrics_pod-N.txt)
was hardcoded to a 5-minute interval. Shadow sims run for seconds-to-minutes
of simulated time, so a single scrape fires at t~=0 (before the mesh forms)
and the run ends before the next one. The captured libp2p_network_bytes
counter is therefore ~0 and useless for bandwidth comparison.

Add METRICS_INTERVAL_S env var (default 300 to preserve k8s behavior).
Shadow runs set it short (e.g. 15s) so the last scrape captures the
post-traffic cumulative byte counts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* shadow: drop superseded k8s-job.yaml, fix stale image comment

k8s-job.yaml was the old single-image by-hand runner manifest pointing at the
retired radiken/dst-shadow-nim image; the runner Job is now generated by
10ksim's builders. Also fix a kaniko comment that still named dst-shadow-nim.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix comments

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-04 10:09:43 +01:00
..

dst-libp2p-test-node for shadow simulator

This directory contains necessary files for running the dst-libp2p-test-nod using the shadow simulator. This can either be done by compiling the main.nim file directly or by extracting the executable from a Docker image. Currently we support nim-libp2p only. Support for other implementations will be added soon. This directory includes utilities for:

  • Creating a shadow.yaml file for describing the network topology.
  • Running the shadow simulation with configurable parameters.
  • Listing latency and bandwidth utilization.
  • Collecting Prometheus metrics from simulated nodes

Getting the Executable Test Node

You have two options for obtaining the main executable:

Extract the pre-built executable from the Docker image without building it locally.

# Pull the Docker image
docker pull <docker_image>

# Create a temporary container from the image
docker create --name temp-container <docker_image>

# Copy the 'main' binary from the container to this directory
docker cp temp-container:/node/main ./main

# Make it executable
chmod +x ./main

# Remove the temporary container
docker rm temp-container

Option 2: Build from Source

Build the main executable directly from the source.

#Navigate to the parent directory
cd ../nim-test-node/

#Compile with required flags
nim c -d:chronicles_colors=None --threads:on -d:metrics -d:libp2p_network_protocols_metrics -d:release main

#Move the executable to the shadow directory
mv ./nim-test-node/main ../shadow/

Running Simulations

Use the run.sh script to create and run simulations with custo parameters

Example

#usage parameters
./run.sh <runs> <nodes> <Message_size> <num_fragment> <num_publishers>
        <min_bandwidth> <max_bandwidth> <min_latency> <max_latency>
        <anchor_stages> <packet_loss> <publisher_id> <publisher_rotation> <inter_message_delay>

#example
./run.sh 1 2000 15000 1 10 50 150 40 130 5 0.0 6 1 1000

#This will output latency/bandwidth and store prometheus metrics (and peer data) in shadow.data directory

Parameters

Parameter Description Example Value
runs Number of times to run a simulation 1
nodes Total number of nodes in the network 2000
message_size Message size in bytes 15000
num_fragments Number of message fragments (1-10). 1 means not fragmented 1
num_publishers Number of messages to publish 10
min_bandwidth Minimum peer bandwidth in Mbps 50
max_bandwidth Maximum peer bandwidth in Mbps 150
min_latency Minimum link latency in ms 40
max_latency Maximum link latency in ms 130
anchor_stages Bandwidth/latency variations 5
packet_loss Packet loss rate (0.0-1.0) 0.0
publisher_id Starting publisher node ID 6
publisher_rotation Rotate publishers (0=no, 1=yes) 1
inter_message_delay Delay between messages in ms 1000

Supported Implementation

= Available 🚧 = In progress

Name Node Latest tested version
nim-libp2p v1.12.0
go-libp2p 🚧 -
rust-libp2p 🚧 -