Listening on 0.0.0.0 makes libp2p announce every local address with loopback first,
so peers spend a dial attempt reaching themselves. That cost nothing while dials
were unbounded; nim-libp2p now bounds a dial and shares one deadline across the
whole address list, so it can exhaust the budget before the real address is tried.
The metrics and publish servers still bind 0.0.0.0, so in-pod probing is unchanged.
(cherry picked from commit 137cf9dad4)
quic closes a connection after 30s of silence and there is nothing to gossip about
before the first message, so an idle cold start dropped every connection outside the
mesh. Ping now covers every connection at 20s, under the timeout, and stops once
messages arrive because gossip reaches each peer every few seconds by then.
Replace the flat STARTSLEEP (default 180s) wait before a normal node dials
the bootstrap with a small per-pod delay (pod index * STARTUP_JITTER_STEP_MS,
default 50ms).
* shadow: add k8s-based runner setup
Adds a containerized Shadow runner so the existing single-machine prototype
(shadow/{topogen.py,run.sh,*.awk}) can execute on a lab worker without SSH.
- shadow/Dockerfile: multi-stage build — nim test-node `main` (mirrors
nim-test-node/Dockerfile_amd64) + Ubuntu 22.04 runner with Shadow v3.3.0
built from source (rustup + cmake/glib deps).
- shadow/k8s-job.yaml: Job pinned to node-05 in zerotesting-shadow, with
seccomp Unconfined + SYS_PTRACE for Shadow's syscall interposition.
- shadow/kaniko-build-job.yaml: in-cluster image build via kaniko on node-04
(avoids 30+ min QEMU emulation on arm64 macs); pushes to
radiken/dst-shadow-nim.
- .dockerignore at repo root: keeps the buildx context lean for the new
root-context Shadow build; per-experiment builds under nim-test-node/ are
unaffected.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: fix kaniko git context branch case
GitHub branch refs are case-sensitive server-side. The branch landed as
Alan/shadow-k8s (capital A, matching team convention), but the manifest
referenced lowercase.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: add python3-yaml to the runner image
topogen.py imports yaml; without python3-yaml the sim aborts at
ModuleNotFoundError before generating shadow.yaml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: build nim test node dynamically for Shadow runner
The k8s test-node build links the nim main fully static (-static -mmusl,
-static-libgcc, -static-libstdc++) for container portability. Shadow won't
accept a static binary because its syscall interposer is LD_PRELOAD-based,
which only hooks dynamically linked ELFs. Smoke test confirmed:
Failed to verify plugin path '/tmp/t2/main'
Caused by: not a dynamically linked ELF
Drop the static linker flags in the Shadow stage of the Dockerfile so the
nim main ends up dynamically linked against the runner image's glibc /
libssl3 / libstdc++. The k8s build (nim-test-node/Dockerfile_amd64) is
unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: split runner image into base + per-experiment test-node
Until now we baked the nim test-node main binary into the Shadow runner
image. That couples the generic Shadow runtime to one specific test node
and means rebuilding the whole image (Rust + Shadow) any time the nim
source changes.
Split into two images:
- radiken/dst-shadow-base: Ubuntu 22.04 + Shadow v3.3.0 + python deps.
No nim binary. Built rarely. shadow/Dockerfile.
- radiken/dst-test-node-shadow: dynamic-linked nim main, two-stage build
with debian:bookworm-slim final stage. nim-test-node/Dockerfile_amd64_shadow.
Per-run Job specs use an init container to copy /node/main out of the
test-node image into an emptyDir, then run shadow from the base image with
the binary + shadow.yaml mounted in. Decouples the Shadow runtime from the
per-experiment binary.
Two kaniko Job manifests (shadow/kaniko-build-job.yaml and
shadow/kaniko-build-test-node-job.yaml), both pinned to node-04.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: bake python3-requests into base for the publisher host
The Shadow runner pod runs traffic_sync.py (mounted via ConfigMap) as a
publisher host inside shadow.yaml. traffic_sync.py uses the requests
library, which isn't in the upstream python3 package. Bake it in once
rather than apt-installing at every run start.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test-node: make storeMetrics scrape interval configurable
storeMetrics (used in Shadow mode to curl /metrics into metrics_pod-N.txt)
was hardcoded to a 5-minute interval. Shadow sims run for seconds-to-minutes
of simulated time, so a single scrape fires at t~=0 (before the mesh forms)
and the run ends before the next one. The captured libp2p_network_bytes
counter is therefore ~0 and useless for bandwidth comparison.
Add METRICS_INTERVAL_S env var (default 300 to preserve k8s behavior).
Shadow runs set it short (e.g. 15s) so the last scrape captures the
post-traffic cumulative byte counts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: drop superseded k8s-job.yaml, fix stale image comment
k8s-job.yaml was the old single-image by-hand runner manifest pointing at the
retired radiken/dst-shadow-nim image; the runner Job is now generated by
10ksim's builders. Also fix a kaniko comment that still named dst-shadow-nim.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix comments
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Alberto Soutullo