mirror of
https://github.com/vacp2p/dst-libp2p-test-node.git
synced 2026-08-27 12:51:12 +00:00
Zhuolun LiandClaude Opus 4.7
ec1d9a9b62
* shadow: add k8s-based runner setup
Adds a containerized Shadow runner so the existing single-machine prototype
(shadow/{topogen.py,run.sh,*.awk}) can execute on a lab worker without SSH.
- shadow/Dockerfile: multi-stage build — nim test-node `main` (mirrors
nim-test-node/Dockerfile_amd64) + Ubuntu 22.04 runner with Shadow v3.3.0
built from source (rustup + cmake/glib deps).
- shadow/k8s-job.yaml: Job pinned to node-05 in zerotesting-shadow, with
seccomp Unconfined + SYS_PTRACE for Shadow's syscall interposition.
- shadow/kaniko-build-job.yaml: in-cluster image build via kaniko on node-04
(avoids 30+ min QEMU emulation on arm64 macs); pushes to
radiken/dst-shadow-nim.
- .dockerignore at repo root: keeps the buildx context lean for the new
root-context Shadow build; per-experiment builds under nim-test-node/ are
unaffected.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: fix kaniko git context branch case
GitHub branch refs are case-sensitive server-side. The branch landed as
Alan/shadow-k8s (capital A, matching team convention), but the manifest
referenced lowercase.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: add python3-yaml to the runner image
topogen.py imports yaml; without python3-yaml the sim aborts at
ModuleNotFoundError before generating shadow.yaml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: build nim test node dynamically for Shadow runner
The k8s test-node build links the nim main fully static (-static -mmusl,
-static-libgcc, -static-libstdc++) for container portability. Shadow won't
accept a static binary because its syscall interposer is LD_PRELOAD-based,
which only hooks dynamically linked ELFs. Smoke test confirmed:
Failed to verify plugin path '/tmp/t2/main'
Caused by: not a dynamically linked ELF
Drop the static linker flags in the Shadow stage of the Dockerfile so the
nim main ends up dynamically linked against the runner image's glibc /
libssl3 / libstdc++. The k8s build (nim-test-node/Dockerfile_amd64) is
unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: split runner image into base + per-experiment test-node
Until now we baked the nim test-node main binary into the Shadow runner
image. That couples the generic Shadow runtime to one specific test node
and means rebuilding the whole image (Rust + Shadow) any time the nim
source changes.
Split into two images:
- radiken/dst-shadow-base: Ubuntu 22.04 + Shadow v3.3.0 + python deps.
No nim binary. Built rarely. shadow/Dockerfile.
- radiken/dst-test-node-shadow: dynamic-linked nim main, two-stage build
with debian:bookworm-slim final stage. nim-test-node/Dockerfile_amd64_shadow.
Per-run Job specs use an init container to copy /node/main out of the
test-node image into an emptyDir, then run shadow from the base image with
the binary + shadow.yaml mounted in. Decouples the Shadow runtime from the
per-experiment binary.
Two kaniko Job manifests (shadow/kaniko-build-job.yaml and
shadow/kaniko-build-test-node-job.yaml), both pinned to node-04.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: bake python3-requests into base for the publisher host
The Shadow runner pod runs traffic_sync.py (mounted via ConfigMap) as a
publisher host inside shadow.yaml. traffic_sync.py uses the requests
library, which isn't in the upstream python3 package. Bake it in once
rather than apt-installing at every run start.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test-node: make storeMetrics scrape interval configurable
storeMetrics (used in Shadow mode to curl /metrics into metrics_pod-N.txt)
was hardcoded to a 5-minute interval. Shadow sims run for seconds-to-minutes
of simulated time, so a single scrape fires at t~=0 (before the mesh forms)
and the run ends before the next one. The captured libp2p_network_bytes
counter is therefore ~0 and useless for bandwidth comparison.
Add METRICS_INTERVAL_S env var (default 300 to preserve k8s behavior).
Shadow runs set it short (e.g. 15s) so the last scrape captures the
post-traffic cumulative byte counts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* shadow: drop superseded k8s-job.yaml, fix stale image comment
k8s-job.yaml was the old single-image by-hand runner manifest pointing at the
retired radiken/dst-shadow-nim image; the runner Job is now generated by
10ksim's builders. Also fix a kaniko comment that still named dst-shadow-nim.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix comments
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
libp2p GossipSub Test Nodes
This repository contains nim-libp2p, go-libp2p and rust-libp2p based implementations of GossipSub test nodes that can run in both Kubernetes (K8s) and Shadow simulator environments.
Overview
These test nodes are designed for performance testing and evaluation of libp2p's GossipSub protocol under various network conditions. The implementations support:
- Configurable test node parameters
- Kubernetes and shadow deployment
- Multi-transport support (Mplex, Yamux, QUIC)
- Mix protocol support (for nim-libp2p)
- Prometheus metrics collection
- HTTP-based message injection for dynamic test configuration
Node/test-specific details are available in corresponding directories.
How It Works
Peers use random (ID-based) peer selection in the shadow simulator, and a DNS-based peer discovery service in K8s to make target number of connections. After required connections are made, each peer exposes an HTTP endpoint for receiving publish commands from a remote message injector.
Every publisher embeds a timestamp before publishing messages. If fragmentation is required, the publisher breaks messages into desired fragments and embeds fragment numbers. The receiver accumulates fragments and logs the elapsed time.
All peers expose Prometheus metrics for detailed insights.
nim-libp2p Test Node
nim-libp2p test node supports mplex, yamux, and quic transports. It also supports mix protocol for improved anonymity. Use the MOUNTSMIX environment variable to mount mix protocol on any test node. Mix nodes also need to know the available number of mix nodes NUMMIX, and need access to their configuration settings using FILEPATH.
Environment Variables
PEERS— Number of peers in the network (default:100)CONNECTTO— Target number of peers to dial (default:10)MUXER— Stream multiplexer: supports QUIC, yamux and mplex (default:yamux)FRAGMENTS— Number of message fragments (default:1)SHADOWENV— Whether running in shadow simulator (default:false)SELFTRIGGER— Self trigger in GossipSub parameters (default:true)SERVICE— K8s service name for peer discovery (default:nimp2p-service)MAXCONNECTIONS— Maximum number of peers to connect with (default:250)MOUNTSMIX— Running as a mix node (default:false)USESMIX— Participate in mix network (default:false)NUMMIX— Number of mix network peers (default:0)MIXD— Number of mix tunnels to traverse (default:4)FILEPATH— Path of mix node configurations (default:./)
Building
git clone https://github.com/vacp2p/dst-libp2p-test-node.git
cd dst-libp2p-test-node
# Build Docker image
docker build -t nim-libp2p-test .
docker tag nim-libp2p-test user/refactored-test-node:vx.x
# Extract binary for Shadow
docker create --name temp nim-libp2p-test
docker cp temp:/node/main shadow/main
docker rm temp
Deployment
Please see K8s deployment utilities for K8s deployments. See shadow directory for shadow simulator experiments.