test: mc2 testing scripts

2026-06-26 12:29:30 +00:00 · 2026-06-24 05:54:40 +02:00 · 2026-06-24 05:54:40 +02:00 · 5447cc39a8
commit 5447cc39a8
parent 6be5fdf424
3 changed files with 711 additions and 0 deletions
--- a/scripts/README.md
+++ b/scripts/README.md
@ -0,0 +1,223 @@
+# Storage Test Scripts
+
+Helpers for running a local Logos Storage node and testing it against the Linode node.
+
+## Remote Node
+
+The Linode node is running as a `systemd` service:
+
+```bash
+ssh storage@172.235.163.25
+systemctl status logos-storage.service
+journalctl -u logos-storage.service -f
+```
+
+Remote node ports:
+
+| Purpose | Address |
+|---|---|
+| P2P TCP | `172.235.163.25:8070` |
+| Discovery UDP | `172.235.163.25:8090` |
+| REST API | `127.0.0.1:8080` on the Linode only |
+
+The REST API is not exposed publicly. Use the SSH tunnel managed by `storage-test.sh`.
+
+## Start A Local Node
+
+Build the local binary first if needed:
+
+```bash
+make -j1 NIMFLAGS="-d:disableMarchNative"
+```
+
+Start a local node in the foreground:
+
+```bash
+scripts/start-local-node.sh
+```
+
+Default local settings:
+
+| Setting | Default |
+|---|---|
+| Binary | `./build/storage` |
+| Data dir | `~/.logos/storage/local-node` |
+| Log level | `info` |
+| P2P TCP | `8071` |
+| Discovery UDP | `8091` |
+| REST API | `127.0.0.1:8080` |
+| Network | `logos.test` |
+
+`info` is a good default log level: it shows startup, networking, and high-level node events without the volume of `debug` or `trace`.
+
+Use `debug` when diagnosing behavior:
+
+```bash
+scripts/start-local-node.sh --log-level debug
+```
+
+Use `trace` only for detailed protocol/debug investigation because it can be noisy:
+
+```bash
+scripts/start-local-node.sh --log-level trace
+```
+
+Show all local-node options:
+
+```bash
+scripts/start-local-node.sh --help
+```
+
+The local node runs in the foreground. Press `Ctrl-C` to stop it.
+
+## Test Helper
+
+Show commands:
+
+```bash
+scripts/storage-test.sh --help
+```
+
+Defaults:
+
+| Setting | Default |
+|---|---|
+| Remote SSH | `storage@172.235.163.25` |
+| Remote API tunnel | `127.0.0.1:18080` |
+| Local API | `127.0.0.1:8080` |
+| CID state file | `~/.logos/storage/test/cids.log` |
+| Generated test files | `~/.logos/storage/test/files/` |
+
+`cids.log` is a simple upload history. Each upload appends the timestamp, target, returned CID, and source file path. It is useful when running frequent tests because you can recover old CIDs and delete them later without scrolling terminal history.
+
+Recover the latest CID from the upload history:
+
+```bash
+scripts/storage-test.sh last-cid
+scripts/storage-test.sh last-cid remote
+```
+
+## SSH Tunnel
+
+Start the tunnel:
+
+```bash
+scripts/storage-test.sh tunnel start
+```
+
+Check it:
+
+```bash
+scripts/storage-test.sh tunnel status
+```
+
+Stop it:
+
+```bash
+scripts/storage-test.sh tunnel stop
+```
+
+You do not have to stop the tunnel after each test. It is safe to leave it running while you are actively testing. Stop it when you are done, when you want to free local port `18080`, or before changing tunnel settings.
+
+Commands that target `remote` start the tunnel automatically if it is not already running.
+
+## Typical Workflow
+
+Terminal 1: start local node and watch logs.
+
+```bash
+scripts/start-local-node.sh --log-level info
+```
+
+Terminal 2: upload random content to the Linode node.
+
+```bash
+CID="$(scripts/storage-test.sh upload-random remote 10M)"
+printf '%s\n' "$CID"
+```
+
+If you prefer copy/paste, you can also run the upload command directly and copy the printed CID:
+
+```bash
+scripts/storage-test.sh upload-random remote 10M
+```
+
+Recover it later from the upload history:
+
+```bash
+CID="$(scripts/storage-test.sh last-cid remote)"
+```
+
+Ask the local node to fetch and store the content from the network:
+
+```bash
+scripts/storage-test.sh fetch-local "$CID" --wait
+```
+
+Or stream the content through the local node without explicitly storing it first:
+
+```bash
+scripts/storage-test.sh stream-local "$CID" /tmp/logos-download.bin
+```
+
+Check local presence:
+
+```bash
+scripts/storage-test.sh exists local "$CID"
+```
+
+List local and remote CIDs:
+
+```bash
+scripts/storage-test.sh list local
+scripts/storage-test.sh list remote
+```
+
+Delete by CID:
+
+```bash
+scripts/storage-test.sh delete local "$CID"
+scripts/storage-test.sh delete remote "$CID"
+```
+
+Delete all local CIDs:
+
+```bash
+scripts/storage-test.sh delete-all local --yes
+```
+
+Delete all remote CIDs:
+
+```bash
+scripts/storage-test.sh delete-all remote --yes
+```
+
+## Useful API Endpoints
+
+| Operation | Endpoint |
+|---|---|
+| Upload | `POST /api/storage/v1/data` |
+| List local content | `GET /api/storage/v1/data` |
+| Delete local content | `DELETE /api/storage/v1/data/{cid}` |
+| Fetch from network into node | `POST /api/storage/v1/data/{cid}/network` |
+| Fetch progress | `GET /api/storage/v1/data/{cid}/network/progress/{downloadId}` |
+| Stream from network | `GET /api/storage/v1/data/{cid}/network/stream` |
+| Local existence check | `GET /api/storage/v1/data/{cid}/exists` |
+| Storage space | `GET /api/storage/v1/space` |
+
+## Cleanup
+
+Stop the local node with `Ctrl-C`.
+
+Stop the SSH tunnel when finished:
+
+```bash
+scripts/storage-test.sh tunnel stop
+```
+
+Remove local test data if desired:
+
+```bash
+rm -rf ~/.logos/storage/local-node
+rm -rf ~/.logos/storage/test
+```
--- a/scripts/start-local-node.sh
+++ b/scripts/start-local-node.sh
@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+BINARY="${STORAGE_BINARY:-${ROOT_DIR}/build/storage}"
+DATA_DIR="${STORAGE_DATA_DIR:-${HOME}/.logos/storage/local-node}"
+LOG_LEVEL="${STORAGE_LOG_LEVEL:-info}"
+LISTEN_PORT="${STORAGE_LISTEN_PORT:-8071}"
+DISC_PORT="${STORAGE_DISC_PORT:-8091}"
+API_BINDADDR="${STORAGE_API_BINDADDR:-127.0.0.1}"
+API_PORT="${STORAGE_API_PORT:-8080}"
+NETWORK="${STORAGE_NETWORK:-logos.test}"
+
+usage() {
+  cat <<EOF
+Usage: $0 [options] [-- extra storage args]
+
+Start a local Logos Storage node in the foreground so logs are visible.
+
+Options:
+  --binary <path>       Storage binary [$BINARY]
+  --data-dir <path>     Data directory [$DATA_DIR]
+  --log-level <level>   Log level: trace, debug, info, notice, warn, error [$LOG_LEVEL]
+  --listen-port <port>  Local libp2p TCP listen port [$LISTEN_PORT]
+  --disc-port <port>    Local discovery UDP port [$DISC_PORT]
+  --api-bindaddr <ip>   REST API bind address [$API_BINDADDR]
+  --api-port <port>     REST API port [$API_PORT]
+  --network <name>      Network preset [$NETWORK]
+  -h, --help            Show this help.
+
+Environment overrides:
+  STORAGE_BINARY, STORAGE_DATA_DIR, STORAGE_LOG_LEVEL, STORAGE_LISTEN_PORT,
+  STORAGE_DISC_PORT, STORAGE_API_BINDADDR, STORAGE_API_PORT, STORAGE_NETWORK
+
+Examples:
+  $0
+  $0 --log-level debug
+  $0 --data-dir /tmp/logos-storage-local --listen-port 8072 --disc-port 8092
+  $0 --log-level trace -- --metrics --metrics-address=127.0.0.1
+
+The node runs in the foreground. Press Ctrl-C to stop it.
+EOF
+}
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --binary)
+      BINARY="${2:-}"
+      shift 2
+      ;;
+    --data-dir)
+      DATA_DIR="${2:-}"
+      shift 2
+      ;;
+    --log-level)
+      LOG_LEVEL="${2:-}"
+      shift 2
+      ;;
+    --listen-port)
+      LISTEN_PORT="${2:-}"
+      shift 2
+      ;;
+    --disc-port)
+      DISC_PORT="${2:-}"
+      shift 2
+      ;;
+    --api-bindaddr)
+      API_BINDADDR="${2:-}"
+      shift 2
+      ;;
+    --api-port)
+      API_PORT="${2:-}"
+      shift 2
+      ;;
+    --network)
+      NETWORK="${2:-}"
+      shift 2
+      ;;
+    -h|--help)
+      usage
+      exit 0
+      ;;
+    --)
+      shift
+      break
+      ;;
+    *)
+      printf 'error: unknown option: %s\n\n' "$1" >&2
+      usage >&2
+      exit 1
+      ;;
+  esac
+done
+
+[[ -n "$BINARY" ]] || { printf 'error: --binary cannot be empty\n' >&2; exit 1; }
+[[ -x "$BINARY" ]] || { printf 'error: storage binary not executable: %s\n' "$BINARY" >&2; exit 1; }
+
+mkdir -p "$DATA_DIR"
+
+printf 'Starting local Logos Storage node\n'
+printf '  binary:      %s\n' "$BINARY"
+printf '  data dir:    %s\n' "$DATA_DIR"
+printf '  log level:   %s\n' "$LOG_LEVEL"
+printf '  listen TCP:  %s\n' "$LISTEN_PORT"
+printf '  discovery:   %s/udp\n' "$DISC_PORT"
+printf '  REST API:    %s:%s\n' "$API_BINDADDR" "$API_PORT"
+printf '  network:     %s\n' "$NETWORK"
+printf '\n'
+
+exec "$BINARY" \
+  --data-dir="$DATA_DIR" \
+  --log-level="$LOG_LEVEL" \
+  --listen-port="$LISTEN_PORT" \
+  --disc-port="$DISC_PORT" \
+  --api-bindaddr="$API_BINDADDR" \
+  --api-port="$API_PORT" \
+  --network="$NETWORK" \
+  "$@"
--- a/scripts/storage-test.sh
+++ b/scripts/storage-test.sh
@ -0,0 +1,369 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REMOTE_SSH_HOST="${REMOTE_SSH_HOST:-storage@172.235.163.25}"
+REMOTE_API_PORT="${REMOTE_API_PORT:-18080}"
+LOCAL_API_PORT="${LOCAL_API_PORT:-8080}"
+TUNNEL_CONTROL_PATH="${TUNNEL_CONTROL_PATH:-/tmp/logos-storage-tunnel-${USER:-user}.ctl}"
+CID_STATE_FILE="${CID_STATE_FILE:-${HOME}/.logos/storage/test/cids.log}"
+TEST_FILES_DIR="${TEST_FILES_DIR:-${HOME}/.logos/storage/test/files}"
+
+LOCAL_API="http://127.0.0.1:${LOCAL_API_PORT}/api/storage/v1"
+REMOTE_API="http://127.0.0.1:${REMOTE_API_PORT}/api/storage/v1"
+
+usage() {
+  cat <<EOF
+Usage: $0 <command> [options]
+
+Environment:
+  REMOTE_SSH_HOST       SSH host for the remote node [$REMOTE_SSH_HOST]
+  REMOTE_API_PORT       Local tunnel port for remote API [$REMOTE_API_PORT]
+  LOCAL_API_PORT        Local node API port [$LOCAL_API_PORT]
+  TUNNEL_CONTROL_PATH   SSH control socket path [$TUNNEL_CONTROL_PATH]
+  CID_STATE_FILE        Upload history log [$CID_STATE_FILE]
+  TEST_FILES_DIR        Default generated test file directory [$TEST_FILES_DIR]
+
+Targets:
+  local                 API at $LOCAL_API
+  remote                API through SSH tunnel at $REMOTE_API
+
+Commands:
+  help
+      Show this help.
+
+  tunnel start|stop|status
+      Manage SSH tunnel to the remote node API.
+
+  make-file <size> [output-file]
+      Create random content with dd. Example: make-file 10M /tmp/logos-10M.bin
+
+  upload <target> <file>
+      Upload a file to target node and print the returned CID. Appends CID to
+      CID_STATE_FILE. Example: upload remote /tmp/logos-10M.bin
+
+  upload-random <target> <size> [--keep]
+      Create a temporary random file, upload it, print CID. Deletes temp file
+      unless --keep is passed. Example: upload-random remote 10M
+
+  last-cid [target]
+      Print the most recent CID from CID_STATE_FILE, optionally filtered by
+      target.
+
+  list <target>
+      List manifest CIDs stored locally by target node.
+
+  delete <target> <cid>
+      Delete CID from target node local storage.
+
+  delete-all <target> --yes
+      Delete every CID returned by list from target node local storage.
+
+  exists <target> <cid>
+      Check whether target node has CID locally.
+
+  space <target>
+      Show target node storage space information.
+
+  peerid <target>
+      Show target node peer ID.
+
+  fetch-local <cid> [--wait]
+      Ask local node to fetch CID from the network. With --wait, poll progress
+      until the background download is inactive or complete.
+
+  stream-local <cid> <output-file>
+      Stream CID from the network through local node into output-file.
+
+Examples:
+  $0 tunnel start
+  $0 upload-random remote 10M
+  $0 fetch-local <CID> --wait
+  $0 stream-local <CID> /tmp/downloaded.bin
+  $0 delete remote <CID>
+  $0 delete-all local --yes
+EOF
+}
+
+die() {
+  printf 'error: %s\n' "$*" >&2
+  exit 1
+}
+
+need() {
+  command -v "$1" >/dev/null 2>&1 || die "missing required command: $1"
+}
+
+check_common_deps() {
+  need curl
+  need jq
+}
+
+tunnel_status() {
+  ssh -S "$TUNNEL_CONTROL_PATH" -O check "$REMOTE_SSH_HOST" >/dev/null 2>&1
+}
+
+tunnel_start() {
+  need ssh
+  if tunnel_status; then
+    printf 'tunnel already running: 127.0.0.1:%s -> %s:127.0.0.1:8080\n' \
+      "$REMOTE_API_PORT" "$REMOTE_SSH_HOST"
+    return 0
+  fi
+
+  ssh \
+    -M \
+    -S "$TUNNEL_CONTROL_PATH" \
+    -fN \
+    -L "127.0.0.1:${REMOTE_API_PORT}:127.0.0.1:8080" \
+    "$REMOTE_SSH_HOST"
+
+  printf 'started tunnel: 127.0.0.1:%s -> %s:127.0.0.1:8080\n' \
+    "$REMOTE_API_PORT" "$REMOTE_SSH_HOST"
+}
+
+tunnel_stop() {
+  need ssh
+  if tunnel_status; then
+    ssh -S "$TUNNEL_CONTROL_PATH" -O exit "$REMOTE_SSH_HOST" >/dev/null
+    printf 'stopped tunnel\n'
+  else
+    printf 'tunnel not running\n'
+  fi
+}
+
+target_api() {
+  case "${1:-}" in
+    local)
+      printf '%s\n' "$LOCAL_API"
+      ;;
+    remote)
+      tunnel_start >/dev/null
+      printf '%s\n' "$REMOTE_API"
+      ;;
+    *)
+      die "target must be 'local' or 'remote'"
+      ;;
+  esac
+}
+
+make_file() {
+  local size="${1:-}"
+  local out="${2:-}"
+  [[ -n "$size" ]] || die 'make-file requires <size>'
+  if [[ -z "$out" ]]; then
+    mkdir -p "$TEST_FILES_DIR"
+    out="${TEST_FILES_DIR}/logos-test-${size}-$(date -u +%Y%m%dT%H%M%SZ).bin"
+  fi
+  dd if=/dev/urandom of="$out" bs="$size" count=1 status=progress
+  printf '%s\n' "$out"
+}
+
+upload_file() {
+  check_common_deps
+  local target="${1:-}"
+  local file="${2:-}"
+  [[ -n "$file" ]] || die 'upload requires <target> <file>'
+  [[ -f "$file" ]] || die "file not found: $file"
+  [[ -z "${3:-}" ]] || die 'upload does not accept extra options'
+
+  local api cid
+  api="$(target_api "$target")"
+  cid="$(curl -fsS \
+    -H 'Content-Type: application/octet-stream' \
+    --data-binary "@${file}" \
+    "${api}/data")"
+
+  printf '%s\n' "$cid"
+  mkdir -p "$(dirname "$CID_STATE_FILE")"
+  printf '%s %s %s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$target" "$cid" "$file" >> "$CID_STATE_FILE"
+}
+
+upload_random() {
+  local target="${1:-}"
+  local size="${2:-}"
+  local keep=false
+  [[ -n "$target" && -n "$size" ]] || die 'upload-random requires <target> <size> [--keep]'
+
+  shift 2
+  while [[ $# -gt 0 ]]; do
+    case "$1" in
+      --keep)
+        keep=true
+        ;;
+      *)
+        die "unknown upload-random option: $1"
+        ;;
+    esac
+    shift
+  done
+
+  local tmp cid
+  tmp="$(mktemp "${TMPDIR:-/tmp}/logos-storage-test.XXXXXX")"
+  dd if=/dev/urandom of="$tmp" bs="$size" count=1 status=progress >&2
+  cid="$(upload_file "$target" "$tmp")"
+  printf '%s\n' "$cid"
+
+  if [[ "$keep" == true ]]; then
+    printf 'kept file: %s\n' "$tmp" >&2
+  else
+    rm -f "$tmp"
+  fi
+}
+
+last_cid() {
+  local target=""
+
+  while [[ $# -gt 0 ]]; do
+    case "$1" in
+      local|remote)
+        [[ -z "$target" ]] || die 'target specified more than once'
+        target="$1"
+        ;;
+      *)
+        die "unknown last-cid option: $1"
+        ;;
+    esac
+    shift
+  done
+
+  [[ -f "$CID_STATE_FILE" ]] || die "CID state file not found: $CID_STATE_FILE"
+
+  local cid
+  if [[ -n "$target" ]]; then
+    cid="$(awk -v target="$target" '$2 == target { cid = $3 } END { print cid }' "$CID_STATE_FILE")"
+  else
+    cid="$(awk 'NF >= 3 { cid = $3 } END { print cid }' "$CID_STATE_FILE")"
+  fi
+
+  [[ -n "$cid" ]] || die 'no matching CID found'
+  printf '%s\n' "$cid"
+}
+
+list_cids() {
+  check_common_deps
+  local api
+  api="$(target_api "${1:-}")"
+  curl -fsS "${api}/data" | jq -r '.content[]?.cid'
+}
+
+delete_cid() {
+  check_common_deps
+  local target="${1:-}"
+  local cid="${2:-}"
+  [[ -n "$cid" ]] || die 'delete requires <target> <cid>'
+  local api
+  api="$(target_api "$target")"
+  curl -fsS -X DELETE "${api}/data/${cid}" >/dev/null
+  printf 'deleted %s from %s\n' "$cid" "$target"
+}
+
+delete_all() {
+  local target="${1:-}"
+  local yes="${2:-}"
+  [[ "$yes" == '--yes' ]] || die 'delete-all requires --yes'
+
+  local cid count=0
+  while IFS= read -r cid; do
+    [[ -n "$cid" ]] || continue
+    delete_cid "$target" "$cid"
+    count=$((count + 1))
+  done < <(list_cids "$target")
+  printf 'deleted %d CID(s) from %s\n' "$count" "$target"
+}
+
+simple_get() {
+  check_common_deps
+  local target="${1:-}"
+  local path="${2:-}"
+  local api
+  api="$(target_api "$target")"
+  curl -fsS "${api}/${path}"
+  printf '\n'
+}
+
+exists_cid() {
+  local target="${1:-}"
+  local cid="${2:-}"
+  [[ -n "$cid" ]] || die 'exists requires <target> <cid>'
+  simple_get "$target" "data/${cid}/exists"
+}
+
+fetch_local() {
+  check_common_deps
+  local cid="${1:-}"
+  local wait="${2:-}"
+  [[ -n "$cid" ]] || die 'fetch-local requires <cid> [--wait]'
+  [[ -z "$wait" || "$wait" == '--wait' ]] || die 'only supported option is --wait'
+
+  local response download_id
+  response="$(curl -fsS -X POST "${LOCAL_API}/data/${cid}/network")"
+  printf '%s\n' "$response" | jq .
+
+  if [[ "$wait" != '--wait' ]]; then
+    return 0
+  fi
+
+  download_id="$(printf '%s\n' "$response" | jq -r '.downloadId // empty')"
+  [[ -n "$download_id" ]] || die 'response did not contain downloadId'
+
+  while true; do
+    local progress active received total
+    progress="$(curl -fsS "${LOCAL_API}/data/${cid}/network/progress/${download_id}")"
+    printf '%s\n' "$progress" | jq .
+    active="$(printf '%s\n' "$progress" | jq -r '.active')"
+    received="$(printf '%s\n' "$progress" | jq -r '.received // 0')"
+    total="$(printf '%s\n' "$progress" | jq -r '.total // 0')"
+    if [[ "$active" != 'true' || ( "$total" != '0' && "$received" == "$total" ) ]]; then
+      break
+    fi
+    sleep 2
+  done
+}
+
+stream_local() {
+  local cid="${1:-}"
+  local out="${2:-}"
+  [[ -n "$cid" && -n "$out" ]] || die 'stream-local requires <cid> <output-file>'
+  curl -fL "${LOCAL_API}/data/${cid}/network/stream" -o "$out"
+  printf '%s\n' "$out"
+}
+
+cmd="${1:-help}"
+shift || true
+
+case "$cmd" in
+  help|--help|-h)
+    usage
+    ;;
+  tunnel)
+    case "${1:-}" in
+      start) tunnel_start ;;
+      stop) tunnel_stop ;;
+      status)
+        if tunnel_status; then
+          printf 'tunnel running\n'
+        else
+          printf 'tunnel stopped\n'
+          exit 1
+        fi
+        ;;
+      *) die 'usage: tunnel start|stop|status' ;;
+    esac
+    ;;
+  make-file) make_file "$@" ;;
+  upload) upload_file "$@" ;;
+  upload-random) upload_random "$@" ;;
+  last-cid) last_cid "$@" ;;
+  list) list_cids "$@" ;;
+  delete) delete_cid "$@" ;;
+  delete-all) delete_all "$@" ;;
+  exists) exists_cid "$@" ;;
+  space) simple_get "${1:-}" 'space' ;;
+  peerid) simple_get "${1:-}" 'peerid' ;;
+  fetch-local) fetch_local "$@" ;;
+  stream-local) stream_local "$@" ;;
+  *)
+    usage >&2
+    exit 1
+    ;;
+esac