From 95a0e962440d8cbe4ef808d35987d88e9bdb8c3c Mon Sep 17 00:00:00 2001 From: Marcin Czenko Date: Thu, 25 Jun 2026 12:40:20 +0200 Subject: [PATCH] chore: record context in AGENTS.md about storage_test and libstorage_cpp --- tools/libstorage-cpp/AGENTS.md | 167 +++++++++++++++++++++++++++++ tools/storage-test/AGENTS.md | 189 +++++++++++++++++++++++++++++++++ 2 files changed, 356 insertions(+) create mode 100644 tools/libstorage-cpp/AGENTS.md create mode 100644 tools/storage-test/AGENTS.md diff --git a/tools/libstorage-cpp/AGENTS.md b/tools/libstorage-cpp/AGENTS.md new file mode 100644 index 00000000..1ce40386 --- /dev/null +++ b/tools/libstorage-cpp/AGENTS.md @@ -0,0 +1,167 @@ +# Libstorage C++ Tooling Notes + +This folder contains small C++ tools built on top of `library/libstorage.h`. + +## Components + +- `storage_client.hpp`: shared wrapper declarations and result structs. +- `storage_client.cpp`: shared wrapper implementation around the async libstorage C API. +- `storage_lib_cli.cpp`: one-shot command-line tool that starts a libstorage node, runs one command, and exits. +- `storage_lib.cpp`: long-running libstorage daemon with Unix socket IPC. +- `storage_lib_ctl.cpp`: control client that sends one IPC command to `storage_lib`. +- `Makefile`: builds `storage_lib_cli`, `storage_lib`, and `storage_lib_ctl` against `../../build/libstorage.so`. +- `README.md`: operator-facing usage notes. + +`tools/storage-test/` consumes the daemon target through `storage_lib_ctl`. + +## Build Notes + +Build libstorage first. On the current AMD Ryzen AI 9 HX 370 machine, use: + +```bash +make libstorage NIMFLAGS="-d:disableMarchNative" +``` + +This avoids the known GCC/secp256k1 x86_64 asm failure caused by `-march=native`. + +Then build the C++ tools: + +```bash +cd tools/libstorage-cpp +make +``` + +The Makefile links explicitly against `../../build/libstorage.so`. If only `libstorage.a` exists, build the shared library first with `make libstorage`. + +## StorageClient Behavior + +`StorageClient` wraps the async libstorage C API with blocking helper methods. + +Important callback behavior: + +- `RET_PROGRESS` is intermediate and must not complete a wait. +- `RET_OK` and `RET_ERR` are terminal. +- Callbacks run on libstorage worker context; keep callback logic fast and non-blocking. + +Timeout behavior: + +- Positive `timeout_ms` waits up to that duration. +- `timeout_ms == 0` waits indefinitely. + +Follow the existing pattern of returning structured results instead of throwing. Keep the C++ wrapper small and direct. + +## CLI vs Daemon Responsibilities + +`storage_lib_cli` is for one-shot checks and manual experiments. + +`storage_lib` is a primitive daemon that keeps one libstorage node alive across commands. + +Keep daemon commands low-level: + +- `info` +- `version` +- `revision` +- `repo` +- `peer-id` +- `spr` +- `debug` +- `metrics` +- `list` +- `space` +- `upload` +- `download` +- `exists` +- `delete` +- `fetch` +- `manifest` +- `connect` +- `shutdown` + +Do not add scenario commands like `roundtrip`, `many`, or cross-target tests to the daemon unless explicitly requested. Put scenario orchestration in `tools/storage-test/storage-test.sh` so the same scenario can run against REST targets and the libstorage daemon target. + +## Daemon IPC + +`storage_lib` listens on a Unix domain socket. + +Protocol: + +- One client connection per command. +- Request is one whitespace-separated line. +- Response is one JSON line. +- Success shape: `{"ok":true,"result":"..."}`. +- Failure shape: `{"ok":false,"error":"..."}`. + +The protocol currently does not support paths or arguments containing spaces. If that becomes important, prefer moving to JSON requests rather than adding ad-hoc escaping. + +`storage_lib_ctl` socket resolution order: + +1. `--socket ` +2. `STORAGE_LIB_SOCKET` +3. `~/.logos/storage/libstorage/storage_lib.sock` + +`storage_lib_ctl` should return nonzero when the daemon responds with `"ok":false`. + +File paths sent over IPC are resolved by the daemon process. Callers that run from a different working directory, such as `tools/storage-test/storage-test.sh`, should send absolute paths. + +## Daemon Options + +Current daemon options include: + +- `--data-dir` +- `--socket` +- `--log-level` +- `--listen-port` +- `--disc-port` +- `--network` +- `--chunk-size` +- `--timeout-ms` + +Use a distinct data directory from any running standard storage node to avoid datastore locking conflicts. + +Default libstorage daemon paths used by storage-test tooling: + +- Data dir: `~/.logos/storage/libstorage/node` +- Socket: `~/.logos/storage/libstorage/storage_lib.sock` + +## Verification + +Lightweight build and smoke-check flow: + +```bash +make libstorage NIMFLAGS="-d:disableMarchNative" +cd tools/libstorage-cpp +make +``` + +Run daemon from repo root or with absolute paths: + +```bash +tools/libstorage-cpp/storage_lib --data-dir /tmp/storage-lib-data --socket /tmp/storage-lib.sock --log-level FATAL --timeout-ms 0 +``` + +In another shell: + +```bash +tools/libstorage-cpp/storage_lib_ctl --socket /tmp/storage-lib.sock info +tools/libstorage-cpp/storage_lib_ctl --socket /tmp/storage-lib.sock peer-id +tools/libstorage-cpp/storage_lib_ctl --socket /tmp/storage-lib.sock shutdown +``` + +Storage-test integration smoke checks: + +```bash +tools/storage-test/start-local-node.sh --client lib --data-dir /tmp/storage-lib-data --socket /tmp/storage-lib.sock --log-level FATAL --timeout-ms 0 +STORAGE_LIB_SOCKET=/tmp/storage-lib.sock tools/storage-test/storage-test.sh lib peerid +STORAGE_LIB_SOCKET=/tmp/storage-lib.sock tools/libstorage-cpp/storage_lib_ctl shutdown +``` + +Full `tools/storage-test/storage-test.sh lib test` contacts the configured remote Linode and performs real uploads/downloads. Run it deliberately. + +## Style Guidance + +- Keep the wrapper minimal and boring. +- Prefer small local functions over broad abstractions. +- Do not add backward-compatible command aliases unless there is an explicit need. +- Keep scenario logic out of `storage_lib`. +- Preserve JSON responses for daemon IPC so shell scripts can parse success/failure reliably. +- Avoid blocking work in libstorage callbacks. diff --git a/tools/storage-test/AGENTS.md b/tools/storage-test/AGENTS.md new file mode 100644 index 00000000..b9290650 --- /dev/null +++ b/tools/storage-test/AGENTS.md @@ -0,0 +1,189 @@ +# Storage Test Tooling Notes + +This folder contains local/Linode test helpers for comparing the standard REST node with the libstorage C++ daemon target. + +## Current Layout + +- `start-local-node.sh`: foreground launcher for either the standard storage binary or the libstorage daemon. +- `storage-test.sh`: target-first test helper for local REST, remote REST, and libstorage daemon targets. +- `README.md`: operator-facing usage guide. +- `SETUP_STORAGE_NODE.md`: Linode/headless node setup recipe. + +Related C++ libstorage tools live in `../libstorage-cpp/`: + +- `storage_client.hpp` / `storage_client.cpp`: shared C++ wrapper around `library/libstorage.h`. +- `storage_lib_cli`: one-shot libstorage CLI. +- `storage_lib`: daemon-like libstorage process using Unix socket IPC. +- `storage_lib_ctl`: control client that sends one command to `storage_lib`. + +## Build Notes + +On the current AMD Ryzen AI 9 HX 370 machine, local Nim builds should use: + +```bash +make NIMFLAGS="-d:disableMarchNative" +make libstorage NIMFLAGS="-d:disableMarchNative" +``` + +This avoids the known GCC/secp256k1 x86_64 asm failure caused by `-march=native`. + +Build libstorage C++ tools with: + +```bash +cd tools/libstorage-cpp +make +``` + +Generated C++ binaries, daemon data dirs, sockets, logs, and report files are ignored/cleaned by the relevant Makefile or `.gitignore` entries. + +## Launcher Model + +`start-local-node.sh` supports two clients: + +```bash +tools/storage-test/start-local-node.sh --client storage +tools/storage-test/start-local-node.sh --client lib +``` + +Shared config between both clients: + +- `--data-dir` +- `--log-level` +- `--listen-port` +- `--disc-port` +- `--network` + +Standard storage-only config: + +- `--binary` +- `--api-bindaddr` +- `--api-port` + +Lib daemon-only config: + +- `--lib-binary` +- `--socket` +- `--timeout-ms` +- `--chunk-size` + +Default data dirs intentionally differ to avoid datastore locking conflicts: + +- Standard REST node: `~/.logos/storage/local-node` +- Libstorage daemon: `~/.logos/storage/libstorage/node` + +## storage-test.sh Command Model + +The script is target-first. Do not reintroduce command-first compatibility unless explicitly requested. + +```bash +tools/storage-test/storage-test.sh [args] +``` + +Targets: + +- `local`: local standard REST node at `127.0.0.1:${LOCAL_API_PORT}`. +- `remote`: Linode REST node through SSH tunnel at `127.0.0.1:${REMOTE_API_PORT}`. +- `lib`: local `storage_lib` daemon through `storage_lib_ctl` and Unix socket. + +Global commands remain targetless: + +- `help` +- `tunnel start|stop|status` +- `make-file` +- `last-cid [target]` + +Common target commands: + +- `upload ` +- `upload-random [--keep]` +- `download [--local]` +- `fetch [--wait]` +- `list` +- `delete ` +- `delete-all --yes` +- `exists ` +- `space` +- `peerid` +- `test` + +Lib-only target commands: + +- `spr` +- `debug` +- `manifest ` +- `connect [addr...]` + +## Lib Target Details + +`storage-test.sh lib` invokes `storage_lib_ctl`. Socket resolution for `storage_lib_ctl` is: + +1. `--socket ` +2. `STORAGE_LIB_SOCKET` +3. `~/.logos/storage/libstorage/storage_lib.sock` + +`storage-test.sh` passes `--socket "$STORAGE_LIB_SOCKET"` explicitly. + +The daemon IPC protocol is intentionally simple: + +- Request: one whitespace-separated command line per connection. +- Response: one JSON line like `{"ok":true,"result":"..."}` or `{"ok":false,"error":"..."}`. +- Paths with spaces are not supported yet. + +Important: lib daemon file paths are resolved in the daemon process working directory. `storage-test.sh` converts upload/download paths to absolute paths before sending them to `storage_lib_ctl`. + +`--timeout-ms 0` means wait indefinitely in libstorage C++ tooling. + +## Test Scenario + +`storage-test.sh test` currently runs the remote-to-local scenario: + +1. Generate random files with sizes from `TEST_FILE_SIZES`. +2. Upload each file to the `remote` Linode REST node. +3. Download through selected target: `local` REST or `lib` daemon. +4. Validate SHA-256 of source and downloaded file. +5. Delete involved CIDs from the selected local target and `remote`. +6. Remove temp workspace unless `TEST_KEEP_FILES=1`. +7. Write a Markdown report in the caller's current directory: `./report-YYYY-MM-DD_HH-MM-SS.md`. + +Defaults: + +- `TEST_FILE_SIZES="4K 1M 10M"` +- `TEST_KEEP_FILES=0` +- Max per-file test size is 10 MB. + +The report includes timestamps, duration, target, file size list, workspace path, per-file CID/hash/path details, and cleanup results. `report-*.md` is ignored by the root `.gitignore`. + +## Verification Commands + +Use lightweight checks before/after edits: + +```bash +bash -n tools/storage-test/storage-test.sh +bash -n tools/storage-test/start-local-node.sh +tools/storage-test/storage-test.sh --help +tools/storage-test/start-local-node.sh --help +``` + +For lib daemon smoke tests: + +```bash +cd tools/libstorage-cpp +make storage_lib storage_lib_ctl +cd ../.. +tools/storage-test/start-local-node.sh --client lib --data-dir /tmp/storage-lib-data --socket /tmp/storage-lib.sock --log-level FATAL --timeout-ms 0 +``` + +In another shell: + +```bash +STORAGE_LIB_SOCKET=/tmp/storage-lib.sock tools/storage-test/storage-test.sh lib peerid +STORAGE_LIB_SOCKET=/tmp/storage-lib.sock tools/libstorage-cpp/storage_lib_ctl shutdown +``` + +Full `local test` and `lib test` contact the Linode remote and perform real uploads/downloads; run them deliberately. + +## Future Work + +- Add reverse scenario: selected local/lib target uploads, remote downloads/fetches, validates hashes, and cleans up. +- Consider changing daemon IPC to JSON requests if argument quoting or paths with spaces become important. +- Keep long-running/scenario behavior in `storage-test.sh`, not in the daemon, so scenarios can run against all targets.