mirror of
https://github.com/logos-storage/logos-storage-nim.git
synced 2026-07-25 02:53:17 +00:00
chore: record context in AGENTS.md about storage_test and libstorage_cpp
This commit is contained in:
Marcin Czenko
parent
ebe5777c88
commit
95a0e96244
167
tools/libstorage-cpp/AGENTS.md
Normal file
167
tools/libstorage-cpp/AGENTS.md
Normal file
@ -0,0 +1,167 @@
|
||||
# Libstorage C++ Tooling Notes
|
||||
|
||||
This folder contains small C++ tools built on top of `library/libstorage.h`.
|
||||
|
||||
## Components
|
||||
|
||||
- `storage_client.hpp`: shared wrapper declarations and result structs.
|
||||
- `storage_client.cpp`: shared wrapper implementation around the async libstorage C API.
|
||||
- `storage_lib_cli.cpp`: one-shot command-line tool that starts a libstorage node, runs one command, and exits.
|
||||
- `storage_lib.cpp`: long-running libstorage daemon with Unix socket IPC.
|
||||
- `storage_lib_ctl.cpp`: control client that sends one IPC command to `storage_lib`.
|
||||
- `Makefile`: builds `storage_lib_cli`, `storage_lib`, and `storage_lib_ctl` against `../../build/libstorage.so`.
|
||||
- `README.md`: operator-facing usage notes.
|
||||
|
||||
`tools/storage-test/` consumes the daemon target through `storage_lib_ctl`.
|
||||
|
||||
## Build Notes
|
||||
|
||||
Build libstorage first. On the current AMD Ryzen AI 9 HX 370 machine, use:
|
||||
|
||||
```bash
|
||||
make libstorage NIMFLAGS="-d:disableMarchNative"
|
||||
```
|
||||
|
||||
This avoids the known GCC/secp256k1 x86_64 asm failure caused by `-march=native`.
|
||||
|
||||
Then build the C++ tools:
|
||||
|
||||
```bash
|
||||
cd tools/libstorage-cpp
|
||||
make
|
||||
```
|
||||
|
||||
The Makefile links explicitly against `../../build/libstorage.so`. If only `libstorage.a` exists, build the shared library first with `make libstorage`.
|
||||
|
||||
## StorageClient Behavior
|
||||
|
||||
`StorageClient` wraps the async libstorage C API with blocking helper methods.
|
||||
|
||||
Important callback behavior:
|
||||
|
||||
- `RET_PROGRESS` is intermediate and must not complete a wait.
|
||||
- `RET_OK` and `RET_ERR` are terminal.
|
||||
- Callbacks run on libstorage worker context; keep callback logic fast and non-blocking.
|
||||
|
||||
Timeout behavior:
|
||||
|
||||
- Positive `timeout_ms` waits up to that duration.
|
||||
- `timeout_ms == 0` waits indefinitely.
|
||||
|
||||
Follow the existing pattern of returning structured results instead of throwing. Keep the C++ wrapper small and direct.
|
||||
|
||||
## CLI vs Daemon Responsibilities
|
||||
|
||||
`storage_lib_cli` is for one-shot checks and manual experiments.
|
||||
|
||||
`storage_lib` is a primitive daemon that keeps one libstorage node alive across commands.
|
||||
|
||||
Keep daemon commands low-level:
|
||||
|
||||
- `info`
|
||||
- `version`
|
||||
- `revision`
|
||||
- `repo`
|
||||
- `peer-id`
|
||||
- `spr`
|
||||
- `debug`
|
||||
- `metrics`
|
||||
- `list`
|
||||
- `space`
|
||||
- `upload`
|
||||
- `download`
|
||||
- `exists`
|
||||
- `delete`
|
||||
- `fetch`
|
||||
- `manifest`
|
||||
- `connect`
|
||||
- `shutdown`
|
||||
|
||||
Do not add scenario commands like `roundtrip`, `many`, or cross-target tests to the daemon unless explicitly requested. Put scenario orchestration in `tools/storage-test/storage-test.sh` so the same scenario can run against REST targets and the libstorage daemon target.
|
||||
|
||||
## Daemon IPC
|
||||
|
||||
`storage_lib` listens on a Unix domain socket.
|
||||
|
||||
Protocol:
|
||||
|
||||
- One client connection per command.
|
||||
- Request is one whitespace-separated line.
|
||||
- Response is one JSON line.
|
||||
- Success shape: `{"ok":true,"result":"..."}`.
|
||||
- Failure shape: `{"ok":false,"error":"..."}`.
|
||||
|
||||
The protocol currently does not support paths or arguments containing spaces. If that becomes important, prefer moving to JSON requests rather than adding ad-hoc escaping.
|
||||
|
||||
`storage_lib_ctl` socket resolution order:
|
||||
|
||||
1. `--socket <path>`
|
||||
2. `STORAGE_LIB_SOCKET`
|
||||
3. `~/.logos/storage/libstorage/storage_lib.sock`
|
||||
|
||||
`storage_lib_ctl` should return nonzero when the daemon responds with `"ok":false`.
|
||||
|
||||
File paths sent over IPC are resolved by the daemon process. Callers that run from a different working directory, such as `tools/storage-test/storage-test.sh`, should send absolute paths.
|
||||
|
||||
## Daemon Options
|
||||
|
||||
Current daemon options include:
|
||||
|
||||
- `--data-dir`
|
||||
- `--socket`
|
||||
- `--log-level`
|
||||
- `--listen-port`
|
||||
- `--disc-port`
|
||||
- `--network`
|
||||
- `--chunk-size`
|
||||
- `--timeout-ms`
|
||||
|
||||
Use a distinct data directory from any running standard storage node to avoid datastore locking conflicts.
|
||||
|
||||
Default libstorage daemon paths used by storage-test tooling:
|
||||
|
||||
- Data dir: `~/.logos/storage/libstorage/node`
|
||||
- Socket: `~/.logos/storage/libstorage/storage_lib.sock`
|
||||
|
||||
## Verification
|
||||
|
||||
Lightweight build and smoke-check flow:
|
||||
|
||||
```bash
|
||||
make libstorage NIMFLAGS="-d:disableMarchNative"
|
||||
cd tools/libstorage-cpp
|
||||
make
|
||||
```
|
||||
|
||||
Run daemon from repo root or with absolute paths:
|
||||
|
||||
```bash
|
||||
tools/libstorage-cpp/storage_lib --data-dir /tmp/storage-lib-data --socket /tmp/storage-lib.sock --log-level FATAL --timeout-ms 0
|
||||
```
|
||||
|
||||
In another shell:
|
||||
|
||||
```bash
|
||||
tools/libstorage-cpp/storage_lib_ctl --socket /tmp/storage-lib.sock info
|
||||
tools/libstorage-cpp/storage_lib_ctl --socket /tmp/storage-lib.sock peer-id
|
||||
tools/libstorage-cpp/storage_lib_ctl --socket /tmp/storage-lib.sock shutdown
|
||||
```
|
||||
|
||||
Storage-test integration smoke checks:
|
||||
|
||||
```bash
|
||||
tools/storage-test/start-local-node.sh --client lib --data-dir /tmp/storage-lib-data --socket /tmp/storage-lib.sock --log-level FATAL --timeout-ms 0
|
||||
STORAGE_LIB_SOCKET=/tmp/storage-lib.sock tools/storage-test/storage-test.sh lib peerid
|
||||
STORAGE_LIB_SOCKET=/tmp/storage-lib.sock tools/libstorage-cpp/storage_lib_ctl shutdown
|
||||
```
|
||||
|
||||
Full `tools/storage-test/storage-test.sh lib test` contacts the configured remote Linode and performs real uploads/downloads. Run it deliberately.
|
||||
|
||||
## Style Guidance
|
||||
|
||||
- Keep the wrapper minimal and boring.
|
||||
- Prefer small local functions over broad abstractions.
|
||||
- Do not add backward-compatible command aliases unless there is an explicit need.
|
||||
- Keep scenario logic out of `storage_lib`.
|
||||
- Preserve JSON responses for daemon IPC so shell scripts can parse success/failure reliably.
|
||||
- Avoid blocking work in libstorage callbacks.
|
||||
189
tools/storage-test/AGENTS.md
Normal file
189
tools/storage-test/AGENTS.md
Normal file
@ -0,0 +1,189 @@
|
||||
# Storage Test Tooling Notes
|
||||
|
||||
This folder contains local/Linode test helpers for comparing the standard REST node with the libstorage C++ daemon target.
|
||||
|
||||
## Current Layout
|
||||
|
||||
- `start-local-node.sh`: foreground launcher for either the standard storage binary or the libstorage daemon.
|
||||
- `storage-test.sh`: target-first test helper for local REST, remote REST, and libstorage daemon targets.
|
||||
- `README.md`: operator-facing usage guide.
|
||||
- `SETUP_STORAGE_NODE.md`: Linode/headless node setup recipe.
|
||||
|
||||
Related C++ libstorage tools live in `../libstorage-cpp/`:
|
||||
|
||||
- `storage_client.hpp` / `storage_client.cpp`: shared C++ wrapper around `library/libstorage.h`.
|
||||
- `storage_lib_cli`: one-shot libstorage CLI.
|
||||
- `storage_lib`: daemon-like libstorage process using Unix socket IPC.
|
||||
- `storage_lib_ctl`: control client that sends one command to `storage_lib`.
|
||||
|
||||
## Build Notes
|
||||
|
||||
On the current AMD Ryzen AI 9 HX 370 machine, local Nim builds should use:
|
||||
|
||||
```bash
|
||||
make NIMFLAGS="-d:disableMarchNative"
|
||||
make libstorage NIMFLAGS="-d:disableMarchNative"
|
||||
```
|
||||
|
||||
This avoids the known GCC/secp256k1 x86_64 asm failure caused by `-march=native`.
|
||||
|
||||
Build libstorage C++ tools with:
|
||||
|
||||
```bash
|
||||
cd tools/libstorage-cpp
|
||||
make
|
||||
```
|
||||
|
||||
Generated C++ binaries, daemon data dirs, sockets, logs, and report files are ignored/cleaned by the relevant Makefile or `.gitignore` entries.
|
||||
|
||||
## Launcher Model
|
||||
|
||||
`start-local-node.sh` supports two clients:
|
||||
|
||||
```bash
|
||||
tools/storage-test/start-local-node.sh --client storage
|
||||
tools/storage-test/start-local-node.sh --client lib
|
||||
```
|
||||
|
||||
Shared config between both clients:
|
||||
|
||||
- `--data-dir`
|
||||
- `--log-level`
|
||||
- `--listen-port`
|
||||
- `--disc-port`
|
||||
- `--network`
|
||||
|
||||
Standard storage-only config:
|
||||
|
||||
- `--binary`
|
||||
- `--api-bindaddr`
|
||||
- `--api-port`
|
||||
|
||||
Lib daemon-only config:
|
||||
|
||||
- `--lib-binary`
|
||||
- `--socket`
|
||||
- `--timeout-ms`
|
||||
- `--chunk-size`
|
||||
|
||||
Default data dirs intentionally differ to avoid datastore locking conflicts:
|
||||
|
||||
- Standard REST node: `~/.logos/storage/local-node`
|
||||
- Libstorage daemon: `~/.logos/storage/libstorage/node`
|
||||
|
||||
## storage-test.sh Command Model
|
||||
|
||||
The script is target-first. Do not reintroduce command-first compatibility unless explicitly requested.
|
||||
|
||||
```bash
|
||||
tools/storage-test/storage-test.sh <target> <command> [args]
|
||||
```
|
||||
|
||||
Targets:
|
||||
|
||||
- `local`: local standard REST node at `127.0.0.1:${LOCAL_API_PORT}`.
|
||||
- `remote`: Linode REST node through SSH tunnel at `127.0.0.1:${REMOTE_API_PORT}`.
|
||||
- `lib`: local `storage_lib` daemon through `storage_lib_ctl` and Unix socket.
|
||||
|
||||
Global commands remain targetless:
|
||||
|
||||
- `help`
|
||||
- `tunnel start|stop|status`
|
||||
- `make-file`
|
||||
- `last-cid [target]`
|
||||
|
||||
Common target commands:
|
||||
|
||||
- `upload <file>`
|
||||
- `upload-random <size> [--keep]`
|
||||
- `download <cid> <output-file> [--local]`
|
||||
- `fetch <cid> [--wait]`
|
||||
- `list`
|
||||
- `delete <cid>`
|
||||
- `delete-all --yes`
|
||||
- `exists <cid>`
|
||||
- `space`
|
||||
- `peerid`
|
||||
- `test`
|
||||
|
||||
Lib-only target commands:
|
||||
|
||||
- `spr`
|
||||
- `debug`
|
||||
- `manifest <cid>`
|
||||
- `connect <peer-id> [addr...]`
|
||||
|
||||
## Lib Target Details
|
||||
|
||||
`storage-test.sh lib` invokes `storage_lib_ctl`. Socket resolution for `storage_lib_ctl` is:
|
||||
|
||||
1. `--socket <path>`
|
||||
2. `STORAGE_LIB_SOCKET`
|
||||
3. `~/.logos/storage/libstorage/storage_lib.sock`
|
||||
|
||||
`storage-test.sh` passes `--socket "$STORAGE_LIB_SOCKET"` explicitly.
|
||||
|
||||
The daemon IPC protocol is intentionally simple:
|
||||
|
||||
- Request: one whitespace-separated command line per connection.
|
||||
- Response: one JSON line like `{"ok":true,"result":"..."}` or `{"ok":false,"error":"..."}`.
|
||||
- Paths with spaces are not supported yet.
|
||||
|
||||
Important: lib daemon file paths are resolved in the daemon process working directory. `storage-test.sh` converts upload/download paths to absolute paths before sending them to `storage_lib_ctl`.
|
||||
|
||||
`--timeout-ms 0` means wait indefinitely in libstorage C++ tooling.
|
||||
|
||||
## Test Scenario
|
||||
|
||||
`storage-test.sh <local|lib> test` currently runs the remote-to-local scenario:
|
||||
|
||||
1. Generate random files with sizes from `TEST_FILE_SIZES`.
|
||||
2. Upload each file to the `remote` Linode REST node.
|
||||
3. Download through selected target: `local` REST or `lib` daemon.
|
||||
4. Validate SHA-256 of source and downloaded file.
|
||||
5. Delete involved CIDs from the selected local target and `remote`.
|
||||
6. Remove temp workspace unless `TEST_KEEP_FILES=1`.
|
||||
7. Write a Markdown report in the caller's current directory: `./report-YYYY-MM-DD_HH-MM-SS.md`.
|
||||
|
||||
Defaults:
|
||||
|
||||
- `TEST_FILE_SIZES="4K 1M 10M"`
|
||||
- `TEST_KEEP_FILES=0`
|
||||
- Max per-file test size is 10 MB.
|
||||
|
||||
The report includes timestamps, duration, target, file size list, workspace path, per-file CID/hash/path details, and cleanup results. `report-*.md` is ignored by the root `.gitignore`.
|
||||
|
||||
## Verification Commands
|
||||
|
||||
Use lightweight checks before/after edits:
|
||||
|
||||
```bash
|
||||
bash -n tools/storage-test/storage-test.sh
|
||||
bash -n tools/storage-test/start-local-node.sh
|
||||
tools/storage-test/storage-test.sh --help
|
||||
tools/storage-test/start-local-node.sh --help
|
||||
```
|
||||
|
||||
For lib daemon smoke tests:
|
||||
|
||||
```bash
|
||||
cd tools/libstorage-cpp
|
||||
make storage_lib storage_lib_ctl
|
||||
cd ../..
|
||||
tools/storage-test/start-local-node.sh --client lib --data-dir /tmp/storage-lib-data --socket /tmp/storage-lib.sock --log-level FATAL --timeout-ms 0
|
||||
```
|
||||
|
||||
In another shell:
|
||||
|
||||
```bash
|
||||
STORAGE_LIB_SOCKET=/tmp/storage-lib.sock tools/storage-test/storage-test.sh lib peerid
|
||||
STORAGE_LIB_SOCKET=/tmp/storage-lib.sock tools/libstorage-cpp/storage_lib_ctl shutdown
|
||||
```
|
||||
|
||||
Full `local test` and `lib test` contact the Linode remote and perform real uploads/downloads; run them deliberately.
|
||||
|
||||
## Future Work
|
||||
|
||||
- Add reverse scenario: selected local/lib target uploads, remote downloads/fetches, validates hashes, and cleans up.
|
||||
- Consider changing daemon IPC to JSON requests if argument quoting or paths with spaces become important.
|
||||
- Keep long-running/scenario behavior in `storage-test.sh`, not in the daemon, so scenarios can run against all targets.
|
||||
Loading…
x
Reference in New Issue
Block a user