logos-storage-module/doctests/storage-module-runtime.test.yaml

name: "Running This Storage Module Against logoscore"
output: storage-module-runtime.md
release: ""

intro: |
  `logos-storage-module` is a Logos `core` module that wraps the
  [libstorage](https://github.com/logos-storage/logos-storage-nim) C library to
  run a decentralised storage node — upload, download, and data-management
  operations over a libp2p network. This doc-test exercises **this**
  storage-module commit end-to-end through the headless `logoscore` runtime:

  1. Build the `logoscore` CLI and the `lgpm` local package manager from their
     published flakes. `logoscore` is the headless frontend for `logos-liblogos`,
     so building it brings in the whole module-runtime stack (`logos_host`,
     `liblogos_core`, the IPC layer).
  2. Build **this** storage module as an installable `.lgx` package straight from
     its own flake's `#lgx` output, **pinned to the commit under test** — so the
     module you run is built from exactly what is checked out here, not the latest
     published release.
  3. Install the `.lgx` into a `./modules` directory with `lgpm`.
  4. Start `logoscore` in daemon mode (`-D`), load `storage_module`, introspect
     it with `module-info`, and drive a real node lifecycle: start it from a
     config, read its identity, upload a local file,
     download it back, and stop it again — verifying the module actually runs and
     round-trips real values through libstorage.

  Because the module is built from the commit under test and then loaded and called
  through a real `logoscore` daemon, a green run is real evidence that this change
  keeps the storage module loadable and callable.

what_you_build: "This `storage_module`, packaged as `.lgx`, installed with `lgpm`, and driven through a `logoscore` daemon — node start, a local file upload, a download round-trip, and shutdown."

what_you_learn:
  - How to build the `logoscore` runtime and the `lgpm` package manager from their flakes
  - How a module's flake exposes a ready-to-install `.lgx` via its `#lgx` output
  - How to install an `.lgx` into a modules directory with `lgpm`
  - How to start the `logoscore` daemon, load a module, introspect it, and call its methods
  - How to configure, start, exercise, and stop a libstorage node headlessly
  - How to round-trip a file through `uploadUrl` and `downloadToUrl`
  - How to shut the daemon down and confirm it has exited

prerequisites:
  - |
    **Nix** with flakes enabled. Install from [nixos.org](https://nixos.org/download.html), then enable flakes:

    ```bash
    mkdir -p ~/.config/nix
    echo 'experimental-features = nix-command flakes' >> ~/.config/nix/nix.conf
    ```

    Verify: `nix flake --help >/dev/null 2>&1 && echo "Flakes enabled"`
  - "**A Linux or macOS machine.**"
  - "**`jq`** on your `PATH` — used to pull the uploaded CID out of the `manifests` JSON. Verify: `jq --version`"

sections:
  - title: "Build logoscore"
    step: true
    text: |
      Build the `logoscore` CLI from its published flake. The result is symlinked to
      `./logos/`. `logoscore` is the headless frontend for `logos-liblogos`, so this
      one build brings in the whole module-runtime stack the daemon needs.
    steps:
      - title: "Build the CLI"
        run: "nix build 'github:logos-co/logos-logoscore-cli' --out-link ./logos"
        code_block: |
          nix build 'github:logos-co/logos-logoscore-cli' --out-link ./logos
        check_file: "logos/bin/logoscore"
        post_text: |
          The build produces `logos/bin/logoscore` plus bundled runtime libraries
          and a `logos/modules/` directory containing the built-in
          `capability_module` (required for the auth handshake when loading
          modules).

  - title: "Build the lgpm package manager"
    step: true
    text: |
      `lgpm` installs `.lgx` packages into a modules directory and scans what is
      installed. Build it from the `logos-package-manager` flake and link it as
      `./lgpm`.
    steps:
      - title: "Build lgpm"
        run: "nix build 'github:logos-co/logos-package-manager#cli' -o lgpm"
        check_file: "lgpm/bin/lgpm"
        post_text: "The executable is at `./lgpm/bin/lgpm`."

  - title: "Build and install this storage module"
    step: true
    text: |
      Build **this** storage module's `.lgx` straight from its flake's `#lgx`
      output and install it into a local `./modules` directory with `lgpm`. Every
      module built with
      [`logos-module-builder`](https://github.com/logos-co/logos-module-builder)
      exposes a ready-to-install `#lgx`.

    steps:
      - title: "Build the module's .lgx"
        text: |
          Build the `#lgx` output and link it as `./storage-lgx`. (This compiles
          the module and its libstorage dependency through Nix, so the first build
          is slow.)
        run: "nix build 'github:logos-co/logos-storage-module{release}#lgx' -o storage-lgx"
        code_block: |
          # From inside the clone this is simply: nix build '.#lgx'
          nix build 'github:logos-co/logos-storage-module{release}#lgx' -o storage-lgx
        post_text: "The `.lgx` package is now under `./storage-lgx/`:"
        extra_run:
          run: "ls storage-lgx/*.lgx"

      - title: "Seed the modules directory with the bundled capability module"
        text: |
          `storage_module` is loaded through the host's capability layer, so the
          modules directory also needs the `capability_module` that ships with
          `logoscore`. Copy it across first.
        run: |
          mkdir -p modules
          cp -RL ./logos/modules/. ./modules/
        check_file: "modules/capability_module/manifest.json"

      - title: "Install the .lgx with lgpm"
        text: |
          Install the freshly-built package into `./modules`. `storage_module` is
          a `core` module, so it goes to `--modules-dir`. The package is unsigned
          (a local dev build), so we pass `--allow-unsigned`.
        run: "./lgpm/bin/lgpm --modules-dir ./modules --allow-unsigned install --file storage-lgx/*.lgx"
        expect_contains:
          - "Installed to:"

      - title: "Confirm the install"
        text: "Scan the directory and confirm the module landed:"
        run: "./lgpm/bin/lgpm --modules-dir ./modules list"
        expect_contains:
          - "storage_module"
        check_file: "modules/storage_module/manifest.json"

  - title: "Run the daemon and call the module"
    step: true
    text: |
      Start `logoscore` in daemon mode pointed at `./modules`, then use the client
      subcommands to load `storage_module`, introspect it, and drive a node
      lifecycle. Daemon output is captured in `logs.txt`.
    steps:
      - title: "Start the daemon"
        text: |
          Start logoscore in daemon mode in the background, capturing output to
          `logs.txt`:
        run: "sh -c './logos/bin/logoscore -D -m ./modules > logs.txt 2>&1 &'"
        code_block: "logoscore -D -m ./modules > logs.txt &"
        post_text: |
          The `-D` flag starts the daemon. The client subcommands below connect to
          this running process via the config written under `~/.logoscore/`.

      - run: "sleep 3"

      - title: "Inspect the startup log"
        text: "Review the daemon's startup output:"
        run: "cat logs.txt"

      - title: "Check daemon status"
        text: "Verify the daemon is running:"
        run: "./logos/bin/logoscore status"
        code_block: "logoscore status"

      - title: "List discovered modules"
        text: "`storage_module` should be visible in the scan directory:"
        run: "./logos/bin/logoscore list-modules"
        code_block: "logoscore list-modules"
        expect_contains:
          - "storage_module"

      - title: "Load the module"
        text: "Load `storage_module` into the running daemon:"
        run: "./logos/bin/logoscore load-module storage_module"
        code_block: "logoscore load-module storage_module"
        expect_contains:
          - "storage_module"

      - title: "Confirm the module is loaded"
        text: |
          Re-run `status`; the module that was `not_loaded` before now reports
          `loaded`:
        run: "./logos/bin/logoscore status"
        code_block: "logoscore status"
        expect_contains:
          - "storage_module"
          - '"status":"loaded"'

      - title: "Introspect the module with module-info"
        text: |
          `module-info` lists the `Q_INVOKABLE` methods the module exposes — the
          same methods you can `call`:
        run: "./logos/bin/logoscore module-info storage_module"
        code_block: "logoscore module-info storage_module"
        expect_contains:
          - "storage_module"
          - "version"
          - "uploadUrl"
          - "space"

      - title: "Write the node configuration"
        text: |
          `storage_module.start` takes a JSON configuration string. We keep it
          minimal and let libstorage fill in sensible defaults for everything else
          (listen addresses, repo kind, quota, discovery) — that already yields a
          fully isolated, working node:

          - `data-dir` — the node's on-disk repo. We use an **absolute** path and
            create the directory first, because in daemon mode the module runs as
            its own process whose working directory may differ from this one, and
            libstorage opens the repo at exactly the path given.
          - `log-level` / `log-file` — send the node's logs to a file in that dir.

          The directory is created and the config written in one step (using
          `$(pwd)` so the paths are absolute):
        run: |
          mkdir -p "$(pwd)/storage-data"
          cat > config.json <<EOF
          {
              "data-dir": "$(pwd)/storage-data",
              "log-level": "DEBUG",
              "log-file": "$(pwd)/storage-data/storage.log",
              "nat": "none"
          }
          EOF
        code_block: |
          mkdir -p "$(pwd)/storage-data"
          cat > config.json <<EOF
          {
              "data-dir": "$(pwd)/storage-data",
              "log-level": "DEBUG",
              "log-file": "$(pwd)/storage-data/storage.log",
              "nat": "none"
          }
          EOF
        check_file: "config.json"

      - title: "Start the node"
        text: |
          `start` creates and configures the libstorage node from the JSON config,
          then brings the libp2p node online. The `@config.json` syntax loads the
          file's contents as the argument. The return value confirms the start
          command was accepted; the real outcome is delivered as a `storageStart`
          event in the daemon log. The remaining calls all run against this
          started node.
        run: "./logos/bin/logoscore call storage_module start @config.json"
        code_block: "logoscore call storage_module start @config.json"
        expect_contains:
          - '"result":true'

      - title: "Wait for the node to come up"
        text: |
          Starting a libp2p node takes a moment. Give it a few seconds before
          querying the node, then inspect the log for the `storageStart` event:
        run: "sleep 5"

      - run: "cat logs.txt"
        code_block: "cat logs.txt"
        post_text: |
          The emitted `storageStart` event carries `{ "success": true, ... }`.

      - title: "Inspect the node with debug"
        text: |
          `debug` returns a JSON object describing the running node. It contains
          a lot of useful information like the peerId, spr ...etc.
          We assert with `jq` that the node's `id` and `spr` came back non-empty:
        run: |
          ./logos/bin/logoscore call storage_module debug \
            | jq -e '(.result.value.id // "") != "" and (.result.value.spr // "") != ""'
        code_block: "logoscore call storage_module debug"

      - title: "List manifests (empty baseline)"
        text: |
          `manifests` lists everything stored locally. On a fresh node this is an
          empty array — we'll call it again after an upload to see it change:
        run: "./logos/bin/logoscore call storage_module manifests"
        code_block: "logoscore call storage_module manifests"
        expect_contains:
          - '"result"'

      - title: "Create the file to upload"
        text: |
          Create a small file in the working directory. We upload it in the next
          step:
        file:
          path: hello.txt
          content: |
            Hello from the logos-storage-module doc-test.

      - title: "Upload a local file"
        text: |
          Upload the file with `uploadUrl`. It takes an **absolute** path (the
          daemon resolves it from its own working directory) and a chunk size in
          bytes, and returns a session ID immediately; the upload itself runs in
          the background. On a fresh `fs` node with no peers the blocks are stored
          locally, so this is a real, fully-offline round-trip. We assert on
          `"success":true` so a rejected upload fails here rather than silently
          later:
        run: './logos/bin/logoscore call storage_module uploadUrl "$(pwd)/hello.txt" 65536'
        code_block: 'logoscore call storage_module uploadUrl "$(pwd)/hello.txt" 65536'
        expect_contains:
          - '"success":true'

      - title: "Wait for the upload to complete"
        text: |
          The upload runs in the background, so give it a few seconds before
          checking that the file landed, then inspect the log for the
          `storageUploadDone` event and its CID:
        run: "sleep 1"

      - run: "cat logs.txt"
        code_block: "cat logs.txt"
        post_text: |
          The emitted `storageUploadDone` event carries the new content's `cid` —
          proof the file was chunked, stored, and a manifest written.

      - title: "List manifests (now populated)"
        text: |
          Call `manifests` again. The uploaded file now appears as a stored
          manifest — we assert on presence rather than the (non-deterministic) CID:
        run: "./logos/bin/logoscore call storage_module manifests"
        code_block: "logoscore call storage_module manifests"
        expect_contains:
          - '"result"'

      - title: "Capture the uploaded CID"
        text: |
          `downloadToUrl` needs the content's CID. We pull it out of the first
          `manifests` entry with `jq` and save it to `cid.txt`. Each command runs in
          its own shell, so we pass the value to the next step through a file rather
          than a shell variable:
        run: |
          ./logos/bin/logoscore call storage_module manifests \
            | jq -er '.result.value[0].cid' > cid.txt
          cat cid.txt
        code_block: |
          logoscore call storage_module manifests \
            | jq -er '.result.value[0].cid' > cid.txt
        check_file: "cid.txt"

      - title: "Download the file back"
        text: |
          `downloadToUrl` fetches the content for a CID and writes it to a local
          file. It takes the CID, an **absolute** destination path, a `local` flag,
          and a chunk size in bytes. We pass `local` as `true`: the upload stored
          the blocks in this node's own repo, so the download reads them straight
          back with no network. Like `uploadUrl` it is asynchronous and returns a
          session ID immediately; completion arrives as a `storageDownloadDone`
          event in the log.
        run: './logos/bin/logoscore call storage_module downloadToUrl "$(cat cid.txt)" "$(pwd)/downloaded.txt" true 65536'
        code_block: 'logoscore call storage_module downloadToUrl "$(cat cid.txt)" "$(pwd)/downloaded.txt" true 65536'
        expect_contains:
          - '"result"'

      - title: "Wait for the download to complete"
        text: |
          The download runs in the background, so give it a few seconds, then
          inspect the log for the `storageDownloadDone` event:
        run: "sleep 1"

      - run: "cat logs.txt"
        code_block: "cat logs.txt"
        post_text: |
          The emitted `storageDownloadDone` event carries `{ "success": true, ... }`.

      - title: "Verify the round-trip"
        text: |
          Read the downloaded file back. Its contents are identical to the file we
          uploaded — proof the upload/download round-trip preserved the data
          exactly:
        run: 'cat "$(pwd)/downloaded.txt"'
        code_block: "cat downloaded.txt"
        expect_contains:
          - "Hello from the logos-storage-module doc-test."

      - title: "Check the content exists locally"
        text: |
          `exists` reports whether the content for a CID is in local storage. After
          the upload it returns `true`:
        run: './logos/bin/logoscore call storage_module exists "$(cat cid.txt)"'
        code_block: 'logoscore call storage_module exists "$(cat cid.txt)"'
        expect_contains:
          - '"value":true'

      - title: "Watch for the manifest event"
        text: |
          Because `downloadManifest` is asynchronous, we need to watch for the
          `storageDownloadManifestDone` event before triggering the fetch.
        run: "sh -c './logos/bin/logoscore watch storage_module --event storageDownloadManifestDone --json > manifest-event.txt 2>&1 & echo $! > watch.pid'"
        code_block: "logoscore watch storage_module --event storageDownloadManifestDone --json"

      - run: "sleep 2"

      - title: "Fetch the manifest"
        text: |
          Call `downloadManifest`. It returns immediately; the real result is
          delivered to the watcher started above:
        run: './logos/bin/logoscore call storage_module downloadManifest "$(cat cid.txt)"'
        code_block: 'logoscore call storage_module downloadManifest "$(cat cid.txt)"'
        expect_contains:
          - '"result"'

      - title: "Confirm the manifest event arrived"
        text: |
          Give the event a moment to land, then inspect what the watcher
          captured. The `storageDownloadManifestDone` event carries
          `{ "success": true, "cid": ..., "manifest": { ... } }` — the metadata
          describing how the content is stored:
        run: |
          sleep 3
          kill "$(cat watch.pid)" 2>/dev/null || true
          cat manifest-event.txt
        code_block: "cat manifest-event.txt"
        expect_contains:
          - "storageDownloadManifestDone"
          - "hello.txt"

      - title: "Remove the content"
        text: |
          `remove` deletes the content for a CID from local storage. The delete
          may touch the network and can take a while, so it runs in the
          background: the call returns immediately and the outcome arrives as a
          `storageRemoveDone` event in the log.
        run: './logos/bin/logoscore call storage_module remove "$(cat cid.txt)"'
        code_block: 'logoscore call storage_module remove "$(cat cid.txt)"'
        expect_contains:
          - '"success":true'

      - title: "Wait for the removal to complete"
        text: |
          The removal runs in the background, so give it a moment, then inspect
          the log for the `storageRemoveDone` event:
        run: "sleep 1"

      - title: "Confirm the content is gone"
        text: "Call `exists` again; with the content removed it now returns `false`:"
        run: './logos/bin/logoscore call storage_module exists "$(cat cid.txt)"'
        code_block: 'logoscore call storage_module exists "$(cat cid.txt)"'
        expect_contains:
          - '"value":false'

      - title: "Stop the node"
        text: |
          `stop` shuts the libp2p node down and releases the libstorage context.
          Like `start` it is asynchronous; the return confirms the stop command
          was sent, and a `storageStop` event follows in the log. The node can be
          started again later by calling `start @config.json`.
        run: "./logos/bin/logoscore call storage_module stop"
        code_block: "logoscore call storage_module stop"
        expect_contains:
          - '"success":true'

      - run: "sleep 2"

      - title: "Stop the daemon"
        text: "Shut the daemon down cleanly:"
        run: "./logos/bin/logoscore stop"
        code_block: "logoscore stop"
        post_text: |
          The daemon removes its state file and exits.

      - run: "sleep 2"

      - title: "Confirm the daemon has stopped"
        text: |
          With no daemon running, the client reports `not_running` and exits
          non-zero, so we add `|| true` to let the doc-test assert on the output:
        run: "./logos/bin/logoscore status || true"
        code_block: "logoscore status"
        expect_contains:
          - '"status":"not_running"'