logos-storage-go/.github/copilot-instructions.md

Codex Client Go Library - AI Agent Guide

Project Overview

A lightweight Go client library for interacting with Codex decentralized storage nodes. The project provides:

• Core Library: communities/codex_client.go - HTTP client wrapping Codex REST API
• CLI Tools: cmd/upload/ and cmd/download/ - Command-line utilities for file operations

Quickstart (build • run • test)

• Build CLIs: go build -o bin/codex-upload ./cmd/upload && go build -o bin/codex-download ./cmd/download
• Upload: ./bin/codex-upload -file test-data.bin -host localhost -port 8080
• Download: ./bin/codex-download -cid <CID> -file out.bin -host localhost -port 8080
• Unit tests: go test -v ./communities
• Integration test: go test -v -tags=integration ./communities -run Integration

Architecture & Design Patterns

Client Structure

The CodexClient type in communities/codex_client.go is the central abstraction:

• Wraps Codex HTTP API (/api/codex/v1/data/*) with Go-idiomatic methods
• Supports both network downloads (/network/stream) and local downloads (direct CID access)
• Context-aware operations for cancellation support (DownloadWithContext, TriggerDownloadWithContext)
• All uploads use application/octet-stream with Content-Disposition header for filenames

Data Flow Pattern

1. Upload: file → io.Reader → HTTP POST → CID string
2. Download: CID string → HTTP GET → io.Writer → file
3. Network vs Local: Network download uses /network/stream endpoint (retrieves from DHT), local uses direct /data/{cid}

Key Implementation Details

• Streaming with Cancellation: Custom copyWithContext (64KB buffer) checks context cancellation between chunks
• Default Timeout: 60s for HTTP operations (configurable via SetRequestTimeout)
• Error Handling: Errors wrapped with %w; HTTP errors include status code and body
• CID Format: CIDv1 + multibase base58btc (prefix z), using the raw codec (Dv). Example: zDvZRwzm...

CID format cheatsheet

• Multibase prefix z = base58btc; Dv denotes CIDv1 + raw codec
• CIDs commonly start with zDv... in Codex (e.g., zDvZRwzmRigWseNB...)
• Upload responses may include a trailing newline; client trims via strings.TrimSpace

CLI Commands & Usage

Build Commands

# Build upload tool
go build -o codex-upload ./cmd/upload

# Build download tool  
go build -o codex-download ./cmd/download

# Or build both
go build ./cmd/...

Running CLI Tools

# Upload a file (defaults to localhost:8080)
go run ./cmd/upload -file test-data.bin -host localhost -port 8080

# Download by CID
go run ./cmd/download -cid <CID> -file output.bin -host localhost -port 8080

CLI notes:

• Download writes to an io.Writer (a created file). On failure, the tool deletes the partial file.
• Flags follow Go flag conventions; -h/--help prints usage.

Development Conventions

Module & Import Pattern

• Module name: go-codex-client (defined in go.mod)
• Internal imports use: "go-codex-client/communities"
• No external dependencies beyond Go stdlib (HTTP, context, io operations)

Error Handling Style

• Use log.Fatalf in CLI tools for unrecoverable errors
• Return wrapped errors from library functions: fmt.Errorf("context: %w", err)
• HTTP errors include status codes and response body in error messages

Flag Usage Pattern (CLI)

Standard pattern across CLI tools:

var (
    host = flag.String("host", "localhost", "Codex host")
    port = flag.String("port", "8080", "Codex port")
    // tool-specific flags...
)
flag.Parse()

Status & Known Issues

• Download CLI fixed: creates the output file and passes it as io.Writer to client.Download; removes the file on error.

Integration Notes

• Codex API Version: Uses v1 endpoints (/api/codex/v1/)
• Content Type: Always application/octet-stream for uploads
• CID Response: Upload returns a CID string (often newline-terminated); client trims whitespace
• Network Operations: Assumes Codex node is running and accessible at specified host:port

Common failure modes (and fixes)

• 404 on download: CID not found or node can’t retrieve from network; verify CID and node connectivity
• 500/Bad Gateway: upstream node error; retry or check Codex node logs
• Timeouts: increase via client.SetRequestTimeout(...) or set CODEX_TIMEOUT_MS in integration tests
• Partial file on download error: download CLI already deletes the file on failure

Testing & Debugging

• Unit tests (stdlib) in communities/codex_client_test.go cover:
  • Upload success (headers validated) returning CID
  • Download success to a bytes.Buffer
  • HasCid existence checks with JSON parsing
  • RemoveCid deletion with 204 status validation
  • TriggerDownload async operations with manifest parsing
  • LocalDownload direct local storage access
  • Cancellation: streaming handler exits on client cancel; fast and warning-free
• Run: go test -v ./communities
• Integration test (requires Codex node): communities/codex_client_integration_test.go
  • Build tag-gated: run with go test -v -tags=integration ./communities -run Integration
  • Env: CODEX_HOST (default localhost), CODEX_API_PORT (default 8080), optional CODEX_TIMEOUT_MS
  • Uses random 1KB payload, logs hex preview, uploads, downloads, and verifies equality
  • Includes TriggerDownload test: uploads→triggers async download→polls HasCid (10s timeout)→verifies content
  • All tests use RemoveCid cleanup in defer blocks to prevent storage accumulation
• Debug by observing HTTP responses and Codex node logs; client timeout defaults to 60s

Repo Meta

• .gitignore excludes build artifacts (bin/, binaries), coverage outputs, and IDE files
• README.md documents build/run/test workflows
• COPYRIGHT.md provides MIT license