initial commit

This commit is contained in:
Marcin Czenko 2025-10-20 02:46:50 +02:00
commit bdde71e871
No known key found for this signature in database
GPG Key ID: A0449219BDBA98AE
12 changed files with 741 additions and 0 deletions

120
.github/copilot-instructions.md vendored Normal file
View File

@ -0,0 +1,120 @@
# Codex Client Go Library - AI Agent Guide
## Project Overview
A lightweight Go client library for interacting with Codex decentralized storage nodes. The project provides:
- **Core Library**: `communities/codex_client.go` - HTTP client wrapping Codex REST API
- **CLI Tools**: `cmd/upload/` and `cmd/download/` - Command-line utilities for file operations
## Quickstart (build • run • test)
- Build CLIs: `go build -o bin/codex-upload ./cmd/upload && go build -o bin/codex-download ./cmd/download`
- Upload: `./bin/codex-upload -file test-data.bin -host localhost -port 8080`
- Download: `./bin/codex-download -cid <CID> -file out.bin -host localhost -port 8080`
- Unit tests: `go test -v ./communities`
- Integration test: `go test -v -tags=integration ./communities -run Integration`
## Architecture & Design Patterns
### Client Structure
The `CodexClient` type in `communities/codex_client.go` is the central abstraction:
- Wraps Codex HTTP API (`/api/codex/v1/data/*`) with Go-idiomatic methods
- Supports both network downloads (`/network/stream`) and local downloads (direct CID access)
- Context-aware operations for cancellation support (`DownloadWithContext`, `LocalDownloadWithContext`)
- All uploads use `application/octet-stream` with `Content-Disposition` header for filenames
### Data Flow Pattern
1. **Upload**: `file → io.Reader → HTTP POST → CID string`
2. **Download**: `CID string → HTTP GET → io.Writer → file`
3. Network vs Local: Network download uses `/network/stream` endpoint (retrieves from DHT), local uses direct `/data/{cid}`
### Key Implementation Details
- **Streaming with Cancellation**: Custom `copyWithContext` (64KB buffer) checks context cancellation between chunks
- **Default Timeout**: 60s for HTTP operations (configurable via `SetRequestTimeout`)
- **Error Handling**: Errors wrapped with `%w`; HTTP errors include status code and body
- **CID Format**: CIDv1 + multibase base58btc (prefix `z`), using the `raw` codec (`Dv`). Example: `zDvZRwzm...`
### CID format cheatsheet
- Multibase prefix `z` = base58btc; `Dv` denotes CIDv1 + raw codec
- CIDs commonly start with `zDv...` in Codex (e.g., `zDvZRwzmRigWseNB...`)
- Upload responses may include a trailing newline; client trims via `strings.TrimSpace`
## CLI Commands & Usage
### Build Commands
```bash
# Build upload tool
go build -o codex-upload ./cmd/upload
# Build download tool
go build -o codex-download ./cmd/download
# Or build both
go build ./cmd/...
```
### Running CLI Tools
```bash
# Upload a file (defaults to localhost:8080)
go run ./cmd/upload -file test-data.bin -host localhost -port 8080
# Download by CID
go run ./cmd/download -cid <CID> -file output.bin -host localhost -port 8080
```
CLI notes:
- Download writes to an `io.Writer` (a created file). On failure, the tool deletes the partial file.
- Flags follow Go `flag` conventions; `-h/--help` prints usage.
## Development Conventions
### Module & Import Pattern
- Module name: `go-codex-client` (defined in `go.mod`)
- Internal imports use: `"go-codex-client/communities"`
- No external dependencies beyond Go stdlib (HTTP, context, io operations)
### Error Handling Style
- Use `log.Fatalf` in CLI tools for unrecoverable errors
- Return wrapped errors from library functions: `fmt.Errorf("context: %w", err)`
- HTTP errors include status codes and response body in error messages
### Flag Usage Pattern (CLI)
Standard pattern across CLI tools:
```go
var (
host = flag.String("host", "localhost", "Codex host")
port = flag.String("port", "8080", "Codex port")
// tool-specific flags...
)
flag.Parse()
```
## Status & Known Issues
- Download CLI fixed: creates the output file and passes it as `io.Writer` to `client.Download`; removes the file on error.
## Integration Notes
- **Codex API Version**: Uses v1 endpoints (`/api/codex/v1/`)
- **Content Type**: Always `application/octet-stream` for uploads
- **CID Response**: Upload returns a CID string (often newline-terminated); client trims whitespace
- **Network Operations**: Assumes Codex node is running and accessible at specified host:port
## Common failure modes (and fixes)
- 404 on download: CID not found or node cant retrieve from network; verify CID and node connectivity
- 500/Bad Gateway: upstream node error; retry or check Codex node logs
- Timeouts: increase via `client.SetRequestTimeout(...)` or set `CODEX_TIMEOUT_MS` in integration tests
- Partial file on download error: download CLI already deletes the file on failure
## Testing & Debugging
- Unit tests (stdlib) in `communities/codex_client_test.go` cover:
- Upload success (headers validated) returning CID
- Download success to a `bytes.Buffer`
- Cancellation: streaming handler exits on client cancel; fast and warning-free
- Run: `go test -v ./communities`
- Integration test (requires Codex node): `communities/codex_client_integration_test.go`
- Build tag-gated: run with `go test -v -tags=integration ./communities -run Integration`
- Env: `CODEX_HOST` (default `localhost`), `CODEX_API_PORT` (default `8080`), optional `CODEX_TIMEOUT_MS`
- Uses random 1KB payload, logs hex preview, uploads, downloads, and verifies equality
- Debug by observing HTTP responses and Codex node logs; client timeout defaults to 60s
## Repo Meta
- `.gitignore` excludes build artifacts (`bin/`, binaries), coverage outputs, and IDE files
- `README.md` documents build/run/test workflows
- `COPYRIGHT.md` provides MIT license

21
.gitignore vendored Normal file
View File

@ -0,0 +1,21 @@
# Go build artifacts
/codex-upload
/codex-download
/bin/
/dist/
/build/
/out/
output.bin
# Test binaries and coverage
*.test
*.out
*.coverprofile
coverage*.txt
coverage*.out
# OS files
.DS_Store
# Logs
*.log

5
.vscode/settings.json vendored Normal file
View File

@ -0,0 +1,5 @@
{
"gopls": {
"buildFlags": ["-tags=integration"]
}
}

21
COPYRIGHT.md Normal file
View File

@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025 go-codex-client contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

100
README.md Normal file
View File

@ -0,0 +1,100 @@
# go-codex-client
A lightweight Go client utility for interacting with Codex client.
## Project layout
- `communities/codex_client.go` — core HTTP client (upload/download, context-aware streaming)
- `cmd/upload/` — CLI to upload a file to Codex
- `cmd/download/` — CLI to download a file by CID
- `.github/copilot-instructions.md` — guidance for AI coding agents
We will be running codex client, and then use a small testing utility to check if the low level abstraction - CodexClient - correctly uploads and downloads the content.
### Running CodexClient
I often remove some logging noise, by slightly changing the build
params in `build.nims` (nim-codex):
```nim
task codex, "build codex binary":
buildBinary "codex",
# params = "-d:chronicles_runtime_filtering -d:chronicles_log_level=TRACE"
params =
"-d:chronicles_runtime_filtering -d:chronicles_log_level=TRACE -d:chronicles_enabled_topics:restapi:TRACE,node:TRACE"
```
You see a slightly more selective `params` in the `codex` task.
To run the client I use the following command:
```bash
./build/codex --data-dir=./data-1 --listen-addrs=/ip4/127.0.0.1/tcp/8081 --api-port=8001 --nat=none --disc-port=8091 --log-level=TRACE
```
### Building codex-upload and codex-download utilities
Use the following command to build the `codex-upload` and `codex-download` utilities:
```bash
go build -o bin/codex-upload ./cmd/upload
go build -o bin/codex-download ./cmd/download
```
### Uploading content to Codex
Now, using the `codex-upload` utility, we can upload the content to Codex as follows:
```bash
~/code/local/go-codex-client
./bin/codex-upload -file test-data.bin -host localhost -port 8001
Uploading test-data.bin (43 bytes) to Codex at localhost:8001...
✅ Upload successful!
CID: zDvZRwzm8K7bcyPeBXcZzWD7AWc4VqNuseduDr3VsuYA1yXej49V
```
### Downloading content from Codex
Now, having the content uploaded to Codex - let's get it back using the `codex-download` utility:
```bash
~/code/local/go-codex-client
./bin/codex-download -cid zDvZRwzm8K7bcyPeBXcZzWD7AWc4VqNuseduDr3VsuYA1yXej49V -file output.bin -host localhost -port 8001
Downloading CID zDvZRwzm8K7bcyPeBXcZzWD7AWc4VqNuseduDr3VsuYA1yXej49V from Codex at localhost:8001...
✅ Download successful!
Saved to: output.bin
```
You can easily compare that the downloaded content matches the original using:
```bash
~/code/local/go-codex-client
openssl sha256 test-data.bin
SHA2-256(test-data.bin)= c74ce73165c288348b168baffc477b6db38af3c629b42a7725c35d99d400d992
~/code/local/go-codex-client
openssl sha256 output.bin
SHA2-256(output.bin)= c74ce73165c288348b168baffc477b6db38af3c629b42a7725c35d99d400d992
```
### Running tests
There are a couple of basic tests, including one integration test.
To run the unit tests:
```bash
go test -v ./communities
=== RUN TestUpload_Success
--- PASS: TestUpload_Success (0.00s)
=== RUN TestDownload_Success
--- PASS: TestDownload_Success (0.00s)
=== RUN TestDownloadWithContext_Cancel
--- PASS: TestDownloadWithContext_Cancel (0.04s)
PASS
ok go-codex-client/communities 0.044s
```
To run the integration test, use `integration` tag and narrow the scope using `-run Integration`:
```bash
go test -v -tags=integration ./communities -run Integration -timeout 15s
```

48
cmd/download/main.go Normal file
View File

@ -0,0 +1,48 @@
package main
import (
"flag"
"fmt"
"log"
"os"
"go-codex-client/communities" // Import the local communities package
)
func main() {
var (
host = flag.String("host", "localhost", "Codex host")
port = flag.String("port", "8080", "Codex port")
cid = flag.String("cid", "", "CID of the file to download")
file = flag.String("file", "downloaded-file.bin", "File to save the downloaded data")
)
flag.Parse()
if *cid == "" {
log.Fatal("CID is required (use -cid flag)")
}
// Create Codex client
client := communities.NewCodexClient(*host, *port)
// Create output file
outputFile, err := os.Create(*file)
if err != nil {
log.Fatalf("Failed to create output file %s: %v", *file, err)
}
defer outputFile.Close()
fmt.Printf("Downloading CID %s from Codex at %s:%s...\n", *cid, *host, *port)
// Download data - pass the io.Writer (outputFile), not the string
err = client.Download(*cid, outputFile)
if err != nil {
// Clean up the failed/partial file
os.Remove(*file)
log.Fatalf("Download failed: %v", err)
}
fmt.Printf("✅ Download successful!\n")
fmt.Printf("Saved to: %s\n", *file)
}

45
cmd/upload/main.go Normal file
View File

@ -0,0 +1,45 @@
package main
import (
"bytes"
"flag"
"fmt"
"log"
"os"
"go-codex-client/communities" // Import the local communities package
)
func main() {
var (
host = flag.String("host", "localhost", "Codex host")
port = flag.String("port", "8080", "Codex port")
file = flag.String("file", "test-data.bin", "File to upload")
filename = flag.String("name", "", "Filename to use in upload (defaults to actual filename)")
)
flag.Parse()
// Read file data
data, err := os.ReadFile(*file)
if err != nil {
log.Fatalf("Failed to read file %s: %v", *file, err)
}
// Use actual filename if name not specified
uploadName := *filename
if uploadName == "" {
uploadName = *file
}
fmt.Printf("Uploading %s (%d bytes) to Codex at %s:%s...\n", *file, len(data), *host, *port)
// Create Codex client and upload
client := communities.NewCodexClient(*host, *port)
cid, err := client.Upload(bytes.NewReader(data), uploadName)
if err != nil {
log.Fatalf("Upload failed: %v", err)
}
fmt.Printf("✅ Upload successful!\n")
fmt.Printf("CID: %s\n", cid)
}

164
communities/codex_client.go Normal file
View File

@ -0,0 +1,164 @@
/* Package communities
*
* Provides a CodexClient type that you can use to conveniently
* upload buffers to Codex.
*
*/
package communities
import (
"bytes"
"context"
"fmt"
"io"
"net/http"
"strings"
"time"
)
// CodexClient handles basic upload/download operations with Codex storage
type CodexClient struct {
BaseURL string
Client *http.Client
}
// NewCodexClient creates a new Codex client
func NewCodexClient(host string, port string) *CodexClient {
return &CodexClient{
BaseURL: fmt.Sprintf("http://%s:%s", host, port),
Client: &http.Client{Timeout: 60 * time.Second},
}
}
// Upload uploads data from a reader to Codex and returns the CID
func (c *CodexClient) Upload(data io.Reader, filename string) (string, error) {
url := fmt.Sprintf("%s/api/codex/v1/data", c.BaseURL)
// Create the HTTP request
req, err := http.NewRequest("POST", url, data)
if err != nil {
return "", fmt.Errorf("failed to create request: %w", err)
}
// Set headers
req.Header.Set("Content-Type", "application/octet-stream")
req.Header.Set("Content-Disposition", fmt.Sprintf(`filename="%s"`, filename))
// Send request
resp, err := c.Client.Do(req)
if err != nil {
return "", fmt.Errorf("failed to upload to codex: %w", err)
}
defer resp.Body.Close()
// Check if request was successful
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(resp.Body)
return "", fmt.Errorf("codex upload failed with status %d: %s", resp.StatusCode, string(body))
}
// Read the CID response
cidBytes, err := io.ReadAll(resp.Body)
if err != nil {
return "", fmt.Errorf("failed to read response: %w", err)
}
cid := strings.TrimSpace(string(cidBytes))
return cid, nil
}
// Download downloads data from Codex by CID and writes it to the provided writer
func (c *CodexClient) Download(cid string, output io.Writer) error {
return c.DownloadWithContext(context.Background(), cid, output)
}
func (c *CodexClient) LocalDownload(cid string, output io.Writer) error {
return c.LocalDownloadWithContext(context.Background(), cid, output)
}
// DownloadWithContext downloads data from Codex by CID with cancellation support
func (c *CodexClient) DownloadWithContext(ctx context.Context, cid string, output io.Writer) error {
url := fmt.Sprintf("%s/api/codex/v1/data/%s/network/stream", c.BaseURL, cid)
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
if err != nil {
return fmt.Errorf("failed to create request: %w", err)
}
resp, err := c.Client.Do(req)
if err != nil {
return fmt.Errorf("failed to download from codex: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(resp.Body)
return fmt.Errorf("codex download failed with status %d: %s", resp.StatusCode, string(body))
}
// Use context-aware copy for cancellable streaming
return c.copyWithContext(ctx, output, resp.Body)
}
func (c *CodexClient) LocalDownloadWithContext(ctx context.Context, cid string, output io.Writer) error {
url := fmt.Sprintf("%s/api/codex/v1/data/%s", c.BaseURL, cid)
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
if err != nil {
return fmt.Errorf("failed to create request: %w", err)
}
resp, err := c.Client.Do(req)
if err != nil {
return fmt.Errorf("failed to download from codex: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(resp.Body)
return fmt.Errorf("codex download failed with status %d: %s", resp.StatusCode, string(body))
}
// Use context-aware copy for cancellable streaming
return c.copyWithContext(ctx, output, resp.Body)
}
// copyWithContext performs io.Copy but respects context cancellation
func (c *CodexClient) copyWithContext(ctx context.Context, dst io.Writer, src io.Reader) error {
// Create a buffer for chunked copying
buf := make([]byte, 64*1024) // 64KB buffer
for {
select {
case <-ctx.Done():
return ctx.Err() // Return cancellation error
default:
}
// Read a chunk
n, err := src.Read(buf)
if n > 0 {
// Write the chunk
if _, writeErr := dst.Write(buf[:n]); writeErr != nil {
return fmt.Errorf("failed to write data: %w", writeErr)
}
}
if err == io.EOF {
return nil // Successful completion
}
if err != nil {
return fmt.Errorf("failed to read data: %w", err)
}
}
}
// SetRequestTimeout sets the HTTP client timeout for requests
func (c *CodexClient) SetRequestTimeout(timeout time.Duration) {
c.Client.Timeout = timeout
}
// UploadArchive is a convenience method for uploading archive data
func (c *CodexClient) UploadArchive(encodedArchive []byte) (string, error) {
return c.Upload(bytes.NewReader(encodedArchive), "archive-data.bin")
}

View File

@ -0,0 +1,70 @@
//go:build integration
// +build integration
package communities
import (
"bytes"
"context"
"crypto/rand"
"encoding/hex"
"os"
"testing"
"time"
)
// This test exercises real network calls against a running Codex node.
// It is disabled by default via the "integration" build tag.
// Run with:
//
// go test -v -tags=integration ./communities -run Integration
//
// Required env vars (with defaults):
//
// CODEX_HOST (default: localhost)
// CODEX_API_PORT (default: 8080)
// CODEX_TIMEOUT_MS (optional; default: 60000)
func TestIntegration_UploadAndDownload(t *testing.T) {
host := getenv("CODEX_HOST", "localhost")
port := getenv("CODEX_API_PORT", "8080")
client := NewCodexClient(host, port)
// Optional request timeout override
if ms := os.Getenv("CODEX_TIMEOUT_MS"); ms != "" {
if d, err := time.ParseDuration(ms + "ms"); err == nil {
client.SetRequestTimeout(d)
}
}
// Generate random payload to ensure proper round-trip verification
payload := make([]byte, 1024)
if _, err := rand.Read(payload); err != nil {
t.Fatalf("failed to generate random payload: %v", err)
}
t.Logf("Generated payload (first 32 bytes hex): %s", hex.EncodeToString(payload[:32]))
cid, err := client.Upload(bytes.NewReader(payload), "it.bin")
if err != nil {
t.Fatalf("upload failed: %v", err)
}
t.Logf("Upload successful, CID: %s", cid)
// Download via network stream with a context timeout to avoid hanging
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
var buf bytes.Buffer
if err := client.DownloadWithContext(ctx, cid, &buf); err != nil {
t.Fatalf("download failed: %v", err)
}
if got := buf.Bytes(); !bytes.Equal(got, payload) {
t.Fatalf("payload mismatch: got %q want %q", string(got), string(payload))
}
}
func getenv(k, def string) string {
if v := os.Getenv(k); v != "" {
return v
}
return def
}

View File

@ -0,0 +1,143 @@
package communities
import (
"bytes"
"context"
"errors"
"io"
"net/http"
"net/http/httptest"
"testing"
"time"
)
func TestUpload_Success(t *testing.T) {
// Arrange a fake Codex server that validates headers and returns a CID
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
if r.URL.Path != "/api/codex/v1/data" {
w.WriteHeader(http.StatusNotFound)
return
}
if ct := r.Header.Get("Content-Type"); ct != "application/octet-stream" {
w.WriteHeader(http.StatusBadRequest)
return
}
if cd := r.Header.Get("Content-Disposition"); cd != "filename=\"hello.txt\"" {
w.WriteHeader(http.StatusBadRequest)
return
}
_, _ = io.ReadAll(r.Body) // consume body
_ = r.Body.Close()
w.WriteHeader(http.StatusOK)
// Codex returns CIDv1 base58btc
// prefix: zDv
// - z = multibase prefix for base58btc
// - Dv = CIDv1 prefix for raw codex
// we add a newline to simulate real response
_, _ = w.Write([]byte("zDvZRwzmTestCID123\n"))
}))
defer server.Close()
client := NewCodexClient("localhost", "8080")
client.BaseURL = server.URL
// Act
cid, err := client.Upload(bytes.NewReader([]byte("payload")), "hello.txt")
// Assert
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
// Codex uses CIDv1 with base58btc encoding (prefix: zDv)
if cid != "zDvZRwzmTestCID123" {
t.Fatalf("unexpected cid: %q", cid)
}
}
func TestDownload_Success(t *testing.T) {
const wantCID = "zDvZRwzm"
const payload = "hello from codex"
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
if r.URL.Path != "/api/codex/v1/data/"+wantCID+"/network/stream" {
w.WriteHeader(http.StatusNotFound)
return
}
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte(payload))
}))
defer server.Close()
client := NewCodexClient("localhost", "8080")
client.BaseURL = server.URL
var buf bytes.Buffer
if err := client.Download(wantCID, &buf); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got := buf.String(); got != payload {
t.Fatalf("unexpected payload: %q", got)
}
}
func TestDownloadWithContext_Cancel(t *testing.T) {
const cid = "zDvZRwzm"
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/api/codex/v1/data/"+cid+"/network/stream" {
w.WriteHeader(http.StatusNotFound)
return
}
w.Header().Set("Content-Type", "application/octet-stream")
flusher, _ := w.(http.Flusher)
w.WriteHeader(http.StatusOK)
// Stream data slowly so the request can be canceled
for i := 0; i < 1000; i++ {
select {
case <-r.Context().Done():
return
default:
}
if _, err := w.Write([]byte("x")); err != nil {
// Client likely went away; stop writing
return
}
if flusher != nil {
flusher.Flush()
}
time.Sleep(10 * time.Millisecond)
}
}))
defer server.Close()
client := NewCodexClient("localhost", "8080")
client.BaseURL = server.URL
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond)
defer cancel()
err := client.DownloadWithContext(ctx, cid, io.Discard)
if err == nil {
t.Fatalf("expected cancellation error, got nil")
}
// Accept either canceled or deadline exceeded depending on timing
if !errors.Is(err, context.Canceled) && !errors.Is(err, context.DeadlineExceeded) {
// net/http may wrap the context error; check error string as a fallback
es := err.Error()
if !(es == context.Canceled.Error() || es == context.DeadlineExceeded.Error()) {
t.Fatalf("expected context cancellation, got: %v", err)
}
}
}

3
go.mod Normal file
View File

@ -0,0 +1,3 @@
module go-codex-client
go 1.21

1
test-data.bin Normal file
View File

@ -0,0 +1 @@
Hello, Codex! This is test data for upload.