mirror of
https://github.com/logos-storage/logos-storage-go.git
synced 2026-01-02 13:23:11 +00:00
initial commit
This commit is contained in:
commit
bdde71e871
120
.github/copilot-instructions.md
vendored
Normal file
120
.github/copilot-instructions.md
vendored
Normal file
@ -0,0 +1,120 @@
|
||||
# Codex Client Go Library - AI Agent Guide
|
||||
|
||||
## Project Overview
|
||||
A lightweight Go client library for interacting with Codex decentralized storage nodes. The project provides:
|
||||
- **Core Library**: `communities/codex_client.go` - HTTP client wrapping Codex REST API
|
||||
- **CLI Tools**: `cmd/upload/` and `cmd/download/` - Command-line utilities for file operations
|
||||
|
||||
## Quickstart (build • run • test)
|
||||
- Build CLIs: `go build -o bin/codex-upload ./cmd/upload && go build -o bin/codex-download ./cmd/download`
|
||||
- Upload: `./bin/codex-upload -file test-data.bin -host localhost -port 8080`
|
||||
- Download: `./bin/codex-download -cid <CID> -file out.bin -host localhost -port 8080`
|
||||
- Unit tests: `go test -v ./communities`
|
||||
- Integration test: `go test -v -tags=integration ./communities -run Integration`
|
||||
|
||||
## Architecture & Design Patterns
|
||||
|
||||
### Client Structure
|
||||
The `CodexClient` type in `communities/codex_client.go` is the central abstraction:
|
||||
- Wraps Codex HTTP API (`/api/codex/v1/data/*`) with Go-idiomatic methods
|
||||
- Supports both network downloads (`/network/stream`) and local downloads (direct CID access)
|
||||
- Context-aware operations for cancellation support (`DownloadWithContext`, `LocalDownloadWithContext`)
|
||||
- All uploads use `application/octet-stream` with `Content-Disposition` header for filenames
|
||||
|
||||
### Data Flow Pattern
|
||||
1. **Upload**: `file → io.Reader → HTTP POST → CID string`
|
||||
2. **Download**: `CID string → HTTP GET → io.Writer → file`
|
||||
3. Network vs Local: Network download uses `/network/stream` endpoint (retrieves from DHT), local uses direct `/data/{cid}`
|
||||
|
||||
### Key Implementation Details
|
||||
- **Streaming with Cancellation**: Custom `copyWithContext` (64KB buffer) checks context cancellation between chunks
|
||||
- **Default Timeout**: 60s for HTTP operations (configurable via `SetRequestTimeout`)
|
||||
- **Error Handling**: Errors wrapped with `%w`; HTTP errors include status code and body
|
||||
- **CID Format**: CIDv1 + multibase base58btc (prefix `z`), using the `raw` codec (`Dv`). Example: `zDvZRwzm...`
|
||||
|
||||
### CID format cheatsheet
|
||||
- Multibase prefix `z` = base58btc; `Dv` denotes CIDv1 + raw codec
|
||||
- CIDs commonly start with `zDv...` in Codex (e.g., `zDvZRwzmRigWseNB...`)
|
||||
- Upload responses may include a trailing newline; client trims via `strings.TrimSpace`
|
||||
|
||||
## CLI Commands & Usage
|
||||
|
||||
### Build Commands
|
||||
```bash
|
||||
# Build upload tool
|
||||
go build -o codex-upload ./cmd/upload
|
||||
|
||||
# Build download tool
|
||||
go build -o codex-download ./cmd/download
|
||||
|
||||
# Or build both
|
||||
go build ./cmd/...
|
||||
```
|
||||
|
||||
### Running CLI Tools
|
||||
```bash
|
||||
# Upload a file (defaults to localhost:8080)
|
||||
go run ./cmd/upload -file test-data.bin -host localhost -port 8080
|
||||
|
||||
# Download by CID
|
||||
go run ./cmd/download -cid <CID> -file output.bin -host localhost -port 8080
|
||||
```
|
||||
|
||||
CLI notes:
|
||||
- Download writes to an `io.Writer` (a created file). On failure, the tool deletes the partial file.
|
||||
- Flags follow Go `flag` conventions; `-h/--help` prints usage.
|
||||
|
||||
## Development Conventions
|
||||
|
||||
### Module & Import Pattern
|
||||
- Module name: `go-codex-client` (defined in `go.mod`)
|
||||
- Internal imports use: `"go-codex-client/communities"`
|
||||
- No external dependencies beyond Go stdlib (HTTP, context, io operations)
|
||||
|
||||
### Error Handling Style
|
||||
- Use `log.Fatalf` in CLI tools for unrecoverable errors
|
||||
- Return wrapped errors from library functions: `fmt.Errorf("context: %w", err)`
|
||||
- HTTP errors include status codes and response body in error messages
|
||||
|
||||
### Flag Usage Pattern (CLI)
|
||||
Standard pattern across CLI tools:
|
||||
```go
|
||||
var (
|
||||
host = flag.String("host", "localhost", "Codex host")
|
||||
port = flag.String("port", "8080", "Codex port")
|
||||
// tool-specific flags...
|
||||
)
|
||||
flag.Parse()
|
||||
```
|
||||
|
||||
## Status & Known Issues
|
||||
- Download CLI fixed: creates the output file and passes it as `io.Writer` to `client.Download`; removes the file on error.
|
||||
|
||||
## Integration Notes
|
||||
- **Codex API Version**: Uses v1 endpoints (`/api/codex/v1/`)
|
||||
- **Content Type**: Always `application/octet-stream` for uploads
|
||||
- **CID Response**: Upload returns a CID string (often newline-terminated); client trims whitespace
|
||||
- **Network Operations**: Assumes Codex node is running and accessible at specified host:port
|
||||
|
||||
## Common failure modes (and fixes)
|
||||
- 404 on download: CID not found or node can’t retrieve from network; verify CID and node connectivity
|
||||
- 500/Bad Gateway: upstream node error; retry or check Codex node logs
|
||||
- Timeouts: increase via `client.SetRequestTimeout(...)` or set `CODEX_TIMEOUT_MS` in integration tests
|
||||
- Partial file on download error: download CLI already deletes the file on failure
|
||||
|
||||
## Testing & Debugging
|
||||
- Unit tests (stdlib) in `communities/codex_client_test.go` cover:
|
||||
- Upload success (headers validated) returning CID
|
||||
- Download success to a `bytes.Buffer`
|
||||
- Cancellation: streaming handler exits on client cancel; fast and warning-free
|
||||
- Run: `go test -v ./communities`
|
||||
- Integration test (requires Codex node): `communities/codex_client_integration_test.go`
|
||||
- Build tag-gated: run with `go test -v -tags=integration ./communities -run Integration`
|
||||
- Env: `CODEX_HOST` (default `localhost`), `CODEX_API_PORT` (default `8080`), optional `CODEX_TIMEOUT_MS`
|
||||
- Uses random 1KB payload, logs hex preview, uploads, downloads, and verifies equality
|
||||
- Debug by observing HTTP responses and Codex node logs; client timeout defaults to 60s
|
||||
|
||||
## Repo Meta
|
||||
- `.gitignore` excludes build artifacts (`bin/`, binaries), coverage outputs, and IDE files
|
||||
- `README.md` documents build/run/test workflows
|
||||
- `COPYRIGHT.md` provides MIT license
|
||||
21
.gitignore
vendored
Normal file
21
.gitignore
vendored
Normal file
@ -0,0 +1,21 @@
|
||||
# Go build artifacts
|
||||
/codex-upload
|
||||
/codex-download
|
||||
/bin/
|
||||
/dist/
|
||||
/build/
|
||||
/out/
|
||||
output.bin
|
||||
|
||||
# Test binaries and coverage
|
||||
*.test
|
||||
*.out
|
||||
*.coverprofile
|
||||
coverage*.txt
|
||||
coverage*.out
|
||||
|
||||
# OS files
|
||||
.DS_Store
|
||||
|
||||
# Logs
|
||||
*.log
|
||||
5
.vscode/settings.json
vendored
Normal file
5
.vscode/settings.json
vendored
Normal file
@ -0,0 +1,5 @@
|
||||
{
|
||||
"gopls": {
|
||||
"buildFlags": ["-tags=integration"]
|
||||
}
|
||||
}
|
||||
21
COPYRIGHT.md
Normal file
21
COPYRIGHT.md
Normal file
@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2025 go-codex-client contributors
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
100
README.md
Normal file
100
README.md
Normal file
@ -0,0 +1,100 @@
|
||||
# go-codex-client
|
||||
|
||||
A lightweight Go client utility for interacting with Codex client.
|
||||
|
||||
## Project layout
|
||||
- `communities/codex_client.go` — core HTTP client (upload/download, context-aware streaming)
|
||||
- `cmd/upload/` — CLI to upload a file to Codex
|
||||
- `cmd/download/` — CLI to download a file by CID
|
||||
- `.github/copilot-instructions.md` — guidance for AI coding agents
|
||||
|
||||
We will be running codex client, and then use a small testing utility to check if the low level abstraction - CodexClient - correctly uploads and downloads the content.
|
||||
|
||||
### Running CodexClient
|
||||
|
||||
I often remove some logging noise, by slightly changing the build
|
||||
params in `build.nims` (nim-codex):
|
||||
|
||||
```nim
|
||||
task codex, "build codex binary":
|
||||
buildBinary "codex",
|
||||
# params = "-d:chronicles_runtime_filtering -d:chronicles_log_level=TRACE"
|
||||
params =
|
||||
"-d:chronicles_runtime_filtering -d:chronicles_log_level=TRACE -d:chronicles_enabled_topics:restapi:TRACE,node:TRACE"
|
||||
```
|
||||
|
||||
You see a slightly more selective `params` in the `codex` task.
|
||||
|
||||
To run the client I use the following command:
|
||||
|
||||
```bash
|
||||
./build/codex --data-dir=./data-1 --listen-addrs=/ip4/127.0.0.1/tcp/8081 --api-port=8001 --nat=none --disc-port=8091 --log-level=TRACE
|
||||
```
|
||||
|
||||
### Building codex-upload and codex-download utilities
|
||||
|
||||
Use the following command to build the `codex-upload` and `codex-download` utilities:
|
||||
|
||||
```bash
|
||||
go build -o bin/codex-upload ./cmd/upload
|
||||
go build -o bin/codex-download ./cmd/download
|
||||
```
|
||||
### Uploading content to Codex
|
||||
|
||||
Now, using the `codex-upload` utility, we can upload the content to Codex as follows:
|
||||
|
||||
```bash
|
||||
~/code/local/go-codex-client
|
||||
❯ ./bin/codex-upload -file test-data.bin -host localhost -port 8001
|
||||
Uploading test-data.bin (43 bytes) to Codex at localhost:8001...
|
||||
✅ Upload successful!
|
||||
CID: zDvZRwzm8K7bcyPeBXcZzWD7AWc4VqNuseduDr3VsuYA1yXej49V
|
||||
```
|
||||
|
||||
### Downloading content from Codex
|
||||
|
||||
Now, having the content uploaded to Codex - let's get it back using the `codex-download` utility:
|
||||
|
||||
```bash
|
||||
~/code/local/go-codex-client
|
||||
❯ ./bin/codex-download -cid zDvZRwzm8K7bcyPeBXcZzWD7AWc4VqNuseduDr3VsuYA1yXej49V -file output.bin -host localhost -port 8001
|
||||
Downloading CID zDvZRwzm8K7bcyPeBXcZzWD7AWc4VqNuseduDr3VsuYA1yXej49V from Codex at localhost:8001...
|
||||
✅ Download successful!
|
||||
Saved to: output.bin
|
||||
```
|
||||
|
||||
You can easily compare that the downloaded content matches the original using:
|
||||
|
||||
```bash
|
||||
~/code/local/go-codex-client
|
||||
❯ openssl sha256 test-data.bin
|
||||
SHA2-256(test-data.bin)= c74ce73165c288348b168baffc477b6db38af3c629b42a7725c35d99d400d992
|
||||
|
||||
~/code/local/go-codex-client
|
||||
❯ openssl sha256 output.bin
|
||||
SHA2-256(output.bin)= c74ce73165c288348b168baffc477b6db38af3c629b42a7725c35d99d400d992
|
||||
```
|
||||
|
||||
### Running tests
|
||||
|
||||
There are a couple of basic tests, including one integration test.
|
||||
|
||||
To run the unit tests:
|
||||
|
||||
```bash
|
||||
❯ go test -v ./communities
|
||||
=== RUN TestUpload_Success
|
||||
--- PASS: TestUpload_Success (0.00s)
|
||||
=== RUN TestDownload_Success
|
||||
--- PASS: TestDownload_Success (0.00s)
|
||||
=== RUN TestDownloadWithContext_Cancel
|
||||
--- PASS: TestDownloadWithContext_Cancel (0.04s)
|
||||
PASS
|
||||
ok go-codex-client/communities 0.044s
|
||||
```
|
||||
|
||||
To run the integration test, use `integration` tag and narrow the scope using `-run Integration`:
|
||||
|
||||
```bash
|
||||
go test -v -tags=integration ./communities -run Integration -timeout 15s
|
||||
```
|
||||
48
cmd/download/main.go
Normal file
48
cmd/download/main.go
Normal file
@ -0,0 +1,48 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
|
||||
"go-codex-client/communities" // Import the local communities package
|
||||
)
|
||||
|
||||
func main() {
|
||||
var (
|
||||
host = flag.String("host", "localhost", "Codex host")
|
||||
port = flag.String("port", "8080", "Codex port")
|
||||
cid = flag.String("cid", "", "CID of the file to download")
|
||||
file = flag.String("file", "downloaded-file.bin", "File to save the downloaded data")
|
||||
)
|
||||
|
||||
flag.Parse()
|
||||
|
||||
if *cid == "" {
|
||||
log.Fatal("CID is required (use -cid flag)")
|
||||
}
|
||||
|
||||
// Create Codex client
|
||||
client := communities.NewCodexClient(*host, *port)
|
||||
|
||||
// Create output file
|
||||
outputFile, err := os.Create(*file)
|
||||
if err != nil {
|
||||
log.Fatalf("Failed to create output file %s: %v", *file, err)
|
||||
}
|
||||
defer outputFile.Close()
|
||||
|
||||
fmt.Printf("Downloading CID %s from Codex at %s:%s...\n", *cid, *host, *port)
|
||||
|
||||
// Download data - pass the io.Writer (outputFile), not the string
|
||||
err = client.Download(*cid, outputFile)
|
||||
if err != nil {
|
||||
// Clean up the failed/partial file
|
||||
os.Remove(*file)
|
||||
log.Fatalf("Download failed: %v", err)
|
||||
}
|
||||
|
||||
fmt.Printf("✅ Download successful!\n")
|
||||
fmt.Printf("Saved to: %s\n", *file)
|
||||
}
|
||||
45
cmd/upload/main.go
Normal file
45
cmd/upload/main.go
Normal file
@ -0,0 +1,45 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
|
||||
"go-codex-client/communities" // Import the local communities package
|
||||
)
|
||||
|
||||
func main() {
|
||||
var (
|
||||
host = flag.String("host", "localhost", "Codex host")
|
||||
port = flag.String("port", "8080", "Codex port")
|
||||
file = flag.String("file", "test-data.bin", "File to upload")
|
||||
filename = flag.String("name", "", "Filename to use in upload (defaults to actual filename)")
|
||||
)
|
||||
flag.Parse()
|
||||
|
||||
// Read file data
|
||||
data, err := os.ReadFile(*file)
|
||||
if err != nil {
|
||||
log.Fatalf("Failed to read file %s: %v", *file, err)
|
||||
}
|
||||
|
||||
// Use actual filename if name not specified
|
||||
uploadName := *filename
|
||||
if uploadName == "" {
|
||||
uploadName = *file
|
||||
}
|
||||
|
||||
fmt.Printf("Uploading %s (%d bytes) to Codex at %s:%s...\n", *file, len(data), *host, *port)
|
||||
// Create Codex client and upload
|
||||
client := communities.NewCodexClient(*host, *port)
|
||||
|
||||
cid, err := client.Upload(bytes.NewReader(data), uploadName)
|
||||
if err != nil {
|
||||
log.Fatalf("Upload failed: %v", err)
|
||||
}
|
||||
|
||||
fmt.Printf("✅ Upload successful!\n")
|
||||
fmt.Printf("CID: %s\n", cid)
|
||||
}
|
||||
164
communities/codex_client.go
Normal file
164
communities/codex_client.go
Normal file
@ -0,0 +1,164 @@
|
||||
/* Package communities
|
||||
*
|
||||
* Provides a CodexClient type that you can use to conveniently
|
||||
* upload buffers to Codex.
|
||||
*
|
||||
*/
|
||||
package communities
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// CodexClient handles basic upload/download operations with Codex storage
|
||||
type CodexClient struct {
|
||||
BaseURL string
|
||||
Client *http.Client
|
||||
}
|
||||
|
||||
// NewCodexClient creates a new Codex client
|
||||
func NewCodexClient(host string, port string) *CodexClient {
|
||||
return &CodexClient{
|
||||
BaseURL: fmt.Sprintf("http://%s:%s", host, port),
|
||||
Client: &http.Client{Timeout: 60 * time.Second},
|
||||
}
|
||||
}
|
||||
|
||||
// Upload uploads data from a reader to Codex and returns the CID
|
||||
func (c *CodexClient) Upload(data io.Reader, filename string) (string, error) {
|
||||
url := fmt.Sprintf("%s/api/codex/v1/data", c.BaseURL)
|
||||
|
||||
// Create the HTTP request
|
||||
req, err := http.NewRequest("POST", url, data)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to create request: %w", err)
|
||||
}
|
||||
|
||||
// Set headers
|
||||
req.Header.Set("Content-Type", "application/octet-stream")
|
||||
req.Header.Set("Content-Disposition", fmt.Sprintf(`filename="%s"`, filename))
|
||||
|
||||
// Send request
|
||||
resp, err := c.Client.Do(req)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to upload to codex: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
// Check if request was successful
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
return "", fmt.Errorf("codex upload failed with status %d: %s", resp.StatusCode, string(body))
|
||||
}
|
||||
|
||||
// Read the CID response
|
||||
cidBytes, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to read response: %w", err)
|
||||
}
|
||||
|
||||
cid := strings.TrimSpace(string(cidBytes))
|
||||
return cid, nil
|
||||
}
|
||||
|
||||
// Download downloads data from Codex by CID and writes it to the provided writer
|
||||
func (c *CodexClient) Download(cid string, output io.Writer) error {
|
||||
return c.DownloadWithContext(context.Background(), cid, output)
|
||||
}
|
||||
|
||||
func (c *CodexClient) LocalDownload(cid string, output io.Writer) error {
|
||||
return c.LocalDownloadWithContext(context.Background(), cid, output)
|
||||
}
|
||||
|
||||
// DownloadWithContext downloads data from Codex by CID with cancellation support
|
||||
func (c *CodexClient) DownloadWithContext(ctx context.Context, cid string, output io.Writer) error {
|
||||
url := fmt.Sprintf("%s/api/codex/v1/data/%s/network/stream", c.BaseURL, cid)
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create request: %w", err)
|
||||
}
|
||||
|
||||
resp, err := c.Client.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to download from codex: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
return fmt.Errorf("codex download failed with status %d: %s", resp.StatusCode, string(body))
|
||||
}
|
||||
|
||||
// Use context-aware copy for cancellable streaming
|
||||
return c.copyWithContext(ctx, output, resp.Body)
|
||||
}
|
||||
|
||||
func (c *CodexClient) LocalDownloadWithContext(ctx context.Context, cid string, output io.Writer) error {
|
||||
url := fmt.Sprintf("%s/api/codex/v1/data/%s", c.BaseURL, cid)
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create request: %w", err)
|
||||
}
|
||||
|
||||
resp, err := c.Client.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to download from codex: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
return fmt.Errorf("codex download failed with status %d: %s", resp.StatusCode, string(body))
|
||||
}
|
||||
|
||||
// Use context-aware copy for cancellable streaming
|
||||
return c.copyWithContext(ctx, output, resp.Body)
|
||||
}
|
||||
|
||||
// copyWithContext performs io.Copy but respects context cancellation
|
||||
func (c *CodexClient) copyWithContext(ctx context.Context, dst io.Writer, src io.Reader) error {
|
||||
// Create a buffer for chunked copying
|
||||
buf := make([]byte, 64*1024) // 64KB buffer
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err() // Return cancellation error
|
||||
default:
|
||||
}
|
||||
|
||||
// Read a chunk
|
||||
n, err := src.Read(buf)
|
||||
if n > 0 {
|
||||
// Write the chunk
|
||||
if _, writeErr := dst.Write(buf[:n]); writeErr != nil {
|
||||
return fmt.Errorf("failed to write data: %w", writeErr)
|
||||
}
|
||||
}
|
||||
|
||||
if err == io.EOF {
|
||||
return nil // Successful completion
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to read data: %w", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// SetRequestTimeout sets the HTTP client timeout for requests
|
||||
func (c *CodexClient) SetRequestTimeout(timeout time.Duration) {
|
||||
c.Client.Timeout = timeout
|
||||
}
|
||||
|
||||
// UploadArchive is a convenience method for uploading archive data
|
||||
func (c *CodexClient) UploadArchive(encodedArchive []byte) (string, error) {
|
||||
return c.Upload(bytes.NewReader(encodedArchive), "archive-data.bin")
|
||||
}
|
||||
70
communities/codex_client_integration_test.go
Normal file
70
communities/codex_client_integration_test.go
Normal file
@ -0,0 +1,70 @@
|
||||
//go:build integration
|
||||
// +build integration
|
||||
|
||||
package communities
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// This test exercises real network calls against a running Codex node.
|
||||
// It is disabled by default via the "integration" build tag.
|
||||
// Run with:
|
||||
//
|
||||
// go test -v -tags=integration ./communities -run Integration
|
||||
//
|
||||
// Required env vars (with defaults):
|
||||
//
|
||||
// CODEX_HOST (default: localhost)
|
||||
// CODEX_API_PORT (default: 8080)
|
||||
// CODEX_TIMEOUT_MS (optional; default: 60000)
|
||||
func TestIntegration_UploadAndDownload(t *testing.T) {
|
||||
host := getenv("CODEX_HOST", "localhost")
|
||||
port := getenv("CODEX_API_PORT", "8080")
|
||||
client := NewCodexClient(host, port)
|
||||
|
||||
// Optional request timeout override
|
||||
if ms := os.Getenv("CODEX_TIMEOUT_MS"); ms != "" {
|
||||
if d, err := time.ParseDuration(ms + "ms"); err == nil {
|
||||
client.SetRequestTimeout(d)
|
||||
}
|
||||
}
|
||||
|
||||
// Generate random payload to ensure proper round-trip verification
|
||||
payload := make([]byte, 1024)
|
||||
if _, err := rand.Read(payload); err != nil {
|
||||
t.Fatalf("failed to generate random payload: %v", err)
|
||||
}
|
||||
t.Logf("Generated payload (first 32 bytes hex): %s", hex.EncodeToString(payload[:32]))
|
||||
|
||||
cid, err := client.Upload(bytes.NewReader(payload), "it.bin")
|
||||
if err != nil {
|
||||
t.Fatalf("upload failed: %v", err)
|
||||
}
|
||||
t.Logf("Upload successful, CID: %s", cid)
|
||||
|
||||
// Download via network stream with a context timeout to avoid hanging
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
var buf bytes.Buffer
|
||||
if err := client.DownloadWithContext(ctx, cid, &buf); err != nil {
|
||||
t.Fatalf("download failed: %v", err)
|
||||
}
|
||||
if got := buf.Bytes(); !bytes.Equal(got, payload) {
|
||||
t.Fatalf("payload mismatch: got %q want %q", string(got), string(payload))
|
||||
}
|
||||
}
|
||||
|
||||
func getenv(k, def string) string {
|
||||
if v := os.Getenv(k); v != "" {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
143
communities/codex_client_test.go
Normal file
143
communities/codex_client_test.go
Normal file
@ -0,0 +1,143 @@
|
||||
package communities
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestUpload_Success(t *testing.T) {
|
||||
// Arrange a fake Codex server that validates headers and returns a CID
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodPost {
|
||||
w.WriteHeader(http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
if r.URL.Path != "/api/codex/v1/data" {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
if ct := r.Header.Get("Content-Type"); ct != "application/octet-stream" {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
if cd := r.Header.Get("Content-Disposition"); cd != "filename=\"hello.txt\"" {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
_, _ = io.ReadAll(r.Body) // consume body
|
||||
_ = r.Body.Close()
|
||||
|
||||
w.WriteHeader(http.StatusOK)
|
||||
// Codex returns CIDv1 base58btc
|
||||
// prefix: zDv
|
||||
// - z = multibase prefix for base58btc
|
||||
// - Dv = CIDv1 prefix for raw codex
|
||||
// we add a newline to simulate real response
|
||||
_, _ = w.Write([]byte("zDvZRwzmTestCID123\n"))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client := NewCodexClient("localhost", "8080")
|
||||
client.BaseURL = server.URL
|
||||
|
||||
// Act
|
||||
cid, err := client.Upload(bytes.NewReader([]byte("payload")), "hello.txt")
|
||||
|
||||
// Assert
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
// Codex uses CIDv1 with base58btc encoding (prefix: zDv)
|
||||
if cid != "zDvZRwzmTestCID123" {
|
||||
t.Fatalf("unexpected cid: %q", cid)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDownload_Success(t *testing.T) {
|
||||
const wantCID = "zDvZRwzm"
|
||||
const payload = "hello from codex"
|
||||
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet {
|
||||
w.WriteHeader(http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
if r.URL.Path != "/api/codex/v1/data/"+wantCID+"/network/stream" {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write([]byte(payload))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client := NewCodexClient("localhost", "8080")
|
||||
client.BaseURL = server.URL
|
||||
|
||||
var buf bytes.Buffer
|
||||
if err := client.Download(wantCID, &buf); err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if got := buf.String(); got != payload {
|
||||
t.Fatalf("unexpected payload: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDownloadWithContext_Cancel(t *testing.T) {
|
||||
const cid = "zDvZRwzm"
|
||||
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/api/codex/v1/data/"+cid+"/network/stream" {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/octet-stream")
|
||||
flusher, _ := w.(http.Flusher)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
// Stream data slowly so the request can be canceled
|
||||
for i := 0; i < 1000; i++ {
|
||||
select {
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
default:
|
||||
}
|
||||
if _, err := w.Write([]byte("x")); err != nil {
|
||||
// Client likely went away; stop writing
|
||||
return
|
||||
}
|
||||
if flusher != nil {
|
||||
flusher.Flush()
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
client := NewCodexClient("localhost", "8080")
|
||||
client.BaseURL = server.URL
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
err := client.DownloadWithContext(ctx, cid, io.Discard)
|
||||
if err == nil {
|
||||
t.Fatalf("expected cancellation error, got nil")
|
||||
}
|
||||
// Accept either canceled or deadline exceeded depending on timing
|
||||
if !errors.Is(err, context.Canceled) && !errors.Is(err, context.DeadlineExceeded) {
|
||||
// net/http may wrap the context error; check error string as a fallback
|
||||
es := err.Error()
|
||||
if !(es == context.Canceled.Error() || es == context.DeadlineExceeded.Error()) {
|
||||
t.Fatalf("expected context cancellation, got: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
1
test-data.bin
Normal file
1
test-data.bin
Normal file
@ -0,0 +1 @@
|
||||
Hello, Codex! This is test data for upload.
|
||||
Loading…
x
Reference in New Issue
Block a user