mirror of
https://github.com/logos-storage/logos-storage-go.git
synced 2026-01-04 06:13:07 +00:00
initial commit
This commit is contained in:
commit
bdde71e871
120
.github/copilot-instructions.md
vendored
Normal file
120
.github/copilot-instructions.md
vendored
Normal file
@ -0,0 +1,120 @@
|
|||||||
|
# Codex Client Go Library - AI Agent Guide
|
||||||
|
|
||||||
|
## Project Overview
|
||||||
|
A lightweight Go client library for interacting with Codex decentralized storage nodes. The project provides:
|
||||||
|
- **Core Library**: `communities/codex_client.go` - HTTP client wrapping Codex REST API
|
||||||
|
- **CLI Tools**: `cmd/upload/` and `cmd/download/` - Command-line utilities for file operations
|
||||||
|
|
||||||
|
## Quickstart (build • run • test)
|
||||||
|
- Build CLIs: `go build -o bin/codex-upload ./cmd/upload && go build -o bin/codex-download ./cmd/download`
|
||||||
|
- Upload: `./bin/codex-upload -file test-data.bin -host localhost -port 8080`
|
||||||
|
- Download: `./bin/codex-download -cid <CID> -file out.bin -host localhost -port 8080`
|
||||||
|
- Unit tests: `go test -v ./communities`
|
||||||
|
- Integration test: `go test -v -tags=integration ./communities -run Integration`
|
||||||
|
|
||||||
|
## Architecture & Design Patterns
|
||||||
|
|
||||||
|
### Client Structure
|
||||||
|
The `CodexClient` type in `communities/codex_client.go` is the central abstraction:
|
||||||
|
- Wraps Codex HTTP API (`/api/codex/v1/data/*`) with Go-idiomatic methods
|
||||||
|
- Supports both network downloads (`/network/stream`) and local downloads (direct CID access)
|
||||||
|
- Context-aware operations for cancellation support (`DownloadWithContext`, `LocalDownloadWithContext`)
|
||||||
|
- All uploads use `application/octet-stream` with `Content-Disposition` header for filenames
|
||||||
|
|
||||||
|
### Data Flow Pattern
|
||||||
|
1. **Upload**: `file → io.Reader → HTTP POST → CID string`
|
||||||
|
2. **Download**: `CID string → HTTP GET → io.Writer → file`
|
||||||
|
3. Network vs Local: Network download uses `/network/stream` endpoint (retrieves from DHT), local uses direct `/data/{cid}`
|
||||||
|
|
||||||
|
### Key Implementation Details
|
||||||
|
- **Streaming with Cancellation**: Custom `copyWithContext` (64KB buffer) checks context cancellation between chunks
|
||||||
|
- **Default Timeout**: 60s for HTTP operations (configurable via `SetRequestTimeout`)
|
||||||
|
- **Error Handling**: Errors wrapped with `%w`; HTTP errors include status code and body
|
||||||
|
- **CID Format**: CIDv1 + multibase base58btc (prefix `z`), using the `raw` codec (`Dv`). Example: `zDvZRwzm...`
|
||||||
|
|
||||||
|
### CID format cheatsheet
|
||||||
|
- Multibase prefix `z` = base58btc; `Dv` denotes CIDv1 + raw codec
|
||||||
|
- CIDs commonly start with `zDv...` in Codex (e.g., `zDvZRwzmRigWseNB...`)
|
||||||
|
- Upload responses may include a trailing newline; client trims via `strings.TrimSpace`
|
||||||
|
|
||||||
|
## CLI Commands & Usage
|
||||||
|
|
||||||
|
### Build Commands
|
||||||
|
```bash
|
||||||
|
# Build upload tool
|
||||||
|
go build -o codex-upload ./cmd/upload
|
||||||
|
|
||||||
|
# Build download tool
|
||||||
|
go build -o codex-download ./cmd/download
|
||||||
|
|
||||||
|
# Or build both
|
||||||
|
go build ./cmd/...
|
||||||
|
```
|
||||||
|
|
||||||
|
### Running CLI Tools
|
||||||
|
```bash
|
||||||
|
# Upload a file (defaults to localhost:8080)
|
||||||
|
go run ./cmd/upload -file test-data.bin -host localhost -port 8080
|
||||||
|
|
||||||
|
# Download by CID
|
||||||
|
go run ./cmd/download -cid <CID> -file output.bin -host localhost -port 8080
|
||||||
|
```
|
||||||
|
|
||||||
|
CLI notes:
|
||||||
|
- Download writes to an `io.Writer` (a created file). On failure, the tool deletes the partial file.
|
||||||
|
- Flags follow Go `flag` conventions; `-h/--help` prints usage.
|
||||||
|
|
||||||
|
## Development Conventions
|
||||||
|
|
||||||
|
### Module & Import Pattern
|
||||||
|
- Module name: `go-codex-client` (defined in `go.mod`)
|
||||||
|
- Internal imports use: `"go-codex-client/communities"`
|
||||||
|
- No external dependencies beyond Go stdlib (HTTP, context, io operations)
|
||||||
|
|
||||||
|
### Error Handling Style
|
||||||
|
- Use `log.Fatalf` in CLI tools for unrecoverable errors
|
||||||
|
- Return wrapped errors from library functions: `fmt.Errorf("context: %w", err)`
|
||||||
|
- HTTP errors include status codes and response body in error messages
|
||||||
|
|
||||||
|
### Flag Usage Pattern (CLI)
|
||||||
|
Standard pattern across CLI tools:
|
||||||
|
```go
|
||||||
|
var (
|
||||||
|
host = flag.String("host", "localhost", "Codex host")
|
||||||
|
port = flag.String("port", "8080", "Codex port")
|
||||||
|
// tool-specific flags...
|
||||||
|
)
|
||||||
|
flag.Parse()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Status & Known Issues
|
||||||
|
- Download CLI fixed: creates the output file and passes it as `io.Writer` to `client.Download`; removes the file on error.
|
||||||
|
|
||||||
|
## Integration Notes
|
||||||
|
- **Codex API Version**: Uses v1 endpoints (`/api/codex/v1/`)
|
||||||
|
- **Content Type**: Always `application/octet-stream` for uploads
|
||||||
|
- **CID Response**: Upload returns a CID string (often newline-terminated); client trims whitespace
|
||||||
|
- **Network Operations**: Assumes Codex node is running and accessible at specified host:port
|
||||||
|
|
||||||
|
## Common failure modes (and fixes)
|
||||||
|
- 404 on download: CID not found or node can’t retrieve from network; verify CID and node connectivity
|
||||||
|
- 500/Bad Gateway: upstream node error; retry or check Codex node logs
|
||||||
|
- Timeouts: increase via `client.SetRequestTimeout(...)` or set `CODEX_TIMEOUT_MS` in integration tests
|
||||||
|
- Partial file on download error: download CLI already deletes the file on failure
|
||||||
|
|
||||||
|
## Testing & Debugging
|
||||||
|
- Unit tests (stdlib) in `communities/codex_client_test.go` cover:
|
||||||
|
- Upload success (headers validated) returning CID
|
||||||
|
- Download success to a `bytes.Buffer`
|
||||||
|
- Cancellation: streaming handler exits on client cancel; fast and warning-free
|
||||||
|
- Run: `go test -v ./communities`
|
||||||
|
- Integration test (requires Codex node): `communities/codex_client_integration_test.go`
|
||||||
|
- Build tag-gated: run with `go test -v -tags=integration ./communities -run Integration`
|
||||||
|
- Env: `CODEX_HOST` (default `localhost`), `CODEX_API_PORT` (default `8080`), optional `CODEX_TIMEOUT_MS`
|
||||||
|
- Uses random 1KB payload, logs hex preview, uploads, downloads, and verifies equality
|
||||||
|
- Debug by observing HTTP responses and Codex node logs; client timeout defaults to 60s
|
||||||
|
|
||||||
|
## Repo Meta
|
||||||
|
- `.gitignore` excludes build artifacts (`bin/`, binaries), coverage outputs, and IDE files
|
||||||
|
- `README.md` documents build/run/test workflows
|
||||||
|
- `COPYRIGHT.md` provides MIT license
|
||||||
21
.gitignore
vendored
Normal file
21
.gitignore
vendored
Normal file
@ -0,0 +1,21 @@
|
|||||||
|
# Go build artifacts
|
||||||
|
/codex-upload
|
||||||
|
/codex-download
|
||||||
|
/bin/
|
||||||
|
/dist/
|
||||||
|
/build/
|
||||||
|
/out/
|
||||||
|
output.bin
|
||||||
|
|
||||||
|
# Test binaries and coverage
|
||||||
|
*.test
|
||||||
|
*.out
|
||||||
|
*.coverprofile
|
||||||
|
coverage*.txt
|
||||||
|
coverage*.out
|
||||||
|
|
||||||
|
# OS files
|
||||||
|
.DS_Store
|
||||||
|
|
||||||
|
# Logs
|
||||||
|
*.log
|
||||||
5
.vscode/settings.json
vendored
Normal file
5
.vscode/settings.json
vendored
Normal file
@ -0,0 +1,5 @@
|
|||||||
|
{
|
||||||
|
"gopls": {
|
||||||
|
"buildFlags": ["-tags=integration"]
|
||||||
|
}
|
||||||
|
}
|
||||||
21
COPYRIGHT.md
Normal file
21
COPYRIGHT.md
Normal file
@ -0,0 +1,21 @@
|
|||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2025 go-codex-client contributors
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
100
README.md
Normal file
100
README.md
Normal file
@ -0,0 +1,100 @@
|
|||||||
|
# go-codex-client
|
||||||
|
|
||||||
|
A lightweight Go client utility for interacting with Codex client.
|
||||||
|
|
||||||
|
## Project layout
|
||||||
|
- `communities/codex_client.go` — core HTTP client (upload/download, context-aware streaming)
|
||||||
|
- `cmd/upload/` — CLI to upload a file to Codex
|
||||||
|
- `cmd/download/` — CLI to download a file by CID
|
||||||
|
- `.github/copilot-instructions.md` — guidance for AI coding agents
|
||||||
|
|
||||||
|
We will be running codex client, and then use a small testing utility to check if the low level abstraction - CodexClient - correctly uploads and downloads the content.
|
||||||
|
|
||||||
|
### Running CodexClient
|
||||||
|
|
||||||
|
I often remove some logging noise, by slightly changing the build
|
||||||
|
params in `build.nims` (nim-codex):
|
||||||
|
|
||||||
|
```nim
|
||||||
|
task codex, "build codex binary":
|
||||||
|
buildBinary "codex",
|
||||||
|
# params = "-d:chronicles_runtime_filtering -d:chronicles_log_level=TRACE"
|
||||||
|
params =
|
||||||
|
"-d:chronicles_runtime_filtering -d:chronicles_log_level=TRACE -d:chronicles_enabled_topics:restapi:TRACE,node:TRACE"
|
||||||
|
```
|
||||||
|
|
||||||
|
You see a slightly more selective `params` in the `codex` task.
|
||||||
|
|
||||||
|
To run the client I use the following command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./build/codex --data-dir=./data-1 --listen-addrs=/ip4/127.0.0.1/tcp/8081 --api-port=8001 --nat=none --disc-port=8091 --log-level=TRACE
|
||||||
|
```
|
||||||
|
|
||||||
|
### Building codex-upload and codex-download utilities
|
||||||
|
|
||||||
|
Use the following command to build the `codex-upload` and `codex-download` utilities:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go build -o bin/codex-upload ./cmd/upload
|
||||||
|
go build -o bin/codex-download ./cmd/download
|
||||||
|
```
|
||||||
|
### Uploading content to Codex
|
||||||
|
|
||||||
|
Now, using the `codex-upload` utility, we can upload the content to Codex as follows:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
~/code/local/go-codex-client
|
||||||
|
❯ ./bin/codex-upload -file test-data.bin -host localhost -port 8001
|
||||||
|
Uploading test-data.bin (43 bytes) to Codex at localhost:8001...
|
||||||
|
✅ Upload successful!
|
||||||
|
CID: zDvZRwzm8K7bcyPeBXcZzWD7AWc4VqNuseduDr3VsuYA1yXej49V
|
||||||
|
```
|
||||||
|
|
||||||
|
### Downloading content from Codex
|
||||||
|
|
||||||
|
Now, having the content uploaded to Codex - let's get it back using the `codex-download` utility:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
~/code/local/go-codex-client
|
||||||
|
❯ ./bin/codex-download -cid zDvZRwzm8K7bcyPeBXcZzWD7AWc4VqNuseduDr3VsuYA1yXej49V -file output.bin -host localhost -port 8001
|
||||||
|
Downloading CID zDvZRwzm8K7bcyPeBXcZzWD7AWc4VqNuseduDr3VsuYA1yXej49V from Codex at localhost:8001...
|
||||||
|
✅ Download successful!
|
||||||
|
Saved to: output.bin
|
||||||
|
```
|
||||||
|
|
||||||
|
You can easily compare that the downloaded content matches the original using:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
~/code/local/go-codex-client
|
||||||
|
❯ openssl sha256 test-data.bin
|
||||||
|
SHA2-256(test-data.bin)= c74ce73165c288348b168baffc477b6db38af3c629b42a7725c35d99d400d992
|
||||||
|
|
||||||
|
~/code/local/go-codex-client
|
||||||
|
❯ openssl sha256 output.bin
|
||||||
|
SHA2-256(output.bin)= c74ce73165c288348b168baffc477b6db38af3c629b42a7725c35d99d400d992
|
||||||
|
```
|
||||||
|
|
||||||
|
### Running tests
|
||||||
|
|
||||||
|
There are a couple of basic tests, including one integration test.
|
||||||
|
|
||||||
|
To run the unit tests:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
❯ go test -v ./communities
|
||||||
|
=== RUN TestUpload_Success
|
||||||
|
--- PASS: TestUpload_Success (0.00s)
|
||||||
|
=== RUN TestDownload_Success
|
||||||
|
--- PASS: TestDownload_Success (0.00s)
|
||||||
|
=== RUN TestDownloadWithContext_Cancel
|
||||||
|
--- PASS: TestDownloadWithContext_Cancel (0.04s)
|
||||||
|
PASS
|
||||||
|
ok go-codex-client/communities 0.044s
|
||||||
|
```
|
||||||
|
|
||||||
|
To run the integration test, use `integration` tag and narrow the scope using `-run Integration`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
go test -v -tags=integration ./communities -run Integration -timeout 15s
|
||||||
|
```
|
||||||
48
cmd/download/main.go
Normal file
48
cmd/download/main.go
Normal file
@ -0,0 +1,48 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"log"
|
||||||
|
"os"
|
||||||
|
|
||||||
|
"go-codex-client/communities" // Import the local communities package
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
var (
|
||||||
|
host = flag.String("host", "localhost", "Codex host")
|
||||||
|
port = flag.String("port", "8080", "Codex port")
|
||||||
|
cid = flag.String("cid", "", "CID of the file to download")
|
||||||
|
file = flag.String("file", "downloaded-file.bin", "File to save the downloaded data")
|
||||||
|
)
|
||||||
|
|
||||||
|
flag.Parse()
|
||||||
|
|
||||||
|
if *cid == "" {
|
||||||
|
log.Fatal("CID is required (use -cid flag)")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create Codex client
|
||||||
|
client := communities.NewCodexClient(*host, *port)
|
||||||
|
|
||||||
|
// Create output file
|
||||||
|
outputFile, err := os.Create(*file)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("Failed to create output file %s: %v", *file, err)
|
||||||
|
}
|
||||||
|
defer outputFile.Close()
|
||||||
|
|
||||||
|
fmt.Printf("Downloading CID %s from Codex at %s:%s...\n", *cid, *host, *port)
|
||||||
|
|
||||||
|
// Download data - pass the io.Writer (outputFile), not the string
|
||||||
|
err = client.Download(*cid, outputFile)
|
||||||
|
if err != nil {
|
||||||
|
// Clean up the failed/partial file
|
||||||
|
os.Remove(*file)
|
||||||
|
log.Fatalf("Download failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Printf("✅ Download successful!\n")
|
||||||
|
fmt.Printf("Saved to: %s\n", *file)
|
||||||
|
}
|
||||||
45
cmd/upload/main.go
Normal file
45
cmd/upload/main.go
Normal file
@ -0,0 +1,45 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"log"
|
||||||
|
"os"
|
||||||
|
|
||||||
|
"go-codex-client/communities" // Import the local communities package
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
var (
|
||||||
|
host = flag.String("host", "localhost", "Codex host")
|
||||||
|
port = flag.String("port", "8080", "Codex port")
|
||||||
|
file = flag.String("file", "test-data.bin", "File to upload")
|
||||||
|
filename = flag.String("name", "", "Filename to use in upload (defaults to actual filename)")
|
||||||
|
)
|
||||||
|
flag.Parse()
|
||||||
|
|
||||||
|
// Read file data
|
||||||
|
data, err := os.ReadFile(*file)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("Failed to read file %s: %v", *file, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Use actual filename if name not specified
|
||||||
|
uploadName := *filename
|
||||||
|
if uploadName == "" {
|
||||||
|
uploadName = *file
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Printf("Uploading %s (%d bytes) to Codex at %s:%s...\n", *file, len(data), *host, *port)
|
||||||
|
// Create Codex client and upload
|
||||||
|
client := communities.NewCodexClient(*host, *port)
|
||||||
|
|
||||||
|
cid, err := client.Upload(bytes.NewReader(data), uploadName)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("Upload failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Printf("✅ Upload successful!\n")
|
||||||
|
fmt.Printf("CID: %s\n", cid)
|
||||||
|
}
|
||||||
164
communities/codex_client.go
Normal file
164
communities/codex_client.go
Normal file
@ -0,0 +1,164 @@
|
|||||||
|
/* Package communities
|
||||||
|
*
|
||||||
|
* Provides a CodexClient type that you can use to conveniently
|
||||||
|
* upload buffers to Codex.
|
||||||
|
*
|
||||||
|
*/
|
||||||
|
package communities
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// CodexClient handles basic upload/download operations with Codex storage
|
||||||
|
type CodexClient struct {
|
||||||
|
BaseURL string
|
||||||
|
Client *http.Client
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewCodexClient creates a new Codex client
|
||||||
|
func NewCodexClient(host string, port string) *CodexClient {
|
||||||
|
return &CodexClient{
|
||||||
|
BaseURL: fmt.Sprintf("http://%s:%s", host, port),
|
||||||
|
Client: &http.Client{Timeout: 60 * time.Second},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Upload uploads data from a reader to Codex and returns the CID
|
||||||
|
func (c *CodexClient) Upload(data io.Reader, filename string) (string, error) {
|
||||||
|
url := fmt.Sprintf("%s/api/codex/v1/data", c.BaseURL)
|
||||||
|
|
||||||
|
// Create the HTTP request
|
||||||
|
req, err := http.NewRequest("POST", url, data)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("failed to create request: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Set headers
|
||||||
|
req.Header.Set("Content-Type", "application/octet-stream")
|
||||||
|
req.Header.Set("Content-Disposition", fmt.Sprintf(`filename="%s"`, filename))
|
||||||
|
|
||||||
|
// Send request
|
||||||
|
resp, err := c.Client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("failed to upload to codex: %w", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
// Check if request was successful
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
body, _ := io.ReadAll(resp.Body)
|
||||||
|
return "", fmt.Errorf("codex upload failed with status %d: %s", resp.StatusCode, string(body))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read the CID response
|
||||||
|
cidBytes, err := io.ReadAll(resp.Body)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("failed to read response: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
cid := strings.TrimSpace(string(cidBytes))
|
||||||
|
return cid, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Download downloads data from Codex by CID and writes it to the provided writer
|
||||||
|
func (c *CodexClient) Download(cid string, output io.Writer) error {
|
||||||
|
return c.DownloadWithContext(context.Background(), cid, output)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *CodexClient) LocalDownload(cid string, output io.Writer) error {
|
||||||
|
return c.LocalDownloadWithContext(context.Background(), cid, output)
|
||||||
|
}
|
||||||
|
|
||||||
|
// DownloadWithContext downloads data from Codex by CID with cancellation support
|
||||||
|
func (c *CodexClient) DownloadWithContext(ctx context.Context, cid string, output io.Writer) error {
|
||||||
|
url := fmt.Sprintf("%s/api/codex/v1/data/%s/network/stream", c.BaseURL, cid)
|
||||||
|
|
||||||
|
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to create request: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
resp, err := c.Client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to download from codex: %w", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
body, _ := io.ReadAll(resp.Body)
|
||||||
|
return fmt.Errorf("codex download failed with status %d: %s", resp.StatusCode, string(body))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Use context-aware copy for cancellable streaming
|
||||||
|
return c.copyWithContext(ctx, output, resp.Body)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *CodexClient) LocalDownloadWithContext(ctx context.Context, cid string, output io.Writer) error {
|
||||||
|
url := fmt.Sprintf("%s/api/codex/v1/data/%s", c.BaseURL, cid)
|
||||||
|
|
||||||
|
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to create request: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
resp, err := c.Client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to download from codex: %w", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
body, _ := io.ReadAll(resp.Body)
|
||||||
|
return fmt.Errorf("codex download failed with status %d: %s", resp.StatusCode, string(body))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Use context-aware copy for cancellable streaming
|
||||||
|
return c.copyWithContext(ctx, output, resp.Body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// copyWithContext performs io.Copy but respects context cancellation
|
||||||
|
func (c *CodexClient) copyWithContext(ctx context.Context, dst io.Writer, src io.Reader) error {
|
||||||
|
// Create a buffer for chunked copying
|
||||||
|
buf := make([]byte, 64*1024) // 64KB buffer
|
||||||
|
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err() // Return cancellation error
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read a chunk
|
||||||
|
n, err := src.Read(buf)
|
||||||
|
if n > 0 {
|
||||||
|
// Write the chunk
|
||||||
|
if _, writeErr := dst.Write(buf[:n]); writeErr != nil {
|
||||||
|
return fmt.Errorf("failed to write data: %w", writeErr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if err == io.EOF {
|
||||||
|
return nil // Successful completion
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to read data: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetRequestTimeout sets the HTTP client timeout for requests
|
||||||
|
func (c *CodexClient) SetRequestTimeout(timeout time.Duration) {
|
||||||
|
c.Client.Timeout = timeout
|
||||||
|
}
|
||||||
|
|
||||||
|
// UploadArchive is a convenience method for uploading archive data
|
||||||
|
func (c *CodexClient) UploadArchive(encodedArchive []byte) (string, error) {
|
||||||
|
return c.Upload(bytes.NewReader(encodedArchive), "archive-data.bin")
|
||||||
|
}
|
||||||
70
communities/codex_client_integration_test.go
Normal file
70
communities/codex_client_integration_test.go
Normal file
@ -0,0 +1,70 @@
|
|||||||
|
//go:build integration
|
||||||
|
// +build integration
|
||||||
|
|
||||||
|
package communities
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"crypto/rand"
|
||||||
|
"encoding/hex"
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// This test exercises real network calls against a running Codex node.
|
||||||
|
// It is disabled by default via the "integration" build tag.
|
||||||
|
// Run with:
|
||||||
|
//
|
||||||
|
// go test -v -tags=integration ./communities -run Integration
|
||||||
|
//
|
||||||
|
// Required env vars (with defaults):
|
||||||
|
//
|
||||||
|
// CODEX_HOST (default: localhost)
|
||||||
|
// CODEX_API_PORT (default: 8080)
|
||||||
|
// CODEX_TIMEOUT_MS (optional; default: 60000)
|
||||||
|
func TestIntegration_UploadAndDownload(t *testing.T) {
|
||||||
|
host := getenv("CODEX_HOST", "localhost")
|
||||||
|
port := getenv("CODEX_API_PORT", "8080")
|
||||||
|
client := NewCodexClient(host, port)
|
||||||
|
|
||||||
|
// Optional request timeout override
|
||||||
|
if ms := os.Getenv("CODEX_TIMEOUT_MS"); ms != "" {
|
||||||
|
if d, err := time.ParseDuration(ms + "ms"); err == nil {
|
||||||
|
client.SetRequestTimeout(d)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Generate random payload to ensure proper round-trip verification
|
||||||
|
payload := make([]byte, 1024)
|
||||||
|
if _, err := rand.Read(payload); err != nil {
|
||||||
|
t.Fatalf("failed to generate random payload: %v", err)
|
||||||
|
}
|
||||||
|
t.Logf("Generated payload (first 32 bytes hex): %s", hex.EncodeToString(payload[:32]))
|
||||||
|
|
||||||
|
cid, err := client.Upload(bytes.NewReader(payload), "it.bin")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("upload failed: %v", err)
|
||||||
|
}
|
||||||
|
t.Logf("Upload successful, CID: %s", cid)
|
||||||
|
|
||||||
|
// Download via network stream with a context timeout to avoid hanging
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
if err := client.DownloadWithContext(ctx, cid, &buf); err != nil {
|
||||||
|
t.Fatalf("download failed: %v", err)
|
||||||
|
}
|
||||||
|
if got := buf.Bytes(); !bytes.Equal(got, payload) {
|
||||||
|
t.Fatalf("payload mismatch: got %q want %q", string(got), string(payload))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func getenv(k, def string) string {
|
||||||
|
if v := os.Getenv(k); v != "" {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
return def
|
||||||
|
}
|
||||||
143
communities/codex_client_test.go
Normal file
143
communities/codex_client_test.go
Normal file
@ -0,0 +1,143 @@
|
|||||||
|
package communities
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestUpload_Success(t *testing.T) {
|
||||||
|
// Arrange a fake Codex server that validates headers and returns a CID
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.Method != http.MethodPost {
|
||||||
|
w.WriteHeader(http.StatusMethodNotAllowed)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if r.URL.Path != "/api/codex/v1/data" {
|
||||||
|
w.WriteHeader(http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if ct := r.Header.Get("Content-Type"); ct != "application/octet-stream" {
|
||||||
|
w.WriteHeader(http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if cd := r.Header.Get("Content-Disposition"); cd != "filename=\"hello.txt\"" {
|
||||||
|
w.WriteHeader(http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
_, _ = io.ReadAll(r.Body) // consume body
|
||||||
|
_ = r.Body.Close()
|
||||||
|
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
// Codex returns CIDv1 base58btc
|
||||||
|
// prefix: zDv
|
||||||
|
// - z = multibase prefix for base58btc
|
||||||
|
// - Dv = CIDv1 prefix for raw codex
|
||||||
|
// we add a newline to simulate real response
|
||||||
|
_, _ = w.Write([]byte("zDvZRwzmTestCID123\n"))
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
client := NewCodexClient("localhost", "8080")
|
||||||
|
client.BaseURL = server.URL
|
||||||
|
|
||||||
|
// Act
|
||||||
|
cid, err := client.Upload(bytes.NewReader([]byte("payload")), "hello.txt")
|
||||||
|
|
||||||
|
// Assert
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
// Codex uses CIDv1 with base58btc encoding (prefix: zDv)
|
||||||
|
if cid != "zDvZRwzmTestCID123" {
|
||||||
|
t.Fatalf("unexpected cid: %q", cid)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDownload_Success(t *testing.T) {
|
||||||
|
const wantCID = "zDvZRwzm"
|
||||||
|
const payload = "hello from codex"
|
||||||
|
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.Method != http.MethodGet {
|
||||||
|
w.WriteHeader(http.StatusMethodNotAllowed)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if r.URL.Path != "/api/codex/v1/data/"+wantCID+"/network/stream" {
|
||||||
|
w.WriteHeader(http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
_, _ = w.Write([]byte(payload))
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
client := NewCodexClient("localhost", "8080")
|
||||||
|
client.BaseURL = server.URL
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
if err := client.Download(wantCID, &buf); err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
if got := buf.String(); got != payload {
|
||||||
|
t.Fatalf("unexpected payload: %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDownloadWithContext_Cancel(t *testing.T) {
|
||||||
|
const cid = "zDvZRwzm"
|
||||||
|
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.URL.Path != "/api/codex/v1/data/"+cid+"/network/stream" {
|
||||||
|
w.WriteHeader(http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/octet-stream")
|
||||||
|
flusher, _ := w.(http.Flusher)
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
// Stream data slowly so the request can be canceled
|
||||||
|
for i := 0; i < 1000; i++ {
|
||||||
|
select {
|
||||||
|
case <-r.Context().Done():
|
||||||
|
return
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
if _, err := w.Write([]byte("x")); err != nil {
|
||||||
|
// Client likely went away; stop writing
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if flusher != nil {
|
||||||
|
flusher.Flush()
|
||||||
|
}
|
||||||
|
time.Sleep(10 * time.Millisecond)
|
||||||
|
}
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
client := NewCodexClient("localhost", "8080")
|
||||||
|
client.BaseURL = server.URL
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
err := client.DownloadWithContext(ctx, cid, io.Discard)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("expected cancellation error, got nil")
|
||||||
|
}
|
||||||
|
// Accept either canceled or deadline exceeded depending on timing
|
||||||
|
if !errors.Is(err, context.Canceled) && !errors.Is(err, context.DeadlineExceeded) {
|
||||||
|
// net/http may wrap the context error; check error string as a fallback
|
||||||
|
es := err.Error()
|
||||||
|
if !(es == context.Canceled.Error() || es == context.DeadlineExceeded.Error()) {
|
||||||
|
t.Fatalf("expected context cancellation, got: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
1
test-data.bin
Normal file
1
test-data.bin
Normal file
@ -0,0 +1 @@
|
|||||||
|
Hello, Codex! This is test data for upload.
|
||||||
Loading…
x
Reference in New Issue
Block a user