logos-storage-nim-cs-dist-tests/docs/RunningReleaseTests.md

Running Release Tests

This guide covers running the release tests both locally (against a local Kubernetes cluster) and on Digital Ocean (or any remote Kubernetes cluster).

---

1. Running Locally (Docker Desktop Kubernetes)

Prerequisites

1. Install .NET 10.0+. (If you install a newer version, update all net10.0 references in the .csproj files to match.)
2. Install Docker Desktop.
3. Enable Kubernetes in Docker Desktop: Settings → Kubernetes → Enable Kubernetes (kubeadm) → Apply & Restart. This may take a few minutes.

Note on Kubernetes client compatibility: The KubernetesClient package version in the KubernetesWorkflow project must be compatible with the Kubernetes version that kubeadm exposes. For example, if kubeadm exposes Kubernetes 1.34.1, then KubernetesClient must be version 18.x.

How it works

When you run dotnet test from your machine, the framework detects it is running outside the cluster (by checking whether the KUBERNETES_PORT and KUBERNETES_SERVICE_HOST environment variables are set). In this mode it connects to the cluster via your local ~/.kube/config, which Docker Desktop automatically configures.

Each test creates its own isolated namespace in the cluster, starts the required Storage nodes as pods, runs the test, then tears everything down.

Verify the cluster is working

Before running tests, confirm Docker Desktop's Kubernetes context is active:

kubectl config current-context   # should show "docker-desktop"
kubectl get nodes                # should show a Ready node

If the current context is not docker-desktop (e.g. it points to a remote cluster), switch it:

kubectl config use-context docker-desktop

Run the tests

Most IDEs let you run individual tests or test fixtures directly from the code file. To run from the command line:

cd /path/to/logos-storage-nim-cs-dist-tests

# Run all release tests
dotnet test Tests/LogosStorageReleaseTests

# Run a specific test by name
dotnet test Tests/LogosStorageReleaseTests --filter=OneClientTest

# Run with verbose output
dotnet test Tests/LogosStorageReleaseTests --logger="console;verbosity=detailed"

Useful environment variables

Variable	Default	Description
STORAGEDOCKERIMAGE	logosstorage/logos-storage-nim:latest-dist-tests	Storage node image to test
KUBECONFIG	~/.kube/config	Path to kubeconfig file (optional when using Docker Desktop)
LOGPATH	LogosStorageTestLogs (relative)	Directory for test logs
DATAFILEPATH	TestDataFiles (relative)	Directory for test data files
ALWAYS_LOGS	(unset)	Set to any non-empty value to always download container logs (not just on failure)
TEST_TYPE	(unset)	Set to release-tests only when running inside the cluster (see §2c). Do not set this for local runs — it activates long in-cluster timeouts.

Example — run against a specific Storage image:

STORAGEDOCKERIMAGE=logosstorage/logos-storage-nim:v0.1.8 dotnet test Tests/LogosStorageReleaseTests

Troubleshooting

NullReferenceException at DistTest..ctor()

If every test fails immediately with a stack trace ending at DistTest.GetWebCallTimeSet() or DistTest.GetK8sTimeSet(), check whether TEST_TYPE is set to release-tests in your shell:

echo $TEST_TYPE

If it is, unset it before running locally:

unset TEST_TYPE
dotnet test Tests/LogosStorageReleaseTests

Setting TEST_TYPE=release-tests triggers in-cluster detection which tries to log a message through an object that hasn't been constructed yet, causing the crash.

Tests fail with kubectl errors / wrong cluster

Ensure the active context is docker-desktop, not a remote cluster:

kubectl config current-context      # confirm "docker-desktop"
kubectl config use-context docker-desktop   # switch if needed
kubectl get nodes                   # should show a Ready node

Image not found

The STORAGEDOCKERIMAGE must be pullable from Docker Desktop. Either use a published image or build and push locally:

docker pull logosstorage/logos-storage-nim:latest-dist-tests

---

2. Running on Digital Ocean (Remote Kubernetes)

Overview

On a remote cluster the test runner itself must run inside the cluster, because the framework needs direct pod-to-pod networking. The CI workflow does this automatically by creating a Kubernetes Job that runs the test runner image. You can also do it manually.

Prerequisites

• A running Digital Ocean Kubernetes (DOKS) cluster
• kubectl configured to talk to that cluster
• The cluster must be pre-configured (see below)

2a. Cluster pre-configuration

Do these steps once per cluster.

1. Label the worker nodes that will run the test runner pod

kubectl label node <node-name> workload-type=tests-runners-ci

The job manifest uses a nodeSelector for this label, so at least one node must have it.

2. Create the kubeconfig secret for the test runner

The test runner pod itself needs a kubeconfig to manage pods inside the cluster. Use a static service-account-based kubeconfig — avoid copying your local ~/.kube/config directly, as it likely uses an exec credential plugin (e.g. doctl) that won't be available inside the pod.

# Create a service account and grant it cluster-admin access
kubectl create serviceaccount dist-tests-app
kubectl create clusterrolebinding dist-tests-app \
  --clusterrole=cluster-admin \
  --serviceaccount=default:dist-tests-app

# Create a long-lived static token
kubectl create token dist-tests-app --duration=8760h > /tmp/sa-token.txt

# Build a static kubeconfig using the token
CLUSTER=$(kubectl config view --minify -o jsonpath='{.clusters[0].name}')
SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
kubectl config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 --decode > /tmp/ca.crt

kubectl --kubeconfig=/tmp/static-kubeconfig.yaml config set-cluster "$CLUSTER" \
  --server="$SERVER" \
  --certificate-authority=/tmp/ca.crt \
  --embed-certs=true
kubectl --kubeconfig=/tmp/static-kubeconfig.yaml config set-credentials dist-tests-app \
  --token=$(cat /tmp/sa-token.txt)
kubectl --kubeconfig=/tmp/static-kubeconfig.yaml config set-context default \
  --cluster="$CLUSTER" --user=dist-tests-app
kubectl --kubeconfig=/tmp/static-kubeconfig.yaml config use-context default

# Store it as a secret
kubectl create secret generic storage-dist-tests-app-kubeconfig \
  --from-file=kubeconfig.yaml=/tmp/static-kubeconfig.yaml \
  -n default

The pod mounts this at /opt/kubeconfig.yaml and passes it via KUBECONFIG.

3. (If not already present) Create the system-node-critical priority class

The job manifest requests priorityClassName: system-node-critical. On most clusters this exists already; check with:

kubectl get priorityclass system-node-critical

If missing, create it or change the priority class name in docker/job-release-tests.yaml.

2b. Running via GitHub Actions (recommended)

This is the standard automated path.

Required GitHub secret:

Secret	Value
KUBE_CONFIG	Base64-encoded kubeconfig with permissions to create Jobs/Pods in the cluster

Encode your kubeconfig:

base64 -i ~/.kube/do-cluster-config | pbcopy   # macOS
base64 -w0 ~/.kube/do-cluster-config           # Linux

Trigger the workflow:

Go to Actions → Run Release Tests → Run workflow and provide:

Input	Description
storagedockerimage	Image to test, e.g. logosstorage/logos-storage-nim:v0.1.8-dist-tests

Optional inputs (for non-default branches/repos):

Input	Description
source	Repository URL (defaults to current repo)
branch	Branch to clone for running tests (defaults to current branch)

What happens:

1. GitHub Actions decodes KUBE_CONFIG and runs kubectl
2. Creates a Kubernetes Job from docker/job-release-tests.yaml with substituted variables
3. The Job runs the logosstorage/logos-storage-dist-tests:latest image
4. The entrypoint clones this repo, runs dotnet test Tests/LogosStorageReleaseTests
5. The workflow streams pod logs and fails if the Job does not complete successfully

2c. Running manually with kubectl

Useful for debugging or one-off runs without CI.

Set the required variables:

export NAMESPACE=default
export NAMEPREFIX=r-tests-manual
export RUNID=$(date +%Y%m%d-%H%M%S)
export TESTID=$(git rev-parse --short HEAD)
export TEST_TYPE=release-tests
export SOURCE=https://github.com/logos-storage/logos-storage-nim-cs-dist-tests.git
export BRANCH=master
export STORAGEDOCKERIMAGE=logosstorage/logos-storage-nim:latest-dist-tests
export COMMAND='["dotnet","test","Tests/LogosStorageReleaseTests"]'

Apply the job:

envsubst < docker/job-release-tests.yaml | kubectl apply -f -

Follow the logs:

# Wait for pod to start
kubectl get pod --selector job-name=$NAMEPREFIX -w

# Stream logs
POD=$(kubectl get pod --selector job-name=$NAMEPREFIX -o jsonpath='{.items[0].metadata.name}')
kubectl logs $POD -f

# Check final job status
kubectl get job $NAMEPREFIX -o jsonpath='{.status.conditions[0].type}'
# Should print "Complete"

Cleanup:

Jobs are auto-deleted after 24 hours (TTL configured in the manifest). To delete immediately:

kubectl delete job $NAMEPREFIX

Key differences: local vs. remote

	Local (Docker Desktop)	Remote (Digital Ocean)
Runner location	Your machine (external to cluster)	Inside a pod in the cluster
Kubeconfig	~/.kube/config (auto)	Mounted secret storage-dist-tests-app-kubeconfig
Network access to pods	Via kubectl port-forward / node IP	Direct pod-to-pod
RUNNERLOCATION detection	ExternalToCluster (automatic)	InternalToCluster (automatic inside pod)
How to run	dotnet test on your machine	Kubernetes Job
Image required	No (builds from source)	logosstorage/logos-storage-dist-tests:latest