3.9 KiB
Run tests with Docker in Kubernetes
We may run tests on local or remote Kubernetes cluster. Local cluster flow uses direct access to nodes ports and this is why we introduced a different way to check services ports, for more information please see Tests run modes. Configuration option RUNNERLOCATION
is responsible for that.
For local run it is easier to install .Net and run tests on Docker Desktop Kubernetes cluster. In case of remote run we do not expose services via Ingress Controller and we can't access cluster nodes, this is why we should run tests only inside the Kubernetes.
We can run tests on remote cluster in the following ways
Run pod inside the cluster using generic .Net image
steps
-
Create dist-tests-runner.yaml
-- apiVersion: v1 kind: Pod metadata: name: dist-tests-runner namespace: default spec: containers: - name: dotnet image: mcr.microsoft.com/dotnet/sdk:7.0 command: ["sleep", "infinity"]
-
Deploy pod in the cluster
kubectl apply -f dist-tests-runner.yaml
-
Copy kubeconfig to the pod
kubectl cp kubeconfig.yaml dist-tests-runner:/opt
-
Exec into the pod via kubectl or OpenLens
kubectl exec -it dist-tests-runner -- bash
-
Clone repository inside the pod
git clone https://github.com/codex-storage/cs-codex-dist-tests.git
-
Update kubeconfig option in config file
cd cs-codex-dist-tests vi DistTestCore/Configuration.cs
GetNullableEnvVarOrDefault("KUBECONFIG", "/opt/kubeconfig.yaml")
-
Run tests
dotnet test Tests
-
Check the results and analyze the logs
Run pod inside the cluster using prepared Docker images
Before the run we should create some objects inside the cluster
- Namespace where we will run the image
- Service Account to run tests inside the cluster
- Secret with kubeconfig for created SA
- Configmap with custom app config if required
For more information please see Manual run inside Kubernetes via Job
Then we need to create a manifest to run the pod
runner.yaml
---
apiVersion: v1
kind: Pod
metadata:
name: dist-tests-runner
namespace: cs-codex-dist-tests
labels:
name: cs-codex-dist-tests
spec:
containers:
- name: cs-codex-dist-tests
image: codexstorage/cs-codex-dist-tests:sha-671ee4e
env:
- name: RUNNERLOCATION
value: InternalToCluster
- name: KUBECONFIG
value: /opt/kubeconfig.yaml
- name: CONFIG
value: "/opt/Configuration.cs"
- name: CONFIG_SHOW
value: "true"
volumeMounts:
- name: kubeconfig
mountPath: /opt/kubeconfig.yaml
subPath: kubeconfig.yaml
- name: config
mountPath: /opt/Configuration.cs
subPath: Configuration.cs
- name: logs
mountPath: /var/log/cs-codex-dist-tests
# command:
# - "dotnet"
# - "test"
# - "Tests"
restartPolicy: Never
volumes:
- name: kubeconfig
secret:
secretName: cs-codex-dist-tests-app-kubeconfig
- name: config
configMap:
name: cs-codex-dist-tests
- name: logs
hostPath:
path: /var/log/cs-codex-dist-tests
For more information about pod variables please see job.yaml.
And then apply it
kubectl apply -f runner.yaml
After the pod run, custom entrypoint will do the following
- Clone repository
- Switch to the specific branch -
master
by default - Run all tests -
dotnet test