139 lines
3.9 KiB
Markdown
139 lines
3.9 KiB
Markdown
# Run tests with Docker in Kubernetes
|
|
|
|
We may [run tests on local](../LOCALSETUP.MD) or remote Kubernetes cluster. Local cluster flow uses direct access to nodes ports and this is why we introduced a different way to check services ports, for more information please see [Tests run modes](../../../issues/20). Configuration option `RUNNERLOCATION` is responsible for that.
|
|
|
|
|
|
For local run it is easier to install .Net and run tests on Docker Desktop Kubernetes cluster. In case of remote run we do not expose services via Ingress Controller and we can't access cluster nodes, this is why we should run tests only inside the Kubernetes.
|
|
|
|
We can run tests on remote cluster in the following ways
|
|
|
|
#### Run pod inside the cluster using generic .Net image
|
|
|
|
<details>
|
|
<summary>steps</summary>
|
|
|
|
1. Create dist-tests-runner.yaml
|
|
```yaml
|
|
--
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
name: dist-tests-runner
|
|
namespace: default
|
|
spec:
|
|
containers:
|
|
- name: dotnet
|
|
image: mcr.microsoft.com/dotnet/sdk:7.0
|
|
command: ["sleep", "infinity"]
|
|
```
|
|
2. Deploy pod in the cluster
|
|
```shell
|
|
kubectl apply -f dist-tests-runner.yaml
|
|
```
|
|
|
|
3. Copy kubeconfig to the pod
|
|
```shell
|
|
kubectl cp kubeconfig.yaml dist-tests-runner:/opt
|
|
```
|
|
|
|
4. Exec into the pod via kubectl or [OpenLens](https://github.com/MuhammedKalkan/OpenLens)
|
|
```shell
|
|
kubectl exec -it dist-tests-runner -- bash
|
|
```
|
|
|
|
5. Clone repository inside the pod
|
|
```shell
|
|
git clone https://github.com/codex-storage/cs-codex-dist-tests.git
|
|
```
|
|
|
|
6. Update kubeconfig option in config file
|
|
```shell
|
|
cd cs-codex-dist-tests
|
|
vi DistTestCore/Configuration.cs
|
|
```
|
|
```dotnet
|
|
GetNullableEnvVarOrDefault("KUBECONFIG", "/opt/kubeconfig.yaml")
|
|
```
|
|
|
|
7. Run tests
|
|
```shell
|
|
dotnet test Tests
|
|
```
|
|
|
|
8. Check the results and analyze the logs
|
|
</details>
|
|
|
|
#### Run pod inside the cluster using [prepared Docker images](https://hub.docker.com/r/codexstorage/cs-codex-dist-tests/tags)
|
|
|
|
Before the run we should create some objects inside the cluster
|
|
1. Namespace where we will run the image
|
|
2. [Service Account to run tests inside the cluster](https://github.com/codex-storage/cs-codex-dist-tests/issues/21)
|
|
3. Secret with kubeconfig for created SA
|
|
4. Configmap with custom app config if required
|
|
|
|
For more information please see [Manual run inside Kubernetes via Job](../../../issues/7)
|
|
|
|
Then we need to create a manifest to run the pod
|
|
<details>
|
|
<summary>runner.yaml</summary>
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
name: dist-tests-runner
|
|
namespace: cs-codex-dist-tests
|
|
labels:
|
|
name: cs-codex-dist-tests
|
|
spec:
|
|
containers:
|
|
- name: cs-codex-dist-tests
|
|
image: codexstorage/cs-codex-dist-tests:sha-671ee4e
|
|
env:
|
|
- name: RUNNERLOCATION
|
|
value: InternalToCluster
|
|
- name: KUBECONFIG
|
|
value: /opt/kubeconfig.yaml
|
|
- name: CONFIG
|
|
value: "/opt/Configuration.cs"
|
|
- name: CONFIG_SHOW
|
|
value: "true"
|
|
volumeMounts:
|
|
- name: kubeconfig
|
|
mountPath: /opt/kubeconfig.yaml
|
|
subPath: kubeconfig.yaml
|
|
- name: config
|
|
mountPath: /opt/Configuration.cs
|
|
subPath: Configuration.cs
|
|
- name: logs
|
|
mountPath: /var/log/cs-codex-dist-tests
|
|
# command:
|
|
# - "dotnet"
|
|
# - "test"
|
|
# - "Tests"
|
|
restartPolicy: Never
|
|
volumes:
|
|
- name: kubeconfig
|
|
secret:
|
|
secretName: cs-codex-dist-tests-app-kubeconfig
|
|
- name: config
|
|
configMap:
|
|
name: cs-codex-dist-tests
|
|
- name: logs
|
|
hostPath:
|
|
path: /var/log/cs-codex-dist-tests
|
|
```
|
|
For more information about pod variables please see [job.yaml](job.yaml).
|
|
</details>
|
|
|
|
And then apply it
|
|
```shell
|
|
kubectl apply -f runner.yaml
|
|
```
|
|
|
|
After the pod run, custom [entrypoint](docker-entrypoint.sh) will do the following
|
|
1. Clone repository
|
|
2. Switch to the specific branch - `master` by default
|
|
3. Run all tests - `dotnet test`
|