cs-codex-dist-tests/docker

Run tests with Docker in Kubernetes

We may run tests on local or remote Kubernetes cluster. Local cluster flow uses direct access to nodes ports and this is why we introduced a different way to check services ports, for more information please see Tests run modes. Configuration option RUNNERLOCATION is responsible for that.

For local run it is easier to install .Net and run tests on Docker Desktop Kubernetes cluster. In case of remote run we do not expose services via Ingress Controller and we can't access cluster nodes, this is why we should run tests only inside the Kubernetes.

We can run tests on remote cluster in the following ways

Run pod inside the cluster using generic .Net image

steps

1. Create dist-tests-runner.yaml

      --
      apiVersion: v1
      kind: Pod
      metadata:
      name: dist-tests-runner
      namespace: default
      spec:
      containers:
      - name: dotnet
         image: mcr.microsoft.com/dotnet/sdk:7.0
         command: ["sleep", "infinity"]
   

2. Deploy pod in the cluster

   kubectl apply -f dist-tests-runner.yaml
   

3. Copy kubeconfig to the pod

   kubectl cp kubeconfig.yaml dist-tests-runner:/opt
   

4. Exec into the pod via kubectl or OpenLens

   kubectl exec -it dist-tests-runner -- bash
   

5. Clone repository inside the pod

   git clone https://github.com/codex-storage/cs-codex-dist-tests.git
   

6. Update kubeconfig option in config file

   cd cs-codex-dist-tests
   vi DistTestCore/Configuration.cs
   

   GetNullableEnvVarOrDefault("KUBECONFIG", "/opt/kubeconfig.yaml")
   

7. Run tests

   dotnet test Tests
   

8. Check the results and analyze the logs

Run pod inside the cluster using prepared Docker images

Before the run we should create some objects inside the cluster

1. Namespace where we will run the image
2. Service Account to run tests inside the cluster
3. Secret with kubeconfig for created SA
4. Configmap with custom app config if required

For more information please see Manual run inside Kubernetes via Job

Then we need to create a manifest to run the pod

runner.yaml

---
apiVersion: v1
kind: Pod
metadata:
  name: dist-tests-runner
  namespace: cs-codex-dist-tests
  labels:
    name: cs-codex-dist-tests
spec:
  containers:
  - name: cs-codex-dist-tests
    image: codexstorage/cs-codex-dist-tests:sha-671ee4e
    env:
    - name: RUNNERLOCATION
      value: InternalToCluster
    - name: KUBECONFIG
      value: /opt/kubeconfig.yaml
    - name: CONFIG
      value: "/opt/Configuration.cs"
    - name: CONFIG_SHOW
      value: "true"
    volumeMounts:
    - name: kubeconfig
      mountPath: /opt/kubeconfig.yaml
      subPath: kubeconfig.yaml
    - name: config
      mountPath: /opt/Configuration.cs
      subPath: Configuration.cs
    - name: logs
      mountPath: /var/log/cs-codex-dist-tests
    # command:
    # - "dotnet"
    # - "test"
    # - "Tests"
  restartPolicy: Never
  volumes:
    - name: kubeconfig
      secret:
        secretName: cs-codex-dist-tests-app-kubeconfig
    - name: config
      configMap:
        name: cs-codex-dist-tests
    - name: logs
      hostPath:
        path: /var/log/cs-codex-dist-tests

For more information about pod variables please see job.yaml.

And then apply it

kubectl apply -f runner.yaml

After the pod run, custom entrypoint will do the following

1. Clone repository
2. Switch to the specific branch - master by default
3. Run all tests - dotnet test