Merge branch 'master' into feature/codex-net-deployer

2023-07-12 14:40:14 +02:00 · 2023-07-12 14:40:14 +02:00 · 512aeb79a0
parent 42acb862b6 1f2635ea7d
commit 512aeb79a0
2 changed files with 132 additions and 78 deletions
--- a/.github/workflows/docker.yml
+++ b/.github/workflows/docker.yml
@ -78,31 +78,7 @@ jobs:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

-      - name: Docker - Build and export to Docker
-        uses: docker/build-push-action@v4
-        with:
-          context: .
-          file: ${{ env.DOCKER_FILE }}
-          platforms: ${{ env.PLATFORM }}
-          load: true
-          tags: ${{ steps.meta.outputs.tags }}
-          labels: ${{ steps.meta.outputs.labels }}
-
-      - name: Docker - Minify image
-        uses: kitabisa/docker-slim-action@v1
-        id: slim
-        env:
-          DSLIM_HTTP_PROBE: false
-        with:
-          target: ${{ steps.meta.outputs.tags }}
-          overwrite: true
-
-      - name: Docker - Show slim report
-        run: echo "${REPORT}" | jq -r
-        env:
-          REPORT: ${{ steps.slim.outputs.report }}
-
-      - name: Docker - Push to Docker registry
+      - name: Docker - Build and Push
        uses: docker/build-push-action@v4
        with:
          context: .
@ -145,4 +121,3 @@ jobs:
          inputs: ${{ env.TAGS }}
          images: ${{ needs.build.outputs.tags-linux-amd64 }},${{ needs.build.outputs.tags-linux-arm64 }}
          push: true
-
--- a/docker/README.md
+++ b/docker/README.md
@ -1,59 +1,138 @@
 # Run tests with Docker in Kubernetes

-We may [run tests localy](../LOCALSETUP.MD) using installed Dotnet and inside Kubernetes we may use a [prepared Docker images](https://hub.docker.com/r/codexstorage/cs-codex-dist-tests/tags).
+ We may [run tests on local](../LOCALSETUP.MD) or remote Kubernetes cluster. Local cluster flow uses direct access to nodes ports and this is why we introduced a different way to check services ports, for more information please see [Tests run modes](../../../issues/20). Configuration option `RUNNERLOCATION` is responsible for that.


-Custom [entrypoint](docker-entrypoint.sh) will do the following
+ For local run it is easier to install .Net and run tests on Docker Desktop Kubernetes cluster. In case of remote run we do not expose services via Ingress Controller and we can't access cluster nodes, this is why we should run tests only inside the Kubernetes.
+
+ We can run tests on remote cluster in the following ways
+
+#### Run pod inside the cluster using generic .Net image
+
+<details>
+<summary>steps</summary>
+
+1. Create dist-tests-runner.yaml
+   ```yaml
+      --
+      apiVersion: v1
+      kind: Pod
+      metadata:
+      name: dist-tests-runner
+      namespace: default
+      spec:
+      containers:
+      - name: dotnet
+         image: mcr.microsoft.com/dotnet/sdk:7.0
+         command: ["sleep", "infinity"]
+   ```
+2. Deploy pod in the cluster
+   ```shell
+   kubectl apply -f dist-tests-runner.yaml
+   ```
+
+3. Copy kubeconfig to the pod
+   ```shell
+   kubectl cp kubeconfig.yaml dist-tests-runner:/opt
+   ```
+
+4. Exec into the pod via kubectl or [OpenLens](https://github.com/MuhammedKalkan/OpenLens)
+   ```shell
+   kubectl exec -it dist-tests-runner -- bash
+   ```
+
+5. Clone repository inside the pod
+   ```shell
+   git clone https://github.com/codex-storage/cs-codex-dist-tests.git
+   ```
+
+6. Update kubeconfig option in config file
+   ```shell
+   cd cs-codex-dist-tests
+   vi DistTestCore/Configuration.cs
+   ```
+   ```dotnet
+   GetNullableEnvVarOrDefault("KUBECONFIG", "/opt/kubeconfig.yaml")
+   ```
+
+7. Run tests
+   ```shell
+   dotnet test Tests
+   ```
+
+8. Check the results and analyze the logs
+</details>
+
+#### Run pod inside the cluster using [prepared Docker images](https://hub.docker.com/r/codexstorage/cs-codex-dist-tests/tags)
+
+ Before the run we should create some objects inside the cluster
+ 1. Namespace where we will run the image
+ 2. [Service Account to run tests inside the cluster](https://github.com/codex-storage/cs-codex-dist-tests/issues/21)
+ 3. Secret with kubeconfig for created SA
+ 4. Configmap with custom app config if required
+
+ For more information please see [Manual run inside Kubernetes via Job](../../../issues/7)
+
+ Then we need to create a manifest to run the pod
+ <details>
+ <summary>runner.yaml</summary>
+
+ ```yaml
+ ---
+ apiVersion: v1
+ kind: Pod
+ metadata:
+   name: dist-tests-runner
+   namespace: cs-codex-dist-tests
+   labels:
+     name: cs-codex-dist-tests
+ spec:
+   containers:
+   - name: cs-codex-dist-tests
+     image: codexstorage/cs-codex-dist-tests:sha-671ee4e
+     env:
+     - name: RUNNERLOCATION
+       value: InternalToCluster
+     - name: KUBECONFIG
+       value: /opt/kubeconfig.yaml
+     - name: CONFIG
+       value: "/opt/Configuration.cs"
+     - name: CONFIG_SHOW
+       value: "true"
+     volumeMounts:
+     - name: kubeconfig
+       mountPath: /opt/kubeconfig.yaml
+       subPath: kubeconfig.yaml
+     - name: config
+       mountPath: /opt/Configuration.cs
+       subPath: Configuration.cs
+     - name: logs
+       mountPath: /var/log/cs-codex-dist-tests
+     # command:
+     # - "dotnet"
+     # - "test"
+     # - "Tests"
+   restartPolicy: Never
+   volumes:
+     - name: kubeconfig
+       secret:
+         secretName: cs-codex-dist-tests-app-kubeconfig
+     - name: config
+       configMap:
+         name: cs-codex-dist-tests
+     - name: logs
+       hostPath:
+         path: /var/log/cs-codex-dist-tests
+ ```
+ For more information about pod variables please see [job.yaml](job.yaml).
+ </details>
+
+ And then apply it
+ ```shell
+ kubectl apply -f runner.yaml
+ ```
+
+ After the pod run, custom [entrypoint](docker-entrypoint.sh) will do the following
 1. Clone repository
 2. Switch to the specific branch - `master` by default
 3. Run all tests - `dotnet test`
-
-**Run with defaults**
-```bash
-docker run \
-  --rm \
-  --name cs-codex-dist-tests \
-  codexstorage/cs-codex-dist-tests:sha-686757e
-```
-
-**Just short tests**
-```bash
-docker run \
-  --rm \
-  --name cs-codex-dist-tests \
-  codexstorage/cs-codex-dist-tests:sha-686757e \
-  dotnet test Tests
-```
-
-**Custom branch**
-```bash
-docker run \
-  --rm \
-  --name cs-codex-dist-tests \
-  --env BRANCH=feature/tests \
-  codexstorage/cs-codex-dist-tests:sha-686757e
-```
-
-**Custom local config**
-```bash
-docker run \
-  --rm \
-  --name cs-codex-dist-tests \
-  --env CONFIG=/opt/Configuration.cs \
-  --env CONFIG_SHOW=true \
-  --volume $PWD/DistTestCore/Configuration.cs:/opt/Configuration.cs \
-  codexstorage/cs-codex-dist-tests:sha-686757e
-```
-
-**Local kubeconfig with custom local config**
-```bash
-docker run \
-  --rm \
-  --name cs-codex-dist-tests \
-  --env CONFIG=/opt/Configuration.cs \
-  --env CONFIG_SHOW=true \
-  --env SOURCE=https://github.com/codex-storage/cs-codex-dist-tests.git \
-  --volume $PWD/DistTestCore/Configuration.cs:/opt/Configuration.cs \
-  --volume $PWD/kubeconfig.yml:/opt/kubeconfig.yml \
-  codexstorage/cs-codex-dist-tests:sha-686757e
-```