From 085c4431a28ba9fa610a2113a62b4b945283e1ed Mon Sep 17 00:00:00 2001 From: benbierens Date: Mon, 1 May 2023 16:26:26 +0200 Subject: [PATCH 1/6] Setup and contributing instructions --- CONTRIBUTINGTESTS.MD | 24 ++++++++++++++++++++ LOCALSETUP.MD | 18 +++++++++++++++ README.md | 28 ++++++++++++++++-------- Tests/DurabilityTests/DurabilityTests.cs | 6 ++--- 4 files changed, 64 insertions(+), 12 deletions(-) create mode 100644 CONTRIBUTINGTESTS.MD create mode 100644 LOCALSETUP.MD diff --git a/CONTRIBUTINGTESTS.MD b/CONTRIBUTINGTESTS.MD new file mode 100644 index 00000000..e2e3ac43 --- /dev/null +++ b/CONTRIBUTINGTESTS.MD @@ -0,0 +1,24 @@ +# Distributed System Tests for Nim-Codex + +## Contributing tests +Do you want to write some tests for Codex using this distributed test setup? Great! Here's what you do. + +1. Create a branch. Name it something descriptive, but start it with `tests/` please. [Example: `tests/data-redundancy`.] +1. Checkout your branch, and decide if your tests will be 'short' tests (minutes to hours), or 'long' tests (hours to days), or both! Create a folder for your tests in the matching folders (`/Tests`, `/LongTests`) and don't worry: You can always move your tests later if you like. [Example, short: `/Tests/DataRedundancy/`, long: `/LongTests/DataRedundancy/`] +1. Create one or more code files in your new folder, and write some tests! Here are some tips to help you get started. You can always take a look at the example tests found in [`/Tests/BasicTests/ExampleTests.cs`](/Tests/BasicTests/ExampleTests.cs) + 1. Set up a standard NUnit test fixture. + 1. Inherrit from `DistTest` or `AutoBootstrapDistTest`. + 1. When using `DistTest`: + 1. You must start your own Codex bootstrap node. You can use `SetupCodexBootstrapNode(...)` for this. + 1. When you start other Codex nodes with `SetupCodexNodes(...)` you can pass the bootstrap node by adding the `.WithBootstrapNode(...)` option. + 1. When using `AutoBootstrapDistTest`: + 1. The test-infra creates the bootstrap node for you, and automatically passes it to each Codex node you create in your tests. Handy for keeping your tests clean and to-the-point. + 1. When using the auto-bootstrap, you have no control over the bootstrap node from your tests. You can't (for example) shut it down during the course of the test. If you need this level of control for your scenario, use the `DistTest` instead. + 1. You can generate files of random test data by calling `GenerateTestFile(...)`. + 1. If your test needs a long time to run, add the `[UseLongTimeouts]` function attribute. This will greatly increase maximum time-out values for operations like for example uploading and downloading files. + 1. You can enable access to the Codex node metrics by adding the option `.EnableMetrics()`. Enabling metrics will make the test-infra download and save all Codex metrics in case of a test failure. (The metrics are stored as CSV, in the same location as the test log file.) + 1. You can enable access to the blockchain marketplace by adding the option `.EnableMarketplace(...)`. + 1. Enabling metrics and/or enabling the marketplace takes extra resources from the test-infra and increases the time needed during Codex node setup. Please don't enable these features unless your tests need them. + 1. Tip: Codex nodes can be named. Use the option `WithName(...)` and make reading your test logs a little nicer! + 1. Tip: Commit often. +1. Once you're happy with your tests, please create a pull-request and ask (another) Codex core contributor to review your changes. diff --git a/LOCALSETUP.MD b/LOCALSETUP.MD new file mode 100644 index 00000000..06976a07 --- /dev/null +++ b/LOCALSETUP.MD @@ -0,0 +1,18 @@ +# Distributed System Tests for Nim-Codex + +## Local setup +These steps will help you set up everything you need to run and debug the tests on your local system. + +### Installing the requirements. +1. Install dotnet v6.0 or newer. (If you install a newer version, consider updating the .csproj files by replacing all mention of `net6.0` with your version.) +1. Set up a nice C# IDE or plugin for your current IDE. +1. Install docker desktop. +1. In the docker-desktop settings, enable kubernetes. (This might take a few minutes.) + +### Configure to taste. +The tests should run as-is. You can change the configuration. The items below explain the what and how. +1. Open `DistTestCore/Configuration.cs`. +1. `k8sNamespace` defines the Kubernetes namespace the tests will use. All Kubernetes resources used during the test will be created in it. At the beginning of a test run and at the end of each test, the namespace and all resources in it will be deletes. +1. `kubeConfigFile`. If you are using the Kubernetes cluster created in docker desktop, this field should be set to null. If you wish to use a different cluster, set this field to the path (absolute or relative) of your KubeConfig file. +1. `LogConfig(path, debugEnabled)`. Path defines the path (absolute or relative) where the tests logs will be saved. The debug flag allows you to enable additional logging. This is mostly useful when something's wrong with the test infra. +1. `FileManagerFolder` defines the path (absolute or relative) where the test infra will generate and store test data files. This folder will be deleted at the end of every test run. diff --git a/README.md b/README.md index 1c0fb45c..024197af 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,28 @@ # Distributed System Tests for Nim-Codex +Using a common dotnet unit-test framework and a few other libraries, this project allows you to write tests that use multiple Codex node instances in various configurations to test the distributed system in a controlled, reproducable environment. + Nim-Codex: https://github.com/status-im/nim-codex +Dotnet: v6.0 +Kubernetes: v1.25.4 +Dotnet-kubernetes SDK: v10.1.4 https://github.com/kubernetes-client/csharp +Nethereum: v4.14.0 -Tests are built on dotnet v6.0 and Kubernetes v1.25.4, using dotnet-kubernetes SDK: https://github.com/kubernetes-client/csharp +## Tests +Tests are devided into two assemblies: `/Tests` and `/LongTests`. +`/Tests` is to be used for tests that take several minutes to hours to execute. +`/LongTests` is to be used for tests that take hours to days to execute. -## Requirement +TODO: All tests will eventually be running as part of a dedicated CI pipeline and kubernetes cluster. Currently, we're developing these tests and the infra-code to support it by running the whole thing locally. -At this moment, the tests require a local kubernetes cluster to be installed. +## Test logs +Because tests potentially take a long time to run, logging is in place to help you investigate failures afterwards. Should a test fail, all Codex terminal output (as well as metrics if they have been enabled) will be downloaded and stored along with a details, step-by-step log of the test. If something's gone wrong and you're here to discover the details, head for the logs. -## Run +## How to contribute tests +An important goal of the test infra is to provide an simple, accessible way for developers to write their tests. If you want to contribute tests for Codex, please follow the steps [HERE](/CONTRIBUTINGTESTS.MD). -Short tests: These tests may take minutes to an hour. -`dotnet test Tests` - -Long tests: These may takes hours to days. -`dotnet test LongTests` +## Run the tests on your machine +Creating tests is much easier when you can debug them on your local system. This is possible, but requires some set-up. If you want to be able to run the tests on your local system, follow the steps [HERE](/LOCALSETUP.MD). Please note that tests which require explicit node locations cannot be executed locally. (Well, you could comment out the location statements and then it would probably work. But that might impact the validity/usefulness of the test.) +## Missing functionality +Surely the test-infra doesn't do everything we'll need it to do. If you're running into a limitation and would like to request a new feature for the test-infra, please create an issue. diff --git a/Tests/DurabilityTests/DurabilityTests.cs b/Tests/DurabilityTests/DurabilityTests.cs index 584d1c66..008799b9 100644 --- a/Tests/DurabilityTests/DurabilityTests.cs +++ b/Tests/DurabilityTests/DurabilityTests.cs @@ -11,12 +11,12 @@ namespace Tests.DurabilityTests [Test] public void BootstrapNodeDisappearsTest() { - var bootstrapNode = SetupCodexNode(); + var bootstrapNode = SetupCodexBootstrapNode(); var group = SetupCodexNodes(2, s => s.WithBootstrapNode(bootstrapNode)); var primary = group[0]; var secondary = group[1]; - // There is 1 minute of time for the nodes to connect to each other. + // There is 1 minute of time f or the nodes to connect to each other. // (Should be easy, they're in the same pod.) Time.Sleep(TimeSpan.FromMinutes(6)); bootstrapNode.BringOffline(); @@ -31,7 +31,7 @@ namespace Tests.DurabilityTests [Test] public void DataRetentionTest() { - var bootstrapNode = SetupCodexNode(s => s.WithLogLevel(CodexLogLevel.Trace)); + var bootstrapNode = SetupCodexBootstrapNode(s => s.WithLogLevel(CodexLogLevel.Trace)); var startGroup = SetupCodexNodes(2, s => s.WithLogLevel(CodexLogLevel.Trace).WithBootstrapNode(bootstrapNode)); var finishGroup = SetupCodexNodes(10, s => s.WithLogLevel(CodexLogLevel.Trace).WithBootstrapNode(bootstrapNode)); From c1f1daa6fe3aec039488f6c88d1c00b8391f9483 Mon Sep 17 00:00:00 2001 From: benbierens Date: Tue, 2 May 2023 07:46:17 +0200 Subject: [PATCH 2/6] Fixed some typos --- LOCALSETUP.MD | 5 ++++- README.md | 4 ++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/LOCALSETUP.MD b/LOCALSETUP.MD index 06976a07..862995c6 100644 --- a/LOCALSETUP.MD +++ b/LOCALSETUP.MD @@ -12,7 +12,10 @@ These steps will help you set up everything you need to run and debug the tests ### Configure to taste. The tests should run as-is. You can change the configuration. The items below explain the what and how. 1. Open `DistTestCore/Configuration.cs`. -1. `k8sNamespace` defines the Kubernetes namespace the tests will use. All Kubernetes resources used during the test will be created in it. At the beginning of a test run and at the end of each test, the namespace and all resources in it will be deletes. +1. `k8sNamespace` defines the Kubernetes namespace the tests will use. All Kubernetes resources used during the test will be created in it. At the beginning of a test run and at the end of each test, the namespace and all resources in it will be deleted. 1. `kubeConfigFile`. If you are using the Kubernetes cluster created in docker desktop, this field should be set to null. If you wish to use a different cluster, set this field to the path (absolute or relative) of your KubeConfig file. 1. `LogConfig(path, debugEnabled)`. Path defines the path (absolute or relative) where the tests logs will be saved. The debug flag allows you to enable additional logging. This is mostly useful when something's wrong with the test infra. 1. `FileManagerFolder` defines the path (absolute or relative) where the test infra will generate and store test data files. This folder will be deleted at the end of every test run. + +### Running the tests +Most IDEs will let you run individual tests or test fixtures straight from the code file. If you want to run all the tests, you can use `dotnet test`. You can control which tests to run by specifying which folder of tests to run. `dotnet test Tests` will run only the tests in `/Tests` and exclude the long tests. diff --git a/README.md b/README.md index 024197af..34b3ab19 100644 --- a/README.md +++ b/README.md @@ -16,10 +16,10 @@ Tests are devided into two assemblies: `/Tests` and `/LongTests`. TODO: All tests will eventually be running as part of a dedicated CI pipeline and kubernetes cluster. Currently, we're developing these tests and the infra-code to support it by running the whole thing locally. ## Test logs -Because tests potentially take a long time to run, logging is in place to help you investigate failures afterwards. Should a test fail, all Codex terminal output (as well as metrics if they have been enabled) will be downloaded and stored along with a details, step-by-step log of the test. If something's gone wrong and you're here to discover the details, head for the logs. +Because tests potentially take a long time to run, logging is in place to help you investigate failures afterwards. Should a test fail, all Codex terminal output (as well as metrics if they have been enabled) will be downloaded and stored along with a detailed, step-by-step log of the test. If something's gone wrong and you're here to discover the details, head for the logs. ## How to contribute tests -An important goal of the test infra is to provide an simple, accessible way for developers to write their tests. If you want to contribute tests for Codex, please follow the steps [HERE](/CONTRIBUTINGTESTS.MD). +An important goal of the test infra is to provide a simple, accessible way for developers to write their tests. If you want to contribute tests for Codex, please follow the steps [HERE](/CONTRIBUTINGTESTS.MD). ## Run the tests on your machine Creating tests is much easier when you can debug them on your local system. This is possible, but requires some set-up. If you want to be able to run the tests on your local system, follow the steps [HERE](/LOCALSETUP.MD). Please note that tests which require explicit node locations cannot be executed locally. (Well, you could comment out the location statements and then it would probably work. But that might impact the validity/usefulness of the test.) From 3c9cb9b7daa31f6613305589790a0073803af2f6 Mon Sep 17 00:00:00 2001 From: benbierens Date: Fri, 5 May 2023 08:35:13 +0200 Subject: [PATCH 3/6] Turns off test cases for two-client tests --- Tests/BasicTests/TwoClientTests.cs | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/Tests/BasicTests/TwoClientTests.cs b/Tests/BasicTests/TwoClientTests.cs index 461dfa85..d4edbedd 100644 --- a/Tests/BasicTests/TwoClientTests.cs +++ b/Tests/BasicTests/TwoClientTests.cs @@ -7,23 +7,15 @@ namespace Tests.BasicTests [TestFixture] public class TwoClientTests : DistTest { - [TestCase(1)] - [TestCase(2)] - [TestCase(3)] - [TestCase(4)] - [TestCase(5)] - [TestCase(6)] - [TestCase(7)] - [TestCase(8)] - [TestCase(9)] - public void TwoClientsOnePodTest(int size) + [Test] + public void TwoClientsOnePodTest() { var group = SetupCodexNodes(2); var primary = group[0]; var secondary = group[1]; - PerformTwoClientTest(primary, secondary, size.MB()); + PerformTwoClientTest(primary, secondary); } [Test] From 9c6f00dbce922928a3f8a51b8b2b8540334b96e8 Mon Sep 17 00:00:00 2001 From: benbierens Date: Fri, 5 May 2023 08:47:20 +0200 Subject: [PATCH 4/6] Disable net-isolation tests --- Tests/BasicTests/NetworkIsolationTest.cs | 1 + 1 file changed, 1 insertion(+) diff --git a/Tests/BasicTests/NetworkIsolationTest.cs b/Tests/BasicTests/NetworkIsolationTest.cs index f3406d4b..e1c6e7b3 100644 --- a/Tests/BasicTests/NetworkIsolationTest.cs +++ b/Tests/BasicTests/NetworkIsolationTest.cs @@ -8,6 +8,7 @@ namespace Tests.BasicTests // This is a test to check network-isolation in the test-infrastructure. // It requires parallelism(2) or greater to run. [TestFixture] + [Ignore("Disabled until a solution is implemented.")] public class NetworkIsolationTest : DistTest { private IOnlineCodexNode? node = null; From 2e7bdc94ac43ac0cd5f8f7f9e4847ec65e04258c Mon Sep 17 00:00:00 2001 From: Veaceslav Doina <20563034+veaceslavdoina@users.noreply.github.com> Date: Sun, 28 May 2023 22:48:33 +0300 Subject: [PATCH 5/6] Add Docker builds --- .github/workflows/docker.yml | 148 +++++++++++++++++++++++++++++++++++ docker/Dockerfile | 7 ++ docker/README.md | 59 ++++++++++++++ docker/docker-entrypoint.sh | 19 +++++ 4 files changed, 233 insertions(+) create mode 100644 .github/workflows/docker.yml create mode 100644 docker/Dockerfile create mode 100644 docker/README.md create mode 100644 docker/docker-entrypoint.sh diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml new file mode 100644 index 00000000..5bd71146 --- /dev/null +++ b/.github/workflows/docker.yml @@ -0,0 +1,148 @@ +name: Docker + + +on: + push: + branches: + - master + tags: + - 'v*.*.*' + paths: + - docker/Dockerfile + - docker/docker-entrypoint.sh + - .github/workflows/docker.yml + workflow_dispatch: + + +env: + DOCKER_FILE: docker/Dockerfile + DOCKER_REPO: codexstorage/cs-codex-dist-tests + + +jobs: + # Build platform specific image + build: + strategy: + fail-fast: true + matrix: + target: + - os: linux + arch: amd64 + - os: linux + arch: arm64 + include: + - target: + os: linux + arch: amd64 + builder: ubuntu-22.04 + - target: + os: linux + arch: arm64 + builder: buildjet-4vcpu-ubuntu-2204-arm + + name: Build ${{ matrix.target.os }}/${{ matrix.target.arch }} + runs-on: ${{ matrix.builder }} + outputs: + tags-linux-amd64: ${{ steps.tags.outputs.tags-linux-amd64 }} + tags-linux-arm64: ${{ steps.tags.outputs.tags-linux-arm64 }} + env: + PLATFORM: ${{ format('{0}/{1}', 'linux', matrix.target.arch) }} + SUFFIX: ${{ format('{0}-{1}', 'linux', matrix.target.arch) }} + steps: + - name: Checkout + uses: actions/checkout@v3 + + - name: Docker - Meta + id: meta + uses: docker/metadata-action@v4 + with: + images: ${{ env.DOCKER_REPO }} + flavor: | + latest=false + tags: | + type=semver,pattern={{version}},suffix=-${{ env.SUFFIX }} + type=sha,suffix=-${{ env.SUFFIX }},enable=${{ !startsWith(github.ref, 'refs/tags/') }} + + - name: Docker - Set tags output + id: tags + run: | + if [[ '${{ matrix.target.os }}' == 'linux' && '${{ matrix.target.arch }}' == 'amd64' ]]; then + echo "tags-linux-amd64=${{ steps.meta.outputs.tags }}" >> "$GITHUB_OUTPUT" + elif [[ '${{ matrix.target.os }}' == 'linux' && '${{ matrix.target.arch }}' == 'arm64' ]]; then + echo "tags-linux-arm64=${{ steps.meta.outputs.tags }}" >> "$GITHUB_OUTPUT" + fi + + - name: Docker - Login to Docker Hub + uses: docker/login-action@v2 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Docker - Build and export to Docker + uses: docker/build-push-action@v4 + with: + context: . + file: ${{ env.DOCKER_FILE }} + platforms: ${{ env.PLATFORM }} + load: true + tags: ${{ steps.meta.outputs.tags }} + labels: ${{ steps.meta.outputs.labels }} + + - name: Docker - Minify image + uses: kitabisa/docker-slim-action@v1 + id: slim + env: + DSLIM_HTTP_PROBE: false + with: + target: ${{ steps.meta.outputs.tags }} + overwrite: true + + - name: Docker - Show slim report + run: echo "${REPORT}" | jq -r + env: + REPORT: ${{ steps.slim.outputs.report }} + + - name: Docker - Push to Docker registry + uses: docker/build-push-action@v4 + with: + context: . + file: ${{ env.DOCKER_FILE }} + platforms: ${{ env.PLATFORM }} + push: true + tags: ${{ steps.meta.outputs.tags }} + labels: ${{ steps.meta.outputs.labels }} + + # Publish single image + publish: + name: Push single image + runs-on: ubuntu-latest + needs: build + steps: + - name: Docker - Meta + id: meta + uses: docker/metadata-action@v4 + with: + images: ${{ env.DOCKER_REPO }} + tags: | + type=semver,pattern={{version}} + type=sha,enable=${{ !startsWith(github.ref, 'refs/tags/') }} + + - name: Docker - Set tags + run: | + # Transform multi-line tags in to the comma-seperated + TAGS=$(echo "${{ steps.meta.outputs.tags }}" | tr '\n' ',' | awk '{gsub(/,$/,"");}1') + echo "TAGS=${TAGS}" >>$GITHUB_ENV + + - name: Docker - Login to Docker Hub + uses: docker/login-action@v2 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Docker - Create and push manifest images + uses: Noelware/docker-manifest-action@master + with: + inputs: ${{ env.TAGS }} + images: ${{ needs.build.outputs.tags-linux-amd64 }},${{ needs.build.outputs.tags-linux-arm64 }} + push: true + diff --git a/docker/Dockerfile b/docker/Dockerfile new file mode 100644 index 00000000..606a07f5 --- /dev/null +++ b/docker/Dockerfile @@ -0,0 +1,7 @@ +FROM mcr.microsoft.com/dotnet/sdk:7.0 + +COPY --chmod=0755 docker/docker-entrypoint.sh / + +ENTRYPOINT ["/docker-entrypoint.sh"] +CMD ["dotnet", "test"] + diff --git a/docker/README.md b/docker/README.md new file mode 100644 index 00000000..f626802e --- /dev/null +++ b/docker/README.md @@ -0,0 +1,59 @@ +# Run tests with Docker in Kubernetes + +We may [run tests localy](../LOCALSETUP.MD) using installed Dotnet and inside Kubernetes we may use a [prepared Docker images](https://hub.docker.com/repository/docker/codexstorage/cs-codex-dist-tests). + + +Custom [entrypoint](docker-entrypoint.sh) will do the following + 1. Clone repository + 2. Switch to the specific branch - `master` by default + 3. Run all tests - `dotnet test` + +**Run with defaults** +```bash +docker run \ + --rm \ + --name cs-codex-dist-tests \ + codexstorage/cs-codex-dist-tests:sha-686757e +``` + +**Just short tests** +```bash +docker run \ + --rm \ + --name cs-codex-dist-tests \ + codexstorage/cs-codex-dist-tests:sha-686757e \ + dotnet test Tests +``` + +**Custom branch** +```bash +docker run \ + --rm \ + --name cs-codex-dist-tests \ + --env BRANCH=feature/tests \ + codexstorage/cs-codex-dist-tests:sha-686757e +``` + +**Custom local config** +```bash +docker run \ + --rm \ + --name cs-codex-dist-tests \ + --env CONFIG=/opt/Configuration.cs \ + --env CONFIG_SHOW=true \ + --volume $PWD/DistTestCore/Configuration.cs:/opt/Configuration.cs \ + codexstorage/cs-codex-dist-tests:sha-686757e +``` + +**Local kubeconfig with custom local config** +```bash +docker run \ + --rm \ + --name cs-codex-dist-tests \ + --env CONFIG=/opt/Configuration.cs \ + --env CONFIG_SHOW=true \ + --env SOURCE=https://github.com/codex-storage/cs-codex-dist-tests.git \ + --volume $PWD/DistTestCore/Configuration.cs:/opt/Configuration.cs \ + --volume $PWD/kubeconfig.yml:/opt/kubeconfig.yml \ + codexstorage/cs-codex-dist-tests:sha-686757e +``` diff --git a/docker/docker-entrypoint.sh b/docker/docker-entrypoint.sh new file mode 100644 index 00000000..10bc1d15 --- /dev/null +++ b/docker/docker-entrypoint.sh @@ -0,0 +1,19 @@ +#!/bin/bash + +# Variables +SOURCE="${SOURCE:-https://github.com/codex-storage/cs-codex-dist-tests.git}" +BRANCH="${BRANCH:-master}" +FOLDER="${FOLDER:-/opt/dist-tests}" + + +# Get tests +echo "Clone ${SOURCE}" +git clone -b "${BRANCH}" "${SOURCE}" "${FOLDER}" +[[ -n "${CONFIG}" ]] && { echo Link config "${CONFIG}"; ln --symbolic --force "${CONFIG}" "${FOLDER}/DistTestCore/Configuration.cs"; } +[[ "${CONFIG_SHOW}" == "true" ]] && { echo Show config "${CONFIG}"; cat "${FOLDER}/DistTestCore/Configuration.cs"; } +cd "${FOLDER}" + +# Run +echo "Run tests on branch '`git branch --show-current`' ..." +exec "$@" + From 1b490c475e48fca995801b55fa59cd93a6151bdd Mon Sep 17 00:00:00 2001 From: Slava <20563034+veaceslavdoina@users.noreply.github.com> Date: Mon, 29 May 2023 11:46:26 +0300 Subject: [PATCH 6/6] Fix Docker repository link (#6) --- docker/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker/README.md b/docker/README.md index f626802e..020d0281 100644 --- a/docker/README.md +++ b/docker/README.md @@ -1,6 +1,6 @@ # Run tests with Docker in Kubernetes -We may [run tests localy](../LOCALSETUP.MD) using installed Dotnet and inside Kubernetes we may use a [prepared Docker images](https://hub.docker.com/repository/docker/codexstorage/cs-codex-dist-tests). +We may [run tests localy](../LOCALSETUP.MD) using installed Dotnet and inside Kubernetes we may use a [prepared Docker images](https://hub.docker.com/r/codexstorage/cs-codex-dist-tests/tags). Custom [entrypoint](docker-entrypoint.sh) will do the following