diff --git a/.github/workflows/dist-tests.yaml b/.github/workflows/dist-tests.yaml index 2fbf048d..1453f000 100644 --- a/.github/workflows/dist-tests.yaml +++ b/.github/workflows/dist-tests.yaml @@ -41,7 +41,7 @@ env: SOURCE: ${{ format('{0}/{1}', github.server_url, github.repository) }} NAMEPREFIX: codex-dist-tests NAMESPACE: default - COMMAND: dotnet test Tests + COMMAND: dotnet test Tests/CodexTests JOB_MANIFEST: docker/job.yaml KUBE_CONFIG: ${{ secrets.KUBE_CONFIG }} KUBE_VERSION: v1.26.1 diff --git a/CONTRIBUTINGPLUGINS.MD b/CONTRIBUTINGPLUGINS.MD new file mode 100644 index 00000000..71619fb8 --- /dev/null +++ b/CONTRIBUTINGPLUGINS.MD @@ -0,0 +1,86 @@ +# Distributed System Tests for Nim-Codex + +## Contributing plugins +The testing framework was created for testing Codex. However, it's been designed such that other distributed/containerized projects can 'easily' be added. In order to add your project to the framework you must: +1. Create a library assembly in the project plugins folder. +1. It must contain a type that implements the `IProjectPlugin` interface from the `Core` assembly. +1. If your plugin wants to expose any specific methods or objects to the code using the framework (the tests and tools), it must implement extensions for the `CoreInterface` type. + +## Constructors & Tools +Your implementation of `IProjectPlugin` must have a public constructor with a single argument of type `IPluginTools`, for example: +```C# + public class MyPlugin : IProjectPlugin + { + public MyPlugin(IPluginTools tools) + { + ... + } + + ... + } +``` + +`IPluginTools` provides your plugin access to all framework functionality, such as logging, tracked file management, container lifecycle management, and a means to create HTTP clients for containers. (Without having to figure out addresses manually.) + +## Plugin Interfaces +The `IProjectPlugin` interface requires the implementation of two methods. +1. `Announce` - It is considered polite to use the logging functionality provided by the `IPluginTools` to announce that your plugin has been loaded. You may also want to log some manner of version information at this time if applicable. +1. `Decommission` - Should your plugin have any active system resources, free them in this method. + +There are a few optional interfaces your plugin may choose to implement. The framework will automatically use these interfaces. +1. `IHasLogPrefix` - Implementing this interface allows you to provide a string with will be prepended to all log statements made by your plugin. +1. `IHasMetadata` - This allows you to provide metadata in the form of key/value pairs. This metadata can be accessed by code that uses your plugin. + +## Core Interface +Any functionality your plugin wants to expose to code which uses the framework will have to be added on to the `CoreInterface` type. You can accomplish this by using C# extension methods. The framework provides a `GetPlugin` method to access your plugin instance from the `CoreInterface` type: +```C# + public static class CoreInterfaceExtensions + { + public static MyPluginReturnType DoSomethingCool(this CoreInterface ci, string someArgument) + { + return Plugin(ci).SomethingCool(someArgument); + } + + private static MyPlugin Plugin(CoreInterface ci) + { + return ci.GetPlugin(); + } + } +``` + +While technically you can build whatever you like on top of the `CoreInterface` and your own plugin types, I recommend that you follow the approach explained below. + +## Deploying, Wrapping, and Starting +When building a plugin, it is important to make as few assumptions as possible about how it will be used by whoever is going to use the framework. For this reason, I recommend you expose three kinds of methods using your `CoreInterface` extensions: +1. Deploy - This kind of method should deploy your project, creating and configuring containers as needed and returning containers as a result. If your project requires additional information, you can create a new class type to contain both it and the containers created. +1. Wrap - This kind of method should, when given the previously mentioned container information, create some kind of convenient accessor or interactor object. This object should abstract away for example details of a REST API of your project, allowing users of your plugin to write their code using a set of methods and types that nicely model your project's domain. +1. Start - This kind of method does both, simply calling a Deploy method first, then a Wrap method, and returns the result. + +Here's an example: +```C# +public static class CoreInterfaceExtensions + { + public static RunningContainers DeployMyProject(this CoreInterface ci, string someArgument) + { + // `RunningContainers` is a framework type. It contains all necessary information about a deployed container. It is serializable. + // Should you need to return any additional information, create a new type that contains it as well as the container information. Make sure it is serializable. + return Plugin(ci).DeployMyProjectContainer(someArgument); // <-- This method should use the `PluginTools.CreateWorkflow()` tool to deploy a container with a configuration that matches someArguments. + } + + public static IMyProjectNode WrapMyProjectContainer(this CoreInterface ci, RunningContainers container) + { + return Plugin(ci).WrapMyContainerProject(container); // <-- This method probably will use the 'PluginTools.CreateHttp()` tool to create an HTTP client for the container, then wrap it in an object that + // represents the API of your project. + } + + public static IMyProjectNode StartMyProject(this CoreInterface ci, string someArgument) + { + // Start is now nothing more than a convenience method, combining the previous two. + var rc = ci.DeployMyProject(someArgument); + return WrapMyProjectContainer(ci, rc); + } + } +``` + +The primary reason to decouple deploying and wrapping functionalities is that some use cases require these steps to be performed by separate applications, and different moments in time. For this reason, whatever is returned by the deploy methods should be serializable. After deserialization at some later time, it should then be valid input for the wrap method. The Codex continuous tests system is a clear example of this use case: The `CodexNetDeployer` tool uses deploy methods to create Codex nodes. Then it writes the returned objects to a JSON file. Some time later, the `CodexContinousTests` application uses this JSON file to reconstruct the objects created by the deploy methods. It then uses the wrap methods to create accessors and interactors, which are used for testing. + diff --git a/CONTRIBUTINGTESTS.MD b/CONTRIBUTINGTESTS.MD index e2e3ac43..070e192a 100644 --- a/CONTRIBUTINGTESTS.MD +++ b/CONTRIBUTINGTESTS.MD @@ -1,24 +1,40 @@ # Distributed System Tests for Nim-Codex ## Contributing tests -Do you want to write some tests for Codex using this distributed test setup? Great! Here's what you do. +Do you want to write some tests using this distributed test setup? Great! Here's what you do. 1. Create a branch. Name it something descriptive, but start it with `tests/` please. [Example: `tests/data-redundancy`.] -1. Checkout your branch, and decide if your tests will be 'short' tests (minutes to hours), or 'long' tests (hours to days), or both! Create a folder for your tests in the matching folders (`/Tests`, `/LongTests`) and don't worry: You can always move your tests later if you like. [Example, short: `/Tests/DataRedundancy/`, long: `/LongTests/DataRedundancy/`] -1. Create one or more code files in your new folder, and write some tests! Here are some tips to help you get started. You can always take a look at the example tests found in [`/Tests/BasicTests/ExampleTests.cs`](/Tests/BasicTests/ExampleTests.cs) - 1. Set up a standard NUnit test fixture. - 1. Inherrit from `DistTest` or `AutoBootstrapDistTest`. - 1. When using `DistTest`: - 1. You must start your own Codex bootstrap node. You can use `SetupCodexBootstrapNode(...)` for this. - 1. When you start other Codex nodes with `SetupCodexNodes(...)` you can pass the bootstrap node by adding the `.WithBootstrapNode(...)` option. - 1. When using `AutoBootstrapDistTest`: - 1. The test-infra creates the bootstrap node for you, and automatically passes it to each Codex node you create in your tests. Handy for keeping your tests clean and to-the-point. - 1. When using the auto-bootstrap, you have no control over the bootstrap node from your tests. You can't (for example) shut it down during the course of the test. If you need this level of control for your scenario, use the `DistTest` instead. - 1. You can generate files of random test data by calling `GenerateTestFile(...)`. - 1. If your test needs a long time to run, add the `[UseLongTimeouts]` function attribute. This will greatly increase maximum time-out values for operations like for example uploading and downloading files. - 1. You can enable access to the Codex node metrics by adding the option `.EnableMetrics()`. Enabling metrics will make the test-infra download and save all Codex metrics in case of a test failure. (The metrics are stored as CSV, in the same location as the test log file.) - 1. You can enable access to the blockchain marketplace by adding the option `.EnableMarketplace(...)`. - 1. Enabling metrics and/or enabling the marketplace takes extra resources from the test-infra and increases the time needed during Codex node setup. Please don't enable these features unless your tests need them. - 1. Tip: Codex nodes can be named. Use the option `WithName(...)` and make reading your test logs a little nicer! - 1. Tip: Commit often. -1. Once you're happy with your tests, please create a pull-request and ask (another) Codex core contributor to review your changes. +1. Checkout your branch. +1. Create a new assembly in the `/Tests` folder. This can be an NUnit test assembly or simply a console app. +1. Add Project references to `Core`, as well as any project plugin you'll be using. +1. Write tests! Use existing tests for inspiration. + +## Tips for writing tests for Codex +### Transient tests +1. Add new code files to `Tests/CodexTests` +1. Inherrit from `CodexDistTest` or `AutoBootstrapDistTest`. +1. When using `CodexDistTest`: + 1. You must start your own Codex bootstrap node. You can use `AddCodex(...)` for this. + 1. When you start other Codex nodes with `AddCodex(...)` you can pass the bootstrap node by adding the `.WithBootstrapNode(...)` option. +1. When using `AutoBootstrapDistTest`: + 1. The test-infra creates the bootstrap node for you, and automatically passes it to each Codex node you create in your tests. Handy for keeping your tests clean and to-the-point. + 1. When using the auto-bootstrap, you have no control over the bootstrap node from your tests. You can't (for example) shut it down during the course of the test. If you need this level of control for your scenario, use the `CodexDistTest` instead. +1. If your test needs a long time to run, add the `[UseLongTimeouts]` function attribute. This will greatly increase maximum time-out values for operations like for example uploading and downloading files. +### Continuous tests +1. Add new code files to `Tests/CodexContinousTests/Tests` +1. Inherrit from `ContinuousTest` +1. Define one or more methods and decorate them with the `[TestMoment(...)]` attribute. +1. The TestMoment takes a number of seconds as argument. Each moment will be executed by the continuous test runner applying the given seconds as delay. (Non-cumulative. So two moments at T:10 will be executed one after another without delay, in this case the order of execution should not be depended upon.) +1. Continuous tests automatically receive access to the Codex nodes that the tests are being run against. +1. Additionally, Continuous tests can start their own transient Codex nodes and bootstrap them against the persistent nodes. + +### Tips for either type of test +1. You can generate files of random test data by calling `GenerateTestFile(...)`. +1. You can enable access to the Codex node metrics by adding the option `.EnableMetrics()`. Enabling metrics will make the test-infra download and save all Codex metrics in case of a test failure. (The metrics are stored as CSV, in the same location as the test log file.) +1. You can enable access to the blockchain marketplace by adding the option `.EnableMarketplace(...)`. +1. Enabling metrics and/or enabling the marketplace takes extra resources from the test-infra and increases the time needed during Codex node setup. Please don't enable these features unless your tests need them. +1. Tip: Codex nodes can be named. Use the option `WithName(...)` and make reading your test logs a little nicer! +1. Tip: Commit often. + +## Don't forget +1. Once you're happy with your tests, please create a pull-request and ask a Codex core contributor to review your changes. diff --git a/CodexNetDeployer/CodexNodeStarter.cs b/CodexNetDeployer/CodexNodeStarter.cs deleted file mode 100644 index c6b38502..00000000 --- a/CodexNetDeployer/CodexNodeStarter.cs +++ /dev/null @@ -1,127 +0,0 @@ -using DistTestCore; -using DistTestCore.Codex; -using DistTestCore.Marketplace; -using KubernetesWorkflow; -using Utils; - -namespace CodexNetDeployer -{ - public class CodexNodeStarter - { - private readonly Configuration config; - private readonly TestLifecycle lifecycle; - private readonly GethStartResult gethResult; - private string bootstrapSpr = ""; - private int validatorsLeft; - - public CodexNodeStarter(Configuration config, TestLifecycle lifecycle, GethStartResult gethResult, int numberOfValidators) - { - this.config = config; - this.lifecycle = lifecycle; - this.gethResult = gethResult; - validatorsLeft = numberOfValidators; - } - - public CodexNodeStartResult? Start(int i) - { - Console.Write($" - {i} = "); - var workflow = lifecycle.WorkflowCreator.CreateWorkflow(); - var workflowStartup = new StartupConfig(); - workflowStartup.Add(gethResult); - workflowStartup.Add(CreateCodexStartupConfig(bootstrapSpr, i, validatorsLeft)); - workflowStartup.NameOverride = GetCodexContainerName(i); - - var containers = workflow.Start(1, Location.Unspecified, new CodexContainerRecipe(), workflowStartup); - - var container = containers.Containers.First(); - var codexAccess = new CodexAccess(lifecycle.Log, container, lifecycle.TimeSet, lifecycle.Configuration.GetAddress(container)); - var account = gethResult.MarketplaceNetwork.Bootstrap.AllAccounts.Accounts[i]; - var tokenAddress = gethResult.MarketplaceNetwork.Marketplace.TokenAddress; - var marketAccess = new MarketplaceAccess(lifecycle, gethResult.MarketplaceNetwork, account, codexAccess); - - try - { - var debugInfo = codexAccess.GetDebugInfo(); - if (!string.IsNullOrWhiteSpace(debugInfo.spr)) - { - Console.Write("Online\t"); - - var interaction = gethResult.MarketplaceNetwork.Bootstrap.StartInteraction(lifecycle); - interaction.MintTestTokens(new[] { account.Account }, config.InitialTestTokens, tokenAddress); - Console.Write("Tokens minted\t"); - - var response = marketAccess.MakeStorageAvailable( - totalSpace: config.StorageSell!.Value.MB(), - minPriceForTotalSpace: config.MinPrice.TestTokens(), - maxCollateral: config.MaxCollateral.TestTokens(), - maxDuration: TimeSpan.FromSeconds(config.MaxDuration)); - - if (!string.IsNullOrEmpty(response)) - { - Console.Write("Storage available\tOK" + Environment.NewLine); - - if (string.IsNullOrEmpty(bootstrapSpr)) bootstrapSpr = debugInfo.spr; - validatorsLeft--; - return new CodexNodeStartResult(workflow, container, codexAccess); - } - } - } - catch (Exception ex) - { - Console.WriteLine("Exception:" + ex.ToString()); - } - - Console.Write("Unknown failure. Downloading container log." + Environment.NewLine); - lifecycle.DownloadLog(container); - - return null; - } - - private string GetCodexContainerName(int i) - { - if (i == 0) return "BOOTSTRAP"; - return "CODEX" + i; - } - - private CodexStartupConfig CreateCodexStartupConfig(string bootstrapSpr, int i, int validatorsLeft) - { - var codexStart = new CodexStartupConfig(config.CodexLogLevel); - - if (!string.IsNullOrEmpty(bootstrapSpr)) codexStart.BootstrapSpr = bootstrapSpr; - codexStart.StorageQuota = config.StorageQuota!.Value.MB(); - var marketplaceConfig = new MarketplaceInitialConfig(100000.Eth(), 0.TestTokens(), validatorsLeft > 0); - marketplaceConfig.AccountIndexOverride = i; - codexStart.MarketplaceConfig = marketplaceConfig; - codexStart.MetricsMode = config.Metrics; - - if (config.BlockTTL != Configuration.SecondsIn1Day) - { - codexStart.BlockTTL = config.BlockTTL; - } - if (config.BlockMI != Configuration.TenMinutes) - { - codexStart.BlockMaintenanceInterval = TimeSpan.FromSeconds(config.BlockMI); - } - if (config.BlockMN != 1000) - { - codexStart.BlockMaintenanceNumber = config.BlockMN; - } - - return codexStart; - } - } - - public class CodexNodeStartResult - { - public CodexNodeStartResult(StartupWorkflow workflow, RunningContainer container, CodexAccess access) - { - Workflow = workflow; - Container = container; - Access = access; - } - - public StartupWorkflow Workflow { get; } - public RunningContainer Container { get; } - public CodexAccess Access { get; } - } -} diff --git a/CodexNetDeployer/Deployer.cs b/CodexNetDeployer/Deployer.cs deleted file mode 100644 index eb329fc9..00000000 --- a/CodexNetDeployer/Deployer.cs +++ /dev/null @@ -1,156 +0,0 @@ -using DistTestCore; -using DistTestCore.Codex; -using KubernetesWorkflow; -using Logging; -using Utils; - -namespace CodexNetDeployer -{ - public class Deployer - { - private readonly Configuration config; - private readonly DefaultTimeSet timeset; - private readonly PeerConnectivityChecker peerConnectivityChecker; - - public Deployer(Configuration config) - { - this.config = config; - timeset = new DefaultTimeSet(); - peerConnectivityChecker = new PeerConnectivityChecker(); - } - - public CodexDeployment Deploy() - { - Log("Initializing..."); - var lifecycle = CreateTestLifecycle(); - - Log("Preparing configuration..."); - // We trick the Geth companion node into unlocking all of its accounts, by saying we want to start 999 codex nodes. - var setup = new CodexSetup(999, config.CodexLogLevel); - setup.WithStorageQuota(config.StorageQuota!.Value.MB()).EnableMarketplace(0.TestTokens()); - setup.MetricsMode = config.Metrics; - - Log("Creating Geth instance and deploying contracts..."); - var gethStarter = new GethStarter(lifecycle); - var gethResults = gethStarter.BringOnlineMarketplaceFor(setup); - - Log("Geth started. Codex contracts deployed."); - Log("Warning: It can take up to 45 minutes for the Geth node to finish unlocking all if its 1000 preconfigured accounts."); - - // It takes a second for the geth node to unlock a single account. Let's assume 3. - // We can't start the codex nodes until their accounts are definitely unlocked. So - // We wait: - Thread.Sleep(TimeSpan.FromSeconds(3.0 * config.NumberOfCodexNodes!.Value)); - - Log("Starting Codex nodes..."); - - // Each node must have its own IP, so it needs it own pod. Start them 1 at a time. - var codexStarter = new CodexNodeStarter(config, lifecycle, gethResults, config.NumberOfValidators!.Value); - var startResults = new List(); - for (var i = 0; i < config.NumberOfCodexNodes; i++) - { - var result = codexStarter.Start(i); - if (result != null) startResults.Add(result); - } - - var (prometheusContainer, grafanaStartInfo) = StartMetricsService(lifecycle, setup, startResults.Select(r => r.Container)); - - CheckPeerConnectivity(startResults); - CheckContainerRestarts(startResults); - - return new CodexDeployment(gethResults, startResults.Select(r => r.Container).ToArray(), prometheusContainer, grafanaStartInfo, CreateMetadata()); - } - - private TestLifecycle CreateTestLifecycle() - { - var kubeConfig = GetKubeConfig(config.KubeConfigFile); - - var lifecycleConfig = new DistTestCore.Configuration - ( - kubeConfigFile: kubeConfig, - logPath: "null", - logDebug: false, - dataFilesPath: "notUsed", - codexLogLevel: config.CodexLogLevel, - k8sNamespacePrefix: config.KubeNamespace - ); - - var lifecycle = new TestLifecycle(new NullLog(), lifecycleConfig, timeset, string.Empty); - DefaultContainerRecipe.TestsType = config.TestsTypePodLabel; - DefaultContainerRecipe.ApplicationIds = lifecycle.GetApplicationIds(); - return lifecycle; - } - - private (RunningContainer?, GrafanaStartInfo?) StartMetricsService(TestLifecycle lifecycle, CodexSetup setup, IEnumerable codexContainers) - { - if (setup.MetricsMode == DistTestCore.Metrics.MetricsMode.None) return (null, null); - - Log("Starting metrics service..."); - var runningContainers = new[] { new RunningContainers(null!, null!, codexContainers.ToArray()) }; - var prometheusContainer = lifecycle.PrometheusStarter.CollectMetricsFor(runningContainers).Containers.Single(); - - if (setup.MetricsMode == DistTestCore.Metrics.MetricsMode.Record) return (prometheusContainer, null); - - Log("Starting dashboard service..."); - var grafanaStartInfo = lifecycle.GrafanaStarter.StartDashboard(prometheusContainer, setup); - return (prometheusContainer, grafanaStartInfo); - } - - private string? GetKubeConfig(string kubeConfigFile) - { - if (string.IsNullOrEmpty(kubeConfigFile) || kubeConfigFile.ToLowerInvariant() == "null") return null; - return kubeConfigFile; - } - - private void CheckPeerConnectivity(List codexContainers) - { - if (!config.CheckPeerConnection) return; - - Log("Starting peer-connectivity check for deployed nodes..."); - peerConnectivityChecker.CheckConnectivity(codexContainers); - Log("Check passed."); - } - - private void CheckContainerRestarts(List startResults) - { - var crashes = new List(); - foreach (var startResult in startResults) - { - var watcher = startResult.Workflow.CreateCrashWatcher(startResult.Container); - if (watcher.HasContainerCrashed()) crashes.Add(startResult.Container); - } - - if (!crashes.Any()) - { - Log("Container restart check passed."); - } - else - { - Log($"Deployment failed. The following containers have crashed: {string.Join(",", crashes.Select(c => c.Name))}"); - throw new Exception("Deployment failed: One or more containers crashed."); - } - } - - private DeploymentMetadata CreateMetadata() - { - return new DeploymentMetadata( - kubeNamespace: config.KubeNamespace, - numberOfCodexNodes: config.NumberOfCodexNodes!.Value, - numberOfValidators: config.NumberOfValidators!.Value, - storageQuotaMB: config.StorageQuota!.Value, - codexLogLevel: config.CodexLogLevel, - initialTestTokens: config.InitialTestTokens, - minPrice: config.MinPrice, - maxCollateral: config.MaxCollateral, - maxDuration: config.MaxDuration, - blockTTL: config.BlockTTL, - blockMI: config.BlockMI, - blockMN: config.BlockMN); - } - - private void Log(string msg) - { - Console.WriteLine(msg); - } - } -} diff --git a/CodexNetDeployer/PeerConnectivityChecker.cs b/CodexNetDeployer/PeerConnectivityChecker.cs deleted file mode 100644 index 6e7eddb7..00000000 --- a/CodexNetDeployer/PeerConnectivityChecker.cs +++ /dev/null @@ -1,34 +0,0 @@ -using DistTestCore.Helpers; -using Logging; - -namespace CodexNetDeployer -{ - public class PeerConnectivityChecker - { - public void CheckConnectivity(List startResults) - { - var log = new ConsoleLog(); - var checker = new PeerConnectionTestHelpers(log); - var access = startResults.Select(r => r.Access); - - checker.AssertFullyConnected(access); - } - } - - public class ConsoleLog : BaseLog - { - public ConsoleLog() : base(false) - { - } - - protected override string GetFullName() - { - return "CONSOLE"; - } - - public override void Log(string message) - { - Console.WriteLine(message); - } - } -} diff --git a/CodexNetDownloader/CodexNetDownloader.csproj b/CodexNetDownloader/CodexNetDownloader.csproj deleted file mode 100644 index 0df12235..00000000 --- a/CodexNetDownloader/CodexNetDownloader.csproj +++ /dev/null @@ -1,16 +0,0 @@ - - - - Exe - net7.0 - enable - enable - - - - - - - - - diff --git a/CodexNetDownloader/Configuration.cs b/CodexNetDownloader/Configuration.cs deleted file mode 100644 index a42136a4..00000000 --- a/CodexNetDownloader/Configuration.cs +++ /dev/null @@ -1,19 +0,0 @@ -using ArgsUniform; -using DistTestCore.Codex; - -namespace CodexNetDownloader -{ - public class Configuration - { - [Uniform("output-path", "o", "OUTPUT", true, "Path where files will be written.")] - public string OutputPath { get; set; } = "output"; - - [Uniform("codex-deployment", "c", "CODEXDEPLOYMENT", true, "Path to codex-deployment JSON file.")] - public string CodexDeploymentJson { get; set; } = string.Empty; - - [Uniform("kube-config", "kc", "KUBECONFIG", true, "Path to Kubeconfig file. Use 'null' (default) to use local cluster.")] - public string KubeConfigFile { get; set; } = "null"; - - public CodexDeployment CodexDeployment { get; set; } = null!; - } -} diff --git a/CodexNetDownloader/Program.cs b/CodexNetDownloader/Program.cs deleted file mode 100644 index 5744e71e..00000000 --- a/CodexNetDownloader/Program.cs +++ /dev/null @@ -1,48 +0,0 @@ -using ArgsUniform; -using ContinuousTests; -using DistTestCore; -using DistTestCore.Codex; -using Logging; -using Newtonsoft.Json; - -public class Program -{ - public static void Main(string[] args) - { - var nl = Environment.NewLine; - Console.WriteLine("CodexNetDownloader" + nl); - - var uniformArgs = new ArgsUniform(PrintHelp, args); - var config = uniformArgs.Parse(true); - - config.CodexDeployment = ParseCodexDeploymentJson(config.CodexDeploymentJson); - - if (!Directory.Exists(config.OutputPath)) Directory.CreateDirectory(config.OutputPath); - - var k8sFactory = new K8sFactory(); - var lifecycle = k8sFactory.CreateTestLifecycle(config.KubeConfigFile, config.OutputPath, "dataPath", config.CodexDeployment.Metadata.KubeNamespace, new DefaultTimeSet(), new NullLog()); - - foreach (var container in config.CodexDeployment.CodexContainers) - { - lifecycle.DownloadLog(container); - } - - Console.WriteLine("Done!"); - } - - private static CodexDeployment ParseCodexDeploymentJson(string filename) - { - var d = JsonConvert.DeserializeObject(File.ReadAllText(filename))!; - if (d == null) throw new Exception("Unable to parse " + filename); - return d; - } - - private static void PrintHelp() - { - var nl = Environment.NewLine; - Console.WriteLine("CodexNetDownloader lets you download all container logs given a codex-deployment.json file." + nl); - - Console.WriteLine("CodexNetDownloader assumes you are running this tool from *inside* the Kubernetes cluster. " + - "If you are not running this from a container inside the cluster, add the argument '--external'." + nl); - } -} diff --git a/ContinuousTests/CodexAccessFactory.cs b/ContinuousTests/CodexAccessFactory.cs deleted file mode 100644 index 78e90693..00000000 --- a/ContinuousTests/CodexAccessFactory.cs +++ /dev/null @@ -1,20 +0,0 @@ -using DistTestCore; -using DistTestCore.Codex; -using KubernetesWorkflow; -using Logging; - -namespace ContinuousTests -{ - public class CodexAccessFactory - { - public CodexAccess[] Create(Configuration config, RunningContainer[] containers, BaseLog log, ITimeSet timeSet) - { - return containers.Select(container => - { - var address = container.ClusterExternalAddress; - if (config.RunnerLocation == RunnerLocation.InternalToCluster) address = container.ClusterInternalAddress; - return new CodexAccess(log, container, timeSet, address); - }).ToArray(); - } - } -} diff --git a/ContinuousTests/ContinuousTest.cs b/ContinuousTests/ContinuousTest.cs deleted file mode 100644 index 3e76298c..00000000 --- a/ContinuousTests/ContinuousTest.cs +++ /dev/null @@ -1,119 +0,0 @@ -using DistTestCore; -using DistTestCore.Codex; -using DistTestCore.Logs; -using KubernetesWorkflow; -using Logging; - -namespace ContinuousTests -{ - public abstract class ContinuousTestLongTimeouts : ContinuousTest - { - public override ITimeSet TimeSet => new LongTimeSet(); - } - - public abstract class ContinuousTest - { - protected const int Zero = 0; - protected const int MinuteOne = 60; - protected const int MinuteFive = MinuteOne * 5; - protected const int HourOne = MinuteOne * 60; - protected const int HourThree = HourOne * 3; - protected const int DayOne = HourOne * 24; - protected const int DayThree = DayOne * 3; - - private const string UploadFailedMessage = "Unable to store block"; - - public void Initialize(CodexAccess[] nodes, BaseLog log, FileManager fileManager, Configuration configuration, CancellationToken cancelToken) - { - Nodes = nodes; - Log = log; - FileManager = fileManager; - Configuration = configuration; - CancelToken = cancelToken; - - if (nodes != null) - { - NodeRunner = new NodeRunner(Nodes, configuration, TimeSet, Log, CustomK8sNamespace, EthereumAccountIndex); - } - else - { - NodeRunner = null!; - } - } - - public CodexAccess[] Nodes { get; private set; } = null!; - public BaseLog Log { get; private set; } = null!; - public IFileManager FileManager { get; private set; } = null!; - public Configuration Configuration { get; private set; } = null!; - public virtual ITimeSet TimeSet { get { return new DefaultTimeSet(); } } - public CancellationToken CancelToken { get; private set; } = new CancellationToken(); - public NodeRunner NodeRunner { get; private set; } = null!; - - public abstract int RequiredNumberOfNodes { get; } - public abstract TimeSpan RunTestEvery { get; } - public abstract TestFailMode TestFailMode { get; } - public virtual int EthereumAccountIndex { get { return -1; } } - public virtual string CustomK8sNamespace { get { return string.Empty; } } - - public string Name - { - get - { - return GetType().Name; - } - } - - public ContentId? UploadFile(CodexAccess node, TestFile file) - { - using var fileStream = File.OpenRead(file.Filename); - - var logMessage = $"Uploading file {file.Describe()}..."; - var response = Stopwatch.Measure(Log, logMessage, () => - { - return node.UploadFile(fileStream); - }); - - if (string.IsNullOrEmpty(response)) return null; - if (response.StartsWith(UploadFailedMessage)) return null; - - Log.Log($"Uploaded file. Received contentId: '{response}'."); - return new ContentId(response); - } - - public TestFile DownloadFile(CodexAccess node, ContentId contentId, string fileLabel = "") - { - var logMessage = $"Downloading for contentId: '{contentId.Id}'..."; - var file = FileManager.CreateEmptyTestFile(fileLabel); - Stopwatch.Measure(Log, logMessage, () => DownloadToFile(node, contentId.Id, file)); - Log.Log($"Downloaded file {file.Describe()} to '{file.Filename}'."); - return file; - } - - public IDownloadedLog DownloadContainerLog(RunningContainer container, int? tailLines = null) - { - var nodeRunner = new NodeRunner(Nodes, Configuration, TimeSet, Log, Configuration.CodexDeployment.Metadata.KubeNamespace, EthereumAccountIndex); - return nodeRunner.DownloadLog(container, tailLines); - } - - private void DownloadToFile(CodexAccess node, string contentId, TestFile file) - { - using var fileStream = File.OpenWrite(file.Filename); - try - { - using var downloadStream = node.DownloadFile(contentId); - downloadStream.CopyTo(fileStream); - } - catch - { - Log.Log($"Failed to download file '{contentId}'."); - throw; - } - } - } - - public enum TestFailMode - { - StopAfterFirstFailure, - AlwaysRunAllMoments - } -} diff --git a/ContinuousTests/K8sFactory.cs b/ContinuousTests/K8sFactory.cs deleted file mode 100644 index 5ae8b1b0..00000000 --- a/ContinuousTests/K8sFactory.cs +++ /dev/null @@ -1,34 +0,0 @@ -using DistTestCore.Codex; -using DistTestCore; -using Logging; - -namespace ContinuousTests -{ - public class K8sFactory - { - public TestLifecycle CreateTestLifecycle(string kubeConfigFile, string logPath, string dataFilePath, string customNamespace, ITimeSet timeSet, BaseLog log) - { - var kubeConfig = GetKubeConfig(kubeConfigFile); - var lifecycleConfig = new DistTestCore.Configuration - ( - kubeConfigFile: kubeConfig, - logPath: logPath, - logDebug: false, - dataFilesPath: dataFilePath, - codexLogLevel: CodexLogLevel.Debug, - k8sNamespacePrefix: customNamespace - ); - - var lifecycle = new TestLifecycle(log, lifecycleConfig, timeSet, string.Empty); - DefaultContainerRecipe.TestsType = "continuous-tests"; - DefaultContainerRecipe.ApplicationIds = lifecycle.GetApplicationIds(); - return lifecycle; - } - - private static string? GetKubeConfig(string kubeConfigFile) - { - if (string.IsNullOrEmpty(kubeConfigFile) || kubeConfigFile.ToLowerInvariant() == "null") return null; - return kubeConfigFile; - } - } -} diff --git a/ContinuousTests/NodeRunner.cs b/ContinuousTests/NodeRunner.cs deleted file mode 100644 index d599c4d3..00000000 --- a/ContinuousTests/NodeRunner.cs +++ /dev/null @@ -1,113 +0,0 @@ -using DistTestCore.Codex; -using DistTestCore.Marketplace; -using DistTestCore; -using KubernetesWorkflow; -using NUnit.Framework; -using Logging; -using Utils; -using DistTestCore.Logs; - -namespace ContinuousTests -{ - public class NodeRunner - { - private readonly K8sFactory k8SFactory = new K8sFactory(); - private readonly CodexAccess[] nodes; - private readonly Configuration config; - private readonly ITimeSet timeSet; - private readonly BaseLog log; - private readonly string customNamespace; - private readonly int ethereumAccountIndex; - - public NodeRunner(CodexAccess[] nodes, Configuration config, ITimeSet timeSet, BaseLog log, string customNamespace, int ethereumAccountIndex) - { - this.nodes = nodes; - this.config = config; - this.timeSet = timeSet; - this.log = log; - this.customNamespace = customNamespace; - this.ethereumAccountIndex = ethereumAccountIndex; - } - - public void RunNode(Action operation) - { - RunNode(nodes.ToList().PickOneRandom(), operation, 0.TestTokens()); - } - - public void RunNode(CodexAccess bootstrapNode, Action operation) - { - RunNode(bootstrapNode, operation, 0.TestTokens()); - } - - public IDownloadedLog DownloadLog(RunningContainer container, int? tailLines = null) - { - var subFile = log.CreateSubfile(); - var description = container.Name; - var handler = new LogDownloadHandler(container, description, subFile); - - log.Log($"Downloading logs for {description} to file '{subFile.FullFilename}'"); - - var lifecycle = CreateTestLifecycle(); - var flow = lifecycle.WorkflowCreator.CreateWorkflow(); - flow.DownloadContainerLog(container, handler, tailLines); - - return new DownloadedLog(subFile, description); - } - - public void RunNode(CodexAccess bootstrapNode, Action operation, TestToken mintTestTokens) - { - var lifecycle = CreateTestLifecycle(); - var flow = lifecycle.WorkflowCreator.CreateWorkflow(); - - try - { - var debugInfo = bootstrapNode.GetDebugInfo(); - Assert.That(!string.IsNullOrEmpty(debugInfo.spr)); - - var startupConfig = new StartupConfig(); - startupConfig.NameOverride = "TransientNode"; - var codexStartConfig = new CodexStartupConfig(CodexLogLevel.Trace); - codexStartConfig.MarketplaceConfig = new MarketplaceInitialConfig(0.Eth(), 0.TestTokens(), false); - codexStartConfig.MarketplaceConfig.AccountIndexOverride = ethereumAccountIndex; - codexStartConfig.BootstrapSpr = debugInfo.spr; - startupConfig.Add(codexStartConfig); - startupConfig.Add(config.CodexDeployment.GethStartResult); - var rc = flow.Start(1, Location.Unspecified, new CodexContainerRecipe(), startupConfig); - - var account = config.CodexDeployment.GethStartResult.CompanionNode.Accounts[ethereumAccountIndex]; - - var marketplaceNetwork = config.CodexDeployment.GethStartResult.MarketplaceNetwork; - if (mintTestTokens.Amount > 0) - { - var tokenAddress = marketplaceNetwork.Marketplace.TokenAddress; - var interaction = marketplaceNetwork.Bootstrap.StartInteraction(lifecycle); - interaction.MintTestTokens(new[] { account.Account }, mintTestTokens.Amount, tokenAddress); - } - - var container = rc.Containers[0]; - var address = lifecycle.Configuration.GetAddress(container); - var codexAccess = new CodexAccess(log, container, lifecycle.TimeSet, address); - var marketAccess = new MarketplaceAccess(lifecycle, marketplaceNetwork, account, codexAccess); - - try - { - operation(codexAccess, marketAccess, lifecycle); - } - catch - { - lifecycle.DownloadLog(container); - throw; - } - } - finally - { - flow.DeleteTestResources(); - } - } - - private TestLifecycle CreateTestLifecycle() - { - return k8SFactory.CreateTestLifecycle(config.KubeConfigFile, config.LogPath, config.DataPath, customNamespace, timeSet, log); - } - } -} diff --git a/ContinuousTests/Tests/ThresholdChecks.cs b/ContinuousTests/Tests/ThresholdChecks.cs deleted file mode 100644 index 6ed6ea29..00000000 --- a/ContinuousTests/Tests/ThresholdChecks.cs +++ /dev/null @@ -1,60 +0,0 @@ -using DistTestCore; -using DistTestCore.Codex; -using NUnit.Framework; - -namespace ContinuousTests.Tests -{ - public class ThresholdChecks : ContinuousTest - { - public override int RequiredNumberOfNodes => 1; - public override TimeSpan RunTestEvery => TimeSpan.FromSeconds(30); - public override TestFailMode TestFailMode => TestFailMode.StopAfterFirstFailure; - - private static readonly List previousBreaches = new List(); - - [TestMoment(t: 0)] - public void CheckAllThresholds() - { - var allNodes = CreateAccessToAllNodes(); - foreach (var n in allNodes) CheckThresholds(n); - } - - private void CheckThresholds(CodexAccess n) - { - var breaches = n.GetDebugThresholdBreaches(); - if (breaches.breaches.Any()) - { - var newBreaches = new List(); - foreach (var b in breaches.breaches) - { - if (!previousBreaches.Contains(b)) - { - newBreaches.Add(b); - previousBreaches.Add(b); - } - } - - if (newBreaches.Any()) - { - Assert.Fail(string.Join(",", newBreaches.Select(b => FormatBreach(n, b)))); - - Program.Cancellation.Cts.Cancel(); - } - } - } - - private string FormatBreach(CodexAccess n, string breach) - { - return $"{n.Container.Name} = '{breach}'"; - } - - private CodexAccess[] CreateAccessToAllNodes() - { - // Normally, a continuous test accesses only a subset of the nodes in the deployment. - // This time, we want to check all of them. - var factory = new CodexAccessFactory(); - var allContainers = Configuration.CodexDeployment.CodexContainers; - return factory.Create(Configuration, allContainers, Log, new DefaultTimeSet()); - } - } -} diff --git a/DistTestCore/AutoBootstrapDistTest.cs b/DistTestCore/AutoBootstrapDistTest.cs deleted file mode 100644 index 25bb1554..00000000 --- a/DistTestCore/AutoBootstrapDistTest.cs +++ /dev/null @@ -1,30 +0,0 @@ -using NUnit.Framework; - -namespace DistTestCore -{ - public class AutoBootstrapDistTest : DistTest - { - public override IOnlineCodexNode SetupCodexBootstrapNode(Action setup) - { - throw new Exception("AutoBootstrapDistTest creates and attaches a single bootstrap node for you. " + - "If you want to control the bootstrap node from your test, please use DistTest instead."); - } - - public override ICodexNodeGroup SetupCodexNodes(int numberOfNodes, Action setup) - { - var codexSetup = CreateCodexSetup(numberOfNodes); - setup(codexSetup); - codexSetup.WithBootstrapNode(BootstrapNode); - return BringOnline(codexSetup); - } - - [SetUp] - public void SetUpBootstrapNode() - { - var setup = CreateCodexSetup(1).WithName("BOOTSTRAP"); - BootstrapNode = BringOnline(setup)[0]; - } - - protected IOnlineCodexNode BootstrapNode { get; private set; } = null!; - } -} diff --git a/DistTestCore/BaseStarter.cs b/DistTestCore/BaseStarter.cs deleted file mode 100644 index 4d106432..00000000 --- a/DistTestCore/BaseStarter.cs +++ /dev/null @@ -1,42 +0,0 @@ -using Logging; - -namespace DistTestCore -{ - public class BaseStarter - { - protected readonly TestLifecycle lifecycle; - private Stopwatch? stopwatch; - - public BaseStarter(TestLifecycle lifecycle) - { - this.lifecycle = lifecycle; - } - - protected void LogStart(string msg) - { - Log(msg); - stopwatch = Stopwatch.Begin(lifecycle.Log, GetClassName()); - } - - protected void LogEnd(string msg) - { - stopwatch!.End(msg); - stopwatch = null; - } - - protected void Log(string msg) - { - lifecycle.Log.Log($"{GetClassName()} {msg}"); - } - - protected void Debug(string msg) - { - lifecycle.Log.Debug($"{GetClassName()} {msg}", 1); - } - - private string GetClassName() - { - return $"({GetType().Name})"; - } - } -} diff --git a/DistTestCore/CodexNodeFactory.cs b/DistTestCore/CodexNodeFactory.cs deleted file mode 100644 index 9b67158b..00000000 --- a/DistTestCore/CodexNodeFactory.cs +++ /dev/null @@ -1,32 +0,0 @@ -using DistTestCore.Codex; -using DistTestCore.Marketplace; -using DistTestCore.Metrics; - -namespace DistTestCore -{ - public interface ICodexNodeFactory - { - OnlineCodexNode CreateOnlineCodexNode(CodexAccess access, CodexNodeGroup group); - } - - public class CodexNodeFactory : ICodexNodeFactory - { - private readonly TestLifecycle lifecycle; - private readonly IMetricsAccessFactory metricsAccessFactory; - private readonly IMarketplaceAccessFactory marketplaceAccessFactory; - - public CodexNodeFactory(TestLifecycle lifecycle, IMetricsAccessFactory metricsAccessFactory, IMarketplaceAccessFactory marketplaceAccessFactory) - { - this.lifecycle = lifecycle; - this.metricsAccessFactory = metricsAccessFactory; - this.marketplaceAccessFactory = marketplaceAccessFactory; - } - - public OnlineCodexNode CreateOnlineCodexNode(CodexAccess access, CodexNodeGroup group) - { - var metricsAccess = metricsAccessFactory.CreateMetricsAccess(access.Container); - var marketplaceAccess = marketplaceAccessFactory.CreateMarketplaceAccess(access); - return new OnlineCodexNode(lifecycle, access, group, metricsAccess, marketplaceAccess); - } - } -} diff --git a/DistTestCore/CodexStarter.cs b/DistTestCore/CodexStarter.cs deleted file mode 100644 index 7a8947b0..00000000 --- a/DistTestCore/CodexStarter.cs +++ /dev/null @@ -1,159 +0,0 @@ -using DistTestCore.Codex; -using DistTestCore.Marketplace; -using DistTestCore.Metrics; -using KubernetesWorkflow; -using Logging; - -namespace DistTestCore -{ - public class CodexStarter : BaseStarter - { - public CodexStarter(TestLifecycle lifecycle) - : base(lifecycle) - { - } - - public List RunningGroups { get; } = new List(); - - public ICodexNodeGroup BringOnline(CodexSetup codexSetup) - { - LogSeparator(); - LogStart($"Starting {codexSetup.Describe()}..."); - var gethStartResult = lifecycle.GethStarter.BringOnlineMarketplaceFor(codexSetup); - gethStartResult = lifecycle.GethStarter.BringOnlineValidatorFor(codexSetup, gethStartResult); - - var startupConfig = CreateStartupConfig(gethStartResult, codexSetup); - - var containers = StartCodexContainers(startupConfig, codexSetup.NumberOfNodes, codexSetup.Location); - - var metricAccessFactory = CollectMetrics(codexSetup, containers); - - var codexNodeFactory = new CodexNodeFactory(lifecycle, metricAccessFactory, gethStartResult.MarketplaceAccessFactory); - - var group = CreateCodexGroup(codexSetup, containers, codexNodeFactory); - lifecycle.SetCodexVersion(group.Version); - - var nl = Environment.NewLine; - var podInfos = string.Join(nl, containers.Containers().Select(c => $"Container: '{c.Name}' runs at '{c.Pod.PodInfo.K8SNodeName}'={c.Pod.PodInfo.Ip}")); - LogEnd($"Started {codexSetup.NumberOfNodes} nodes " + - $"of image '{containers.Containers().First().Recipe.Image}' " + - $"and version '{group.Version}'{nl}" + - podInfos); - LogSeparator(); - - return group; - } - - public void BringOffline(CodexNodeGroup group) - { - LogStart($"Stopping {group.Describe()}..."); - var workflow = CreateWorkflow(); - foreach (var c in group.Containers) - { - StopCrashWatcher(c); - workflow.Stop(c); - } - RunningGroups.Remove(group); - LogEnd("Stopped."); - } - - public void DeleteAllResources() - { - var workflow = CreateWorkflow(); - workflow.DeleteTestResources(); - - RunningGroups.Clear(); - } - - public void DownloadLog(RunningContainer container, ILogHandler logHandler, int? tailLines) - { - var workflow = CreateWorkflow(); - workflow.DownloadContainerLog(container, logHandler, tailLines); - } - - private IMetricsAccessFactory CollectMetrics(CodexSetup codexSetup, RunningContainers[] containers) - { - if (codexSetup.MetricsMode == MetricsMode.None) return new MetricsUnavailableAccessFactory(); - - var runningContainers = lifecycle.PrometheusStarter.CollectMetricsFor(containers); - - if (codexSetup.MetricsMode == MetricsMode.Dashboard) - { - lifecycle.GrafanaStarter.StartDashboard(runningContainers.Containers.First(), codexSetup); - } - - return new CodexNodeMetricsAccessFactory(lifecycle, runningContainers); - } - - private StartupConfig CreateStartupConfig(GethStartResult gethStartResult, CodexSetup codexSetup) - { - var startupConfig = new StartupConfig(); - startupConfig.NameOverride = codexSetup.NameOverride; - startupConfig.Add(codexSetup); - startupConfig.Add(gethStartResult); - return startupConfig; - } - - private RunningContainers[] StartCodexContainers(StartupConfig startupConfig, int numberOfNodes, Location location) - { - var result = new List(); - var recipe = new CodexContainerRecipe(); - for (var i = 0; i < numberOfNodes; i++) - { - var workflow = CreateWorkflow(); - var rc = workflow.Start(1, location, recipe, startupConfig); - CreateCrashWatcher(workflow, rc); - result.Add(rc); - } - return result.ToArray(); - } - - private CodexNodeGroup CreateCodexGroup(CodexSetup codexSetup, RunningContainers[] runningContainers, CodexNodeFactory codexNodeFactory) - { - var group = new CodexNodeGroup(lifecycle, codexSetup, runningContainers, codexNodeFactory); - RunningGroups.Add(group); - - try - { - Stopwatch.Measure(lifecycle.Log, "EnsureOnline", group.EnsureOnline, debug: true); - } - catch - { - CodexNodesNotOnline(runningContainers); - throw; - } - - return group; - } - - private void CodexNodesNotOnline(RunningContainers[] runningContainers) - { - Log("Codex nodes failed to start"); - foreach (var container in runningContainers.Containers()) lifecycle.DownloadLog(container); - } - - private StartupWorkflow CreateWorkflow() - { - return lifecycle.WorkflowCreator.CreateWorkflow(); - } - - private void LogSeparator() - { - Log("----------------------------------------------------------------------------"); - } - - private void CreateCrashWatcher(StartupWorkflow workflow, RunningContainers rc) - { - var c = rc.Containers.Single(); - c.CrashWatcher = workflow.CreateCrashWatcher(c); - } - - private void StopCrashWatcher(RunningContainers containers) - { - foreach (var c in containers.Containers) - { - c.CrashWatcher?.Stop(); - } - } - } -} diff --git a/DistTestCore/Configuration.cs b/DistTestCore/Configuration.cs deleted file mode 100644 index a8170221..00000000 --- a/DistTestCore/Configuration.cs +++ /dev/null @@ -1,144 +0,0 @@ -using DistTestCore.Codex; -using KubernetesWorkflow; -using System.Net.NetworkInformation; -using Utils; - -namespace DistTestCore -{ - public class Configuration - { - private readonly string? kubeConfigFile; - private readonly string logPath; - private readonly bool logDebug; - private readonly string dataFilesPath; - private readonly CodexLogLevel codexLogLevel; - private readonly string k8sNamespacePrefix; - private static RunnerLocation? runnerLocation = null; - - public Configuration() - { - kubeConfigFile = GetNullableEnvVarOrDefault("KUBECONFIG", null); - logPath = GetEnvVarOrDefault("LOGPATH", "CodexTestLogs"); - logDebug = GetEnvVarOrDefault("LOGDEBUG", "false").ToLowerInvariant() == "true"; - dataFilesPath = GetEnvVarOrDefault("DATAFILEPATH", "TestDataFiles"); - codexLogLevel = ParseEnum.Parse(GetEnvVarOrDefault("LOGLEVEL", nameof(CodexLogLevel.Trace))); - k8sNamespacePrefix = "ct-"; - } - - public Configuration(string? kubeConfigFile, string logPath, bool logDebug, string dataFilesPath, CodexLogLevel codexLogLevel, string k8sNamespacePrefix) - { - this.kubeConfigFile = kubeConfigFile; - this.logPath = logPath; - this.logDebug = logDebug; - this.dataFilesPath = dataFilesPath; - this.codexLogLevel = codexLogLevel; - this.k8sNamespacePrefix = k8sNamespacePrefix; - } - - public KubernetesWorkflow.Configuration GetK8sConfiguration(ITimeSet timeSet) - { - return new KubernetesWorkflow.Configuration( - k8sNamespacePrefix: k8sNamespacePrefix, - kubeConfigFile: kubeConfigFile, - operationTimeout: timeSet.K8sOperationTimeout(), - retryDelay: timeSet.WaitForK8sServiceDelay() - ); - } - - public Logging.LogConfig GetLogConfig() - { - return new Logging.LogConfig(logPath, debugEnabled: logDebug); - } - - public string GetFileManagerFolder() - { - return dataFilesPath; - } - - public CodexLogLevel GetCodexLogLevel() - { - return codexLogLevel; - } - - public Address GetAddress(RunningContainer container) - { - if (runnerLocation == null) - { - runnerLocation = RunnerLocationUtils.DetermineRunnerLocation(container); - } - - if (runnerLocation == RunnerLocation.InternalToCluster) - { - return container.ClusterInternalAddress; - } - return container.ClusterExternalAddress; - } - - private static string GetEnvVarOrDefault(string varName, string defaultValue) - { - var v = Environment.GetEnvironmentVariable(varName); - if (v == null) return defaultValue; - return v; - } - - private static string? GetNullableEnvVarOrDefault(string varName, string? defaultValue) - { - var v = Environment.GetEnvironmentVariable(varName); - if (v == null) return defaultValue; - return v; - } - } - - public enum RunnerLocation - { - ExternalToCluster, - InternalToCluster, - } - - public static class RunnerLocationUtils - { - private static bool alreadyDidThat = false; - - public static RunnerLocation DetermineRunnerLocation(RunningContainer container) - { - // We want to be sure we don't ping more often than strictly necessary. - // If we have already determined the location during this application - // lifetime, don't do it again. - if (alreadyDidThat) throw new Exception("We already did that."); - alreadyDidThat = true; - - if (PingHost(container.Pod.PodInfo.Ip)) - { - return RunnerLocation.InternalToCluster; - } - if (PingHost(Format(container.ClusterExternalAddress))) - { - return RunnerLocation.ExternalToCluster; - } - - throw new Exception("Unable to determine runner location."); - } - - private static string Format(Address host) - { - return host.Host - .Replace("http://", "") - .Replace("https://", ""); - } - - private static bool PingHost(string host) - { - try - { - using var pinger = new Ping(); - PingReply reply = pinger.Send(host); - return reply.Status == IPStatus.Success; - } - catch (PingException) - { - } - - return false; - } - } -} diff --git a/DistTestCore/DefaultContainerRecipe.cs b/DistTestCore/DefaultContainerRecipe.cs deleted file mode 100644 index cfb8e57a..00000000 --- a/DistTestCore/DefaultContainerRecipe.cs +++ /dev/null @@ -1,40 +0,0 @@ -using KubernetesWorkflow; -using Logging; - -namespace DistTestCore -{ - public abstract class DefaultContainerRecipe : ContainerRecipeFactory - { - public static string TestsType { get; set; } = "NotSet"; - public static ApplicationIds? ApplicationIds { get; set; } = null; - - protected abstract void InitializeRecipe(StartupConfig config); - - protected override void Initialize(StartupConfig config) - { - Add("tests-type", TestsType); - Add("runid", NameUtils.GetRunId()); - Add("testid", NameUtils.GetTestId()); - Add("category", NameUtils.GetCategoryName()); - Add("fixturename", NameUtils.GetRawFixtureName()); - Add("testname", NameUtils.GetTestMethodName()); - - if (ApplicationIds != null) - { - Add("codexid", ApplicationIds.CodexId); - Add("gethid", ApplicationIds.GethId); - Add("prometheusid", ApplicationIds.PrometheusId); - Add("codexcontractsid", ApplicationIds.CodexContractsId); - Add("grafanaid", ApplicationIds.GrafanaId); - } - Add("app", AppName); - - InitializeRecipe(config); - } - - private void Add(string name, string value) - { - AddPodLabel(name, value); - } - } -} diff --git a/DistTestCore/FileManager.cs b/DistTestCore/FileManager.cs deleted file mode 100644 index ab864737..00000000 --- a/DistTestCore/FileManager.cs +++ /dev/null @@ -1,218 +0,0 @@ -using Logging; -using NUnit.Framework; -using Utils; - -namespace DistTestCore -{ - public interface IFileManager - { - TestFile CreateEmptyTestFile(string label = ""); - TestFile GenerateTestFile(ByteSize size, string label = ""); - void DeleteAllTestFiles(); - void PushFileSet(); - void PopFileSet(); - } - - public class FileManager : IFileManager - { - public const int ChunkSize = 1024 * 1024 * 100; - private static NumberSource folderNumberSource = new NumberSource(0); - private readonly Random random = new Random(); - private readonly BaseLog log; - private readonly string folder; - private readonly List> fileSetStack = new List>(); - - public FileManager(BaseLog log, Configuration configuration) - { - folder = Path.Combine(configuration.GetFileManagerFolder(), folderNumberSource.GetNextNumber().ToString("D5")); - - EnsureDirectory(); - this.log = log; - } - - public TestFile CreateEmptyTestFile(string label = "") - { - var path = Path.Combine(folder, Guid.NewGuid().ToString() + "_test.bin"); - var result = new TestFile(log, path, label); - File.Create(result.Filename).Close(); - if (fileSetStack.Any()) fileSetStack.Last().Add(result); - return result; - } - - public TestFile GenerateTestFile(ByteSize size, string label) - { - var sw = Stopwatch.Begin(log); - var result = GenerateFile(size, label); - sw.End($"Generated file '{result.Describe()}'."); - return result; - } - - public void DeleteAllTestFiles() - { - DeleteDirectory(); - } - - public void PushFileSet() - { - fileSetStack.Add(new List()); - } - - public void PopFileSet() - { - if (!fileSetStack.Any()) return; - var pop = fileSetStack.Last(); - fileSetStack.Remove(pop); - - foreach (var file in pop) - { - try - { - File.Delete(file.Filename); - } - catch { } - } - } - - private TestFile GenerateFile(ByteSize size, string label) - { - var result = CreateEmptyTestFile(label); - CheckSpaceAvailable(result, size); - - GenerateFileBytes(result, size); - return result; - } - - private void CheckSpaceAvailable(TestFile testFile, ByteSize size) - { - var file = new FileInfo(testFile.Filename); - var drive = new DriveInfo(file.Directory!.Root.FullName); - - var spaceAvailable = drive.TotalFreeSpace; - - if (spaceAvailable < size.SizeInBytes) - { - var msg = $"Inconclusive: Not enough disk space to perform test. " + - $"{Formatter.FormatByteSize(size.SizeInBytes)} required. " + - $"{Formatter.FormatByteSize(spaceAvailable)} available."; - - log.Log(msg); - Assert.Inconclusive(msg); - } - } - - private void GenerateFileBytes(TestFile result, ByteSize size) - { - long bytesLeft = size.SizeInBytes; - int chunkSize = ChunkSize; - while (bytesLeft > 0) - { - try - { - var length = Math.Min(bytesLeft, chunkSize); - AppendRandomBytesToFile(result, length); - bytesLeft -= length; - } - catch - { - chunkSize = chunkSize / 2; - if (chunkSize < 1024) throw; - } - } - } - - private void AppendRandomBytesToFile(TestFile result, long length) - { - var bytes = new byte[length]; - random.NextBytes(bytes); - using var stream = new FileStream(result.Filename, FileMode.Append); - stream.Write(bytes, 0, bytes.Length); - } - - private void EnsureDirectory() - { - if (!Directory.Exists(folder)) Directory.CreateDirectory(folder); - } - - private void DeleteDirectory() - { - Directory.Delete(folder, true); - } - } - - public class TestFile - { - private readonly BaseLog log; - - public TestFile(BaseLog log, string filename, string label) - { - this.log = log; - Filename = filename; - Label = label; - } - - public string Filename { get; } - public string Label { get; } - - public void AssertIsEqual(TestFile? actual) - { - var sw = Stopwatch.Begin(log); - try - { - AssertEqual(actual); - } - finally - { - sw.End($"{nameof(TestFile)}.{nameof(AssertIsEqual)}"); - } - } - - public string Describe() - { - var sizePostfix = $" ({Formatter.FormatByteSize(GetFileSize())})"; - if (!string.IsNullOrEmpty(Label)) return Label + sizePostfix; - return $"'{Filename}'{sizePostfix}"; - } - - private void AssertEqual(TestFile? actual) - { - if (actual == null) Assert.Fail("TestFile is null."); - if (actual == this || actual!.Filename == Filename) Assert.Fail("TestFile is compared to itself."); - - Assert.That(actual.GetFileSize(), Is.EqualTo(GetFileSize()), "Files are not of equal length."); - - using var streamExpected = new FileStream(Filename, FileMode.Open, FileAccess.Read); - using var streamActual = new FileStream(actual.Filename, FileMode.Open, FileAccess.Read); - - var bytesExpected = new byte[FileManager.ChunkSize]; - var bytesActual = new byte[FileManager.ChunkSize]; - - var readExpected = 0; - var readActual = 0; - - while (true) - { - readExpected = streamExpected.Read(bytesExpected, 0, FileManager.ChunkSize); - readActual = streamActual.Read(bytesActual, 0, FileManager.ChunkSize); - - if (readExpected == 0 && readActual == 0) - { - log.Log($"OK: '{Describe()}' is equal to '{actual.Describe()}'."); - return; - } - - Assert.That(readActual, Is.EqualTo(readExpected), "Unable to read buffers of equal length."); - - for (var i = 0; i < readActual; i++) - { - if (bytesExpected[i] != bytesActual[i]) Assert.Fail("File contents not equal."); - } - } - } - - private long GetFileSize() - { - var info = new FileInfo(Filename); - return info.Length; - } - } -} diff --git a/DistTestCore/GethStarter.cs b/DistTestCore/GethStarter.cs deleted file mode 100644 index 7df4ba41..00000000 --- a/DistTestCore/GethStarter.cs +++ /dev/null @@ -1,99 +0,0 @@ -using DistTestCore.Marketplace; - -namespace DistTestCore -{ - public class GethStarter : BaseStarter - { - private readonly MarketplaceNetworkCache marketplaceNetworkCache; - private readonly GethCompanionNodeStarter companionNodeStarter; - - public GethStarter(TestLifecycle lifecycle) - : base(lifecycle) - { - marketplaceNetworkCache = new MarketplaceNetworkCache( - new GethBootstrapNodeStarter(lifecycle), - new CodexContractsStarter(lifecycle)); - companionNodeStarter = new GethCompanionNodeStarter(lifecycle); - } - - public GethStartResult BringOnlineMarketplaceFor(CodexSetup codexSetup) - { - if (codexSetup.MarketplaceConfig == null) return CreateMarketplaceUnavailableResult(); - - var marketplaceNetwork = marketplaceNetworkCache.Get(); - var companionNode = StartCompanionNode(codexSetup, marketplaceNetwork); - - LogStart("Setting up initial balance..."); - TransferInitialBalance(marketplaceNetwork, codexSetup.MarketplaceConfig, companionNode); - LogEnd($"Initial balance of {codexSetup.MarketplaceConfig.InitialTestTokens} set for {codexSetup.NumberOfNodes} nodes."); - - return CreateGethStartResult(marketplaceNetwork, companionNode); - } - - public GethStartResult BringOnlineValidatorFor(CodexSetup codexSetup, GethStartResult previousResult) - { - // allow marketplace and validator to be enabled on the same Codex node - if (previousResult.CompanionNode != null || (codexSetup.EnableValidator ?? false) == false) return previousResult; - - var marketplaceNetwork = marketplaceNetworkCache.Get(); - var companionNode = StartCompanionNode(codexSetup, marketplaceNetwork); - - return CreateGethStartResult(marketplaceNetwork, companionNode); - } - - private void TransferInitialBalance(MarketplaceNetwork marketplaceNetwork, MarketplaceInitialConfig marketplaceConfig, GethCompanionNodeInfo companionNode) - { - if (marketplaceConfig.InitialTestTokens.Amount == 0) return; - - var interaction = marketplaceNetwork.StartInteraction(lifecycle); - var tokenAddress = marketplaceNetwork.Marketplace.TokenAddress; - - var accounts = companionNode.Accounts.Select(a => a.Account).ToArray(); - interaction.MintTestTokens(accounts, marketplaceConfig.InitialTestTokens.Amount, tokenAddress); - } - - private GethStartResult CreateGethStartResult(MarketplaceNetwork marketplaceNetwork, GethCompanionNodeInfo companionNode) - { - return new GethStartResult(CreateMarketplaceAccessFactory(marketplaceNetwork), marketplaceNetwork, companionNode); - } - - private GethStartResult CreateMarketplaceUnavailableResult() - { - return new GethStartResult(new MarketplaceUnavailableAccessFactory(), null!, null!); - } - - private IMarketplaceAccessFactory CreateMarketplaceAccessFactory(MarketplaceNetwork marketplaceNetwork) - { - return new GethMarketplaceAccessFactory(lifecycle, marketplaceNetwork); - } - - private GethCompanionNodeInfo StartCompanionNode(CodexSetup codexSetup, MarketplaceNetwork marketplaceNetwork) - { - return companionNodeStarter.StartCompanionNodeFor(codexSetup, marketplaceNetwork); - } - } - - public class MarketplaceNetworkCache - { - private readonly GethBootstrapNodeStarter bootstrapNodeStarter; - private readonly CodexContractsStarter codexContractsStarter; - private MarketplaceNetwork? network; - - public MarketplaceNetworkCache(GethBootstrapNodeStarter bootstrapNodeStarter, CodexContractsStarter codexContractsStarter) - { - this.bootstrapNodeStarter = bootstrapNodeStarter; - this.codexContractsStarter = codexContractsStarter; - } - - public MarketplaceNetwork Get() - { - if (network == null) - { - var bootstrapInfo = bootstrapNodeStarter.StartGethBootstrapNode(); - var marketplaceInfo = codexContractsStarter.Start(bootstrapInfo); - network = new MarketplaceNetwork(bootstrapInfo, marketplaceInfo ); - } - return network; - } - } -} diff --git a/DistTestCore/GrafanaStarter.cs b/DistTestCore/GrafanaStarter.cs deleted file mode 100644 index d4a3cd8f..00000000 --- a/DistTestCore/GrafanaStarter.cs +++ /dev/null @@ -1,193 +0,0 @@ -using DistTestCore.Metrics; -using IdentityModel.Client; -using KubernetesWorkflow; -using Newtonsoft.Json; -using System.Reflection; -using Utils; - -namespace DistTestCore -{ - public class GrafanaStarter : BaseStarter - { - private const string StorageQuotaThresholdReplaceToken = "\"\""; - private const string BytesUsedGraphAxisSoftMaxReplaceToken = "\"\""; - - public GrafanaStarter(TestLifecycle lifecycle) - : base(lifecycle) - { - } - - public GrafanaStartInfo StartDashboard(RunningContainer prometheusContainer, CodexSetup codexSetup) - { - LogStart($"Starting dashboard server"); - - var grafanaContainer = StartGrafanaContainer(); - var grafanaAddress = lifecycle.Configuration.GetAddress(grafanaContainer); - - var http = new Http(lifecycle.Log, new DefaultTimeSet(), grafanaAddress, "api/", AddBasicAuth); - - Log("Connecting datasource..."); - AddDataSource(http, prometheusContainer); - - Log("Uploading dashboard configurations..."); - var jsons = ReadEachDashboardJsonFile(codexSetup); - var dashboardUrls = jsons.Select(j => UploadDashboard(http, grafanaContainer, j)).ToArray(); - - LogEnd("Dashboard server started."); - - return new GrafanaStartInfo(dashboardUrls, grafanaContainer); - } - - private RunningContainer StartGrafanaContainer() - { - var startupConfig = new StartupConfig(); - - var workflow = lifecycle.WorkflowCreator.CreateWorkflow(); - var grafanaContainers = workflow.Start(1, Location.Unspecified, new GrafanaContainerRecipe(), startupConfig); - if (grafanaContainers.Containers.Length != 1) throw new InvalidOperationException("Expected 1 dashboard container to be created."); - - return grafanaContainers.Containers.First(); - } - - private void AddBasicAuth(HttpClient client) - { - client.SetBasicAuthentication( - GrafanaContainerRecipe.DefaultAdminUser, - GrafanaContainerRecipe.DefaultAdminPassword); - } - - private static void AddDataSource(Http http, RunningContainer prometheusContainer) - { - var prometheusAddress = prometheusContainer.ClusterExternalAddress; - var prometheusUrl = prometheusAddress.Host + ":" + prometheusAddress.Port; - var response = http.HttpPostJson("datasources", new GrafanaDataSourceRequest - { - uid = "c89eaad3-9184-429f-ac94-8ba0b1824dbb", - name = "CodexPrometheus", - type = "prometheus", - url = prometheusUrl, - access = "proxy", - basicAuth = false, - jsonData = new GrafanaDataSourceJsonData - { - httpMethod = "POST" - } - }); - - if (response.message != "Datasource added") - { - throw new Exception("Test infra failure: Failed to add datasource to dashboard: " + response.message); - } - } - - public static string UploadDashboard(Http http, RunningContainer grafanaContainer, string dashboardJson) - { - var request = GetDashboardCreateRequest(dashboardJson); - var response = http.HttpPostString("dashboards/db", request); - var jsonResponse = JsonConvert.DeserializeObject(response); - if (jsonResponse == null || string.IsNullOrEmpty(jsonResponse.url)) throw new Exception("Failed to upload dashboard."); - - var grafanaAddress = grafanaContainer.ClusterExternalAddress; - return grafanaAddress.Host + ":" + grafanaAddress.Port + jsonResponse.url; - } - - private static string[] ReadEachDashboardJsonFile(CodexSetup codexSetup) - { - var assembly = Assembly.GetExecutingAssembly(); - var resourceNames = new[] - { - "DistTestCore.Metrics.dashboard.json" - }; - - return resourceNames.Select(r => GetManifestResource(assembly, r, codexSetup)).ToArray(); - } - - private static string GetManifestResource(Assembly assembly, string resourceName, CodexSetup codexSetup) - { - using var stream = assembly.GetManifestResourceStream(resourceName); - if (stream == null) throw new Exception("Unable to find resource " + resourceName); - using var reader = new StreamReader(stream); - return ApplyReplacements(reader.ReadToEnd(), codexSetup); - } - - private static string ApplyReplacements(string input, CodexSetup codexSetup) - { - var quotaString = GetQuotaString(codexSetup); - var softMaxString = GetSoftMaxString(codexSetup); - - return input - .Replace(StorageQuotaThresholdReplaceToken, quotaString) - .Replace(BytesUsedGraphAxisSoftMaxReplaceToken, softMaxString); - } - - private static string GetQuotaString(CodexSetup codexSetup) - { - return GetCodexStorageQuotaInBytes(codexSetup).ToString(); - } - - private static string GetSoftMaxString(CodexSetup codexSetup) - { - var quota = GetCodexStorageQuotaInBytes(codexSetup); - var softMax = Convert.ToInt64(quota * 1.1); // + 10%, for nice viewing. - return softMax.ToString(); - } - - private static long GetCodexStorageQuotaInBytes(CodexSetup codexSetup) - { - if (codexSetup.StorageQuota != null) return codexSetup.StorageQuota.SizeInBytes; - - // Codex default: 8GB - return 8.GB().SizeInBytes; - } - - private static string GetDashboardCreateRequest(string dashboardJson) - { - return $"{{\"dashboard\": {dashboardJson} ,\"message\": \"Default Codex Dashboard\",\"overwrite\": false}}"; - } - } - - public class GrafanaStartInfo - { - public GrafanaStartInfo(string[] dashboardUrls, RunningContainer container) - { - DashboardUrls = dashboardUrls; - Container = container; - } - - public string[] DashboardUrls { get; } - public RunningContainer Container { get; } - } - - public class GrafanaDataSourceRequest - { - public string uid { get; set; } = string.Empty; - public string name { get; set; } = string.Empty; - public string type { get; set; } = string.Empty; - public string url { get; set; } = string.Empty; - public string access { get; set; } = string.Empty; - public bool basicAuth { get; set; } - public GrafanaDataSourceJsonData jsonData { get; set; } = new(); - } - - public class GrafanaDataSourceResponse - { - public int id { get; set; } - public string message { get; set; } = string.Empty; - public string name { get; set; } = string.Empty; - } - - public class GrafanaDataSourceJsonData - { - public string httpMethod { get; set; } = string.Empty; - } - - public class GrafanaPostDashboardResponse - { - public int id { get; set; } - public string slug { get; set; } = string.Empty; - public string status { get; set; } = string.Empty; - public string uid { get; set; } = string.Empty; - public string url { get; set; } = string.Empty; - public int version { get; set; } - } -} diff --git a/DistTestCore/Logs/DontDownloadLogsAndMetricsOnFailureAttribute.cs b/DistTestCore/Logs/DontDownloadLogsAndMetricsOnFailureAttribute.cs deleted file mode 100644 index b95d875a..00000000 --- a/DistTestCore/Logs/DontDownloadLogsAndMetricsOnFailureAttribute.cs +++ /dev/null @@ -1,15 +0,0 @@ -using NUnit.Framework; - -namespace DistTestCore.Logs -{ - [AttributeUsage(AttributeTargets.Method, AllowMultiple = false)] - public class DontDownloadLogsAndMetricsOnFailureAttribute : PropertyAttribute - { - public const string DontDownloadKey = "DontDownloadLogsAndMetrics"; - - public DontDownloadLogsAndMetricsOnFailureAttribute() - : base(DontDownloadKey) - { - } - } -} diff --git a/DistTestCore/Marketplace/CodexContractsContainerConfig.cs b/DistTestCore/Marketplace/CodexContractsContainerConfig.cs deleted file mode 100644 index 3b669a4b..00000000 --- a/DistTestCore/Marketplace/CodexContractsContainerConfig.cs +++ /dev/null @@ -1,16 +0,0 @@ -using KubernetesWorkflow; - -namespace DistTestCore.Marketplace -{ - public class CodexContractsContainerConfig - { - public CodexContractsContainerConfig(string bootstrapNodeIp, Port jsonRpcPort) - { - BootstrapNodeIp = bootstrapNodeIp; - JsonRpcPort = jsonRpcPort; - } - - public string BootstrapNodeIp { get; } - public Port JsonRpcPort { get; } - } -} diff --git a/DistTestCore/Marketplace/GethBootstrapNodeInfo.cs b/DistTestCore/Marketplace/GethBootstrapNodeInfo.cs deleted file mode 100644 index 6825ba58..00000000 --- a/DistTestCore/Marketplace/GethBootstrapNodeInfo.cs +++ /dev/null @@ -1,42 +0,0 @@ -using KubernetesWorkflow; -using NethereumWorkflow; - -namespace DistTestCore.Marketplace -{ - public class GethBootstrapNodeInfo - { - public GethBootstrapNodeInfo(RunningContainers runningContainers, AllGethAccounts allAccounts, string pubKey, Port discoveryPort) - { - RunningContainers = runningContainers; - AllAccounts = allAccounts; - Account = allAccounts.Accounts[0]; - PubKey = pubKey; - DiscoveryPort = discoveryPort; - } - - public RunningContainers RunningContainers { get; } - public AllGethAccounts AllAccounts { get; } - public GethAccount Account { get; } - public string PubKey { get; } - public Port DiscoveryPort { get; } - - public NethereumInteraction StartInteraction(TestLifecycle lifecycle) - { - var address = lifecycle.Configuration.GetAddress(RunningContainers.Containers[0]); - var account = Account; - - var creator = new NethereumInteractionCreator(lifecycle.Log, address.Host, address.Port, account.PrivateKey); - return creator.CreateWorkflow(); - } - } - - public class AllGethAccounts - { - public GethAccount[] Accounts { get; } - - public AllGethAccounts(GethAccount[] accounts) - { - Accounts = accounts; - } - } -} diff --git a/DistTestCore/Marketplace/GethBootstrapNodeStarter.cs b/DistTestCore/Marketplace/GethBootstrapNodeStarter.cs deleted file mode 100644 index d1ebb547..00000000 --- a/DistTestCore/Marketplace/GethBootstrapNodeStarter.cs +++ /dev/null @@ -1,40 +0,0 @@ -using KubernetesWorkflow; - -namespace DistTestCore.Marketplace -{ - public class GethBootstrapNodeStarter : BaseStarter - { - public GethBootstrapNodeStarter(TestLifecycle lifecycle) - : base(lifecycle) - { - } - - public GethBootstrapNodeInfo StartGethBootstrapNode() - { - LogStart("Starting Geth bootstrap node..."); - var startupConfig = CreateBootstrapStartupConfig(); - - var workflow = lifecycle.WorkflowCreator.CreateWorkflow(); - var containers = workflow.Start(1, Location.Unspecified, new GethContainerRecipe(), startupConfig); - if (containers.Containers.Length != 1) throw new InvalidOperationException("Expected 1 Geth bootstrap node to be created. Test infra failure."); - var bootstrapContainer = containers.Containers[0]; - - var extractor = new ContainerInfoExtractor(lifecycle.Log, workflow, bootstrapContainer); - var accounts = extractor.ExtractAccounts(); - var pubKey = extractor.ExtractPubKey(); - var discoveryPort = bootstrapContainer.Recipe.GetPortByTag(GethContainerRecipe.DiscoveryPortTag); - var result = new GethBootstrapNodeInfo(containers, accounts, pubKey, discoveryPort); - - LogEnd($"Geth bootstrap node started with account '{result.Account.Account}'"); - - return result; - } - - private StartupConfig CreateBootstrapStartupConfig() - { - var config = new StartupConfig(); - config.Add(new GethStartupConfig(true, null!, 0, 0)); - return config; - } - } -} diff --git a/DistTestCore/Marketplace/GethCompanionNodeInfo.cs b/DistTestCore/Marketplace/GethCompanionNodeInfo.cs deleted file mode 100644 index 3230d2a5..00000000 --- a/DistTestCore/Marketplace/GethCompanionNodeInfo.cs +++ /dev/null @@ -1,38 +0,0 @@ -using KubernetesWorkflow; -using NethereumWorkflow; - -namespace DistTestCore.Marketplace -{ - public class GethCompanionNodeInfo - { - public GethCompanionNodeInfo(RunningContainer runningContainer, GethAccount[] accounts) - { - RunningContainer = runningContainer; - Accounts = accounts; - } - - public RunningContainer RunningContainer { get; } - public GethAccount[] Accounts { get; } - - public NethereumInteraction StartInteraction(TestLifecycle lifecycle, GethAccount account) - { - var address = lifecycle.Configuration.GetAddress(RunningContainer); - var privateKey = account.PrivateKey; - - var creator = new NethereumInteractionCreator(lifecycle.Log, address.Host, address.Port, privateKey); - return creator.CreateWorkflow(); - } - } - - public class GethAccount - { - public GethAccount(string account, string privateKey) - { - Account = account; - PrivateKey = privateKey; - } - - public string Account { get; } - public string PrivateKey { get; } - } -} diff --git a/DistTestCore/Marketplace/GethCompanionNodeStarter.cs b/DistTestCore/Marketplace/GethCompanionNodeStarter.cs deleted file mode 100644 index 6759e7b4..00000000 --- a/DistTestCore/Marketplace/GethCompanionNodeStarter.cs +++ /dev/null @@ -1,77 +0,0 @@ -using KubernetesWorkflow; -using Utils; - -namespace DistTestCore.Marketplace -{ - public class GethCompanionNodeStarter : BaseStarter - { - private int companionAccountIndex = 0; - - public GethCompanionNodeStarter(TestLifecycle lifecycle) - : base(lifecycle) - { - } - - public GethCompanionNodeInfo StartCompanionNodeFor(CodexSetup codexSetup, MarketplaceNetwork marketplace) - { - LogStart($"Initializing companion for {codexSetup.NumberOfNodes} Codex nodes."); - - var config = CreateCompanionNodeStartupConfig(marketplace.Bootstrap, codexSetup.NumberOfNodes); - - var workflow = lifecycle.WorkflowCreator.CreateWorkflow(); - var containers = workflow.Start(1, Location.Unspecified, new GethContainerRecipe(), CreateStartupConfig(config)); - if (containers.Containers.Length != 1) throw new InvalidOperationException("Expected one Geth companion node to be created. Test infra failure."); - var container = containers.Containers[0]; - - var node = CreateCompanionInfo(container, marketplace, config); - EnsureCompanionNodeIsSynced(node, marketplace); - - LogEnd($"Initialized one companion node for {codexSetup.NumberOfNodes} Codex nodes. Their accounts: [{string.Join(",", node.Accounts.Select(a => a.Account))}]"); - return node; - } - - private GethCompanionNodeInfo CreateCompanionInfo(RunningContainer container, MarketplaceNetwork marketplace, GethStartupConfig config) - { - var accounts = ExtractAccounts(marketplace, config); - return new GethCompanionNodeInfo(container, accounts); - } - - private static GethAccount[] ExtractAccounts(MarketplaceNetwork marketplace, GethStartupConfig config) - { - return marketplace.Bootstrap.AllAccounts.Accounts - .Skip(1 + config.CompanionAccountStartIndex) - .Take(config.NumberOfCompanionAccounts) - .ToArray(); - } - - private void EnsureCompanionNodeIsSynced(GethCompanionNodeInfo node, MarketplaceNetwork marketplace) - { - try - { - Time.WaitUntil(() => - { - var interaction = node.StartInteraction(lifecycle, node.Accounts.First()); - return interaction.IsSynced(marketplace.Marketplace.Address, marketplace.Marketplace.Abi); - }, TimeSpan.FromMinutes(1), TimeSpan.FromSeconds(3)); - } - catch (Exception e) - { - throw new Exception("Geth companion node did not sync within timeout. Test infra failure.", e); - } - } - - private GethStartupConfig CreateCompanionNodeStartupConfig(GethBootstrapNodeInfo bootstrapNode, int numberOfAccounts) - { - var config = new GethStartupConfig(false, bootstrapNode, companionAccountIndex, numberOfAccounts); - companionAccountIndex += numberOfAccounts; - return config; - } - - private StartupConfig CreateStartupConfig(GethStartupConfig gethConfig) - { - var config = new StartupConfig(); - config.Add(gethConfig); - return config; - } - } -} diff --git a/DistTestCore/Marketplace/GethContainerRecipe.cs b/DistTestCore/Marketplace/GethContainerRecipe.cs deleted file mode 100644 index 540b1d33..00000000 --- a/DistTestCore/Marketplace/GethContainerRecipe.cs +++ /dev/null @@ -1,73 +0,0 @@ -using KubernetesWorkflow; - -namespace DistTestCore.Marketplace -{ - public class GethContainerRecipe : DefaultContainerRecipe - { - private const string defaultArgs = "--ipcdisable --syncmode full"; - - public const string HttpPortTag = "http_port"; - public const string DiscoveryPortTag = "disc_port"; - public const string AccountsFilename = "accounts.csv"; - - public override string AppName => "geth"; - public override string Image => "codexstorage/dist-tests-geth:latest"; - - protected override void InitializeRecipe(StartupConfig startupConfig) - { - var config = startupConfig.Get(); - - var args = CreateArgs(config); - - AddEnvVar("GETH_ARGS", args); - } - - private string CreateArgs(GethStartupConfig config) - { - var discovery = AddInternalPort(tag: DiscoveryPortTag); - - if (config.IsBootstrapNode) - { - return CreateBootstapArgs(discovery); - } - - return CreateCompanionArgs(discovery, config); - } - - private string CreateBootstapArgs(Port discovery) - { - AddEnvVar("ENABLE_MINER", "1"); - UnlockAccounts(0, 1); - var exposedPort = AddExposedPort(tag: HttpPortTag); - return $"--http.port {exposedPort.Number} --port {discovery.Number} --discovery.port {discovery.Number} {defaultArgs}"; - } - - private string CreateCompanionArgs(Port discovery, GethStartupConfig config) - { - UnlockAccounts( - config.CompanionAccountStartIndex + 1, - config.NumberOfCompanionAccounts); - - var port = AddInternalPort(); - var authRpc = AddInternalPort(); - var httpPort = AddExposedPort(tag: HttpPortTag); - - var bootPubKey = config.BootstrapNode.PubKey; - var bootIp = config.BootstrapNode.RunningContainers.Containers[0].Pod.PodInfo.Ip; - var bootPort = config.BootstrapNode.DiscoveryPort.Number; - var bootstrapArg = $"--bootnodes enode://{bootPubKey}@{bootIp}:{bootPort} --nat=extip:{bootIp}"; - - return $"--port {port.Number} --discovery.port {discovery.Number} --authrpc.port {authRpc.Number} --http.addr 0.0.0.0 --http.port {httpPort.Number} --ws --ws.addr 0.0.0.0 --ws.port {httpPort.Number} {bootstrapArg} {defaultArgs}"; - } - - private void UnlockAccounts(int startIndex, int numberOfAccounts) - { - if (startIndex < 0) throw new ArgumentException(); - if (numberOfAccounts < 1) throw new ArgumentException(); - if (startIndex + numberOfAccounts > 1000) throw new ArgumentException("Out of accounts!"); - - AddEnvVar("UNLOCK_START_INDEX", startIndex.ToString()); - AddEnvVar("UNLOCK_NUMBER", numberOfAccounts.ToString()); - } - } -} diff --git a/DistTestCore/Marketplace/GethStartResult.cs b/DistTestCore/Marketplace/GethStartResult.cs deleted file mode 100644 index 0ba1e58a..00000000 --- a/DistTestCore/Marketplace/GethStartResult.cs +++ /dev/null @@ -1,19 +0,0 @@ -using Newtonsoft.Json; - -namespace DistTestCore.Marketplace -{ - public class GethStartResult - { - public GethStartResult(IMarketplaceAccessFactory marketplaceAccessFactory, MarketplaceNetwork marketplaceNetwork, GethCompanionNodeInfo companionNode) - { - MarketplaceAccessFactory = marketplaceAccessFactory; - MarketplaceNetwork = marketplaceNetwork; - CompanionNode = companionNode; - } - - [JsonIgnore] - public IMarketplaceAccessFactory MarketplaceAccessFactory { get; } - public MarketplaceNetwork MarketplaceNetwork { get; } - public GethCompanionNodeInfo CompanionNode { get; } - } -} diff --git a/DistTestCore/Marketplace/GethStartupConfig.cs b/DistTestCore/Marketplace/GethStartupConfig.cs deleted file mode 100644 index 7aee0788..00000000 --- a/DistTestCore/Marketplace/GethStartupConfig.cs +++ /dev/null @@ -1,18 +0,0 @@ -namespace DistTestCore.Marketplace -{ - public class GethStartupConfig - { - public GethStartupConfig(bool isBootstrapNode, GethBootstrapNodeInfo bootstrapNode, int companionAccountStartIndex, int numberOfCompanionAccounts) - { - IsBootstrapNode = isBootstrapNode; - BootstrapNode = bootstrapNode; - CompanionAccountStartIndex = companionAccountStartIndex; - NumberOfCompanionAccounts = numberOfCompanionAccounts; - } - - public bool IsBootstrapNode { get; } - public GethBootstrapNodeInfo BootstrapNode { get; } - public int CompanionAccountStartIndex { get; } - public int NumberOfCompanionAccounts { get; } - } -} diff --git a/DistTestCore/Marketplace/MarketplaceAccessFactory.cs b/DistTestCore/Marketplace/MarketplaceAccessFactory.cs deleted file mode 100644 index ac595567..00000000 --- a/DistTestCore/Marketplace/MarketplaceAccessFactory.cs +++ /dev/null @@ -1,41 +0,0 @@ -using DistTestCore.Codex; - -namespace DistTestCore.Marketplace -{ - public interface IMarketplaceAccessFactory - { - IMarketplaceAccess CreateMarketplaceAccess(CodexAccess access); - } - - public class MarketplaceUnavailableAccessFactory : IMarketplaceAccessFactory - { - public IMarketplaceAccess CreateMarketplaceAccess(CodexAccess access) - { - return new MarketplaceUnavailable(); - } - } - - public class GethMarketplaceAccessFactory : IMarketplaceAccessFactory - { - private readonly TestLifecycle lifecycle; - private readonly MarketplaceNetwork marketplaceNetwork; - - public GethMarketplaceAccessFactory(TestLifecycle lifecycle, MarketplaceNetwork marketplaceNetwork) - { - this.lifecycle = lifecycle; - this.marketplaceNetwork = marketplaceNetwork; - } - - public IMarketplaceAccess CreateMarketplaceAccess(CodexAccess access) - { - var companionNode = GetGethCompanionNode(access); - return new MarketplaceAccess(lifecycle, marketplaceNetwork, companionNode, access); - } - - private GethAccount GetGethCompanionNode(CodexAccess access) - { - var account = access.Container.Recipe.Additionals.Single(a => a is GethAccount); - return (GethAccount)account; - } - } -} diff --git a/DistTestCore/Marketplace/MarketplaceInitialConfig.cs b/DistTestCore/Marketplace/MarketplaceInitialConfig.cs deleted file mode 100644 index c51d79f8..00000000 --- a/DistTestCore/Marketplace/MarketplaceInitialConfig.cs +++ /dev/null @@ -1,17 +0,0 @@ -namespace DistTestCore.Marketplace -{ - public class MarketplaceInitialConfig - { - public MarketplaceInitialConfig(Ether initialEth, TestToken initialTestTokens, bool isValidator) - { - InitialEth = initialEth; - InitialTestTokens = initialTestTokens; - IsValidator = isValidator; - } - - public Ether InitialEth { get; } - public TestToken InitialTestTokens { get; } - public bool IsValidator { get; } - public int? AccountIndexOverride { get; set; } - } -} diff --git a/DistTestCore/Marketplace/MarketplaceNetwork.cs b/DistTestCore/Marketplace/MarketplaceNetwork.cs deleted file mode 100644 index bba80a29..00000000 --- a/DistTestCore/Marketplace/MarketplaceNetwork.cs +++ /dev/null @@ -1,21 +0,0 @@ -using NethereumWorkflow; - -namespace DistTestCore.Marketplace -{ - public class MarketplaceNetwork - { - public MarketplaceNetwork(GethBootstrapNodeInfo bootstrap, MarketplaceInfo marketplace) - { - Bootstrap = bootstrap; - Marketplace = marketplace; - } - - public GethBootstrapNodeInfo Bootstrap { get; } - public MarketplaceInfo Marketplace { get; } - - public NethereumInteraction StartInteraction(TestLifecycle lifecycle) - { - return Bootstrap.StartInteraction(lifecycle); - } - } -} diff --git a/DistTestCore/Metrics/GrafanaContainerRecipe.cs b/DistTestCore/Metrics/GrafanaContainerRecipe.cs deleted file mode 100644 index 24ab599b..00000000 --- a/DistTestCore/Metrics/GrafanaContainerRecipe.cs +++ /dev/null @@ -1,25 +0,0 @@ -using KubernetesWorkflow; - -namespace DistTestCore.Metrics -{ - public class GrafanaContainerRecipe : DefaultContainerRecipe - { - public override string AppName => "grafana"; - public override string Image => "grafana/grafana-oss:10.0.3"; - - public const string DefaultAdminUser = "adminium"; - public const string DefaultAdminPassword = "passwordium"; - - protected override void InitializeRecipe(StartupConfig startupConfig) - { - AddExposedPort(3000); - - AddEnvVar("GF_AUTH_ANONYMOUS_ENABLED", "true"); - AddEnvVar("GF_AUTH_ANONYMOUS_ORG_NAME", "Main Org."); - AddEnvVar("GF_AUTH_ANONYMOUS_ORG_ROLE", "Editor"); - - AddEnvVar("GF_SECURITY_ADMIN_USER", DefaultAdminUser); - AddEnvVar("GF_SECURITY_ADMIN_PASSWORD", DefaultAdminPassword); - } - } -} diff --git a/DistTestCore/Metrics/MetricsAccess.cs b/DistTestCore/Metrics/MetricsAccess.cs deleted file mode 100644 index 23b6522d..00000000 --- a/DistTestCore/Metrics/MetricsAccess.cs +++ /dev/null @@ -1,81 +0,0 @@ -using DistTestCore.Helpers; -using KubernetesWorkflow; -using Logging; -using NUnit.Framework; -using NUnit.Framework.Constraints; -using Utils; - -namespace DistTestCore.Metrics -{ - public interface IMetricsAccess - { - void AssertThat(string metricName, IResolveConstraint constraint, string message = ""); - } - - public class MetricsAccess : IMetricsAccess - { - private readonly BaseLog log; - private readonly ITimeSet timeSet; - private readonly MetricsQuery query; - private readonly RunningContainer node; - - public MetricsAccess(BaseLog log, ITimeSet timeSet, MetricsQuery query, RunningContainer node) - { - this.log = log; - this.timeSet = timeSet; - this.query = query; - this.node = node; - } - - public void AssertThat(string metricName, IResolveConstraint constraint, string message = "") - { - AssertHelpers.RetryAssert(constraint, () => - { - var metricSet = GetMetricWithTimeout(metricName); - var metricValue = metricSet.Values[0].Value; - - log.Log($"{node.Name} metric '{metricName}' = {metricValue}"); - return metricValue; - }, message); - } - - public Metrics? GetAllMetrics() - { - return query.GetAllMetricsForNode(node); - } - - private MetricsSet GetMetricWithTimeout(string metricName) - { - var start = DateTime.UtcNow; - - while (true) - { - var mostRecent = GetMostRecent(metricName); - if (mostRecent != null) return mostRecent; - if (DateTime.UtcNow - start > timeSet.WaitForMetricTimeout()) - { - Assert.Fail($"Timeout: Unable to get metric '{metricName}'."); - throw new TimeoutException(); - } - - Time.Sleep(TimeSpan.FromSeconds(2)); - } - } - - private MetricsSet? GetMostRecent(string metricName) - { - var result = query.GetMostRecent(metricName, node); - if (result == null) return null; - return result.Sets.LastOrDefault(); - } - } - - public class MetricsUnavailable : IMetricsAccess - { - public void AssertThat(string metricName, IResolveConstraint constraint, string message = "") - { - Assert.Fail("Incorrect test setup: Metrics were not enabled for this group of Codex nodes. Add 'EnableMetrics()' after 'SetupCodexNodes()' to enable it."); - throw new InvalidOperationException(); - } - } -} diff --git a/DistTestCore/Metrics/MetricsAccessFactory.cs b/DistTestCore/Metrics/MetricsAccessFactory.cs deleted file mode 100644 index 18dae041..00000000 --- a/DistTestCore/Metrics/MetricsAccessFactory.cs +++ /dev/null @@ -1,35 +0,0 @@ -using KubernetesWorkflow; - -namespace DistTestCore.Metrics -{ - public interface IMetricsAccessFactory - { - IMetricsAccess CreateMetricsAccess(RunningContainer codexContainer); - } - - public class MetricsUnavailableAccessFactory : IMetricsAccessFactory - { - public IMetricsAccess CreateMetricsAccess(RunningContainer codexContainer) - { - return new MetricsUnavailable(); - } - } - - public class CodexNodeMetricsAccessFactory : IMetricsAccessFactory - { - private readonly TestLifecycle lifecycle; - private readonly RunningContainers prometheusContainer; - - public CodexNodeMetricsAccessFactory(TestLifecycle lifecycle, RunningContainers prometheusContainer) - { - this.lifecycle = lifecycle; - this.prometheusContainer = prometheusContainer; - } - - public IMetricsAccess CreateMetricsAccess(RunningContainer codexContainer) - { - var query = new MetricsQuery(lifecycle, prometheusContainer); - return new MetricsAccess(lifecycle.Log, lifecycle.TimeSet, query, codexContainer); - } - } -} diff --git a/DistTestCore/Metrics/MetricsMode.cs b/DistTestCore/Metrics/MetricsMode.cs deleted file mode 100644 index 60b4f5ef..00000000 --- a/DistTestCore/Metrics/MetricsMode.cs +++ /dev/null @@ -1,9 +0,0 @@ -namespace DistTestCore.Metrics -{ - public enum MetricsMode - { - None, - Record, - Dashboard - } -} diff --git a/DistTestCore/PrometheusStarter.cs b/DistTestCore/PrometheusStarter.cs deleted file mode 100644 index f19978c3..00000000 --- a/DistTestCore/PrometheusStarter.cs +++ /dev/null @@ -1,52 +0,0 @@ -using DistTestCore.Codex; -using DistTestCore.Metrics; -using KubernetesWorkflow; -using System.Text; - -namespace DistTestCore -{ - public class PrometheusStarter : BaseStarter - { - public PrometheusStarter(TestLifecycle lifecycle) - : base(lifecycle) - { - } - - public RunningContainers CollectMetricsFor(RunningContainers[] containers) - { - LogStart($"Starting metrics server for {containers.Describe()}"); - var startupConfig = new StartupConfig(); - startupConfig.Add(new PrometheusStartupConfig(GeneratePrometheusConfig(containers.Containers()))); - - var workflow = lifecycle.WorkflowCreator.CreateWorkflow(); - var runningContainers = workflow.Start(1, Location.Unspecified, new PrometheusContainerRecipe(), startupConfig); - if (runningContainers.Containers.Length != 1) throw new InvalidOperationException("Expected only 1 Prometheus container to be created."); - - return runningContainers; - } - - private string GeneratePrometheusConfig(RunningContainer[] nodes) - { - var config = ""; - config += "global:\n"; - config += " scrape_interval: 10s\n"; - config += " scrape_timeout: 10s\n"; - config += "\n"; - config += "scrape_configs:\n"; - config += " - job_name: services\n"; - config += " metrics_path: /metrics\n"; - config += " static_configs:\n"; - config += " - targets:\n"; - - foreach (var node in nodes) - { - var ip = node.Pod.PodInfo.Ip; - var port = node.Recipe.GetPortByTag(CodexContainerRecipe.MetricsPortTag).Number; - config += $" - '{ip}:{port}'\n"; - } - - var bytes = Encoding.ASCII.GetBytes(config); - return Convert.ToBase64String(bytes); - } - } -} diff --git a/DistTestCore/TestLifecycle.cs b/DistTestCore/TestLifecycle.cs deleted file mode 100644 index d3f3f0f5..00000000 --- a/DistTestCore/TestLifecycle.cs +++ /dev/null @@ -1,93 +0,0 @@ -using DistTestCore.Codex; -using DistTestCore.Logs; -using DistTestCore.Marketplace; -using DistTestCore.Metrics; -using KubernetesWorkflow; -using Logging; -using Utils; - -namespace DistTestCore -{ - public class TestLifecycle - { - private readonly DateTime testStart; - - public TestLifecycle(BaseLog log, Configuration configuration, ITimeSet timeSet, string testNamespace) - { - Log = log; - Configuration = configuration; - TimeSet = timeSet; - - WorkflowCreator = new WorkflowCreator(log, configuration.GetK8sConfiguration(timeSet), testNamespace); - - FileManager = new FileManager(Log, configuration); - CodexStarter = new CodexStarter(this); - PrometheusStarter = new PrometheusStarter(this); - GrafanaStarter = new GrafanaStarter(this); - GethStarter = new GethStarter(this); - testStart = DateTime.UtcNow; - CodexVersion = null; - - Log.WriteLogTag(); - } - - public BaseLog Log { get; } - public Configuration Configuration { get; } - public ITimeSet TimeSet { get; } - public WorkflowCreator WorkflowCreator { get; } - public FileManager FileManager { get; } - public CodexStarter CodexStarter { get; } - public PrometheusStarter PrometheusStarter { get; } - public GrafanaStarter GrafanaStarter { get; } - public GethStarter GethStarter { get; } - public CodexDebugVersionResponse? CodexVersion { get; private set; } - - public void DeleteAllResources() - { - CodexStarter.DeleteAllResources(); - FileManager.DeleteAllTestFiles(); - } - - public IDownloadedLog DownloadLog(RunningContainer container, int? tailLines = null) - { - var subFile = Log.CreateSubfile(); - var description = container.Name; - var handler = new LogDownloadHandler(container, description, subFile); - - Log.Log($"Downloading logs for {description} to file '{subFile.FullFilename}'"); - CodexStarter.DownloadLog(container, handler, tailLines); - - return new DownloadedLog(subFile, description); - } - - public string GetTestDuration() - { - var testDuration = DateTime.UtcNow - testStart; - return Time.FormatDuration(testDuration); - } - - public void SetCodexVersion(CodexDebugVersionResponse version) - { - if (CodexVersion == null) CodexVersion = version; - } - - public ApplicationIds GetApplicationIds() - { - return new ApplicationIds( - codexId: GetCodexId(), - gethId: new GethContainerRecipe().Image, - prometheusId: new PrometheusContainerRecipe().Image, - codexContractsId: new CodexContractsContainerRecipe().Image, - grafanaId: new GrafanaContainerRecipe().Image - ); - } - - private string GetCodexId() - { - var v = CodexVersion; - if (v == null) return new CodexContainerRecipe().Image; - if (v.version != "untagged build") return v.version; - return v.revision; - } - } -} diff --git a/DistTestCore/Timing.cs b/DistTestCore/Timing.cs deleted file mode 100644 index 38df6d84..00000000 --- a/DistTestCore/Timing.cs +++ /dev/null @@ -1,85 +0,0 @@ -using NUnit.Framework; - -namespace DistTestCore -{ - [AttributeUsage(AttributeTargets.Method, AllowMultiple = false)] - public class UseLongTimeoutsAttribute : PropertyAttribute - { - } - - public interface ITimeSet - { - TimeSpan HttpCallTimeout(); - TimeSpan HttpCallRetryTime(); - TimeSpan HttpCallRetryDelay(); - TimeSpan WaitForK8sServiceDelay(); - TimeSpan K8sOperationTimeout(); - TimeSpan WaitForMetricTimeout(); - } - - public class DefaultTimeSet : ITimeSet - { - public TimeSpan HttpCallTimeout() - { - return TimeSpan.FromMinutes(5); - } - - public TimeSpan HttpCallRetryTime() - { - return TimeSpan.FromMinutes(1); - } - - public TimeSpan HttpCallRetryDelay() - { - return TimeSpan.FromSeconds(1); - } - - public TimeSpan WaitForK8sServiceDelay() - { - return TimeSpan.FromSeconds(10); - } - - public TimeSpan K8sOperationTimeout() - { - return TimeSpan.FromMinutes(30); - } - - public TimeSpan WaitForMetricTimeout() - { - return TimeSpan.FromSeconds(30); - } - } - - public class LongTimeSet : ITimeSet - { - public TimeSpan HttpCallTimeout() - { - return TimeSpan.FromHours(2); - } - - public TimeSpan HttpCallRetryTime() - { - return TimeSpan.FromHours(5); - } - - public TimeSpan HttpCallRetryDelay() - { - return TimeSpan.FromSeconds(2); - } - - public TimeSpan WaitForK8sServiceDelay() - { - return TimeSpan.FromSeconds(10); - } - - public TimeSpan K8sOperationTimeout() - { - return TimeSpan.FromMinutes(15); - } - - public TimeSpan WaitForMetricTimeout() - { - return TimeSpan.FromMinutes(5); - } - } -} diff --git a/ArgsUniform/ArgsUniform.cs b/Framework/ArgsUniform/ArgsUniform.cs similarity index 100% rename from ArgsUniform/ArgsUniform.cs rename to Framework/ArgsUniform/ArgsUniform.cs diff --git a/ArgsUniform/ArgsUniform.csproj b/Framework/ArgsUniform/ArgsUniform.csproj similarity index 100% rename from ArgsUniform/ArgsUniform.csproj rename to Framework/ArgsUniform/ArgsUniform.csproj diff --git a/ArgsUniform/ExampleUser.cs b/Framework/ArgsUniform/ExampleUser.cs similarity index 100% rename from ArgsUniform/ExampleUser.cs rename to Framework/ArgsUniform/ExampleUser.cs diff --git a/ArgsUniform/UniformAttribute.cs b/Framework/ArgsUniform/UniformAttribute.cs similarity index 100% rename from ArgsUniform/UniformAttribute.cs rename to Framework/ArgsUniform/UniformAttribute.cs diff --git a/CodexNetDeployer/CodexNetDeployer.csproj b/Framework/Core/Core.csproj similarity index 57% rename from CodexNetDeployer/CodexNetDeployer.csproj rename to Framework/Core/Core.csproj index 48bca3ad..58c79517 100644 --- a/CodexNetDeployer/CodexNetDeployer.csproj +++ b/Framework/Core/Core.csproj @@ -1,15 +1,14 @@ - Exe net7.0 enable enable - - + + diff --git a/Framework/Core/CoreInterface.cs b/Framework/Core/CoreInterface.cs new file mode 100644 index 00000000..83ea5932 --- /dev/null +++ b/Framework/Core/CoreInterface.cs @@ -0,0 +1,50 @@ +using KubernetesWorkflow; + +namespace Core +{ + public sealed class CoreInterface + { + private readonly EntryPoint entryPoint; + + internal CoreInterface(EntryPoint entryPoint) + { + this.entryPoint = entryPoint; + } + + public T GetPlugin() where T : IProjectPlugin + { + return entryPoint.GetPlugin(); + } + + public IDownloadedLog DownloadLog(IHasContainer containerSource, int? tailLines = null) + { + return DownloadLog(containerSource.Container, tailLines); + } + + public IDownloadedLog DownloadLog(RunningContainer container, int? tailLines = null) + { + var workflow = entryPoint.Tools.CreateWorkflow(); + var file = entryPoint.Tools.GetLog().CreateSubfile(); + entryPoint.Tools.GetLog().Log($"Downloading container log for '{container.Name}' to file '{file.FullFilename}'..."); + var logHandler = new LogDownloadHandler(container.Name, file); + workflow.DownloadContainerLog(container, logHandler, tailLines); + return logHandler.DownloadLog(); + } + + public string ExecuteContainerCommand(IHasContainer containerSource, string command, params string[] args) + { + return ExecuteContainerCommand(containerSource.Container, command, args); + } + + public string ExecuteContainerCommand(RunningContainer container, string command, params string[] args) + { + var workflow = entryPoint.Tools.CreateWorkflow(); + return workflow.ExecuteCommand(container, command, args); + } + } + + public interface IHasContainer + { + RunningContainer Container { get; } + } +} diff --git a/DistTestCore/Logs/DownloadedLog.cs b/Framework/Core/DownloadedLog.cs similarity index 70% rename from DistTestCore/Logs/DownloadedLog.cs rename to Framework/Core/DownloadedLog.cs index 606c4116..9242d9ba 100644 --- a/DistTestCore/Logs/DownloadedLog.cs +++ b/Framework/Core/DownloadedLog.cs @@ -1,27 +1,24 @@ using Logging; -using NUnit.Framework; -namespace DistTestCore.Logs +namespace Core { public interface IDownloadedLog { - void AssertLogContains(string expectedString); + bool DoesLogContain(string expectedString); string[] FindLinesThatContain(params string[] tags); void DeleteFile(); } - public class DownloadedLog : IDownloadedLog + internal class DownloadedLog : IDownloadedLog { private readonly LogFile logFile; - private readonly string owner; - public DownloadedLog(LogFile logFile, string owner) + internal DownloadedLog(LogFile logFile) { this.logFile = logFile; - this.owner = owner; } - public void AssertLogContains(string expectedString) + public bool DoesLogContain(string expectedString) { using var file = File.OpenRead(logFile.FullFilename); using var streamReader = new StreamReader(file); @@ -29,11 +26,12 @@ namespace DistTestCore.Logs var line = streamReader.ReadLine(); while (line != null) { - if (line.Contains(expectedString)) return; + if (line.Contains(expectedString)) return true; line = streamReader.ReadLine(); } - Assert.Fail($"{owner} Unable to find string '{expectedString}' in CodexNode log file {logFile.FullFilename}"); + //Assert.Fail($"{owner} Unable to find string '{expectedString}' in CodexNode log file {logFile.FullFilename}"); + return false; } public string[] FindLinesThatContain(params string[] tags) diff --git a/Framework/Core/EntryPoint.cs b/Framework/Core/EntryPoint.cs new file mode 100644 index 00000000..7977eb3f --- /dev/null +++ b/Framework/Core/EntryPoint.cs @@ -0,0 +1,52 @@ +using KubernetesWorkflow; +using Logging; + +namespace Core +{ + public class EntryPoint + { + private readonly IToolsFactory toolsFactory; + private readonly PluginManager manager = new PluginManager(); + + public EntryPoint(ILog log, Configuration configuration, string fileManagerRootFolder, ITimeSet timeSet) + { + toolsFactory = new ToolsFactory(log, configuration, fileManagerRootFolder, timeSet); + + Tools = toolsFactory.CreateTools(); + manager.InstantiatePlugins(PluginFinder.GetPluginTypes(), toolsFactory); + } + + public EntryPoint(ILog log, Configuration configuration, string fileManagerRootFolder) + : this(log, configuration, fileManagerRootFolder, new DefaultTimeSet()) + { + } + + public IPluginTools Tools { get; } + + public void Announce() + { + manager.AnnouncePlugins(); + } + + public Dictionary GetPluginMetadata() + { + return manager.GatherPluginMetadata().Get(); + } + + public CoreInterface CreateInterface() + { + return new CoreInterface(this); + } + + public void Decommission(bool deleteKubernetesResources, bool deleteTrackedFiles) + { + manager.DecommissionPlugins(deleteKubernetesResources, deleteTrackedFiles); + Tools.Decommission(deleteKubernetesResources, deleteTrackedFiles); + } + + internal T GetPlugin() where T : IProjectPlugin + { + return manager.GetPlugin(); + } + } +} diff --git a/DistTestCore/Http.cs b/Framework/Core/Http.cs similarity index 88% rename from DistTestCore/Http.cs rename to Framework/Core/Http.cs index 773f9a56..5bbd1324 100644 --- a/DistTestCore/Http.cs +++ b/Framework/Core/Http.cs @@ -5,23 +5,35 @@ using System.Net.Http.Headers; using System.Net.Http.Json; using Utils; -namespace DistTestCore +namespace Core { - public class Http + public interface IHttp { - private readonly BaseLog log; + string HttpGetString(string route); + T HttpGetJson(string route); + TResponse HttpPostJson(string route, TRequest body); + string HttpPostJson(string route, TRequest body); + string HttpPostString(string route, string body); + string HttpPostStream(string route, Stream stream); + Stream HttpGetStream(string route); + T TryJsonDeserialize(string json); + } + + internal class Http : IHttp + { + private readonly ILog log; private readonly ITimeSet timeSet; private readonly Address address; private readonly string baseUrl; private readonly Action onClientCreated; private readonly string? logAlias; - public Http(BaseLog log, ITimeSet timeSet, Address address, string baseUrl, string? logAlias = null) + internal Http(ILog log, ITimeSet timeSet, Address address, string baseUrl, string? logAlias = null) : this(log, timeSet, address, baseUrl, DoNothing, logAlias) { } - public Http(BaseLog log, ITimeSet timeSet, Address address, string baseUrl, Action onClientCreated, string? logAlias = null) + internal Http(ILog log, ITimeSet timeSet, Address address, string baseUrl, Action onClientCreated, string? logAlias = null) { this.log = log; this.timeSet = timeSet; diff --git a/DistTestCore/Logs/LogDownloadHandler.cs b/Framework/Core/LogDownloadHandler.cs similarity index 50% rename from DistTestCore/Logs/LogDownloadHandler.cs rename to Framework/Core/LogDownloadHandler.cs index 483e46b9..e1736ed1 100644 --- a/DistTestCore/Logs/LogDownloadHandler.cs +++ b/Framework/Core/LogDownloadHandler.cs @@ -1,25 +1,23 @@ using KubernetesWorkflow; using Logging; -namespace DistTestCore.Logs +namespace Core { - public class LogDownloadHandler : LogHandler, ILogHandler + internal class LogDownloadHandler : LogHandler, ILogHandler { - private readonly RunningContainer container; private readonly LogFile log; - public LogDownloadHandler(RunningContainer container, string description, LogFile log) + internal LogDownloadHandler(string description, LogFile log) { - this.container = container; this.log = log; log.Write($"{description} -->> {log.FullFilename}"); log.WriteRaw(description); } - public DownloadedLog DownloadLog() + internal IDownloadedLog DownloadLog() { - return new DownloadedLog(log, container.Name); + return new DownloadedLog(log); } protected override void ProcessLine(string line) diff --git a/Framework/Core/PluginFinder.cs b/Framework/Core/PluginFinder.cs new file mode 100644 index 00000000..69df0750 --- /dev/null +++ b/Framework/Core/PluginFinder.cs @@ -0,0 +1,46 @@ +using System.Reflection; + +namespace Core +{ + internal static class PluginFinder + { + private static Type[]? pluginTypes = null; + + internal static Type[] GetPluginTypes() + { + if (pluginTypes != null) return pluginTypes; + + // Reflection can be costly. Do this only once. + FindAndLoadPluginAssemblies(); + + var assemblies = AppDomain.CurrentDomain.GetAssemblies(); + pluginTypes = assemblies.SelectMany(a => a.GetTypes().Where(t => + typeof(IProjectPlugin).IsAssignableFrom(t) && + !t.IsAbstract) + ).ToArray(); + + return pluginTypes; + } + + private static void FindAndLoadPluginAssemblies() + { + var files = Directory.GetFiles("."); + foreach (var file in files) + { + var f = file.ToLowerInvariant(); + if (f.Contains("plugin") && f.EndsWith("dll")) + { + var name = Path.GetFileNameWithoutExtension(file); + try + { + Assembly.Load(name); + } + catch (Exception ex) + { + throw new Exception($"Failed to load plugin from file '{name}'.", ex); + } + } + } + } + } +} diff --git a/Framework/Core/PluginManager.cs b/Framework/Core/PluginManager.cs new file mode 100644 index 00000000..e2b2a5cc --- /dev/null +++ b/Framework/Core/PluginManager.cs @@ -0,0 +1,78 @@ +namespace Core +{ + internal class PluginManager + { + private readonly List pairs = new List(); + + internal void InstantiatePlugins(Type[] pluginTypes, IToolsFactory provider) + { + pairs.Clear(); + foreach (var pluginType in pluginTypes) + { + var tools = provider.CreateTools(); + var plugin = InstantiatePlugins(pluginType, tools); + + ApplyLogPrefix(plugin, tools); + } + } + + internal void AnnouncePlugins() + { + foreach (var pair in pairs) pair.Plugin.Announce(); + } + + internal PluginMetadata GatherPluginMetadata() + { + var metadata = new PluginMetadata(); + foreach (var pair in pairs) + { + if (pair.Plugin is IHasMetadata m) + { + m.AddMetadata(metadata); + } + } + return metadata; + } + + internal void DecommissionPlugins(bool deleteKubernetesResources, bool deleteTrackedFiles) + { + foreach (var pair in pairs) + { + pair.Plugin.Decommission(); + pair.Tools.Decommission(deleteKubernetesResources, deleteTrackedFiles); + } + } + + internal T GetPlugin() where T : IProjectPlugin + { + return (T)pairs.Single(p => p.Plugin.GetType() == typeof(T)).Plugin; + } + + private IProjectPlugin InstantiatePlugins(Type pluginType, PluginTools tools) + { + var plugin = (IProjectPlugin)Activator.CreateInstance(pluginType, args: tools)!; + pairs.Add(new PluginToolsPair(plugin, tools)); + return plugin; + } + + private void ApplyLogPrefix(IProjectPlugin plugin, PluginTools tools) + { + if (plugin is IHasLogPrefix hasLogPrefix) + { + tools.ApplyLogPrefix(hasLogPrefix.LogPrefix); + } + } + + private class PluginToolsPair + { + public PluginToolsPair(IProjectPlugin plugin, IPluginTools tools) + { + Plugin = plugin; + Tools = tools; + } + + public IProjectPlugin Plugin { get; } + public IPluginTools Tools { get; } + } + } +} diff --git a/Framework/Core/PluginMetadata.cs b/Framework/Core/PluginMetadata.cs new file mode 100644 index 00000000..7bbe1e17 --- /dev/null +++ b/Framework/Core/PluginMetadata.cs @@ -0,0 +1,27 @@ +namespace Core +{ + internal interface IPluginMetadata + { + Dictionary Get(); + } + + public interface IAddMetadata + { + void Add(string key, string value); + } + + internal class PluginMetadata : IPluginMetadata, IAddMetadata + { + private readonly Dictionary metadata = new Dictionary(); + + public void Add(string key, string value) + { + metadata.Add(key, value); + } + + public Dictionary Get() + { + return new Dictionary(metadata); + } + } +} diff --git a/Framework/Core/PluginTools.cs b/Framework/Core/PluginTools.cs new file mode 100644 index 00000000..5fd0ab86 --- /dev/null +++ b/Framework/Core/PluginTools.cs @@ -0,0 +1,85 @@ +using FileUtils; +using KubernetesWorkflow; +using Logging; +using Utils; + +namespace Core +{ + public interface IPluginTools : IWorkflowTool, ILogTool, IHttpFactoryTool, IFileTool + { + void Decommission(bool deleteKubernetesResources, bool deleteTrackedFiles); + } + + public interface IWorkflowTool + { + IStartupWorkflow CreateWorkflow(string? namespaceOverride = null); + } + + public interface ILogTool + { + ILog GetLog(); + } + + public interface IHttpFactoryTool + { + IHttp CreateHttp(Address address, string baseUrl, Action onClientCreated, string? logAlias = null); + IHttp CreateHttp(Address address, string baseUrl, string? logAlias = null); + } + + public interface IFileTool + { + IFileManager GetFileManager(); + } + + internal class PluginTools : IPluginTools + { + private readonly ITimeSet timeSet; + private readonly WorkflowCreator workflowCreator; + private readonly IFileManager fileManager; + private ILog log; + + internal PluginTools(ILog log, WorkflowCreator workflowCreator, string fileManagerRootFolder, ITimeSet timeSet) + { + this.log = log; + this.workflowCreator = workflowCreator; + this.timeSet = timeSet; + fileManager = new FileManager(log, fileManagerRootFolder); + } + + public void ApplyLogPrefix(string prefix) + { + log = new LogPrefixer(log, prefix); + } + + public IHttp CreateHttp(Address address, string baseUrl, Action onClientCreated, string? logAlias = null) + { + return new Http(log, timeSet, address, baseUrl, onClientCreated, logAlias); + } + + public IHttp CreateHttp(Address address, string baseUrl, string? logAlias = null) + { + return new Http(log, timeSet, address, baseUrl, logAlias); + } + + public IStartupWorkflow CreateWorkflow(string? namespaceOverride = null) + { + return workflowCreator.CreateWorkflow(namespaceOverride); + } + + public void Decommission(bool deleteKubernetesResources, bool deleteTrackedFiles) + { + if (deleteKubernetesResources) CreateWorkflow().DeleteNamespace(); + if (deleteTrackedFiles) fileManager.DeleteAllFiles(); + } + + public IFileManager GetFileManager() + { + return fileManager; + } + + public ILog GetLog() + { + return log; + } + } +} diff --git a/Framework/Core/ProjectPlugin.cs b/Framework/Core/ProjectPlugin.cs new file mode 100644 index 00000000..72f3c16a --- /dev/null +++ b/Framework/Core/ProjectPlugin.cs @@ -0,0 +1,34 @@ +using Utils; + +namespace Core +{ + public interface IProjectPlugin + { + void Announce(); + void Decommission(); + } + + public interface IHasLogPrefix + { + string LogPrefix { get; } + } + + public interface IHasMetadata + { + void AddMetadata(IAddMetadata metadata); + } + + public static class ProjectPlugin + { + /// + /// On some platforms and in some cases, not all required plugin assemblies are automatically loaded into the app domain. + /// In this case, the runtime needs a slight push to load it before the EntryPoint class is instantiated. + /// Used ProjectPlugin.Load<>() before you create an EntryPoint to ensure all plugins you want to use are loaded. + /// + public static void Load() where T : IProjectPlugin + { + var type = typeof(T); + FrameworkAssert.That(type != null, $"Unable to load plugin."); + } + } +} diff --git a/Framework/Core/SerializeGate.cs b/Framework/Core/SerializeGate.cs new file mode 100644 index 00000000..aab0de98 --- /dev/null +++ b/Framework/Core/SerializeGate.cs @@ -0,0 +1,20 @@ +using Newtonsoft.Json; + +namespace Core +{ + public static class SerializeGate + { + /// + /// SerializeGate was added to help ensure deployment objects are serializable + /// and remain viable after deserialization. + /// Tools can be built on top of the core interface that rely on deployment objects being serializable. + /// Insert the serialization gate after deployment but before wrapping to ensure any future changes + /// don't break this requirement. + /// + public static T Gate(T anything) + { + var json = JsonConvert.SerializeObject(anything); + return JsonConvert.DeserializeObject(json)!; + } + } +} diff --git a/Framework/Core/TimeSet.cs b/Framework/Core/TimeSet.cs new file mode 100644 index 00000000..ff7a19ee --- /dev/null +++ b/Framework/Core/TimeSet.cs @@ -0,0 +1,39 @@ +namespace Core +{ + public interface ITimeSet + { + TimeSpan HttpCallTimeout(); + TimeSpan HttpCallRetryTime(); + TimeSpan HttpCallRetryDelay(); + TimeSpan WaitForK8sServiceDelay(); + TimeSpan K8sOperationTimeout(); + } + + public class DefaultTimeSet : ITimeSet + { + public TimeSpan HttpCallTimeout() + { + return TimeSpan.FromMinutes(5); + } + + public TimeSpan HttpCallRetryTime() + { + return TimeSpan.FromMinutes(1); + } + + public TimeSpan HttpCallRetryDelay() + { + return TimeSpan.FromSeconds(1); + } + + public TimeSpan WaitForK8sServiceDelay() + { + return TimeSpan.FromSeconds(10); + } + + public TimeSpan K8sOperationTimeout() + { + return TimeSpan.FromMinutes(30); + } + } +} diff --git a/Framework/Core/ToolsFactory.cs b/Framework/Core/ToolsFactory.cs new file mode 100644 index 00000000..96752b6f --- /dev/null +++ b/Framework/Core/ToolsFactory.cs @@ -0,0 +1,31 @@ +using KubernetesWorkflow; +using Logging; + +namespace Core +{ + internal interface IToolsFactory + { + PluginTools CreateTools(); + } + + internal class ToolsFactory : IToolsFactory + { + private readonly ILog log; + private readonly WorkflowCreator workflowCreator; + private readonly string fileManagerRootFolder; + private readonly ITimeSet timeSet; + + public ToolsFactory(ILog log, Configuration configuration, string fileManagerRootFolder, ITimeSet timeSet) + { + this.log = log; + workflowCreator = new WorkflowCreator(log, configuration); + this.fileManagerRootFolder = fileManagerRootFolder; + this.timeSet = timeSet; + } + + public PluginTools CreateTools() + { + return new PluginTools(log, workflowCreator, fileManagerRootFolder, timeSet); + } + } +} diff --git a/Framework/FileUtils/FileManager.cs b/Framework/FileUtils/FileManager.cs new file mode 100644 index 00000000..9b25c562 --- /dev/null +++ b/Framework/FileUtils/FileManager.cs @@ -0,0 +1,157 @@ +using Logging; +using Utils; + +namespace FileUtils +{ + public interface IFileManager + { + TrackedFile CreateEmptyFile(string label = ""); + TrackedFile GenerateFile(ByteSize size, string label = ""); + void DeleteAllFiles(); + void ScopedFiles(Action action); + T ScopedFiles(Func action); + } + + public class FileManager : IFileManager + { + public const int ChunkSize = 1024 * 1024 * 100; + private static NumberSource folderNumberSource = new NumberSource(0); + private readonly Random random = new Random(); + private readonly ILog log; + private readonly string rootFolder; + private readonly string folder; + private readonly List> fileSetStack = new List>(); + + public FileManager(ILog log, string rootFolder) + { + folder = Path.Combine(rootFolder, folderNumberSource.GetNextNumber().ToString("D5")); + + this.log = log; + this.rootFolder = rootFolder; + } + + public TrackedFile CreateEmptyFile(string label = "") + { + var path = Path.Combine(folder, Guid.NewGuid().ToString() + ".bin"); + EnsureDirectory(); + + var result = new TrackedFile(log, path, label); + File.Create(result.Filename).Close(); + if (fileSetStack.Any()) fileSetStack.Last().Add(result); + return result; + } + + public TrackedFile GenerateFile(ByteSize size, string label) + { + var sw = Stopwatch.Begin(log); + var result = GenerateRandomFile(size, label); + sw.End($"Generated file '{result.Describe()}'."); + return result; + } + + public void DeleteAllFiles() + { + DeleteDirectory(); + } + + public void ScopedFiles(Action action) + { + PushFileSet(); + action(); + PopFileSet(); + } + + public T ScopedFiles(Func action) + { + PushFileSet(); + var result = action(); + PopFileSet(); + return result; + } + + private void PushFileSet() + { + fileSetStack.Add(new List()); + } + + private void PopFileSet() + { + if (!fileSetStack.Any()) return; + var pop = fileSetStack.Last(); + fileSetStack.Remove(pop); + + foreach (var file in pop) + { + File.Delete(file.Filename); + } + + // If the folder is now empty, delete it too. + if (!Directory.GetFiles(folder).Any()) DeleteDirectory(); + } + + private TrackedFile GenerateRandomFile(ByteSize size, string label) + { + var result = CreateEmptyFile(label); + CheckSpaceAvailable(result, size); + + GenerateFileBytes(result, size); + return result; + } + + private void CheckSpaceAvailable(TrackedFile testFile, ByteSize size) + { + var file = new FileInfo(testFile.Filename); + var drive = new DriveInfo(file.Directory!.Root.FullName); + + var spaceAvailable = drive.TotalFreeSpace; + + if (spaceAvailable < size.SizeInBytes) + { + var msg = $"Not enough disk space. " + + $"{Formatter.FormatByteSize(size.SizeInBytes)} required. " + + $"{Formatter.FormatByteSize(spaceAvailable)} available."; + + log.Log(msg); + throw new Exception(msg); + } + } + + private void GenerateFileBytes(TrackedFile result, ByteSize size) + { + long bytesLeft = size.SizeInBytes; + int chunkSize = ChunkSize; + while (bytesLeft > 0) + { + try + { + var length = Math.Min(bytesLeft, chunkSize); + AppendRandomBytesToFile(result, length); + bytesLeft -= length; + } + catch + { + chunkSize = chunkSize / 2; + if (chunkSize < 1024) throw; + } + } + } + + private void AppendRandomBytesToFile(TrackedFile result, long length) + { + var bytes = new byte[length]; + random.NextBytes(bytes); + using var stream = new FileStream(result.Filename, FileMode.Append); + stream.Write(bytes, 0, bytes.Length); + } + + private void EnsureDirectory() + { + Directory.CreateDirectory(folder); + } + + private void DeleteDirectory() + { + if (Directory.Exists(folder)) Directory.Delete(folder, true); + } + } +} diff --git a/Framework/FileUtils/FileUtils.csproj b/Framework/FileUtils/FileUtils.csproj new file mode 100644 index 00000000..10c714ca --- /dev/null +++ b/Framework/FileUtils/FileUtils.csproj @@ -0,0 +1,14 @@ + + + + net7.0 + enable + enable + + + + + + + + diff --git a/Framework/FileUtils/TrackedFile.cs b/Framework/FileUtils/TrackedFile.cs new file mode 100644 index 00000000..5f9b04a8 --- /dev/null +++ b/Framework/FileUtils/TrackedFile.cs @@ -0,0 +1,82 @@ +using Logging; +using Utils; + +namespace FileUtils +{ + public class TrackedFile + { + private readonly ILog log; + + public TrackedFile(ILog log, string filename, string label) + { + this.log = log; + Filename = filename; + Label = label; + } + + public string Filename { get; } + public string Label { get; } + + public void AssertIsEqual(TrackedFile? actual) + { + var sw = Stopwatch.Begin(log); + try + { + AssertEqual(actual); + } + finally + { + sw.End($"{nameof(TrackedFile)}.{nameof(AssertIsEqual)}"); + } + } + + public string Describe() + { + var sizePostfix = $" ({Formatter.FormatByteSize(GetFileSize())})"; + if (!string.IsNullOrEmpty(Label)) return Label + sizePostfix; + return $"'{Filename}'{sizePostfix}"; + } + + private void AssertEqual(TrackedFile? actual) + { + if (actual == null) FrameworkAssert.Fail("TestFile is null."); + if (actual == this || actual!.Filename == Filename) FrameworkAssert.Fail("TestFile is compared to itself."); + + FrameworkAssert.That(actual.GetFileSize() == GetFileSize(), "Files are not of equal length."); + + using var streamExpected = new FileStream(Filename, FileMode.Open, FileAccess.Read); + using var streamActual = new FileStream(actual.Filename, FileMode.Open, FileAccess.Read); + + var bytesExpected = new byte[FileManager.ChunkSize]; + var bytesActual = new byte[FileManager.ChunkSize]; + + var readExpected = 0; + var readActual = 0; + + while (true) + { + readExpected = streamExpected.Read(bytesExpected, 0, FileManager.ChunkSize); + readActual = streamActual.Read(bytesActual, 0, FileManager.ChunkSize); + + if (readExpected == 0 && readActual == 0) + { + log.Log($"OK: '{Describe()}' is equal to '{actual.Describe()}'."); + return; + } + + FrameworkAssert.That(readActual == readExpected, "Unable to read buffers of equal length."); + + for (var i = 0; i < readActual; i++) + { + if (bytesExpected[i] != bytesActual[i]) FrameworkAssert.Fail("File contents not equal."); + } + } + } + + private long GetFileSize() + { + var info = new FileInfo(Filename); + return info.Length; + } + } +} diff --git a/KubernetesWorkflow/ByteSizeExtensions.cs b/Framework/KubernetesWorkflow/ByteSizeExtensions.cs similarity index 100% rename from KubernetesWorkflow/ByteSizeExtensions.cs rename to Framework/KubernetesWorkflow/ByteSizeExtensions.cs diff --git a/KubernetesWorkflow/CommandRunner.cs b/Framework/KubernetesWorkflow/CommandRunner.cs similarity index 100% rename from KubernetesWorkflow/CommandRunner.cs rename to Framework/KubernetesWorkflow/CommandRunner.cs diff --git a/Framework/KubernetesWorkflow/Configuration.cs b/Framework/KubernetesWorkflow/Configuration.cs new file mode 100644 index 00000000..62146820 --- /dev/null +++ b/Framework/KubernetesWorkflow/Configuration.cs @@ -0,0 +1,25 @@ +using Newtonsoft.Json; + +namespace KubernetesWorkflow +{ + public class Configuration + { + public Configuration(string? kubeConfigFile, TimeSpan operationTimeout, TimeSpan retryDelay, string kubernetesNamespace) + { + KubeConfigFile = kubeConfigFile; + OperationTimeout = operationTimeout; + RetryDelay = retryDelay; + KubernetesNamespace = kubernetesNamespace; + } + + public string? KubeConfigFile { get; } + public TimeSpan OperationTimeout { get; } + public TimeSpan RetryDelay { get; } + public string KubernetesNamespace { get; } + public bool AllowNamespaceOverride { get; set; } = true; + public bool AddAppPodLabel { get; set; } = true; + + [JsonIgnore] + public IK8sHooks Hooks { get; set; } = new DoNothingK8sHooks(); + } +} diff --git a/Framework/KubernetesWorkflow/ContainerAdditionals.cs b/Framework/KubernetesWorkflow/ContainerAdditionals.cs new file mode 100644 index 00000000..015e55b9 --- /dev/null +++ b/Framework/KubernetesWorkflow/ContainerAdditionals.cs @@ -0,0 +1,53 @@ +using Newtonsoft.Json.Linq; + +namespace KubernetesWorkflow +{ + public class ContainerAdditionals + { + public ContainerAdditionals(Additional[] additionals) + { + Additionals = additionals; + } + + public static ContainerAdditionals CreateFromUserData(IEnumerable userData) + { + return new ContainerAdditionals(userData.Select(ConvertToAdditional).ToArray()); + } + + public Additional[] Additionals { get; } + + public T? Get() + { + var typeName = GetTypeName(typeof(T)); + var userData = Additionals.SingleOrDefault(a => a.Type == typeName); + if (userData == null) return default(T); + var jobject = (JObject)userData.UserData; + return jobject.ToObject(); + } + + private static Additional ConvertToAdditional(object userData) + { + var typeName = GetTypeName(userData.GetType()); + return new Additional(typeName, userData); + } + + private static string GetTypeName(Type type) + { + var typeName = type.FullName; + if (string.IsNullOrEmpty(typeName)) throw new Exception("Object type fullname is null or empty: " + type); + return typeName; + } + } + + public class Additional + { + public Additional(string type, object userData) + { + Type = type; + UserData = userData; + } + + public string Type { get; } + public object UserData { get; } + } +} diff --git a/KubernetesWorkflow/ContainerRecipe.cs b/Framework/KubernetesWorkflow/ContainerRecipe.cs similarity index 73% rename from KubernetesWorkflow/ContainerRecipe.cs rename to Framework/KubernetesWorkflow/ContainerRecipe.cs index 51ea2e17..215b2280 100644 --- a/KubernetesWorkflow/ContainerRecipe.cs +++ b/Framework/KubernetesWorkflow/ContainerRecipe.cs @@ -2,9 +2,10 @@ { public class ContainerRecipe { - public ContainerRecipe(int number, string image, ContainerResources resources, Port[] exposedPorts, Port[] internalPorts, EnvVar[] envVars, PodLabels podLabels, PodAnnotations podAnnotations, VolumeMount[] volumes, object[] additionals) + public ContainerRecipe(int number, string? nameOverride, string image, ContainerResources resources, Port[] exposedPorts, Port[] internalPorts, EnvVar[] envVars, PodLabels podLabels, PodAnnotations podAnnotations, VolumeMount[] volumes, ContainerAdditionals additionals) { Number = number; + NameOverride = nameOverride; Image = image; Resources = resources; ExposedPorts = exposedPorts; @@ -14,10 +15,20 @@ PodAnnotations = podAnnotations; Volumes = volumes; Additionals = additionals; + + if (NameOverride != null) + { + Name = $"{K8sNameUtils.Format(NameOverride)}-{Number}"; + } + else + { + Name = $"ctnr{Number}"; + } } - public string Name { get { return $"ctnr{Number}"; } } + public string Name { get; } public int Number { get; } + public string? NameOverride { get; } public ContainerResources Resources { get; } public string Image { get; } public Port[] ExposedPorts { get; } @@ -26,11 +37,11 @@ public PodLabels PodLabels { get; } public PodAnnotations PodAnnotations { get; } public VolumeMount[] Volumes { get; } - public object[] Additionals { get; } + public ContainerAdditionals Additionals { get; } - public Port GetPortByTag(string tag) + public Port? GetPortByTag(string tag) { - return ExposedPorts.Concat(InternalPorts).Single(p => p.Tag == tag); + return ExposedPorts.Concat(InternalPorts).SingleOrDefault(p => p.Tag == tag); } public override string ToString() diff --git a/KubernetesWorkflow/ContainerRecipeFactory.cs b/Framework/KubernetesWorkflow/ContainerRecipeFactory.cs similarity index 69% rename from KubernetesWorkflow/ContainerRecipeFactory.cs rename to Framework/KubernetesWorkflow/ContainerRecipeFactory.cs index f8c88ef0..6fb50a57 100644 --- a/KubernetesWorkflow/ContainerRecipeFactory.cs +++ b/Framework/KubernetesWorkflow/ContainerRecipeFactory.cs @@ -12,6 +12,7 @@ namespace KubernetesWorkflow private readonly List volumeMounts = new List(); private readonly List additionals = new List(); private RecipeComponentFactory factory = null!; + private ContainerResources resources = new ContainerResources(); public ContainerRecipe CreateRecipe(int index, int containerNumber, RecipeComponentFactory factory, StartupConfig config) { @@ -21,14 +22,14 @@ namespace KubernetesWorkflow Initialize(config); - var recipe = new ContainerRecipe(containerNumber, Image, Resources, + var recipe = new ContainerRecipe(containerNumber, config.NameOverride, Image, resources, exposedPorts.ToArray(), internalPorts.ToArray(), envVars.ToArray(), podLabels.Clone(), podAnnotations.Clone(), volumeMounts.ToArray(), - additionals.ToArray()); + ContainerAdditionals.CreateFromUserData(additionals)); exposedPorts.Clear(); internalPorts.Clear(); @@ -38,29 +39,25 @@ namespace KubernetesWorkflow volumeMounts.Clear(); additionals.Clear(); this.factory = null!; + resources = new ContainerResources(); return recipe; } public abstract string AppName { get; } public abstract string Image { get; } - public ContainerResources Resources { get; } = new ContainerResources(); protected int ContainerNumber { get; private set; } = 0; protected int Index { get; private set; } = 0; protected abstract void Initialize(StartupConfig config); protected Port AddExposedPort(string tag = "") { - var p = factory.CreatePort(tag); - exposedPorts.Add(p); - return p; + return AddExposedPort(factory.CreatePort(tag)); } protected Port AddExposedPort(int number, string tag = "") { - var p = factory.CreatePort(number, tag); - exposedPorts.Add(p); - return p; + return AddExposedPort(factory.CreatePort(number, tag)); } protected Port AddInternalPort(string tag = "") @@ -112,5 +109,36 @@ namespace KubernetesWorkflow { additionals.Add(userData); } + + protected void SetResourcesRequest(int milliCPUs, ByteSize memory) + { + SetResourcesRequest(new ContainerResourceSet(milliCPUs, memory)); + } + + protected void SetResourceLimits(int milliCPUs, ByteSize memory) + { + SetResourceLimits(new ContainerResourceSet(milliCPUs, memory)); + } + + protected void SetResourcesRequest(ContainerResourceSet requests) + { + resources.Requests = requests; + } + + protected void SetResourceLimits(ContainerResourceSet limits) + { + resources.Limits = limits; + } + + private Port AddExposedPort(Port port) + { + if (exposedPorts.Any()) + { + throw new NotImplementedException("Current implementation only support 1 exposed port per container recipe. " + + $"Methods for determining container addresses in {nameof(StartupWorkflow)} currently rely on this constraint."); + } + exposedPorts.Add(port); + return port; + } } } diff --git a/KubernetesWorkflow/ContainerResources.cs b/Framework/KubernetesWorkflow/ContainerResources.cs similarity index 100% rename from KubernetesWorkflow/ContainerResources.cs rename to Framework/KubernetesWorkflow/ContainerResources.cs diff --git a/KubernetesWorkflow/CrashWatcher.cs b/Framework/KubernetesWorkflow/CrashWatcher.cs similarity index 94% rename from KubernetesWorkflow/CrashWatcher.cs rename to Framework/KubernetesWorkflow/CrashWatcher.cs index 7f8bc71b..8a27da75 100644 --- a/KubernetesWorkflow/CrashWatcher.cs +++ b/Framework/KubernetesWorkflow/CrashWatcher.cs @@ -5,7 +5,7 @@ namespace KubernetesWorkflow { public class CrashWatcher { - private readonly BaseLog log; + private readonly ILog log; private readonly KubernetesClientConfiguration config; private readonly string k8sNamespace; private readonly RunningContainer container; @@ -14,7 +14,7 @@ namespace KubernetesWorkflow private Task? worker; private Exception? workerException; - public CrashWatcher(BaseLog log, KubernetesClientConfiguration config, string k8sNamespace, RunningContainer container) + public CrashWatcher(ILog log, KubernetesClientConfiguration config, string k8sNamespace, RunningContainer container) { this.log = log; this.config = config; diff --git a/KubernetesWorkflow/K8sClient.cs b/Framework/KubernetesWorkflow/K8sClient.cs similarity index 100% rename from KubernetesWorkflow/K8sClient.cs rename to Framework/KubernetesWorkflow/K8sClient.cs diff --git a/KubernetesWorkflow/K8sCluster.cs b/Framework/KubernetesWorkflow/K8sCluster.cs similarity index 100% rename from KubernetesWorkflow/K8sCluster.cs rename to Framework/KubernetesWorkflow/K8sCluster.cs diff --git a/KubernetesWorkflow/K8sController.cs b/Framework/KubernetesWorkflow/K8sController.cs similarity index 94% rename from KubernetesWorkflow/K8sController.cs rename to Framework/KubernetesWorkflow/K8sController.cs index c162bcf8..a3a69fdd 100644 --- a/KubernetesWorkflow/K8sController.cs +++ b/Framework/KubernetesWorkflow/K8sController.cs @@ -7,13 +7,13 @@ namespace KubernetesWorkflow { public class K8sController { - private readonly BaseLog log; + private readonly ILog log; private readonly K8sCluster cluster; private readonly KnownK8sPods knownPods; private readonly WorkflowNumberSource workflowNumberSource; private readonly K8sClient client; - public K8sController(BaseLog log, K8sCluster cluster, KnownK8sPods knownPods, WorkflowNumberSource workflowNumberSource, string testNamespace) + public K8sController(ILog log, K8sCluster cluster, KnownK8sPods knownPods, WorkflowNumberSource workflowNumberSource, string k8sNamespace) { this.log = log; this.cluster = cluster; @@ -21,7 +21,7 @@ namespace KubernetesWorkflow this.workflowNumberSource = workflowNumberSource; client = new K8sClient(cluster.GetK8sClientConfig()); - K8sTestNamespace = cluster.Configuration.K8sNamespacePrefix + testNamespace; + K8sNamespace = k8sNamespace; } public void Dispose() @@ -54,7 +54,7 @@ namespace KubernetesWorkflow public void DownloadPodLog(RunningPod pod, ContainerRecipe recipe, ILogHandler logHandler, int? tailLines) { log.Debug(); - using var stream = client.Run(c => c.ReadNamespacedPodLog(pod.PodInfo.Name, K8sTestNamespace, recipe.Name, tailLines: tailLines)); + using var stream = client.Run(c => c.ReadNamespacedPodLog(pod.PodInfo.Name, K8sNamespace, recipe.Name, tailLines: tailLines)); logHandler.Log(stream); } @@ -63,7 +63,7 @@ namespace KubernetesWorkflow var cmdAndArgs = $"{containerName}: {command} ({string.Join(",", args)})"; log.Debug(cmdAndArgs); - var runner = new CommandRunner(client, K8sTestNamespace, pod, containerName, command, args); + var runner = new CommandRunner(client, K8sNamespace, pod, containerName, command, args); runner.Run(); var result = runner.GetStdOut(); @@ -71,12 +71,12 @@ namespace KubernetesWorkflow return result; } - public void DeleteAllResources() + public void DeleteAllNamespacesStartingWith(string prefix) { log.Debug(); var all = client.Run(c => c.ListNamespace().Items); - var namespaces = all.Select(n => n.Name()).Where(n => n.StartsWith(cluster.Configuration.K8sNamespacePrefix)); + var namespaces = all.Select(n => n.Name()).Where(n => n.StartsWith(prefix)); foreach (var ns in namespaces) { @@ -88,12 +88,12 @@ namespace KubernetesWorkflow } } - public void DeleteTestNamespace() + public void DeleteNamespace() { log.Debug(); if (IsTestNamespaceOnline()) { - client.Run(c => c.DeleteNamespace(K8sTestNamespace, null, null, gracePeriodSeconds: 0)); + client.Run(c => c.DeleteNamespace(K8sNamespace, null, null, gracePeriodSeconds: 0)); } WaitUntilNamespaceDeleted(); } @@ -145,7 +145,7 @@ namespace KubernetesWorkflow #region Namespace management - private string K8sTestNamespace { get; } + private string K8sNamespace { get; } private void EnsureTestNamespace() { @@ -156,8 +156,8 @@ namespace KubernetesWorkflow ApiVersion = "v1", Metadata = new V1ObjectMeta { - Name = K8sTestNamespace, - Labels = new Dictionary { { "name", K8sTestNamespace } } + Name = K8sNamespace, + Labels = new Dictionary { { "name", K8sNamespace } } } }; client.Run(c => c.CreateNamespace(namespaceSpec)); @@ -168,7 +168,7 @@ namespace KubernetesWorkflow private bool IsTestNamespaceOnline() { - return IsNamespaceOnline(K8sTestNamespace); + return IsNamespaceOnline(K8sNamespace); } private bool IsNamespaceOnline(string name) @@ -185,7 +185,7 @@ namespace KubernetesWorkflow Metadata = new V1ObjectMeta { Name = "isolate-policy", - NamespaceProperty = K8sTestNamespace + NamespaceProperty = K8sNamespace }, Spec = new V1NetworkPolicySpec { @@ -314,7 +314,7 @@ namespace KubernetesWorkflow } }; - c.CreateNamespacedNetworkPolicy(body, K8sTestNamespace); + c.CreateNamespacedNetworkPolicy(body, K8sNamespace); }); } @@ -352,7 +352,7 @@ namespace KubernetesWorkflow } }; - client.Run(c => c.CreateNamespacedDeployment(deploymentSpec, K8sTestNamespace)); + client.Run(c => c.CreateNamespacedDeployment(deploymentSpec, K8sNamespace)); WaitUntilDeploymentOnline(deploymentSpec.Metadata.Name); return deploymentSpec.Metadata.Name; @@ -360,7 +360,7 @@ namespace KubernetesWorkflow private void DeleteDeployment(string deploymentName) { - client.Run(c => c.DeleteNamespacedDeployment(deploymentName, K8sTestNamespace)); + client.Run(c => c.DeleteNamespacedDeployment(deploymentName, K8sNamespace)); WaitUntilDeploymentOffline(deploymentName); } @@ -399,8 +399,8 @@ namespace KubernetesWorkflow { return new V1ObjectMeta { - Name = "deploy-" + workflowNumberSource.WorkflowNumber, - NamespaceProperty = K8sTestNamespace, + Name = string.Join('-',containerRecipes.Select(r => r.Name)), + NamespaceProperty = K8sNamespace, Labels = GetSelector(containerRecipes), Annotations = GetAnnotations(containerRecipes) }; @@ -495,7 +495,7 @@ namespace KubernetesWorkflow } } } - }, K8sTestNamespace)); + }, K8sNamespace)); return new V1Volume { @@ -571,7 +571,7 @@ namespace KubernetesWorkflow } }; - client.Run(c => c.CreateNamespacedService(serviceSpec, K8sTestNamespace)); + client.Run(c => c.CreateNamespacedService(serviceSpec, K8sNamespace)); ReadBackServiceAndMapPorts(serviceSpec, containerRecipes, result); @@ -581,7 +581,7 @@ namespace KubernetesWorkflow private void ReadBackServiceAndMapPorts(V1Service serviceSpec, ContainerRecipe[] containerRecipes, List result) { // For each container-recipe, we need to figure out which service-ports it was assigned by K8s. - var readback = client.Run(c => c.ReadNamespacedService(serviceSpec.Metadata.Name, K8sTestNamespace)); + var readback = client.Run(c => c.ReadNamespacedService(serviceSpec.Metadata.Name, K8sNamespace)); foreach (var r in containerRecipes) { if (r.ExposedPorts.Any()) @@ -610,7 +610,7 @@ namespace KubernetesWorkflow private void DeleteService(string serviceName) { - client.Run(c => c.DeleteNamespacedService(serviceName, K8sTestNamespace)); + client.Run(c => c.DeleteNamespacedService(serviceName, K8sNamespace)); } private V1ObjectMeta CreateServiceMetadata() @@ -618,7 +618,7 @@ namespace KubernetesWorkflow return new V1ObjectMeta { Name = "service-" + workflowNumberSource.WorkflowNumber, - NamespaceProperty = K8sTestNamespace + NamespaceProperty = K8sNamespace }; } @@ -672,7 +672,7 @@ namespace KubernetesWorkflow { WaitUntil(() => { - var deployment = client.Run(c => c.ReadNamespacedDeployment(deploymentName, K8sTestNamespace)); + var deployment = client.Run(c => c.ReadNamespacedDeployment(deploymentName, K8sNamespace)); return deployment?.Status.AvailableReplicas != null && deployment.Status.AvailableReplicas > 0; }); } @@ -681,7 +681,7 @@ namespace KubernetesWorkflow { WaitUntil(() => { - var deployments = client.Run(c => c.ListNamespacedDeployment(K8sTestNamespace)); + var deployments = client.Run(c => c.ListNamespacedDeployment(K8sNamespace)); var deployment = deployments.Items.SingleOrDefault(d => d.Metadata.Name == deploymentName); return deployment == null || deployment.Status.AvailableReplicas == 0; }); @@ -691,7 +691,7 @@ namespace KubernetesWorkflow { WaitUntil(() => { - var pods = client.Run(c => c.ListNamespacedPod(K8sTestNamespace)).Items; + var pods = client.Run(c => c.ListNamespacedPod(K8sNamespace)).Items; var pod = pods.SingleOrDefault(p => p.Metadata.Name == podName); return pod == null; }); @@ -714,12 +714,12 @@ namespace KubernetesWorkflow public CrashWatcher CreateCrashWatcher(RunningContainer container) { - return new CrashWatcher(log, cluster.GetK8sClientConfig(), K8sTestNamespace, container); + return new CrashWatcher(log, cluster.GetK8sClientConfig(), K8sNamespace, container); } private PodInfo FetchNewPod() { - var pods = client.Run(c => c.ListNamespacedPod(K8sTestNamespace)).Items; + var pods = client.Run(c => c.ListNamespacedPod(K8sNamespace)).Items; var newPods = pods.Where(p => !knownPods.Contains(p.Name())).ToArray(); if (newPods.Length != 1) throw new InvalidOperationException("Expected only 1 pod to be created. Test infra failure."); diff --git a/Framework/KubernetesWorkflow/K8sHooks.cs b/Framework/KubernetesWorkflow/K8sHooks.cs new file mode 100644 index 00000000..91c25a34 --- /dev/null +++ b/Framework/KubernetesWorkflow/K8sHooks.cs @@ -0,0 +1,24 @@ +namespace KubernetesWorkflow +{ + public interface IK8sHooks + { + void OnContainersStarted(RunningContainers runningContainers); + void OnContainersStopped(RunningContainers runningContainers); + void OnContainerRecipeCreated(ContainerRecipe recipe); + } + + public class DoNothingK8sHooks : IK8sHooks + { + public void OnContainersStarted(RunningContainers runningContainers) + { + } + + public void OnContainersStopped(RunningContainers runningContainers) + { + } + + public void OnContainerRecipeCreated(ContainerRecipe recipe) + { + } + } +} diff --git a/Framework/KubernetesWorkflow/K8sNameUtils.cs b/Framework/KubernetesWorkflow/K8sNameUtils.cs new file mode 100644 index 00000000..c888870b --- /dev/null +++ b/Framework/KubernetesWorkflow/K8sNameUtils.cs @@ -0,0 +1,19 @@ +namespace KubernetesWorkflow +{ + public static class K8sNameUtils + { + public static string Format(string s) + { + var result = s.ToLowerInvariant() + .Replace(" ", "-") + .Replace(":", "-") + .Replace("/", "-") + .Replace("\\", "-") + .Replace("[", "-") + .Replace("]", "-") + .Replace(",", "-"); + + return result.Trim('-'); + } + } +} diff --git a/KubernetesWorkflow/KnownK8sPods.cs b/Framework/KubernetesWorkflow/KnownK8sPods.cs similarity index 100% rename from KubernetesWorkflow/KnownK8sPods.cs rename to Framework/KubernetesWorkflow/KnownK8sPods.cs diff --git a/KubernetesWorkflow/KubernetesWorkflow.csproj b/Framework/KubernetesWorkflow/KubernetesWorkflow.csproj similarity index 88% rename from KubernetesWorkflow/KubernetesWorkflow.csproj rename to Framework/KubernetesWorkflow/KubernetesWorkflow.csproj index cf95d1dc..c9201c37 100644 --- a/KubernetesWorkflow/KubernetesWorkflow.csproj +++ b/Framework/KubernetesWorkflow/KubernetesWorkflow.csproj @@ -9,6 +9,7 @@ + diff --git a/KubernetesWorkflow/Location.cs b/Framework/KubernetesWorkflow/Location.cs similarity index 100% rename from KubernetesWorkflow/Location.cs rename to Framework/KubernetesWorkflow/Location.cs diff --git a/KubernetesWorkflow/PodAnnotations.cs b/Framework/KubernetesWorkflow/PodAnnotations.cs similarity index 100% rename from KubernetesWorkflow/PodAnnotations.cs rename to Framework/KubernetesWorkflow/PodAnnotations.cs diff --git a/KubernetesWorkflow/PodLabels.cs b/Framework/KubernetesWorkflow/PodLabels.cs similarity index 60% rename from KubernetesWorkflow/PodLabels.cs rename to Framework/KubernetesWorkflow/PodLabels.cs index 78aa518d..d8b73331 100644 --- a/KubernetesWorkflow/PodLabels.cs +++ b/Framework/KubernetesWorkflow/PodLabels.cs @@ -6,7 +6,7 @@ public void Add(string key, string value) { - labels.Add(key, Format(value)); + labels.Add(key, K8sNameUtils.Format(value)); } public PodLabels Clone() @@ -21,19 +21,6 @@ labels.Clear(); } - private static string Format(string s) - { - var result = s.ToLowerInvariant() - .Replace(":", "-") - .Replace("/", "-") - .Replace("\\", "-") - .Replace("[", "-") - .Replace("]", "-") - .Replace(",", "-"); - - return result.Trim('-'); - } - internal Dictionary GetLabels() { return labels; diff --git a/KubernetesWorkflow/RecipeComponentFactory.cs b/Framework/KubernetesWorkflow/RecipeComponentFactory.cs similarity index 100% rename from KubernetesWorkflow/RecipeComponentFactory.cs rename to Framework/KubernetesWorkflow/RecipeComponentFactory.cs diff --git a/Framework/KubernetesWorkflow/RunnerLocationUtils.cs b/Framework/KubernetesWorkflow/RunnerLocationUtils.cs new file mode 100644 index 00000000..05d88c49 --- /dev/null +++ b/Framework/KubernetesWorkflow/RunnerLocationUtils.cs @@ -0,0 +1,55 @@ +using System.Net.NetworkInformation; +using Utils; + +namespace KubernetesWorkflow +{ + internal enum RunnerLocation + { + ExternalToCluster, + InternalToCluster, + } + + internal static class RunnerLocationUtils + { + private static RunnerLocation? knownLocation = null; + + internal static RunnerLocation DetermineRunnerLocation(RunningContainer container) + { + if (knownLocation != null) return knownLocation.Value; + + if (PingHost(container.Pod.PodInfo.Ip)) + { + knownLocation = RunnerLocation.InternalToCluster; + } + if (PingHost(Format(container.ClusterExternalAddress))) + { + knownLocation = RunnerLocation.ExternalToCluster; + } + + if (knownLocation == null) throw new Exception("Unable to determine location relative to kubernetes cluster."); + return knownLocation.Value; + } + + private static string Format(Address host) + { + return host.Host + .Replace("http://", "") + .Replace("https://", ""); + } + + private static bool PingHost(string host) + { + try + { + using var pinger = new Ping(); + PingReply reply = pinger.Send(host); + return reply.Status == IPStatus.Success; + } + catch (PingException) + { + } + + return false; + } + } +} diff --git a/KubernetesWorkflow/RunningContainers.cs b/Framework/KubernetesWorkflow/RunningContainers.cs similarity index 84% rename from KubernetesWorkflow/RunningContainers.cs rename to Framework/KubernetesWorkflow/RunningContainers.cs index 3f43a378..cfcae4da 100644 --- a/KubernetesWorkflow/RunningContainers.cs +++ b/Framework/KubernetesWorkflow/RunningContainers.cs @@ -42,7 +42,17 @@ namespace KubernetesWorkflow public Address ClusterInternalAddress { get; } [JsonIgnore] - public CrashWatcher? CrashWatcher { get; set; } + public Address Address + { + get + { + if (RunnerLocationUtils.DetermineRunnerLocation(this) == RunnerLocation.InternalToCluster) + { + return ClusterInternalAddress; + } + return ClusterExternalAddress; + } + } } public static class RunningContainersExtensions diff --git a/KubernetesWorkflow/RunningPod.cs b/Framework/KubernetesWorkflow/RunningPod.cs similarity index 100% rename from KubernetesWorkflow/RunningPod.cs rename to Framework/KubernetesWorkflow/RunningPod.cs diff --git a/KubernetesWorkflow/StartupConfig.cs b/Framework/KubernetesWorkflow/StartupConfig.cs similarity index 100% rename from KubernetesWorkflow/StartupConfig.cs rename to Framework/KubernetesWorkflow/StartupConfig.cs diff --git a/KubernetesWorkflow/StartupWorkflow.cs b/Framework/KubernetesWorkflow/StartupWorkflow.cs similarity index 70% rename from KubernetesWorkflow/StartupWorkflow.cs rename to Framework/KubernetesWorkflow/StartupWorkflow.cs index 7595be21..58f5273f 100644 --- a/KubernetesWorkflow/StartupWorkflow.cs +++ b/Framework/KubernetesWorkflow/StartupWorkflow.cs @@ -3,22 +3,33 @@ using Utils; namespace KubernetesWorkflow { - public class StartupWorkflow + public interface IStartupWorkflow { - private readonly BaseLog log; + RunningContainers Start(int numberOfContainers, Location location, ContainerRecipeFactory recipeFactory, StartupConfig startupConfig); + CrashWatcher CreateCrashWatcher(RunningContainer container); + void Stop(RunningContainers runningContainers); + void DownloadContainerLog(RunningContainer container, ILogHandler logHandler, int? tailLines = null); + string ExecuteCommand(RunningContainer container, string command, params string[] args); + void DeleteNamespace(); + void DeleteNamespacesStartingWith(string namespacePrefix); + } + + public class StartupWorkflow : IStartupWorkflow + { + private readonly ILog log; private readonly WorkflowNumberSource numberSource; private readonly K8sCluster cluster; private readonly KnownK8sPods knownK8SPods; - private readonly string testNamespace; + private readonly string k8sNamespace; private readonly RecipeComponentFactory componentFactory = new RecipeComponentFactory(); - internal StartupWorkflow(BaseLog log, WorkflowNumberSource numberSource, K8sCluster cluster, KnownK8sPods knownK8SPods, string testNamespace) + internal StartupWorkflow(ILog log, WorkflowNumberSource numberSource, K8sCluster cluster, KnownK8sPods knownK8SPods, string k8sNamespace) { this.log = log; this.numberSource = numberSource; this.cluster = cluster; this.knownK8SPods = knownK8SPods; - this.testNamespace = testNamespace; + this.k8sNamespace = k8sNamespace; } public RunningContainers Start(int numberOfContainers, Location location, ContainerRecipeFactory recipeFactory, StartupConfig startupConfig) @@ -26,16 +37,18 @@ namespace KubernetesWorkflow return K8s(controller => { var recipes = CreateRecipes(numberOfContainers, recipeFactory, startupConfig); - var runningPod = controller.BringOnline(recipes, location); + var containers = CreateContainers(runningPod, recipes, startupConfig); - return new RunningContainers(startupConfig, runningPod, CreateContainers(runningPod, recipes, startupConfig)); + var rc = new RunningContainers(startupConfig, runningPod, containers); + cluster.Configuration.Hooks.OnContainersStarted(rc); + return rc; }); } public CrashWatcher CreateCrashWatcher(RunningContainer container) { - return K8s(controller => controller.CreateCrashWatcher(container)); + return K8s(c => c.CreateCrashWatcher(container)); } public void Stop(RunningContainers runningContainers) @@ -43,10 +56,11 @@ namespace KubernetesWorkflow K8s(controller => { controller.Stop(runningContainers.RunningPod); + cluster.Configuration.Hooks.OnContainersStopped(runningContainers); }); } - public void DownloadContainerLog(RunningContainer container, ILogHandler logHandler, int? tailLines) + public void DownloadContainerLog(RunningContainer container, ILogHandler logHandler, int? tailLines = null) { K8s(controller => { @@ -62,19 +76,19 @@ namespace KubernetesWorkflow }); } - public void DeleteAllResources() + public void DeleteNamespace() { K8s(controller => { - controller.DeleteAllResources(); + controller.DeleteNamespace(); }); } - public void DeleteTestResources() + public void DeleteNamespacesStartingWith(string namespacePrefix) { K8s(controller => { - controller.DeleteTestNamespace(); + controller.DeleteAllNamespacesStartingWith(namespacePrefix); }); } @@ -118,11 +132,10 @@ namespace KubernetesWorkflow private Address GetContainerInternalAddress(ContainerRecipe recipe) { var serviceName = "service-" + numberSource.WorkflowNumber; - var namespaceName = cluster.Configuration.K8sNamespacePrefix + testNamespace; var port = GetInternalPort(recipe); return new Address( - $"http://{serviceName}.{namespaceName}.svc.cluster.local", + $"http://{serviceName}.{k8sNamespace}.svc.cluster.local", port); } @@ -144,7 +157,10 @@ namespace KubernetesWorkflow var result = new List(); for (var i = 0; i < numberOfContainers; i++) { - result.Add(recipeFactory.CreateRecipe(i, numberSource.GetContainerNumber(), componentFactory, startupConfig)); + var recipe = recipeFactory.CreateRecipe(i, numberSource.GetContainerNumber(), componentFactory, startupConfig); + if (cluster.Configuration.AddAppPodLabel) recipe.PodLabels.Add("app", recipeFactory.AppName); + cluster.Configuration.Hooks.OnContainerRecipeCreated(recipe); + result.Add(recipe); } return result.ToArray(); @@ -152,14 +168,14 @@ namespace KubernetesWorkflow private void K8s(Action action) { - var controller = new K8sController(log, cluster, knownK8SPods, numberSource, testNamespace); + var controller = new K8sController(log, cluster, knownK8SPods, numberSource, k8sNamespace); action(controller); controller.Dispose(); } private T K8s(Func action) { - var controller = new K8sController(log, cluster, knownK8SPods, numberSource, testNamespace); + var controller = new K8sController(log, cluster, knownK8SPods, numberSource, k8sNamespace); var result = action(controller); controller.Dispose(); return result; diff --git a/Framework/KubernetesWorkflow/WorkflowCreator.cs b/Framework/KubernetesWorkflow/WorkflowCreator.cs new file mode 100644 index 00000000..213db929 --- /dev/null +++ b/Framework/KubernetesWorkflow/WorkflowCreator.cs @@ -0,0 +1,42 @@ +using Logging; +using Utils; + +namespace KubernetesWorkflow +{ + public class WorkflowCreator + { + private readonly NumberSource numberSource = new NumberSource(0); + private readonly NumberSource containerNumberSource = new NumberSource(0); + private readonly KnownK8sPods knownPods = new KnownK8sPods(); + private readonly K8sCluster cluster; + private readonly ILog log; + private readonly Configuration configuration; + private readonly string k8sNamespace; + + public WorkflowCreator(ILog log, Configuration configuration) + { + this.log = log; + this.configuration = configuration; + cluster = new K8sCluster(configuration); + k8sNamespace = configuration.KubernetesNamespace.ToLowerInvariant(); + } + + public IStartupWorkflow CreateWorkflow(string? namespaceOverride = null) + { + var workflowNumberSource = new WorkflowNumberSource(numberSource.GetNextNumber(), + containerNumberSource); + + return new StartupWorkflow(log, workflowNumberSource, cluster, knownPods, GetNamespace(namespaceOverride)); + } + + private string GetNamespace(string? namespaceOverride) + { + if (namespaceOverride != null) + { + if (!configuration.AllowNamespaceOverride) throw new Exception("Namespace override is not allowed."); + return namespaceOverride; + } + return k8sNamespace; + } + } +} diff --git a/KubernetesWorkflow/WorkflowNumberSource.cs b/Framework/KubernetesWorkflow/WorkflowNumberSource.cs similarity index 100% rename from KubernetesWorkflow/WorkflowNumberSource.cs rename to Framework/KubernetesWorkflow/WorkflowNumberSource.cs diff --git a/Logging/ApplicationIds.cs b/Framework/Logging/ApplicationIds.cs similarity index 100% rename from Logging/ApplicationIds.cs rename to Framework/Logging/ApplicationIds.cs diff --git a/Logging/BaseLog.cs b/Framework/Logging/BaseLog.cs similarity index 82% rename from Logging/BaseLog.cs rename to Framework/Logging/BaseLog.cs index d11ecc17..b80b0075 100644 --- a/Logging/BaseLog.cs +++ b/Framework/Logging/BaseLog.cs @@ -2,12 +2,20 @@ namespace Logging { - public abstract class BaseLog + public interface ILog + { + void Log(string message); + void Debug(string message = "", int skipFrames = 0); + void Error(string message); + void AddStringReplace(string from, string to); + LogFile CreateSubfile(string ext = "log"); + } + + public abstract class BaseLog : ILog { private readonly NumberSource subfileNumberSource = new NumberSource(0); private readonly bool debug; private readonly List replacements = new List(); - private bool hasFailed; private LogFile? logFile; protected BaseLog(bool debug) @@ -26,10 +34,6 @@ namespace Logging } } - public virtual void EndTest() - { - } - public virtual void Log(string message) { LogFile.Write(ApplyReplacements(message)); @@ -50,13 +54,6 @@ namespace Logging Log($"[ERROR] {message}"); } - public virtual void MarkAsFailed() - { - if (hasFailed) return; - hasFailed = true; - LogFile.ConcatToFilename("_FAILED"); - } - public virtual void AddStringReplace(string from, string to) { if (string.IsNullOrWhiteSpace(from)) return; @@ -74,14 +71,6 @@ namespace Logging return new LogFile($"{GetFullName()}_{GetSubfileNumber()}", ext); } - public void WriteLogTag() - { - var runId = NameUtils.GetRunId(); - var category = NameUtils.GetCategoryName(); - var name = NameUtils.GetTestMethodName(); - LogFile.WriteRaw($"{runId} {category} {name}"); - } - private string ApplyReplacements(string str) { foreach (var replacement in replacements) diff --git a/Framework/Logging/ConsoleLog.cs b/Framework/Logging/ConsoleLog.cs new file mode 100644 index 00000000..61e61155 --- /dev/null +++ b/Framework/Logging/ConsoleLog.cs @@ -0,0 +1,19 @@ +namespace Logging +{ + public class ConsoleLog : BaseLog + { + public ConsoleLog() : base(false) + { + } + + protected override string GetFullName() + { + return "CONSOLE"; + } + + public override void Log(string message) + { + Console.WriteLine(message); + } + } +} diff --git a/Logging/LogConfig.cs b/Framework/Logging/LogConfig.cs similarity index 100% rename from Logging/LogConfig.cs rename to Framework/Logging/LogConfig.cs diff --git a/Logging/LogFile.cs b/Framework/Logging/LogFile.cs similarity index 100% rename from Logging/LogFile.cs rename to Framework/Logging/LogFile.cs diff --git a/Framework/Logging/LogPrefixer.cs b/Framework/Logging/LogPrefixer.cs new file mode 100644 index 00000000..a3d2f9fb --- /dev/null +++ b/Framework/Logging/LogPrefixer.cs @@ -0,0 +1,39 @@ +namespace Logging +{ + public class LogPrefixer : ILog + { + private readonly ILog backingLog; + private readonly string prefix; + + public LogPrefixer(ILog backingLog, string prefix) + { + this.backingLog = backingLog; + this.prefix = prefix; + } + + public LogFile CreateSubfile(string ext = "log") + { + return backingLog.CreateSubfile(ext); + } + + public void Debug(string message = "", int skipFrames = 0) + { + backingLog.Debug(prefix + message, skipFrames); + } + + public void Error(string message) + { + backingLog.Error(prefix + message); + } + + public void Log(string message) + { + backingLog.Log(prefix + message); + } + + public void AddStringReplace(string from, string to) + { + backingLog.AddStringReplace(from, to); + } + } +} diff --git a/Framework/Logging/LogSplitter.cs b/Framework/Logging/LogSplitter.cs new file mode 100644 index 00000000..6d24d5b3 --- /dev/null +++ b/Framework/Logging/LogSplitter.cs @@ -0,0 +1,42 @@ +namespace Logging +{ + public class LogSplitter : ILog + { + private readonly ILog[] targetLogs; + + public LogSplitter(params ILog[] targetLogs) + { + this.targetLogs = targetLogs; + } + + public void AddStringReplace(string from, string to) + { + OnAll(l => l.AddStringReplace(from, to)); + } + + public LogFile CreateSubfile(string ext = "log") + { + return targetLogs.First().CreateSubfile(ext); + } + + public void Debug(string message = "", int skipFrames = 0) + { + OnAll(l => l.Debug(message, skipFrames + 2)); + } + + public void Error(string message) + { + OnAll(l => l.Error(message)); + } + + public void Log(string message) + { + OnAll(l => l.Log(message)); + } + + private void OnAll(Action action) + { + foreach (var t in targetLogs) action(t); + } + } +} diff --git a/Logging/Logging.csproj b/Framework/Logging/Logging.csproj similarity index 59% rename from Logging/Logging.csproj rename to Framework/Logging/Logging.csproj index defbdccf..0757f272 100644 --- a/Logging/Logging.csproj +++ b/Framework/Logging/Logging.csproj @@ -7,12 +7,6 @@ enable - - - - - - diff --git a/Logging/NullLog.cs b/Framework/Logging/NullLog.cs similarity index 80% rename from Logging/NullLog.cs rename to Framework/Logging/NullLog.cs index 75d43ff6..997e490b 100644 --- a/Logging/NullLog.cs +++ b/Framework/Logging/NullLog.cs @@ -1,8 +1,8 @@ namespace Logging { - public class NullLog : TestLog + public class NullLog : BaseLog { - public NullLog() : base("NULL", false, "NULL") + public NullLog() : base(false) { } @@ -26,10 +26,6 @@ Console.WriteLine("Error: " + message); } - public override void MarkAsFailed() - { - } - public override void AddStringReplace(string from, string to) { } diff --git a/Logging/Stopwatch.cs b/Framework/Logging/Stopwatch.cs similarity index 69% rename from Logging/Stopwatch.cs rename to Framework/Logging/Stopwatch.cs index f62f7f63..830b49a9 100644 --- a/Logging/Stopwatch.cs +++ b/Framework/Logging/Stopwatch.cs @@ -5,25 +5,25 @@ namespace Logging public class Stopwatch { private readonly DateTime start = DateTime.UtcNow; - private readonly BaseLog log; + private readonly ILog log; private readonly string name; private readonly bool debug; - private Stopwatch(BaseLog log, string name, bool debug) + private Stopwatch(ILog log, string name, bool debug) { this.log = log; this.name = name; this.debug = debug; } - public static void Measure(BaseLog log, string name, Action action, bool debug = false) + public static void Measure(ILog log, string name, Action action, bool debug = false) { var sw = Begin(log, name, debug); action(); sw.End(); } - public static T Measure(BaseLog log, string name, Func action, bool debug = false) + public static T Measure(ILog log, string name, Func action, bool debug = false) { var sw = Begin(log, name, debug); var result = action(); @@ -31,22 +31,22 @@ namespace Logging return result; } - public static Stopwatch Begin(BaseLog log) + public static Stopwatch Begin(ILog log) { return Begin(log, ""); } - public static Stopwatch Begin(BaseLog log, string name) + public static Stopwatch Begin(ILog log, string name) { return Begin(log, name, false); } - public static Stopwatch Begin(BaseLog log, bool debug) + public static Stopwatch Begin(ILog log, bool debug) { return Begin(log, "", debug); } - public static Stopwatch Begin(BaseLog log, string name, bool debug) + public static Stopwatch Begin(ILog log, string name, bool debug) { return new Stopwatch(log, name, debug); } diff --git a/Framework/NethereumWorkflow/ConversionExtensions.cs b/Framework/NethereumWorkflow/ConversionExtensions.cs new file mode 100644 index 00000000..22daa7c7 --- /dev/null +++ b/Framework/NethereumWorkflow/ConversionExtensions.cs @@ -0,0 +1,30 @@ +using Nethereum.Hex.HexTypes; +using System.Numerics; + +namespace NethereumWorkflow +{ + public static class ConversionExtensions + { + public static HexBigInteger ToHexBig(this decimal amount) + { + var bigint = ToBig(amount); + var str = bigint.ToString("X"); + return new HexBigInteger(str); + } + + public static BigInteger ToBig(this decimal amount) + { + return new BigInteger(amount); + } + + public static decimal ToDecimal(this HexBigInteger hexBigInteger) + { + return ToDecimal(hexBigInteger.Value); + } + + public static decimal ToDecimal(this BigInteger bigInteger) + { + return (decimal)bigInteger; + } + } +} diff --git a/Framework/NethereumWorkflow/NethereumInteraction.cs b/Framework/NethereumWorkflow/NethereumInteraction.cs new file mode 100644 index 00000000..e3aa5bf4 --- /dev/null +++ b/Framework/NethereumWorkflow/NethereumInteraction.cs @@ -0,0 +1,74 @@ +using Logging; +using Nethereum.Contracts; +using Nethereum.RPC.Eth.DTOs; +using Nethereum.Web3; +using Utils; + +namespace NethereumWorkflow +{ + public class NethereumInteraction + { + private readonly ILog log; + private readonly Web3 web3; + + internal NethereumInteraction(ILog log, Web3 web3) + { + this.log = log; + this.web3 = web3; + } + + public void SendEth(string toAddress, decimal ethAmount) + { + var receipt = Time.Wait(web3.Eth.GetEtherTransferService().TransferEtherAndWaitForReceiptAsync(toAddress, ethAmount)); + if (!receipt.Succeeded()) throw new Exception("Unable to send Eth"); + } + + public decimal GetEthBalance() + { + return GetEthBalance(web3.TransactionManager.Account.Address); + } + + public decimal GetEthBalance(string address) + { + var balance = Time.Wait(web3.Eth.GetBalance.SendRequestAsync(address)); + return Web3.Convert.FromWei(balance.Value); + } + + public TResult Call(string contractAddress, TFunction function) where TFunction : FunctionMessage, new() + { + var handler = web3.Eth.GetContractQueryHandler(); + return Time.Wait(handler.QueryAsync(contractAddress, function)); + } + + public void SendTransaction(string contractAddress, TFunction function) where TFunction : FunctionMessage, new() + { + var handler = web3.Eth.GetContractTransactionHandler(); + var receipt = Time.Wait(handler.SendRequestAndWaitForReceiptAsync(contractAddress, function)); + if (!receipt.Succeeded()) throw new Exception("Unable to perform contract transaction."); + } + + public decimal? GetSyncedBlockNumber() + { + log.Debug(); + var sync = Time.Wait(web3.Eth.Syncing.SendRequestAsync()); + var number = Time.Wait(web3.Eth.Blocks.GetBlockNumber.SendRequestAsync()); + var numberOfBlocks = number.ToDecimal(); + if (sync.IsSyncing) return null; + return numberOfBlocks; + } + + public bool IsContractAvailable(string abi, string contractAddress) + { + log.Debug(); + try + { + var contract = web3.Eth.GetContract(abi, contractAddress); + return contract != null; + } + catch + { + return false; + } + } + } +} diff --git a/Nethereum/NethereumInteractionCreator.cs b/Framework/NethereumWorkflow/NethereumInteractionCreator.cs similarity index 84% rename from Nethereum/NethereumInteractionCreator.cs rename to Framework/NethereumWorkflow/NethereumInteractionCreator.cs index ab5449c3..bad11943 100644 --- a/Nethereum/NethereumInteractionCreator.cs +++ b/Framework/NethereumWorkflow/NethereumInteractionCreator.cs @@ -5,12 +5,12 @@ namespace NethereumWorkflow { public class NethereumInteractionCreator { - private readonly BaseLog log; + private readonly ILog log; private readonly string ip; private readonly int port; private readonly string privateKey; - public NethereumInteractionCreator(BaseLog log, string ip, int port, string privateKey) + public NethereumInteractionCreator(ILog log, string ip, int port, string privateKey) { this.log = log; this.ip = ip; diff --git a/Nethereum/NethereumWorkflow.csproj b/Framework/NethereumWorkflow/NethereumWorkflow.csproj similarity index 100% rename from Nethereum/NethereumWorkflow.csproj rename to Framework/NethereumWorkflow/NethereumWorkflow.csproj diff --git a/Utils/Address.cs b/Framework/Utils/Address.cs similarity index 100% rename from Utils/Address.cs rename to Framework/Utils/Address.cs diff --git a/Utils/ByteSize.cs b/Framework/Utils/ByteSize.cs similarity index 100% rename from Utils/ByteSize.cs rename to Framework/Utils/ByteSize.cs diff --git a/Utils/DebugStack.cs b/Framework/Utils/DebugStack.cs similarity index 100% rename from Utils/DebugStack.cs rename to Framework/Utils/DebugStack.cs diff --git a/Utils/Formatter.cs b/Framework/Utils/Formatter.cs similarity index 100% rename from Utils/Formatter.cs rename to Framework/Utils/Formatter.cs diff --git a/Framework/Utils/FrameworkAssert.cs b/Framework/Utils/FrameworkAssert.cs new file mode 100644 index 00000000..325f7cf2 --- /dev/null +++ b/Framework/Utils/FrameworkAssert.cs @@ -0,0 +1,15 @@ +namespace Utils +{ + public static class FrameworkAssert + { + public static void That(bool condition, string message) + { + if (!condition) Fail(message); + } + + public static void Fail(string message) + { + throw new Exception(message); + } + } +} diff --git a/Utils/NumberSource.cs b/Framework/Utils/NumberSource.cs similarity index 100% rename from Utils/NumberSource.cs rename to Framework/Utils/NumberSource.cs diff --git a/Utils/ParseEnum.cs b/Framework/Utils/ParseEnum.cs similarity index 100% rename from Utils/ParseEnum.cs rename to Framework/Utils/ParseEnum.cs diff --git a/Utils/RandomUtils.cs b/Framework/Utils/RandomUtils.cs similarity index 100% rename from Utils/RandomUtils.cs rename to Framework/Utils/RandomUtils.cs diff --git a/Utils/Time.cs b/Framework/Utils/Time.cs similarity index 100% rename from Utils/Time.cs rename to Framework/Utils/Time.cs diff --git a/Utils/Utils.csproj b/Framework/Utils/Utils.csproj similarity index 100% rename from Utils/Utils.csproj rename to Framework/Utils/Utils.csproj diff --git a/KubernetesWorkflow/Configuration.cs b/KubernetesWorkflow/Configuration.cs deleted file mode 100644 index 53fee79a..00000000 --- a/KubernetesWorkflow/Configuration.cs +++ /dev/null @@ -1,18 +0,0 @@ -namespace KubernetesWorkflow -{ - public class Configuration - { - public Configuration(string k8sNamespacePrefix, string? kubeConfigFile, TimeSpan operationTimeout, TimeSpan retryDelay) - { - K8sNamespacePrefix = k8sNamespacePrefix; - KubeConfigFile = kubeConfigFile; - OperationTimeout = operationTimeout; - RetryDelay = retryDelay; - } - - public string K8sNamespacePrefix { get; } - public string? KubeConfigFile { get; } - public TimeSpan OperationTimeout { get; } - public TimeSpan RetryDelay { get; } - } -} diff --git a/KubernetesWorkflow/WorkflowCreator.cs b/KubernetesWorkflow/WorkflowCreator.cs deleted file mode 100644 index 51f0177d..00000000 --- a/KubernetesWorkflow/WorkflowCreator.cs +++ /dev/null @@ -1,30 +0,0 @@ -using Logging; -using Utils; - -namespace KubernetesWorkflow -{ - public class WorkflowCreator - { - private readonly NumberSource numberSource = new NumberSource(0); - private readonly NumberSource containerNumberSource = new NumberSource(0); - private readonly KnownK8sPods knownPods = new KnownK8sPods(); - private readonly K8sCluster cluster; - private readonly BaseLog log; - private readonly string testNamespace; - - public WorkflowCreator(BaseLog log, Configuration configuration, string testNamespace) - { - cluster = new K8sCluster(configuration); - this.log = log; - this.testNamespace = testNamespace.ToLowerInvariant(); - } - - public StartupWorkflow CreateWorkflow() - { - var workflowNumberSource = new WorkflowNumberSource(numberSource.GetNextNumber(), - containerNumberSource); - - return new StartupWorkflow(log, workflowNumberSource, cluster, knownPods, testNamespace); - } - } -} diff --git a/Logging/StatusLog.cs b/Logging/StatusLog.cs deleted file mode 100644 index f5d58313..00000000 --- a/Logging/StatusLog.cs +++ /dev/null @@ -1,69 +0,0 @@ -using Newtonsoft.Json; - -namespace Logging -{ - public class StatusLog - { - private readonly object fileLock = new object(); - private readonly string fullName; - private readonly string fixtureName; - - public StatusLog(LogConfig config, DateTime start, string name = "") - { - fullName = NameUtils.GetFixtureFullName(config, start, name) + "_STATUS.log"; - fixtureName = NameUtils.GetRawFixtureName(); - } - - public void ConcludeTest(string resultStatus, string testDuration, ApplicationIds applicationIds) - { - Write(new StatusLogJson - { - @timestamp = DateTime.UtcNow.ToString("o"), - runid = NameUtils.GetRunId(), - status = resultStatus, - testid = NameUtils.GetTestId(), - codexid = applicationIds.CodexId, - gethid = applicationIds.GethId, - prometheusid = applicationIds.PrometheusId, - codexcontractsid = applicationIds.CodexContractsId, - grafanaid = applicationIds.GrafanaId, - category = NameUtils.GetCategoryName(), - fixturename = fixtureName, - testname = NameUtils.GetTestMethodName(), - testduration = testDuration - }); - } - - private void Write(StatusLogJson json) - { - try - { - lock (fileLock) - { - File.AppendAllLines(fullName, new[] { JsonConvert.SerializeObject(json) }); - } - } - catch (Exception ex) - { - Console.WriteLine("Unable to write to status log: " + ex); - } - } - } - - public class StatusLogJson - { - public string @timestamp { get; set; } = string.Empty; - public string runid { get; set; } = string.Empty; - public string status { get; set; } = string.Empty; - public string testid { get; set; } = string.Empty; - public string codexid { get; set; } = string.Empty; - public string gethid { get; set; } = string.Empty; - public string prometheusid { get; set; } = string.Empty; - public string codexcontractsid { get; set; } = string.Empty; - public string grafanaid { get; set; } = string.Empty; - public string category { get; set; } = string.Empty; - public string fixturename { get; set; } = string.Empty; - public string testname { get; set; } = string.Empty; - public string testduration { get; set;} = string.Empty; - } -} diff --git a/Logging/TestLog.cs b/Logging/TestLog.cs deleted file mode 100644 index a8d32496..00000000 --- a/Logging/TestLog.cs +++ /dev/null @@ -1,41 +0,0 @@ -using NUnit.Framework; - -namespace Logging -{ - public class TestLog : BaseLog - { - private readonly string methodName; - private readonly string fullName; - - public TestLog(string folder, bool debug, string name = "") - : base(debug) - { - methodName = NameUtils.GetTestMethodName(name); - fullName = Path.Combine(folder, methodName); - - Log($"*** Begin: {methodName}"); - } - - public override void EndTest() - { - var result = TestContext.CurrentContext.Result; - - Log($"*** Finished: {methodName} = {result.Outcome.Status}"); - if (!string.IsNullOrEmpty(result.Message)) - { - Log(result.Message); - Log($"{result.StackTrace}"); - } - - if (result.Outcome.Status == NUnit.Framework.Interfaces.TestStatus.Failed) - { - MarkAsFailed(); - } - } - - protected override string GetFullName() - { - return fullName; - } - } -} diff --git a/Nethereum/NethereumInteraction.cs b/Nethereum/NethereumInteraction.cs deleted file mode 100644 index 624f20c7..00000000 --- a/Nethereum/NethereumInteraction.cs +++ /dev/null @@ -1,146 +0,0 @@ -using Logging; -using Nethereum.ABI.FunctionEncoding.Attributes; -using Nethereum.Contracts; -using Nethereum.Hex.HexTypes; -using Nethereum.Web3; -using System.Numerics; -using Utils; - -namespace NethereumWorkflow -{ - public class NethereumInteraction - { - private readonly BaseLog log; - private readonly Web3 web3; - - internal NethereumInteraction(BaseLog log, Web3 web3) - { - this.log = log; - this.web3 = web3; - } - - public string GetTokenAddress(string marketplaceAddress) - { - log.Debug(marketplaceAddress); - var function = new GetTokenFunction(); - - var handler = web3.Eth.GetContractQueryHandler(); - return Time.Wait(handler.QueryAsync(marketplaceAddress, function)); - } - - public void MintTestTokens(string[] accounts, decimal amount, string tokenAddress) - { - if (amount < 1 || accounts.Length < 1) throw new ArgumentException("Invalid arguments for MintTestTokens"); - - var tasks = accounts.Select(a => MintTokens(a, amount, tokenAddress)); - - Task.WaitAll(tasks.ToArray()); - } - - public decimal GetBalance(string tokenAddress, string account) - { - log.Debug($"({tokenAddress}) {account}"); - var function = new GetTokenBalanceFunction - { - Owner = account - }; - - var handler = web3.Eth.GetContractQueryHandler(); - return ToDecimal(Time.Wait(handler.QueryAsync(tokenAddress, function))); - } - - public bool IsSynced(string marketplaceAddress, string marketplaceAbi) - { - try - { - return IsBlockNumberOK() && IsContractAvailable(marketplaceAddress, marketplaceAbi); - } - catch - { - return false; - } - } - - private Task MintTokens(string account, decimal amount, string tokenAddress) - { - log.Debug($"({tokenAddress}) {amount} --> {account}"); - if (string.IsNullOrEmpty(account)) throw new ArgumentException("Invalid arguments for MintTestTokens"); - - var function = new MintTokensFunction - { - Holder = account, - Amount = ToBig(amount) - }; - - var handler = web3.Eth.GetContractTransactionHandler(); - return handler.SendRequestAndWaitForReceiptAsync(tokenAddress, function); - } - - private bool IsBlockNumberOK() - { - log.Debug(); - var sync = Time.Wait(web3.Eth.Syncing.SendRequestAsync()); - var number = Time.Wait(web3.Eth.Blocks.GetBlockNumber.SendRequestAsync()); - var numberOfBlocks = ToDecimal(number); - return !sync.IsSyncing && numberOfBlocks > 256; - } - - private bool IsContractAvailable(string marketplaceAddress, string marketplaceAbi) - { - log.Debug(); - try - { - var contract = web3.Eth.GetContract(marketplaceAbi, marketplaceAddress); - return contract != null; - } - catch - { - return false; - } - } - - private HexBigInteger ToHexBig(decimal amount) - { - var bigint = ToBig(amount); - var str = bigint.ToString("X"); - return new HexBigInteger(str); - } - - private BigInteger ToBig(decimal amount) - { - return new BigInteger(amount); - } - - private decimal ToDecimal(HexBigInteger hexBigInteger) - { - return ToDecimal(hexBigInteger.Value); - } - - private decimal ToDecimal(BigInteger bigInteger) - { - return (decimal)bigInteger; - } - } - - [Function("token", "address")] - public class GetTokenFunction : FunctionMessage - { - } - - [Function("mint")] - public class MintTokensFunction : FunctionMessage - { - [Parameter("address", "holder", 1)] - public string Holder { get; set; } = string.Empty; - - [Parameter("uint256", "amount", 2)] - public BigInteger Amount { get; set; } - } - - [Function("balanceOf", "uint256")] - public class GetTokenBalanceFunction : FunctionMessage - { - [Parameter("address", "owner", 1)] - public string Owner { get; set; } = string.Empty; - } -} diff --git a/ProjectPlugins/CodexContractsPlugin/CodexContractsAccess.cs b/ProjectPlugins/CodexContractsPlugin/CodexContractsAccess.cs new file mode 100644 index 00000000..f7675e12 --- /dev/null +++ b/ProjectPlugins/CodexContractsPlugin/CodexContractsAccess.cs @@ -0,0 +1,51 @@ +using GethPlugin; +using Logging; + +namespace CodexContractsPlugin +{ + public interface ICodexContracts + { + CodexContractsDeployment Deployment { get; } + + void MintTestTokens(IGethNode gethNode, IHasEthAddress owner, TestToken testTokens); + void MintTestTokens(IGethNode gethNode, EthAddress ethAddress, TestToken testTokens); + TestToken GetTestTokenBalance(IGethNode gethNode, IHasEthAddress owner); + TestToken GetTestTokenBalance(IGethNode gethNode, EthAddress ethAddress); + } + + public class CodexContractsAccess : ICodexContracts + { + private readonly ILog log; + + public CodexContractsAccess(ILog log, CodexContractsDeployment deployment) + { + this.log = log; + Deployment = deployment; + } + + public CodexContractsDeployment Deployment { get; } + + public void MintTestTokens(IGethNode gethNode, IHasEthAddress owner, TestToken testTokens) + { + MintTestTokens(gethNode, owner.EthAddress, testTokens); + } + + public void MintTestTokens(IGethNode gethNode, EthAddress ethAddress, TestToken testTokens) + { + var interaction = new ContractInteractions(log, gethNode); + interaction.MintTestTokens(ethAddress, testTokens.Amount, Deployment.TokenAddress); + } + + public TestToken GetTestTokenBalance(IGethNode gethNode, IHasEthAddress owner) + { + return GetTestTokenBalance(gethNode, owner.EthAddress); + } + + public TestToken GetTestTokenBalance(IGethNode gethNode, EthAddress ethAddress) + { + var interaction = new ContractInteractions(log, gethNode); + var balance = interaction.GetBalance(Deployment.TokenAddress, ethAddress.Address); + return balance.TestTokens(); + } + } +} diff --git a/ProjectPlugins/CodexContractsPlugin/CodexContractsContainerConfig.cs b/ProjectPlugins/CodexContractsPlugin/CodexContractsContainerConfig.cs new file mode 100644 index 00000000..fc742a4f --- /dev/null +++ b/ProjectPlugins/CodexContractsPlugin/CodexContractsContainerConfig.cs @@ -0,0 +1,14 @@ +using GethPlugin; + +namespace CodexContractsPlugin +{ + public class CodexContractsContainerConfig + { + public CodexContractsContainerConfig(IGethNode gethNode) + { + GethNode = gethNode; + } + + public IGethNode GethNode { get; } + } +} diff --git a/DistTestCore/Marketplace/CodexContractsContainerRecipe.cs b/ProjectPlugins/CodexContractsPlugin/CodexContractsContainerRecipe.cs similarity index 56% rename from DistTestCore/Marketplace/CodexContractsContainerRecipe.cs rename to ProjectPlugins/CodexContractsPlugin/CodexContractsContainerRecipe.cs index bd2c906c..6b6aabe1 100644 --- a/DistTestCore/Marketplace/CodexContractsContainerRecipe.cs +++ b/ProjectPlugins/CodexContractsPlugin/CodexContractsContainerRecipe.cs @@ -1,21 +1,23 @@ using KubernetesWorkflow; -namespace DistTestCore.Marketplace +namespace CodexContractsPlugin { - public class CodexContractsContainerRecipe : DefaultContainerRecipe + public class CodexContractsContainerRecipe : ContainerRecipeFactory { + public static string DockerImage { get; } = "codexstorage/codex-contracts-eth:latest-dist-tests"; + public const string MarketplaceAddressFilename = "/hardhat/deployments/codexdisttestnetwork/Marketplace.json"; public const string MarketplaceArtifactFilename = "/hardhat/artifacts/contracts/Marketplace.sol/Marketplace.json"; public override string AppName => "codex-contracts"; - public override string Image => "codexstorage/codex-contracts-eth:latest-dist-tests"; + public override string Image => DockerImage; - protected override void InitializeRecipe(StartupConfig startupConfig) + protected override void Initialize(StartupConfig startupConfig) { var config = startupConfig.Get(); - var ip = config.BootstrapNodeIp; - var port = config.JsonRpcPort.Number; + var ip = config.GethNode.StartResult.Container.Pod.PodInfo.Ip; + var port = config.GethNode.StartResult.HttpPort.Number; AddEnvVar("DISTTEST_NETWORK_URL", $"http://{ip}:{port}"); AddEnvVar("HARDHAT_NETWORK", "codexdisttestnetwork"); diff --git a/ProjectPlugins/CodexContractsPlugin/CodexContractsDeployment.cs b/ProjectPlugins/CodexContractsPlugin/CodexContractsDeployment.cs new file mode 100644 index 00000000..d61857c7 --- /dev/null +++ b/ProjectPlugins/CodexContractsPlugin/CodexContractsDeployment.cs @@ -0,0 +1,16 @@ +namespace CodexContractsPlugin +{ + public class CodexContractsDeployment + { + public CodexContractsDeployment(string marketplaceAddress, string abi, string tokenAddress) + { + MarketplaceAddress = marketplaceAddress; + Abi = abi; + TokenAddress = tokenAddress; + } + + public string MarketplaceAddress { get; } + public string Abi { get; } + public string TokenAddress { get; } + } +} diff --git a/ProjectPlugins/CodexContractsPlugin/CodexContractsPlugin.cs b/ProjectPlugins/CodexContractsPlugin/CodexContractsPlugin.cs new file mode 100644 index 00000000..5c03e335 --- /dev/null +++ b/ProjectPlugins/CodexContractsPlugin/CodexContractsPlugin.cs @@ -0,0 +1,44 @@ +using Core; +using GethPlugin; + +namespace CodexContractsPlugin +{ + public class CodexContractsPlugin : IProjectPlugin, IHasLogPrefix, IHasMetadata + { + private readonly IPluginTools tools; + private readonly CodexContractsStarter starter; + + public CodexContractsPlugin(IPluginTools tools) + { + this.tools = tools; + starter = new CodexContractsStarter(tools); + } + + public string LogPrefix => "(CodexContracts) "; + + public void Announce() + { + tools.GetLog().Log($"Loaded Codex-Marketplace SmartContracts"); + } + + public void AddMetadata(IAddMetadata metadata) + { + metadata.Add("codexcontractsid", CodexContractsContainerRecipe.DockerImage); + } + + public void Decommission() + { + } + + public CodexContractsDeployment DeployContracts(IGethNode gethNode) + { + return starter.Deploy(gethNode); + } + + public ICodexContracts WrapDeploy(CodexContractsDeployment deployment) + { + deployment = SerializeGate.Gate(deployment); + return starter.Wrap(deployment); + } + } +} diff --git a/ProjectPlugins/CodexContractsPlugin/CodexContractsPlugin.csproj b/ProjectPlugins/CodexContractsPlugin/CodexContractsPlugin.csproj new file mode 100644 index 00000000..648b8465 --- /dev/null +++ b/ProjectPlugins/CodexContractsPlugin/CodexContractsPlugin.csproj @@ -0,0 +1,14 @@ + + + + net7.0 + enable + enable + + + + + + + + diff --git a/DistTestCore/Marketplace/CodexContractsStarter.cs b/ProjectPlugins/CodexContractsPlugin/CodexContractsStarter.cs similarity index 56% rename from DistTestCore/Marketplace/CodexContractsStarter.cs rename to ProjectPlugins/CodexContractsPlugin/CodexContractsStarter.cs index 4d628b6f..cc314b7e 100644 --- a/DistTestCore/Marketplace/CodexContractsStarter.cs +++ b/ProjectPlugins/CodexContractsPlugin/CodexContractsStarter.cs @@ -1,22 +1,26 @@ -using KubernetesWorkflow; +using Core; +using GethPlugin; +using KubernetesWorkflow; +using Logging; using Utils; -namespace DistTestCore.Marketplace +namespace CodexContractsPlugin { - public class CodexContractsStarter : BaseStarter + public class CodexContractsStarter { + private readonly IPluginTools tools; - public CodexContractsStarter(TestLifecycle lifecycle) - : base(lifecycle) + public CodexContractsStarter(IPluginTools tools) { + this.tools = tools; } - public MarketplaceInfo Start(GethBootstrapNodeInfo bootstrapNode) + public CodexContractsDeployment Deploy(IGethNode gethNode) { - LogStart("Deploying Codex Marketplace..."); + Log("Deploying Codex SmartContracts..."); - var workflow = lifecycle.WorkflowCreator.CreateWorkflow(); - var startupConfig = CreateStartupConfig(bootstrapNode.RunningContainers.Containers[0]); + var workflow = tools.CreateWorkflow(); + var startupConfig = CreateStartupConfig(gethNode); var containers = workflow.Start(1, Location.Unspecified, new CodexContractsContainerRecipe(), startupConfig); if (containers.Containers.Length != 1) throw new InvalidOperationException("Expected 1 Codex contracts container to be created. Test infra failure."); @@ -24,22 +28,36 @@ namespace DistTestCore.Marketplace WaitUntil(() => { - var logHandler = new ContractsReadyLogHandler(Debug); + var logHandler = new ContractsReadyLogHandler(tools.GetLog()); workflow.DownloadContainerLog(container, logHandler, null); return logHandler.Found; }); Log("Contracts deployed. Extracting addresses..."); - var extractor = new ContainerInfoExtractor(lifecycle.Log, workflow, container); + var extractor = new ContractsContainerInfoExtractor(tools.GetLog(), workflow, container); var marketplaceAddress = extractor.ExtractMarketplaceAddress(); var abi = extractor.ExtractMarketplaceAbi(); - var interaction = bootstrapNode.StartInteraction(lifecycle); + var interaction = new ContractInteractions(tools.GetLog(), gethNode); var tokenAddress = interaction.GetTokenAddress(marketplaceAddress); - LogEnd("Extract completed. Marketplace deployed."); + Log("Extract completed. Checking sync..."); - return new MarketplaceInfo(marketplaceAddress, abi, tokenAddress); + Time.WaitUntil(() => interaction.IsSynced(marketplaceAddress, abi)); + + Log("Synced. Codex SmartContracts deployed."); + + return new CodexContractsDeployment(marketplaceAddress, abi, tokenAddress); + } + + public ICodexContracts Wrap(CodexContractsDeployment deployment) + { + return new CodexContractsAccess(tools.GetLog(), deployment); + } + + private void Log(string msg) + { + tools.GetLog().Log(msg); } private void WaitUntil(Func predicate) @@ -47,41 +65,28 @@ namespace DistTestCore.Marketplace Time.WaitUntil(predicate, TimeSpan.FromMinutes(3), TimeSpan.FromSeconds(2)); } - private StartupConfig CreateStartupConfig(RunningContainer bootstrapContainer) + private StartupConfig CreateStartupConfig(IGethNode gethNode) { var startupConfig = new StartupConfig(); - var contractsConfig = new CodexContractsContainerConfig(bootstrapContainer.Pod.PodInfo.Ip, bootstrapContainer.Recipe.GetPortByTag(GethContainerRecipe.HttpPortTag)); + var contractsConfig = new CodexContractsContainerConfig(gethNode); startupConfig.Add(contractsConfig); return startupConfig; } } - public class MarketplaceInfo - { - public MarketplaceInfo(string address, string abi, string tokenAddress) - { - Address = address; - Abi = abi; - TokenAddress = tokenAddress; - } - - public string Address { get; } - public string Abi { get; } - public string TokenAddress { get; } - } - public class ContractsReadyLogHandler : LogHandler { // Log should contain 'Compiled 15 Solidity files successfully' at some point. private const string RequiredCompiledString = "Solidity files successfully"; // When script is done, it prints the ready-string. private const string ReadyString = "Done! Sleeping indefinitely..."; - private readonly Action debug; + private readonly ILog log; - public ContractsReadyLogHandler(Action debug) + public ContractsReadyLogHandler(ILog log) { - this.debug = debug; - debug($"Looking for '{RequiredCompiledString}' and '{ReadyString}' in container logs..."); + this.log = log; + + log.Debug($"Looking for '{RequiredCompiledString}' and '{ReadyString}' in container logs..."); } public bool SeenCompileString { get; private set; } @@ -89,7 +94,7 @@ namespace DistTestCore.Marketplace protected override void ProcessLine(string line) { - debug(line); + log.Debug(line); if (line.Contains(RequiredCompiledString)) SeenCompileString = true; if (line.Contains(ReadyString)) { diff --git a/ProjectPlugins/CodexContractsPlugin/ContractInteractions.cs b/ProjectPlugins/CodexContractsPlugin/ContractInteractions.cs new file mode 100644 index 00000000..719a6038 --- /dev/null +++ b/ProjectPlugins/CodexContractsPlugin/ContractInteractions.cs @@ -0,0 +1,104 @@ +using GethPlugin; +using Logging; +using Nethereum.ABI.FunctionEncoding.Attributes; +using Nethereum.Contracts; +using NethereumWorkflow; +using System.Numerics; + +namespace CodexContractsPlugin +{ + public class ContractInteractions + { + private readonly ILog log; + private readonly IGethNode gethNode; + + public ContractInteractions(ILog log, IGethNode gethNode) + { + this.log = log; + this.gethNode = gethNode; + } + + public string GetTokenAddress(string marketplaceAddress) + { + log.Debug(marketplaceAddress); + var function = new GetTokenFunction(); + + return gethNode.Call(marketplaceAddress, function); + } + + public void MintTestTokens(EthAddress address, decimal amount, string tokenAddress) + { + MintTokens(address.Address, amount, tokenAddress); + } + + public decimal GetBalance(string tokenAddress, string account) + { + log.Debug($"({tokenAddress}) {account}"); + var function = new GetTokenBalanceFunction + { + Owner = account + }; + + return gethNode.Call(tokenAddress, function).ToDecimal(); + } + + public bool IsSynced(string marketplaceAddress, string marketplaceAbi) + { + try + { + return IsBlockNumberOK() && IsContractAvailable(marketplaceAddress, marketplaceAbi); + } + catch + { + return false; + } + } + + private void MintTokens(string account, decimal amount, string tokenAddress) + { + log.Debug($"({tokenAddress}) {amount} --> {account}"); + if (string.IsNullOrEmpty(account)) throw new ArgumentException("Invalid arguments for MintTestTokens"); + + var function = new MintTokensFunction + { + Holder = account, + Amount = amount.ToBig() + }; + + gethNode.SendTransaction(tokenAddress, function); + } + + private bool IsBlockNumberOK() + { + var n = gethNode.GetSyncedBlockNumber(); + return n != null && n > 256; + } + + private bool IsContractAvailable(string marketplaceAddress, string marketplaceAbi) + { + return gethNode.IsContractAvailable(marketplaceAbi, marketplaceAddress); + } + } + + [Function("token", "address")] + public class GetTokenFunction : FunctionMessage + { + } + + [Function("mint")] + public class MintTokensFunction : FunctionMessage + { + [Parameter("address", "holder", 1)] + public string Holder { get; set; } = string.Empty; + + [Parameter("uint256", "amount", 2)] + public BigInteger Amount { get; set; } + } + + [Function("balanceOf", "uint256")] + public class GetTokenBalanceFunction : FunctionMessage + { + [Parameter("address", "owner", 1)] + public string Owner { get; set; } = string.Empty; + } +} diff --git a/ProjectPlugins/CodexContractsPlugin/ContractsContainerInfoExtractor.cs b/ProjectPlugins/CodexContractsPlugin/ContractsContainerInfoExtractor.cs new file mode 100644 index 00000000..78b3896d --- /dev/null +++ b/ProjectPlugins/CodexContractsPlugin/ContractsContainerInfoExtractor.cs @@ -0,0 +1,66 @@ +using KubernetesWorkflow; +using Logging; +using Newtonsoft.Json; +using Newtonsoft.Json.Linq; +using Utils; + +namespace CodexContractsPlugin +{ + public class ContractsContainerInfoExtractor + { + private readonly ILog log; + private readonly IStartupWorkflow workflow; + private readonly RunningContainer container; + + public ContractsContainerInfoExtractor(ILog log, IStartupWorkflow workflow, RunningContainer container) + { + this.log = log; + this.workflow = workflow; + this.container = container; + } + + public string ExtractMarketplaceAddress() + { + log.Debug(); + var marketplaceAddress = Retry(FetchMarketplaceAddress); + if (string.IsNullOrEmpty(marketplaceAddress)) throw new InvalidOperationException("Unable to fetch marketplace account from codex-contracts node. Test infra failure."); + + return marketplaceAddress; + } + + public string ExtractMarketplaceAbi() + { + log.Debug(); + var marketplaceAbi = Retry(FetchMarketplaceAbi); + if (string.IsNullOrEmpty(marketplaceAbi)) throw new InvalidOperationException("Unable to fetch marketplace artifacts from codex-contracts node. Test infra failure."); + + return marketplaceAbi; + } + + private string FetchMarketplaceAddress() + { + var json = workflow.ExecuteCommand(container, "cat", CodexContractsContainerRecipe.MarketplaceAddressFilename); + var marketplace = JsonConvert.DeserializeObject(json); + return marketplace!.address; + } + + private string FetchMarketplaceAbi() + { + var json = workflow.ExecuteCommand(container, "cat", CodexContractsContainerRecipe.MarketplaceArtifactFilename); + + var artifact = JObject.Parse(json); + var abi = artifact["abi"]; + return abi!.ToString(Formatting.None); + } + + private static string Retry(Func fetch) + { + return Time.Retry(fetch, nameof(ContractsContainerInfoExtractor)); + } + } + + public class MarketplaceJson + { + public string address { get; set; } = string.Empty; + } +} diff --git a/ProjectPlugins/CodexContractsPlugin/CoreInterfaceExtensions.cs b/ProjectPlugins/CodexContractsPlugin/CoreInterfaceExtensions.cs new file mode 100644 index 00000000..d2c3355a --- /dev/null +++ b/ProjectPlugins/CodexContractsPlugin/CoreInterfaceExtensions.cs @@ -0,0 +1,29 @@ +using Core; +using GethPlugin; + +namespace CodexContractsPlugin +{ + public static class CoreInterfaceExtensions + { + public static CodexContractsDeployment DeployCodexContracts(this CoreInterface ci, IGethNode gethNode) + { + return Plugin(ci).DeployContracts(gethNode); + } + + public static ICodexContracts WrapCodexContractsDeployment(this CoreInterface ci, CodexContractsDeployment deployment) + { + return Plugin(ci).WrapDeploy(deployment); + } + + public static ICodexContracts StartCodexContracts(this CoreInterface ci, IGethNode gethNode) + { + var deployment = DeployCodexContracts(ci, gethNode); + return WrapCodexContractsDeployment(ci, deployment); + } + + private static CodexContractsPlugin Plugin(CoreInterface ci) + { + return ci.GetPlugin(); + } + } +} diff --git a/ProjectPlugins/CodexContractsPlugin/TestTokenExtensions.cs b/ProjectPlugins/CodexContractsPlugin/TestTokenExtensions.cs new file mode 100644 index 00000000..a19abde4 --- /dev/null +++ b/ProjectPlugins/CodexContractsPlugin/TestTokenExtensions.cs @@ -0,0 +1,45 @@ +namespace CodexContractsPlugin +{ + public class TestToken : IComparable + { + public TestToken(decimal amount) + { + Amount = amount; + } + + public decimal Amount { get; } + + public int CompareTo(TestToken? other) + { + return Amount.CompareTo(other!.Amount); + } + + public override bool Equals(object? obj) + { + return obj is TestToken token && Amount == token.Amount; + } + + public override int GetHashCode() + { + return HashCode.Combine(Amount); + } + + public override string ToString() + { + return $"{Amount} TestTokens"; + } + } + + public static class TokensIntExtensions + { + public static TestToken TestTokens(this int i) + { + return TestTokens(Convert.ToDecimal(i)); + } + + public static TestToken TestTokens(this decimal i) + { + return new TestToken(i); + } + } +} diff --git a/DistTestCore/Codex/CodexAccess.cs b/ProjectPlugins/CodexPlugin/CodexAccess.cs similarity index 82% rename from DistTestCore/Codex/CodexAccess.cs rename to ProjectPlugins/CodexPlugin/CodexAccess.cs index 67f42635..8e0ac40e 100644 --- a/DistTestCore/Codex/CodexAccess.cs +++ b/ProjectPlugins/CodexPlugin/CodexAccess.cs @@ -1,28 +1,25 @@ -using KubernetesWorkflow; -using Logging; -using Utils; +using Core; +using KubernetesWorkflow; -namespace DistTestCore.Codex +namespace CodexPlugin { public class CodexAccess : ILogHandler { - private readonly BaseLog log; - private readonly ITimeSet timeSet; + private readonly IPluginTools tools; private bool hasContainerCrashed; - public CodexAccess(BaseLog log, RunningContainer container, ITimeSet timeSet, Address address) + public CodexAccess(IPluginTools tools, RunningContainer container, CrashWatcher crashWatcher) { - this.log = log; + this.tools = tools; Container = container; - this.timeSet = timeSet; - Address = address; + CrashWatcher = crashWatcher; hasContainerCrashed = false; - if (container.CrashWatcher != null) container.CrashWatcher.Start(this); + CrashWatcher.Start(this); } public RunningContainer Container { get; } - public Address Address { get; } + public CrashWatcher CrashWatcher { get; } public CodexDebugResponse GetDebugInfo() { @@ -54,6 +51,8 @@ namespace DistTestCore.Codex public string UploadFile(FileStream fileStream) { + // private const string UploadFailedMessage = "Unable to store block"; + return Http().HttpPostStream("upload", fileStream); } @@ -87,9 +86,9 @@ namespace DistTestCore.Codex return Container.Name; } - private Http Http() + private IHttp Http() { - return new Http(log, timeSet, Address, baseUrl: "/api/codex/v1", CheckContainerCrashed, Container.Name); + return tools.CreateHttp(Container.Address, baseUrl: "/api/codex/v1", CheckContainerCrashed, Container.Name); } private void CheckContainerCrashed(HttpClient client) @@ -99,6 +98,7 @@ namespace DistTestCore.Codex public void Log(Stream crashLog) { + var log = tools.GetLog(); var file = log.CreateSubfile(); log.Log($"Container {Container.Name} has crashed. Downloading crash log to '{file.FullFilename}'..."); diff --git a/DistTestCore/Codex/CodexApiTypes.cs b/ProjectPlugins/CodexPlugin/CodexApiTypes.cs similarity index 99% rename from DistTestCore/Codex/CodexApiTypes.cs rename to ProjectPlugins/CodexPlugin/CodexApiTypes.cs index 5944b842..127f4374 100644 --- a/DistTestCore/Codex/CodexApiTypes.cs +++ b/ProjectPlugins/CodexPlugin/CodexApiTypes.cs @@ -1,6 +1,6 @@ using Newtonsoft.Json; -namespace DistTestCore.Codex +namespace CodexPlugin { public class CodexDebugResponse { diff --git a/DistTestCore/Codex/CodexContainerRecipe.cs b/ProjectPlugins/CodexPlugin/CodexContainerRecipe.cs similarity index 72% rename from DistTestCore/Codex/CodexContainerRecipe.cs rename to ProjectPlugins/CodexPlugin/CodexContainerRecipe.cs index 60f1c64b..3f230ba3 100644 --- a/DistTestCore/Codex/CodexContainerRecipe.cs +++ b/ProjectPlugins/CodexPlugin/CodexContainerRecipe.cs @@ -1,11 +1,12 @@ -using DistTestCore.Marketplace; -using KubernetesWorkflow; +using KubernetesWorkflow; using Utils; -namespace DistTestCore.Codex +namespace CodexPlugin { - public class CodexContainerRecipe : DefaultContainerRecipe + public class CodexContainerRecipe : ContainerRecipeFactory { + private readonly MarketplaceStarter marketplaceStarter = new MarketplaceStarter(); + private const string DefaultDockerImage = "codexstorage/nim-codex:latest-dist-tests"; public const string MetricsPortTag = "metrics_port"; public const string DiscoveryPortTag = "discovery-port"; @@ -20,13 +21,13 @@ namespace DistTestCore.Codex public CodexContainerRecipe() { Image = GetDockerImage(); - - Resources.Requests = new ContainerResourceSet(milliCPUs: 100, memory: 100.MB()); - Resources.Limits = new ContainerResourceSet(milliCPUs: 4000, memory: 12.GB()); } - protected override void InitializeRecipe(StartupConfig startupConfig) + protected override void Initialize(StartupConfig startupConfig) { + SetResourcesRequest(milliCPUs: 100, memory: 100.MB()); + SetResourceLimits(milliCPUs: 4000, memory: 12.GB()); + var config = startupConfig.Get(); AddExposedPortAndVar("CODEX_API_PORT"); @@ -65,7 +66,7 @@ namespace DistTestCore.Codex { AddEnvVar("CODEX_BLOCK_MN", config.BlockMaintenanceNumber.ToString()!); } - if (config.MetricsMode != Metrics.MetricsMode.None) + if (config.MetricsEnabled) { var metricsPort = AddInternalPort(MetricsPortTag); AddEnvVar("CODEX_METRICS", "true"); @@ -82,28 +83,34 @@ namespace DistTestCore.Codex if (config.MarketplaceConfig != null) { - var gethConfig = startupConfig.Get(); - var companionNode = gethConfig.CompanionNode; - var companionNodeAccount = companionNode.Accounts[GetAccountIndex(config.MarketplaceConfig)]; - Additional(companionNodeAccount); - - var ip = companionNode.RunningContainer.Pod.PodInfo.Ip; - var port = companionNode.RunningContainer.Recipe.GetPortByTag(GethContainerRecipe.HttpPortTag).Number; + var mconfig = config.MarketplaceConfig; + var gethStart = mconfig.GethNode.StartResult; + var ip = gethStart.Container.Pod.PodInfo.Ip; + var port = gethStart.WsPort.Number; + var marketplaceAddress = mconfig.CodexContracts.Deployment.MarketplaceAddress; AddEnvVar("CODEX_ETH_PROVIDER", $"ws://{ip}:{port}"); - AddEnvVar("CODEX_ETH_ACCOUNT", companionNodeAccount.Account); - AddEnvVar("CODEX_MARKETPLACE_ADDRESS", gethConfig.MarketplaceNetwork.Marketplace.Address); + AddEnvVar("CODEX_MARKETPLACE_ADDRESS", marketplaceAddress); AddEnvVar("CODEX_PERSISTENCE", "true"); + // Custom scripting in the Codex test image will write this variable to a private-key file, + // and pass the correct filename to Codex. + var mStart = marketplaceStarter.Start(); + AddEnvVar("PRIV_KEY", mStart.PrivateKey); + Additional(mStart); + if (config.MarketplaceConfig.IsValidator) { AddEnvVar("CODEX_VALIDATOR", "true"); } } - if(!string.IsNullOrEmpty(config.NameOverride)) { + if(!string.IsNullOrEmpty(config.NameOverride)) + { AddEnvVar("CODEX_NODENAME", config.NameOverride); } + + AddPodLabel("codexid", Image); } private ByteSize GetVolumeCapacity(CodexStartupConfig config) @@ -113,12 +120,6 @@ namespace DistTestCore.Codex return 8.GB().Multiply(1.2); } - private int GetAccountIndex(MarketplaceInitialConfig marketplaceConfig) - { - if (marketplaceConfig.AccountIndexOverride != null) return marketplaceConfig.AccountIndexOverride.Value; - return Index; - } - private string GetDockerImage() { var image = Environment.GetEnvironmentVariable("CODEXDOCKERIMAGE"); diff --git a/DistTestCore/Codex/CodexDeployment.cs b/ProjectPlugins/CodexPlugin/CodexDeployment.cs similarity index 79% rename from DistTestCore/Codex/CodexDeployment.cs rename to ProjectPlugins/CodexPlugin/CodexDeployment.cs index 0595a9a1..6dbf961b 100644 --- a/DistTestCore/Codex/CodexDeployment.cs +++ b/ProjectPlugins/CodexPlugin/CodexDeployment.cs @@ -1,23 +1,21 @@ -using DistTestCore.Marketplace; +using GethPlugin; using KubernetesWorkflow; -namespace DistTestCore.Codex +namespace CodexPlugin { public class CodexDeployment { - public CodexDeployment(GethStartResult gethStartResult, RunningContainer[] codexContainers, RunningContainer? prometheusContainer, GrafanaStartInfo? grafanaStartInfo, DeploymentMetadata metadata) + public CodexDeployment(RunningContainer[] codexContainers, GethDeployment gethDeployment, RunningContainer? prometheusContainer, DeploymentMetadata metadata) { - GethStartResult = gethStartResult; CodexContainers = codexContainers; + GethDeployment = gethDeployment; PrometheusContainer = prometheusContainer; - GrafanaStartInfo = grafanaStartInfo; Metadata = metadata; } - public GethStartResult GethStartResult { get; } public RunningContainer[] CodexContainers { get; } + public GethDeployment GethDeployment { get; } public RunningContainer? PrometheusContainer { get; } - public GrafanaStartInfo? GrafanaStartInfo { get; } public DeploymentMetadata Metadata { get; } } diff --git a/DistTestCore/Codex/CodexLogLevel.cs b/ProjectPlugins/CodexPlugin/CodexLogLevel.cs similarity index 78% rename from DistTestCore/Codex/CodexLogLevel.cs rename to ProjectPlugins/CodexPlugin/CodexLogLevel.cs index cde0eb7a..a859a0c4 100644 --- a/DistTestCore/Codex/CodexLogLevel.cs +++ b/ProjectPlugins/CodexPlugin/CodexLogLevel.cs @@ -1,4 +1,4 @@ -namespace DistTestCore.Codex +namespace CodexPlugin { public enum CodexLogLevel { diff --git a/DistTestCore/OnlineCodexNode.cs b/ProjectPlugins/CodexPlugin/CodexNode.cs similarity index 58% rename from DistTestCore/OnlineCodexNode.cs rename to ProjectPlugins/CodexPlugin/CodexNode.cs index 52fb81fc..f4ff8027 100644 --- a/DistTestCore/OnlineCodexNode.cs +++ b/ProjectPlugins/CodexPlugin/CodexNode.cs @@ -1,49 +1,67 @@ -using DistTestCore.Codex; -using DistTestCore.Logs; -using DistTestCore.Marketplace; -using DistTestCore.Metrics; +using Core; +using FileUtils; +using GethPlugin; +using KubernetesWorkflow; using Logging; -using NUnit.Framework; +using MetricsPlugin; using Utils; -namespace DistTestCore +namespace CodexPlugin { - public interface IOnlineCodexNode + public interface ICodexNode : IHasContainer, IHasMetricsScrapeTarget, IHasEthAddress { string GetName(); CodexDebugResponse GetDebugInfo(); CodexDebugPeerResponse GetDebugPeer(string peerId); - ContentId UploadFile(TestFile file); - TestFile? DownloadContent(ContentId contentId, string fileLabel = ""); - void ConnectToPeer(IOnlineCodexNode node); - IDownloadedLog DownloadLog(int? tailLines = null); - IMetricsAccess Metrics { get; } - IMarketplaceAccess Marketplace { get; } + ContentId UploadFile(TrackedFile file); + TrackedFile? DownloadContent(ContentId contentId, string fileLabel = ""); + void ConnectToPeer(ICodexNode node); CodexDebugVersionResponse Version { get; } - ICodexSetup BringOffline(); + IMarketplaceAccess Marketplace { get; } + CrashWatcher CrashWatcher { get; } + void Stop(); } - public class OnlineCodexNode : IOnlineCodexNode + public class CodexNode : ICodexNode { private const string SuccessfullyConnectedMessage = "Successfully connected to peer"; private const string UploadFailedMessage = "Unable to store block"; - private readonly TestLifecycle lifecycle; + private readonly IPluginTools tools; + private readonly EthAddress? ethAddress; - public OnlineCodexNode(TestLifecycle lifecycle, CodexAccess codexAccess, CodexNodeGroup group, IMetricsAccess metricsAccess, IMarketplaceAccess marketplaceAccess) + public CodexNode(IPluginTools tools, CodexAccess codexAccess, CodexNodeGroup group, IMarketplaceAccess marketplaceAccess, EthAddress? ethAddress) { - this.lifecycle = lifecycle; + this.tools = tools; + this.ethAddress = ethAddress; CodexAccess = codexAccess; Group = group; - Metrics = metricsAccess; Marketplace = marketplaceAccess; Version = new CodexDebugVersionResponse(); } + public RunningContainer Container { get { return CodexAccess.Container; } } public CodexAccess CodexAccess { get; } + public CrashWatcher CrashWatcher { get => CodexAccess.CrashWatcher; } public CodexNodeGroup Group { get; } - public IMetricsAccess Metrics { get; } public IMarketplaceAccess Marketplace { get; } public CodexDebugVersionResponse Version { get; private set; } + public IMetricsScrapeTarget MetricsScrapeTarget + { + get + { + var port = CodexAccess.Container.Recipe.GetPortByTag(CodexContainerRecipe.MetricsPortTag); + if (port == null) throw new Exception("Metrics is not available for this Codex node. Please start it with the option '.EnableMetrics()' to enable it."); + return new MetricsScrapeTarget(CodexAccess.Container, port); + } + } + public EthAddress EthAddress + { + get + { + if (ethAddress == null) throw new Exception("Marketplace is not enabled for this Codex node. Please start it with the option '.EnableMarketplace(...)' to enable it."); + return ethAddress; + } + } public string GetName() { @@ -63,58 +81,53 @@ namespace DistTestCore return CodexAccess.GetDebugPeer(peerId); } - public ContentId UploadFile(TestFile file) + public ContentId UploadFile(TrackedFile file) { using var fileStream = File.OpenRead(file.Filename); var logMessage = $"Uploading file {file.Describe()}..."; Log(logMessage); - var response = Stopwatch.Measure(lifecycle.Log, logMessage, () => + var response = Stopwatch.Measure(tools.GetLog(), logMessage, () => { return CodexAccess.UploadFile(fileStream); }); - if (string.IsNullOrEmpty(response)) Assert.Fail("Received empty response."); - if (response.StartsWith(UploadFailedMessage)) Assert.Fail("Node failed to store block."); + if (string.IsNullOrEmpty(response)) FrameworkAssert.Fail("Received empty response."); + if (response.StartsWith(UploadFailedMessage)) FrameworkAssert.Fail("Node failed to store block."); Log($"Uploaded file. Received contentId: '{response}'."); return new ContentId(response); } - public TestFile? DownloadContent(ContentId contentId, string fileLabel = "") + public TrackedFile? DownloadContent(ContentId contentId, string fileLabel = "") { var logMessage = $"Downloading for contentId: '{contentId.Id}'..."; Log(logMessage); - var file = lifecycle.FileManager.CreateEmptyTestFile(fileLabel); - Stopwatch.Measure(lifecycle.Log, logMessage, () => DownloadToFile(contentId.Id, file)); + var file = tools.GetFileManager().CreateEmptyFile(fileLabel); + Stopwatch.Measure(tools.GetLog(), logMessage, () => DownloadToFile(contentId.Id, file)); Log($"Downloaded file {file.Describe()} to '{file.Filename}'."); return file; } - public void ConnectToPeer(IOnlineCodexNode node) + public void ConnectToPeer(ICodexNode node) { - var peer = (OnlineCodexNode)node; + var peer = (CodexNode)node; Log($"Connecting to peer {peer.GetName()}..."); var peerInfo = node.GetDebugInfo(); var response = CodexAccess.ConnectToPeer(peerInfo.id, GetPeerMultiAddress(peer, peerInfo)); - Assert.That(response, Is.EqualTo(SuccessfullyConnectedMessage), "Unable to connect codex nodes."); + FrameworkAssert.That(response == SuccessfullyConnectedMessage, "Unable to connect codex nodes."); Log($"Successfully connected to peer {peer.GetName()}."); } - public IDownloadedLog DownloadLog(int? tailLines = null) - { - return lifecycle.DownloadLog(CodexAccess.Container, tailLines); - } - - public ICodexSetup BringOffline() + public void Stop() { if (Group.Count() > 1) throw new InvalidOperationException("Codex-nodes that are part of a group cannot be " + "individually shut down. Use 'BringOffline()' on the group object to stop the group. This method is only " + "available for codex-nodes in groups of 1."); - return Group.BringOffline(); + Group.BringOffline(); } public void EnsureOnlineGetVersionResponse() @@ -128,12 +141,12 @@ namespace DistTestCore throw new Exception($"Invalid version information received from Codex node {GetName()}: {debugInfo.codex}"); } - lifecycle.Log.AddStringReplace(nodePeerId, nodeName); - lifecycle.Log.AddStringReplace(debugInfo.table.localNode.nodeId, nodeName); + //lifecycle.Log.AddStringReplace(nodePeerId, nodeName); + //lifecycle.Log.AddStringReplace(debugInfo.table.localNode.nodeId, nodeName); Version = debugInfo.codex; } - private string GetPeerMultiAddress(OnlineCodexNode peer, CodexDebugResponse peerInfo) + private string GetPeerMultiAddress(CodexNode peer, CodexDebugResponse peerInfo) { var multiAddress = peerInfo.addrs.First(); // Todo: Is there a case where First address in list is not the way? @@ -143,7 +156,7 @@ namespace DistTestCore return multiAddress.Replace("0.0.0.0", peer.CodexAccess.Container.Pod.PodInfo.Ip); } - private void DownloadToFile(string contentId, TestFile file) + private void DownloadToFile(string contentId, TrackedFile file) { using var fileStream = File.OpenWrite(file.Filename); try @@ -160,7 +173,7 @@ namespace DistTestCore private void Log(string msg) { - lifecycle.Log.Log($"{GetName()}: {msg}"); + tools.GetLog().Log($"{GetName()}: {msg}"); } } diff --git a/ProjectPlugins/CodexPlugin/CodexNodeFactory.cs b/ProjectPlugins/CodexPlugin/CodexNodeFactory.cs new file mode 100644 index 00000000..4b9dfdda --- /dev/null +++ b/ProjectPlugins/CodexPlugin/CodexNodeFactory.cs @@ -0,0 +1,47 @@ +using Core; +using GethPlugin; +using KubernetesWorkflow; + +namespace CodexPlugin +{ + public interface ICodexNodeFactory + { + CodexNode CreateOnlineCodexNode(CodexAccess access, CodexNodeGroup group); + CrashWatcher CreateCrashWatcher(RunningContainer c); + } + + public class CodexNodeFactory : ICodexNodeFactory + { + private readonly IPluginTools tools; + + public CodexNodeFactory(IPluginTools tools) + { + this.tools = tools; + } + + public CodexNode CreateOnlineCodexNode(CodexAccess access, CodexNodeGroup group) + { + var ethAddress = GetEthAddress(access); + var marketplaceAccess = GetMarketplaceAccess(access, ethAddress); + return new CodexNode(tools, access, group, marketplaceAccess, ethAddress); + } + + private IMarketplaceAccess GetMarketplaceAccess(CodexAccess codexAccess, EthAddress? ethAddress) + { + if (ethAddress == null) return new MarketplaceUnavailable(); + return new MarketplaceAccess(tools.GetLog(), codexAccess); + } + + private EthAddress? GetEthAddress(CodexAccess access) + { + var mStart = access.Container.Recipe.Additionals.Get(); + if (mStart == null) return null; + return mStart.EthAddress; + } + + public CrashWatcher CreateCrashWatcher(RunningContainer c) + { + return tools.CreateWorkflow().CreateCrashWatcher(c); + } + } +} diff --git a/DistTestCore/CodexNodeGroup.cs b/ProjectPlugins/CodexPlugin/CodexNodeGroup.cs similarity index 55% rename from DistTestCore/CodexNodeGroup.cs rename to ProjectPlugins/CodexPlugin/CodexNodeGroup.cs index 7759ea7e..01c56610 100644 --- a/DistTestCore/CodexNodeGroup.cs +++ b/ProjectPlugins/CodexPlugin/CodexNodeGroup.cs @@ -1,29 +1,29 @@ -using DistTestCore.Codex; +using Core; using KubernetesWorkflow; +using MetricsPlugin; using System.Collections; -namespace DistTestCore +namespace CodexPlugin { - public interface ICodexNodeGroup : IEnumerable + public interface ICodexNodeGroup : IEnumerable, IHasManyMetricScrapeTargets { - ICodexSetup BringOffline(); - IOnlineCodexNode this[int index] { get; } + void BringOffline(); + ICodexNode this[int index] { get; } } public class CodexNodeGroup : ICodexNodeGroup { - private readonly TestLifecycle lifecycle; + private readonly CodexStarter starter; - public CodexNodeGroup(TestLifecycle lifecycle, CodexSetup setup, RunningContainers[] containers, ICodexNodeFactory codexNodeFactory) + public CodexNodeGroup(CodexStarter starter, IPluginTools tools, RunningContainers[] containers, ICodexNodeFactory codexNodeFactory) { - this.lifecycle = lifecycle; - Setup = setup; + this.starter = starter; Containers = containers; - Nodes = containers.Containers().Select(c => CreateOnlineCodexNode(c, codexNodeFactory)).ToArray(); + Nodes = containers.Containers().Select(c => CreateOnlineCodexNode(c, tools, codexNodeFactory)).ToArray(); Version = new CodexDebugVersionResponse(); } - public IOnlineCodexNode this[int index] + public ICodexNode this[int index] { get { @@ -31,27 +31,22 @@ namespace DistTestCore } } - public ICodexSetup BringOffline() + public void BringOffline() { - lifecycle.CodexStarter.BringOffline(this); - - var result = Setup; + starter.BringOffline(this); // Clear everything. Prevent accidental use. - Setup = null!; - Nodes = Array.Empty(); + Nodes = Array.Empty(); Containers = null!; - - return result; } - public CodexSetup Setup { get; private set; } public RunningContainers[] Containers { get; private set; } - public OnlineCodexNode[] Nodes { get; private set; } + public CodexNode[] Nodes { get; private set; } public CodexDebugVersionResponse Version { get; private set; } + public IMetricsScrapeTarget[] ScrapeTargets => Nodes.Select(n => n.MetricsScrapeTarget).ToArray(); - public IEnumerator GetEnumerator() + public IEnumerator GetEnumerator() { - return Nodes.Cast().GetEnumerator(); + return Nodes.Cast().GetEnumerator(); } IEnumerator IEnumerable.GetEnumerator() @@ -79,9 +74,10 @@ namespace DistTestCore Version = first; } - private OnlineCodexNode CreateOnlineCodexNode(RunningContainer c, ICodexNodeFactory factory) + private CodexNode CreateOnlineCodexNode(RunningContainer c, IPluginTools tools, ICodexNodeFactory factory) { - var access = new CodexAccess(lifecycle.Log, c, lifecycle.TimeSet, lifecycle.Configuration.GetAddress(c)); + var watcher = factory.CreateCrashWatcher(c); + var access = new CodexAccess(tools, c, watcher); return factory.CreateOnlineCodexNode(access, this); } } diff --git a/ProjectPlugins/CodexPlugin/CodexPlugin.cs b/ProjectPlugins/CodexPlugin/CodexPlugin.cs new file mode 100644 index 00000000..2f670dec --- /dev/null +++ b/ProjectPlugins/CodexPlugin/CodexPlugin.cs @@ -0,0 +1,67 @@ +using Core; +using KubernetesWorkflow; + +namespace CodexPlugin +{ + public class CodexPlugin : IProjectPlugin, IHasLogPrefix, IHasMetadata + { + private readonly CodexStarter codexStarter; + private readonly IPluginTools tools; + private readonly CodexLogLevel defaultLogLevel = CodexLogLevel.Trace; + + public CodexPlugin(IPluginTools tools) + { + codexStarter = new CodexStarter(tools); + this.tools = tools; + } + + public string LogPrefix => "(Codex) "; + + public void Announce() + { + tools.GetLog().Log($"Loaded with Codex ID: '{codexStarter.GetCodexId()}'"); + } + + public void AddMetadata(IAddMetadata metadata) + { + metadata.Add("codexid", codexStarter.GetCodexId()); + } + + public void Decommission() + { + } + + public RunningContainers[] DeployCodexNodes(int numberOfNodes, Action setup) + { + var codexSetup = GetSetup(numberOfNodes, setup); + return codexStarter.BringOnline(codexSetup); + } + + public ICodexNodeGroup WrapCodexContainers(CoreInterface coreInterface, RunningContainers[] containers) + { + containers = containers.Select(c => SerializeGate.Gate(c)).ToArray(); + return codexStarter.WrapCodexContainers(coreInterface, containers); + } + + public void WireUpMarketplace(ICodexNodeGroup result, Action setup) + { + var codexSetup = GetSetup(1, setup); + if (codexSetup.MarketplaceConfig == null) return; + + var mconfig = codexSetup.MarketplaceConfig; + foreach (var node in result) + { + mconfig.GethNode.SendEth(node, mconfig.InitialEth); + mconfig.CodexContracts.MintTestTokens(mconfig.GethNode, node, mconfig.InitialTokens); + } + } + + private CodexSetup GetSetup(int numberOfNodes, Action setup) + { + var codexSetup = new CodexSetup(numberOfNodes); + codexSetup.LogLevel = defaultLogLevel; + setup(codexSetup); + return codexSetup; + } + } +} diff --git a/ProjectPlugins/CodexPlugin/CodexPlugin.csproj b/ProjectPlugins/CodexPlugin/CodexPlugin.csproj new file mode 100644 index 00000000..19c3b60e --- /dev/null +++ b/ProjectPlugins/CodexPlugin/CodexPlugin.csproj @@ -0,0 +1,21 @@ + + + + net7.0 + enable + enable + + + + + + + + + + + + + + + diff --git a/DistTestCore/CodexSetup.cs b/ProjectPlugins/CodexPlugin/CodexSetup.cs similarity index 74% rename from DistTestCore/CodexSetup.cs rename to ProjectPlugins/CodexPlugin/CodexSetup.cs index e421aa9e..d8b25ee5 100644 --- a/DistTestCore/CodexSetup.cs +++ b/ProjectPlugins/CodexPlugin/CodexSetup.cs @@ -1,29 +1,27 @@ -using DistTestCore.Codex; -using DistTestCore.Marketplace; +using CodexContractsPlugin; +using GethPlugin; using KubernetesWorkflow; using Utils; -namespace DistTestCore +namespace CodexPlugin { public interface ICodexSetup { ICodexSetup WithName(string name); ICodexSetup At(Location location); + ICodexSetup WithBootstrapNode(ICodexNode node); ICodexSetup WithLogLevel(CodexLogLevel level); /// /// Sets the log level for codex. The default level is INFO and the /// log level is applied only to the supplied topics. /// ICodexSetup WithLogLevel(CodexLogLevel level, params string[] topics); - ICodexSetup WithBootstrapNode(IOnlineCodexNode node); ICodexSetup WithStorageQuota(ByteSize storageQuota); ICodexSetup WithBlockTTL(TimeSpan duration); ICodexSetup WithBlockMaintenanceInterval(TimeSpan duration); ICodexSetup WithBlockMaintenanceNumber(int numberOfBlocks); ICodexSetup EnableMetrics(); - ICodexSetup EnableMarketplace(TestToken initialBalance); - ICodexSetup EnableMarketplace(TestToken initialBalance, Ether initialEther); - ICodexSetup EnableMarketplace(TestToken initialBalance, Ether initialEther, bool isValidator); + ICodexSetup EnableMarketplace(IGethNode gethNode, ICodexContracts codexContracts, Ether initialEth, TestToken initialTokens, bool isValidator = false); /// /// Provides an invalid proof every N proofs /// @@ -34,8 +32,7 @@ namespace DistTestCore { public int NumberOfNodes { get; } - public CodexSetup(int numberOfNodes, CodexLogLevel logLevel) - : base(logLevel) + public CodexSetup(int numberOfNodes) { NumberOfNodes = numberOfNodes; } @@ -52,7 +49,7 @@ namespace DistTestCore return this; } - public ICodexSetup WithBootstrapNode(IOnlineCodexNode node) + public ICodexSetup WithBootstrapNode(ICodexNode node) { BootstrapSpr = node.GetDebugInfo().spr; return this; @@ -97,23 +94,13 @@ namespace DistTestCore public ICodexSetup EnableMetrics() { - MetricsMode = Metrics.MetricsMode.Record; + MetricsEnabled = true; return this; } - public ICodexSetup EnableMarketplace(TestToken initialBalance) + public ICodexSetup EnableMarketplace(IGethNode gethNode, ICodexContracts codexContracts, Ether initialEth, TestToken initialTokens, bool isValidator = false) { - return EnableMarketplace(initialBalance, 1000.Eth()); - } - - public ICodexSetup EnableMarketplace(TestToken initialBalance, Ether initialEther) - { - return EnableMarketplace(initialBalance, initialEther, false); - } - - public ICodexSetup EnableMarketplace(TestToken initialBalance, Ether initialEther, bool isValidator) - { - MarketplaceConfig = new MarketplaceInitialConfig(initialEther, initialBalance, isValidator); + MarketplaceConfig = new MarketplaceInitialConfig(gethNode, codexContracts, initialEth, initialTokens, isValidator); return this; } diff --git a/ProjectPlugins/CodexPlugin/CodexStarter.cs b/ProjectPlugins/CodexPlugin/CodexStarter.cs new file mode 100644 index 00000000..855fb147 --- /dev/null +++ b/ProjectPlugins/CodexPlugin/CodexStarter.cs @@ -0,0 +1,124 @@ +using Core; +using KubernetesWorkflow; +using Logging; + +namespace CodexPlugin +{ + public class CodexStarter + { + private readonly IPluginTools pluginTools; + private readonly CodexContainerRecipe recipe = new CodexContainerRecipe(); + private CodexDebugVersionResponse? versionResponse; + + public CodexStarter(IPluginTools pluginTools) + { + this.pluginTools = pluginTools; + } + + public RunningContainers[] BringOnline(CodexSetup codexSetup) + { + LogSeparator(); + Log($"Starting {codexSetup.Describe()}..."); + + var startupConfig = CreateStartupConfig(codexSetup); + + var containers = StartCodexContainers(startupConfig, codexSetup.NumberOfNodes, codexSetup.Location); + + var podInfos = string.Join(", ", containers.Containers().Select(c => $"Container: '{c.Name}' runs at '{c.Pod.PodInfo.K8SNodeName}'={c.Pod.PodInfo.Ip}")); + Log($"Started {codexSetup.NumberOfNodes} nodes of image '{containers.Containers().First().Recipe.Image}'. ({podInfos})"); + LogSeparator(); + + return containers; + } + + public ICodexNodeGroup WrapCodexContainers(CoreInterface coreInterface, RunningContainers[] containers) + { + var codexNodeFactory = new CodexNodeFactory(pluginTools); + + var group = CreateCodexGroup(coreInterface, containers, codexNodeFactory); + + Log($"Codex version: {group.Version}"); + versionResponse = group.Version; + + return group; + } + + public void BringOffline(CodexNodeGroup group) + { + Log($"Stopping {group.Describe()}..."); + StopCrashWatcher(group); + var workflow = pluginTools.CreateWorkflow(); + foreach (var c in group.Containers) + { + workflow.Stop(c); + } + Log("Stopped."); + } + + public string GetCodexId() + { + if (versionResponse != null) return versionResponse.version; + return recipe.Image; + } + + private StartupConfig CreateStartupConfig(CodexSetup codexSetup) + { + var startupConfig = new StartupConfig(); + startupConfig.NameOverride = codexSetup.NameOverride; + startupConfig.Add(codexSetup); + return startupConfig; + } + + private RunningContainers[] StartCodexContainers(StartupConfig startupConfig, int numberOfNodes, Location location) + { + var result = new List(); + for (var i = 0; i < numberOfNodes; i++) + { + var workflow = pluginTools.CreateWorkflow(); + result.Add(workflow.Start(1, location, recipe, startupConfig)); + } + return result.ToArray(); + } + + private CodexNodeGroup CreateCodexGroup(CoreInterface coreInterface, RunningContainers[] runningContainers, CodexNodeFactory codexNodeFactory) + { + var group = new CodexNodeGroup(this, pluginTools, runningContainers, codexNodeFactory); + + try + { + Stopwatch.Measure(pluginTools.GetLog(), "EnsureOnline", group.EnsureOnline); + } + catch + { + CodexNodesNotOnline(coreInterface, runningContainers); + throw; + } + + return group; + } + + private void CodexNodesNotOnline(CoreInterface coreInterface, RunningContainers[] runningContainers) + { + Log("Codex nodes failed to start"); + foreach (var container in runningContainers.Containers()) coreInterface.DownloadLog(container); + } + + private void LogSeparator() + { + Log("----------------------------------------------------------------------------"); + } + + private void Log(string message) + { + pluginTools.GetLog().Log(message); + } + + private void StopCrashWatcher(CodexNodeGroup group) + { + foreach (var node in group) + { + node.CrashWatcher.Stop(); + } + } + } +} diff --git a/DistTestCore/Codex/CodexStartupConfig.cs b/ProjectPlugins/CodexPlugin/CodexStartupConfig.cs similarity index 78% rename from DistTestCore/Codex/CodexStartupConfig.cs rename to ProjectPlugins/CodexPlugin/CodexStartupConfig.cs index 0ca738dd..07baec10 100644 --- a/DistTestCore/Codex/CodexStartupConfig.cs +++ b/ProjectPlugins/CodexPlugin/CodexStartupConfig.cs @@ -1,16 +1,23 @@ -using DistTestCore.Marketplace; -using DistTestCore.Metrics; -using KubernetesWorkflow; +using KubernetesWorkflow; using Utils; -namespace DistTestCore.Codex +namespace CodexPlugin { public class CodexStartupConfig { - public CodexStartupConfig(CodexLogLevel logLevel) - { - LogLevel = logLevel; - } + public string? NameOverride { get; set; } + public Location Location { get; set; } + public CodexLogLevel LogLevel { get; set; } + public string[]? LogTopics { get; set; } + public ByteSize? StorageQuota { get; set; } + public bool MetricsEnabled { get; set; } + public MarketplaceInitialConfig? MarketplaceConfig { get; set; } + public string? BootstrapSpr { get; set; } + public int? BlockTTL { get; set; } + public uint? SimulateProofFailures { get; set; } + public bool? EnableValidator { get; set; } + public TimeSpan? BlockMaintenanceInterval { get; set; } + public int? BlockMaintenanceNumber { get; set; } public string LogLevelWithTopics() { @@ -21,19 +28,5 @@ namespace DistTestCore.Codex } return level; } - - public string? NameOverride { get; set; } - public Location Location { get; set; } - public CodexLogLevel LogLevel { get; set; } - public string[]? LogTopics { get; set; } - public ByteSize? StorageQuota { get; set; } - public MetricsMode MetricsMode { get; set; } - public MarketplaceInitialConfig? MarketplaceConfig { get; set; } - public string? BootstrapSpr { get; set; } - public int? BlockTTL { get; set; } - public uint? SimulateProofFailures { get; set; } - public bool? EnableValidator { get; set; } - public TimeSpan? BlockMaintenanceInterval { get; set; } - public int? BlockMaintenanceNumber { get; set; } } } diff --git a/ProjectPlugins/CodexPlugin/CoreInterfaceExtensions.cs b/ProjectPlugins/CodexPlugin/CoreInterfaceExtensions.cs new file mode 100644 index 00000000..bce5fe5d --- /dev/null +++ b/ProjectPlugins/CodexPlugin/CoreInterfaceExtensions.cs @@ -0,0 +1,53 @@ +using Core; +using KubernetesWorkflow; + +namespace CodexPlugin +{ + public static class CoreInterfaceExtensions + { + public static RunningContainers[] DeployCodexNodes(this CoreInterface ci, int number, Action setup) + { + return Plugin(ci).DeployCodexNodes(number, setup); + } + + public static ICodexNodeGroup WrapCodexContainers(this CoreInterface ci, RunningContainer[] containers) + { + // ew, clean this up. + var rcs = new RunningContainers(null!, containers.First().Pod, containers); + return WrapCodexContainers(ci, new[] { rcs }); + } + + public static ICodexNodeGroup WrapCodexContainers(this CoreInterface ci, RunningContainers[] containers) + { + return Plugin(ci).WrapCodexContainers(ci, containers); + } + + public static ICodexNode StartCodexNode(this CoreInterface ci) + { + return ci.StartCodexNodes(1)[0]; + } + + public static ICodexNode StartCodexNode(this CoreInterface ci, Action setup) + { + return ci.StartCodexNodes(1, setup)[0]; + } + + public static ICodexNodeGroup StartCodexNodes(this CoreInterface ci, int number, Action setup) + { + var rc = ci.DeployCodexNodes(number, setup); + var result = ci.WrapCodexContainers(rc); + Plugin(ci).WireUpMarketplace(result, setup); + return result; + } + + public static ICodexNodeGroup StartCodexNodes(this CoreInterface ci, int number) + { + return ci.StartCodexNodes(number, s => { }); + } + + private static CodexPlugin Plugin(CoreInterface ci) + { + return ci.GetPlugin(); + } + } +} diff --git a/DistTestCore/Marketplace/MarketplaceAccess.cs b/ProjectPlugins/CodexPlugin/MarketplaceAccess.cs similarity index 76% rename from DistTestCore/Marketplace/MarketplaceAccess.cs rename to ProjectPlugins/CodexPlugin/MarketplaceAccess.cs index 5c052662..3a40f1c9 100644 --- a/DistTestCore/Marketplace/MarketplaceAccess.cs +++ b/ProjectPlugins/CodexPlugin/MarketplaceAccess.cs @@ -1,34 +1,25 @@ -using DistTestCore.Codex; -using DistTestCore.Helpers; +using CodexContractsPlugin; using Logging; using Newtonsoft.Json; -using NUnit.Framework; -using NUnit.Framework.Constraints; -using System.Numerics; using Utils; +using System.Numerics; -namespace DistTestCore.Marketplace +namespace CodexPlugin { public interface IMarketplaceAccess { - string MakeStorageAvailable(ByteSize size, TestToken minPricePerBytePerSecond, TestToken maxCollateral, TimeSpan maxDuration); + string MakeStorageAvailable(ByteSize size, TestToken minPriceForTotalSpace, TestToken maxCollateral, TimeSpan maxDuration); StoragePurchaseContract RequestStorage(ContentId contentId, TestToken pricePerSlotPerSecond, TestToken requiredCollateral, uint minRequiredNumberOfNodes, int proofProbability, TimeSpan duration); - void AssertThatBalance(IResolveConstraint constraint, string message = ""); - TestToken GetBalance(); } public class MarketplaceAccess : IMarketplaceAccess { - private readonly TestLifecycle lifecycle; - private readonly MarketplaceNetwork marketplaceNetwork; - private readonly GethAccount account; + private readonly ILog log; private readonly CodexAccess codexAccess; - public MarketplaceAccess(TestLifecycle lifecycle, MarketplaceNetwork marketplaceNetwork, GethAccount account, CodexAccess codexAccess) + public MarketplaceAccess(ILog log, CodexAccess codexAccess) { - this.lifecycle = lifecycle; - this.marketplaceNetwork = marketplaceNetwork; - this.account = account; + this.log = log; this.codexAccess = codexAccess; } @@ -61,7 +52,7 @@ namespace DistTestCore.Marketplace Log($"Storage requested successfully. PurchaseId: '{response}'."); - return new StoragePurchaseContract(lifecycle.Log, codexAccess, response, duration); + return new StoragePurchaseContract(log, codexAccess, response, duration); } public string MakeStorageAvailable(ByteSize totalSpace, TestToken minPriceForTotalSpace, TestToken maxCollateral, TimeSpan maxDuration) @@ -99,25 +90,9 @@ namespace DistTestCore.Marketplace return i.ToString("D"); } - public void AssertThatBalance(IResolveConstraint constraint, string message = "") - { - AssertHelpers.RetryAssert(constraint, GetBalance, message); - } - - public TestToken GetBalance() - { - var interaction = marketplaceNetwork.StartInteraction(lifecycle); - var amount = interaction.GetBalance(marketplaceNetwork.Marketplace.TokenAddress, account.Account); - var balance = new TestToken(amount); - - Log($"Balance of {account.Account} is {balance}."); - - return balance; - } - private void Log(string msg) { - lifecycle.Log.Log($"{codexAccess.Container.Name} {msg}"); + log.Log($"{codexAccess.Container.Name} {msg}"); } } @@ -135,31 +110,20 @@ namespace DistTestCore.Marketplace return string.Empty; } - public void AssertThatBalance(IResolveConstraint constraint, string message = "") - { - Unavailable(); - } - - public TestToken GetBalance() - { - Unavailable(); - return new TestToken(0); - } - private void Unavailable() { - Assert.Fail("Incorrect test setup: Marketplace was not enabled for this group of Codex nodes. Add 'EnableMarketplace(...)' after 'SetupCodexNodes()' to enable it."); + FrameworkAssert.Fail("Incorrect test setup: Marketplace was not enabled for this group of Codex nodes. Add 'EnableMarketplace(...)' after 'SetupCodexNodes()' to enable it."); throw new InvalidOperationException(); } } public class StoragePurchaseContract { - private readonly BaseLog log; + private readonly ILog log; private readonly CodexAccess codexAccess; private DateTime? contractStartUtc; - public StoragePurchaseContract(BaseLog log, CodexAccess codexAccess, string purchaseId, TimeSpan contractDuration) + public StoragePurchaseContract(ILog log, CodexAccess codexAccess, string purchaseId, TimeSpan contractDuration) { this.log = log; this.codexAccess = codexAccess; @@ -236,12 +200,12 @@ namespace DistTestCore.Marketplace if (lastState == "errored") { - Assert.Fail("Contract errored: " + statusJson); + FrameworkAssert.Fail("Contract errored: " + statusJson); } if (DateTime.UtcNow - waitStart > timeout) { - Assert.Fail($"Contract did not reach '{desiredState}' within timeout. {statusJson}"); + FrameworkAssert.Fail($"Contract did not reach '{desiredState}' within timeout. {statusJson}"); } } log.Log($"Contract '{desiredState}'."); diff --git a/ProjectPlugins/CodexPlugin/MarketplaceInitialConfig.cs b/ProjectPlugins/CodexPlugin/MarketplaceInitialConfig.cs new file mode 100644 index 00000000..7c925916 --- /dev/null +++ b/ProjectPlugins/CodexPlugin/MarketplaceInitialConfig.cs @@ -0,0 +1,23 @@ +using CodexContractsPlugin; +using GethPlugin; + +namespace CodexPlugin +{ + public class MarketplaceInitialConfig + { + public MarketplaceInitialConfig(IGethNode gethNode, ICodexContracts codexContracts, Ether initialEth, TestToken initialTokens, bool isValidator) + { + GethNode = gethNode; + CodexContracts = codexContracts; + InitialEth = initialEth; + InitialTokens = initialTokens; + IsValidator = isValidator; + } + + public IGethNode GethNode { get; } + public ICodexContracts CodexContracts { get; } + public Ether InitialEth { get; } + public TestToken InitialTokens { get; } + public bool IsValidator { get; } + } +} diff --git a/ProjectPlugins/CodexPlugin/MarketplaceStartResults.cs b/ProjectPlugins/CodexPlugin/MarketplaceStartResults.cs new file mode 100644 index 00000000..4bce517a --- /dev/null +++ b/ProjectPlugins/CodexPlugin/MarketplaceStartResults.cs @@ -0,0 +1,17 @@ +using GethPlugin; + +namespace CodexPlugin +{ + [Serializable] + public class MarketplaceStartResults + { + public MarketplaceStartResults(EthAddress ethAddress, string privateKey) + { + EthAddress = ethAddress; + PrivateKey = privateKey; + } + + public EthAddress EthAddress { get; } + public string PrivateKey { get; } + } +} diff --git a/ProjectPlugins/CodexPlugin/MarketplaceStarter.cs b/ProjectPlugins/CodexPlugin/MarketplaceStarter.cs new file mode 100644 index 00000000..42a5851e --- /dev/null +++ b/ProjectPlugins/CodexPlugin/MarketplaceStarter.cs @@ -0,0 +1,19 @@ +using GethPlugin; +using Nethereum.Hex.HexConvertors.Extensions; +using Nethereum.Web3.Accounts; + +namespace CodexPlugin +{ + public class MarketplaceStarter + { + public MarketplaceStartResults Start() + { + var ecKey = Nethereum.Signer.EthECKey.GenerateKey(); + var privateKey = ecKey.GetPrivateKeyAsBytes().ToHex(); + var account = new Account(privateKey); + var ethAddress = new EthAddress(account.Address); + + return new MarketplaceStartResults(ethAddress, account.PrivateKey); + } + } +} diff --git a/ProjectPlugins/GethPlugin/CoreInterfaceExtensions.cs b/ProjectPlugins/GethPlugin/CoreInterfaceExtensions.cs new file mode 100644 index 00000000..a1d0ecd1 --- /dev/null +++ b/ProjectPlugins/GethPlugin/CoreInterfaceExtensions.cs @@ -0,0 +1,28 @@ +using Core; + +namespace GethPlugin +{ + public static class CoreInterfaceExtensions + { + public static GethDeployment DeployGeth(this CoreInterface ci, Action setup) + { + return Plugin(ci).DeployGeth(setup); + } + + public static IGethNode WrapGethDeployment(this CoreInterface ci, GethDeployment deployment) + { + return Plugin(ci).WrapGethDeployment(deployment); + } + + public static IGethNode StartGethNode(this CoreInterface ci, Action setup) + { + var deploy = DeployGeth(ci, setup); + return WrapGethDeployment(ci, deploy); + } + + private static GethPlugin Plugin(CoreInterface ci) + { + return ci.GetPlugin(); + } + } +} diff --git a/ProjectPlugins/GethPlugin/EthAddress.cs b/ProjectPlugins/GethPlugin/EthAddress.cs new file mode 100644 index 00000000..54c86386 --- /dev/null +++ b/ProjectPlugins/GethPlugin/EthAddress.cs @@ -0,0 +1,17 @@ +namespace GethPlugin +{ + public interface IHasEthAddress + { + EthAddress EthAddress { get; } + } + + public class EthAddress + { + public EthAddress(string address) + { + Address = address; + } + + public string Address { get; } + } +} diff --git a/DistTestCore/Tokens.cs b/ProjectPlugins/GethPlugin/EthTokenExtensions.cs similarity index 50% rename from DistTestCore/Tokens.cs rename to ProjectPlugins/GethPlugin/EthTokenExtensions.cs index 5593ffcf..950a7e17 100644 --- a/DistTestCore/Tokens.cs +++ b/ProjectPlugins/GethPlugin/EthTokenExtensions.cs @@ -1,13 +1,15 @@ -namespace DistTestCore +namespace GethPlugin { public class Ether : IComparable { public Ether(decimal wei) { Wei = wei; + Eth = wei / TokensIntExtensions.WeiPerEth; } public decimal Wei { get; } + public decimal Eth { get; } public int CompareTo(Ether? other) { @@ -30,49 +32,9 @@ } } - public class TestToken : IComparable - { - public TestToken(decimal amount) - { - Amount = amount; - } - - public decimal Amount { get; } - - public int CompareTo(TestToken? other) - { - return Amount.CompareTo(other!.Amount); - } - - public override bool Equals(object? obj) - { - return obj is TestToken token && Amount == token.Amount; - } - - public override int GetHashCode() - { - return HashCode.Combine(Amount); - } - - public override string ToString() - { - return $"{Amount} TestTokens"; - } - } - public static class TokensIntExtensions { - private const decimal weiPerEth = 1000000000000000000; - - public static TestToken TestTokens(this int i) - { - return TestTokens(Convert.ToDecimal(i)); - } - - public static TestToken TestTokens(this decimal i) - { - return new TestToken(i); - } + public const decimal WeiPerEth = 1000000000000000000; public static Ether Eth(this int i) { @@ -86,7 +48,7 @@ public static Ether Eth(this decimal i) { - return new Ether(i * weiPerEth); + return new Ether(i * WeiPerEth); } public static Ether Wei(this decimal i) diff --git a/ProjectPlugins/GethPlugin/GethAccount.cs b/ProjectPlugins/GethPlugin/GethAccount.cs new file mode 100644 index 00000000..10974e20 --- /dev/null +++ b/ProjectPlugins/GethPlugin/GethAccount.cs @@ -0,0 +1,24 @@ +namespace GethPlugin +{ + public class GethAccount + { + public GethAccount(string account, string privateKey) + { + Account = account; + PrivateKey = privateKey; + } + + public string Account { get; } + public string PrivateKey { get; } + } + + public class AllGethAccounts + { + public GethAccount[] Accounts { get; } + + public AllGethAccounts(GethAccount[] accounts) + { + Accounts = accounts; + } + } +} diff --git a/DistTestCore/Marketplace/ContainerInfoExtractor.cs b/ProjectPlugins/GethPlugin/GethContainerInfoExtractor.cs similarity index 62% rename from DistTestCore/Marketplace/ContainerInfoExtractor.cs rename to ProjectPlugins/GethPlugin/GethContainerInfoExtractor.cs index 1eac5cbc..51c8728e 100644 --- a/DistTestCore/Marketplace/ContainerInfoExtractor.cs +++ b/ProjectPlugins/GethPlugin/GethContainerInfoExtractor.cs @@ -1,18 +1,16 @@ using KubernetesWorkflow; using Logging; -using Newtonsoft.Json; -using Newtonsoft.Json.Linq; using Utils; -namespace DistTestCore.Marketplace +namespace GethPlugin { - public class ContainerInfoExtractor + public class GethContainerInfoExtractor { - private readonly BaseLog log; - private readonly StartupWorkflow workflow; + private readonly ILog log; + private readonly IStartupWorkflow workflow; private readonly RunningContainer container; - public ContainerInfoExtractor(BaseLog log, StartupWorkflow workflow, RunningContainer container) + public GethContainerInfoExtractor(ILog log, IStartupWorkflow workflow, RunningContainer container) { this.log = log; this.workflow = workflow; @@ -38,45 +36,11 @@ namespace DistTestCore.Marketplace return pubKey; } - public string ExtractMarketplaceAddress() - { - log.Debug(); - var marketplaceAddress = Retry(FetchMarketplaceAddress); - if (string.IsNullOrEmpty(marketplaceAddress)) throw new InvalidOperationException("Unable to fetch marketplace account from codex-contracts node. Test infra failure."); - - return marketplaceAddress; - } - - public string ExtractMarketplaceAbi() - { - log.Debug(); - var marketplaceAbi = Retry(FetchMarketplaceAbi); - if (string.IsNullOrEmpty(marketplaceAbi)) throw new InvalidOperationException("Unable to fetch marketplace artifacts from codex-contracts node. Test infra failure."); - - return marketplaceAbi; - } - private string FetchAccountsCsv() { return workflow.ExecuteCommand(container, "cat", GethContainerRecipe.AccountsFilename); } - private string FetchMarketplaceAddress() - { - var json = workflow.ExecuteCommand(container, "cat", CodexContractsContainerRecipe.MarketplaceAddressFilename); - var marketplace = JsonConvert.DeserializeObject(json); - return marketplace!.address; - } - - private string FetchMarketplaceAbi() - { - var json = workflow.ExecuteCommand(container, "cat", CodexContractsContainerRecipe.MarketplaceArtifactFilename); - - var artifact = JObject.Parse(json); - var abi = artifact["abi"]; - return abi!.ToString(Formatting.None); - } - private string FetchPubKey() { var enodeFinder = new PubKeyFinder(s => log.Debug(s)); @@ -95,7 +59,7 @@ namespace DistTestCore.Marketplace private static string Retry(Func fetch) { - return Time.Retry(fetch, nameof(ContainerInfoExtractor)); + return Time.Retry(fetch, nameof(GethContainerInfoExtractor)); } } @@ -141,9 +105,4 @@ namespace DistTestCore.Marketplace length: closeIndex - openIndex); } } - - public class MarketplaceJson - { - public string address { get; set; } = string.Empty; - } } diff --git a/ProjectPlugins/GethPlugin/GethContainerRecipe.cs b/ProjectPlugins/GethPlugin/GethContainerRecipe.cs new file mode 100644 index 00000000..5214e0b2 --- /dev/null +++ b/ProjectPlugins/GethPlugin/GethContainerRecipe.cs @@ -0,0 +1,61 @@ +using KubernetesWorkflow; + +namespace GethPlugin +{ + public class GethContainerRecipe : ContainerRecipeFactory + { + public static string DockerImage { get; } = "codexstorage/dist-tests-geth:latest"; + private const string defaultArgs = "--ipcdisable --syncmode full"; + + public const string HttpPortTag = "http_port"; + public const string DiscoveryPortTag = "disc_port"; + public const string wsPortTag = "ws_port"; + public const string AccountsFilename = "accounts.csv"; + + public override string AppName => "geth"; + public override string Image => DockerImage; + + protected override void Initialize(StartupConfig startupConfig) + { + var config = startupConfig.Get(); + + var args = CreateArgs(config); + + AddEnvVar("GETH_ARGS", args); + } + + private string CreateArgs(GethStartupConfig config) + { + var discovery = AddInternalPort(tag: DiscoveryPortTag); + + if (config.IsMiner) AddEnvVar("ENABLE_MINER", "1"); + UnlockAccounts(0, 1); + var httpPort = AddExposedPort(tag: HttpPortTag); + var args = $"--http.addr 0.0.0.0 --http.port {httpPort.Number} --port {discovery.Number} --discovery.port {discovery.Number} {defaultArgs}"; + + var authRpc = AddInternalPort(); + var wsPort = AddInternalPort(tag: wsPortTag); + + if (config.BootstrapNode != null) + { + var bootPubKey = config.BootstrapNode.PublicKey; + var bootIp = config.BootstrapNode.IpAddress; + var bootPort = config.BootstrapNode.Port; + var bootstrapArg = $" --bootnodes enode://{bootPubKey}@{bootIp}:{bootPort} --nat=extip:{bootIp}"; + args += bootstrapArg; + } + + return args + $" --authrpc.port {authRpc.Number} --ws --ws.addr 0.0.0.0 --ws.port {wsPort.Number}"; + } + + private void UnlockAccounts(int startIndex, int numberOfAccounts) + { + if (startIndex < 0) throw new ArgumentException(); + if (numberOfAccounts < 1) throw new ArgumentException(); + if (startIndex + numberOfAccounts > 1000) throw new ArgumentException("Out of accounts!"); + + AddEnvVar("UNLOCK_START_INDEX", startIndex.ToString()); + AddEnvVar("UNLOCK_NUMBER", numberOfAccounts.ToString()); + } + } +} diff --git a/ProjectPlugins/GethPlugin/GethDeployment.cs b/ProjectPlugins/GethPlugin/GethDeployment.cs new file mode 100644 index 00000000..b8d96448 --- /dev/null +++ b/ProjectPlugins/GethPlugin/GethDeployment.cs @@ -0,0 +1,25 @@ +using Core; +using KubernetesWorkflow; + +namespace GethPlugin +{ + public class GethDeployment : IHasContainer + { + public GethDeployment(RunningContainer container, Port discoveryPort, Port httpPort, Port wsPort, AllGethAccounts allAccounts, string pubKey) + { + Container = container; + DiscoveryPort = discoveryPort; + HttpPort = httpPort; + WsPort = wsPort; + AllAccounts = allAccounts; + PubKey = pubKey; + } + + public RunningContainer Container { get; } + public Port DiscoveryPort { get; } + public Port HttpPort { get; } + public Port WsPort { get; } + public AllGethAccounts AllAccounts { get; } + public string PubKey { get; } + } +} diff --git a/ProjectPlugins/GethPlugin/GethNode.cs b/ProjectPlugins/GethPlugin/GethNode.cs new file mode 100644 index 00000000..978a852f --- /dev/null +++ b/ProjectPlugins/GethPlugin/GethNode.cs @@ -0,0 +1,93 @@ +using Core; +using KubernetesWorkflow; +using Logging; +using Nethereum.Contracts; +using NethereumWorkflow; + +namespace GethPlugin +{ + public interface IGethNode : IHasContainer + { + GethDeployment StartResult { get; } + + Ether GetEthBalance(); + Ether GetEthBalance(IHasEthAddress address); + Ether GetEthBalance(EthAddress address); + void SendEth(IHasEthAddress account, Ether eth); + void SendEth(EthAddress account, Ether eth); + TResult Call(string contractAddress, TFunction function) where TFunction : FunctionMessage, new(); + void SendTransaction(string contractAddress, TFunction function) where TFunction : FunctionMessage, new(); + decimal? GetSyncedBlockNumber(); + bool IsContractAvailable(string abi, string contractAddress); + } + + public class GethNode : IGethNode + { + private readonly ILog log; + + public GethNode(ILog log, GethDeployment startResult) + { + this.log = log; + StartResult = startResult; + Account = startResult.AllAccounts.Accounts.First(); + } + + public GethDeployment StartResult { get; } + public GethAccount Account { get; } + public RunningContainer Container => StartResult.Container; + + public Ether GetEthBalance() + { + return StartInteraction().GetEthBalance().Eth(); + } + + public Ether GetEthBalance(IHasEthAddress owner) + { + return GetEthBalance(owner.EthAddress); + } + + public Ether GetEthBalance(EthAddress address) + { + return StartInteraction().GetEthBalance(address.Address).Eth(); + } + + public void SendEth(IHasEthAddress owner, Ether eth) + { + SendEth(owner.EthAddress, eth); + } + + public void SendEth(EthAddress account, Ether eth) + { + StartInteraction().SendEth(account.Address, eth.Eth); + } + + public TResult Call(string contractAddress, TFunction function) where TFunction : FunctionMessage, new() + { + return StartInteraction().Call(contractAddress, function); + } + + public void SendTransaction(string contractAddress, TFunction function) where TFunction : FunctionMessage, new() + { + StartInteraction().SendTransaction(contractAddress, function); + } + + private NethereumInteraction StartInteraction() + { + var address = StartResult.Container.Address; + var account = Account; + + var creator = new NethereumInteractionCreator(log, address.Host, address.Port, account.PrivateKey); + return creator.CreateWorkflow(); + } + + public decimal? GetSyncedBlockNumber() + { + return StartInteraction().GetSyncedBlockNumber(); + } + + public bool IsContractAvailable(string abi, string contractAddress) + { + return StartInteraction().IsContractAvailable(abi, contractAddress); + } + } +} diff --git a/ProjectPlugins/GethPlugin/GethPlugin.cs b/ProjectPlugins/GethPlugin/GethPlugin.cs new file mode 100644 index 00000000..bda6845a --- /dev/null +++ b/ProjectPlugins/GethPlugin/GethPlugin.cs @@ -0,0 +1,45 @@ +using Core; + +namespace GethPlugin +{ + public class GethPlugin : IProjectPlugin, IHasLogPrefix, IHasMetadata + { + private readonly GethStarter starter; + private readonly IPluginTools tools; + + public GethPlugin(IPluginTools tools) + { + starter = new GethStarter(tools); + this.tools = tools; + } + + public string LogPrefix => "(Geth) "; + + public void Announce() + { + tools.GetLog().Log($"Loaded Geth plugin."); + } + + public void AddMetadata(IAddMetadata metadata) + { + metadata.Add("gethid", GethContainerRecipe.DockerImage); + } + + public void Decommission() + { + } + + public GethDeployment DeployGeth(Action setup) + { + var startupConfig = new GethStartupConfig(); + setup(startupConfig); + return starter.StartGeth(startupConfig); + } + + public IGethNode WrapGethDeployment(GethDeployment startResult) + { + startResult = SerializeGate.Gate(startResult); + return starter.WrapGethContainer(startResult); + } + } +} diff --git a/ProjectPlugins/GethPlugin/GethPlugin.csproj b/ProjectPlugins/GethPlugin/GethPlugin.csproj new file mode 100644 index 00000000..73aae77b --- /dev/null +++ b/ProjectPlugins/GethPlugin/GethPlugin.csproj @@ -0,0 +1,15 @@ + + + + net7.0 + enable + enable + + + + + + + + + diff --git a/ProjectPlugins/GethPlugin/GethStarter.cs b/ProjectPlugins/GethPlugin/GethStarter.cs new file mode 100644 index 00000000..2cc9ef73 --- /dev/null +++ b/ProjectPlugins/GethPlugin/GethStarter.cs @@ -0,0 +1,55 @@ +using Core; +using KubernetesWorkflow; + +namespace GethPlugin +{ + public class GethStarter + { + private readonly IPluginTools tools; + + public GethStarter(IPluginTools tools) + { + this.tools = tools; + } + + public GethDeployment StartGeth(GethStartupConfig gethStartupConfig) + { + Log("Starting Geth bootstrap node..."); + + var startupConfig = new StartupConfig(); + startupConfig.Add(gethStartupConfig); + startupConfig.NameOverride = gethStartupConfig.NameOverride; + + var workflow = tools.CreateWorkflow(); + var containers = workflow.Start(1, Location.Unspecified, new GethContainerRecipe(), startupConfig); + if (containers.Containers.Length != 1) throw new InvalidOperationException("Expected 1 Geth bootstrap node to be created. Test infra failure."); + var container = containers.Containers[0]; + + var extractor = new GethContainerInfoExtractor(tools.GetLog(), workflow, container); + var accounts = extractor.ExtractAccounts(); + var pubKey = extractor.ExtractPubKey(); + + var discoveryPort = container.Recipe.GetPortByTag(GethContainerRecipe.DiscoveryPortTag); + if (discoveryPort == null) throw new Exception("Expected discovery port to be created."); + var httpPort = container.Recipe.GetPortByTag(GethContainerRecipe.HttpPortTag); + if (httpPort == null) throw new Exception("Expected http port to be created."); + var wsPort = container.Recipe.GetPortByTag(GethContainerRecipe.wsPortTag); + if (wsPort == null) throw new Exception("Expected ws port to be created."); + + Log($"Geth node started."); + + return new GethDeployment(container, discoveryPort, httpPort, wsPort, accounts, pubKey); + } + + public IGethNode WrapGethContainer(GethDeployment startResult) + { + startResult = SerializeGate.Gate(startResult); + return new GethNode(tools.GetLog(), startResult); + } + + private void Log(string msg) + { + tools.GetLog().Log(msg); + } + } +} diff --git a/ProjectPlugins/GethPlugin/GethStartupConfig.cs b/ProjectPlugins/GethPlugin/GethStartupConfig.cs new file mode 100644 index 00000000..fb1991a8 --- /dev/null +++ b/ProjectPlugins/GethPlugin/GethStartupConfig.cs @@ -0,0 +1,48 @@ +namespace GethPlugin +{ + public interface IGethSetup + { + IGethSetup IsMiner(); + IGethSetup WithBootstrapNode(GethBootstrapNode node); + IGethSetup WithName(string name); + } + + public class GethStartupConfig : IGethSetup + { + public bool IsMiner { get; private set; } + public GethBootstrapNode? BootstrapNode { get; private set; } + public string? NameOverride { get; private set; } + + public IGethSetup WithBootstrapNode(GethBootstrapNode node) + { + BootstrapNode = node; + return this; + } + + public IGethSetup WithName(string name) + { + NameOverride = name; + return this; + } + + IGethSetup IGethSetup.IsMiner() + { + IsMiner = true; + return this; + } + } + + public class GethBootstrapNode + { + public GethBootstrapNode(string publicKey, string ipAddress, int port) + { + PublicKey = publicKey; + IpAddress = ipAddress; + Port = port; + } + + public string PublicKey { get; } + public string IpAddress { get; } + public int Port { get; } + } +} diff --git a/ProjectPlugins/MetricsPlugin/CoreInterfaceExtensions.cs b/ProjectPlugins/MetricsPlugin/CoreInterfaceExtensions.cs new file mode 100644 index 00000000..0c320075 --- /dev/null +++ b/ProjectPlugins/MetricsPlugin/CoreInterfaceExtensions.cs @@ -0,0 +1,50 @@ +using Core; +using KubernetesWorkflow; +using Logging; + +namespace MetricsPlugin +{ + public static class CoreInterfaceExtensions + { + public static RunningContainer DeployMetricsCollector(this CoreInterface ci, params IHasMetricsScrapeTarget[] scrapeTargets) + { + return Plugin(ci).DeployMetricsCollector(scrapeTargets.Select(t => t.MetricsScrapeTarget).ToArray()); + } + + public static RunningContainer DeployMetricsCollector(this CoreInterface ci, params IMetricsScrapeTarget[] scrapeTargets) + { + return Plugin(ci).DeployMetricsCollector(scrapeTargets); + } + + public static IMetricsAccess WrapMetricsCollector(this CoreInterface ci, RunningContainer metricsContainer, IMetricsScrapeTarget scrapeTarget) + { + return Plugin(ci).WrapMetricsCollectorDeployment(metricsContainer, scrapeTarget); + } + + public static IMetricsAccess[] GetMetricsFor(this CoreInterface ci, params IHasManyMetricScrapeTargets[] manyScrapeTargets) + { + return ci.GetMetricsFor(manyScrapeTargets.SelectMany(t => t.ScrapeTargets).ToArray()); + } + + public static IMetricsAccess[] GetMetricsFor(this CoreInterface ci, params IHasMetricsScrapeTarget[] scrapeTargets) + { + return ci.GetMetricsFor(scrapeTargets.Select(t => t.MetricsScrapeTarget).ToArray()); + } + + public static IMetricsAccess[] GetMetricsFor(this CoreInterface ci, params IMetricsScrapeTarget[] scrapeTargets) + { + var rc = ci.DeployMetricsCollector(scrapeTargets); + return scrapeTargets.Select(t => ci.WrapMetricsCollector(rc, t)).ToArray(); + } + + public static LogFile? DownloadAllMetrics(this CoreInterface ci, IMetricsAccess metricsAccess, string targetName) + { + return Plugin(ci).DownloadAllMetrics(metricsAccess, targetName); + } + + private static MetricsPlugin Plugin(CoreInterface ci) + { + return ci.GetPlugin(); + } + } +} diff --git a/ProjectPlugins/MetricsPlugin/MetricsAccess.cs b/ProjectPlugins/MetricsPlugin/MetricsAccess.cs new file mode 100644 index 00000000..ae6470cf --- /dev/null +++ b/ProjectPlugins/MetricsPlugin/MetricsAccess.cs @@ -0,0 +1,64 @@ +using Core; +using KubernetesWorkflow; +using Utils; + +namespace MetricsPlugin +{ + public interface IMetricsAccess : IHasContainer + { + string TargetName { get; } + Metrics? GetAllMetrics(); + MetricsSet GetMetric(string metricName); + MetricsSet GetMetric(string metricName, TimeSpan timeout); + } + + public class MetricsAccess : IMetricsAccess + { + private readonly MetricsQuery query; + private readonly IMetricsScrapeTarget target; + + public MetricsAccess(MetricsQuery query, IMetricsScrapeTarget target) + { + this.query = query; + this.target = target; + TargetName = target.Name; + } + + public string TargetName { get; } + public RunningContainer Container => query.RunningContainer; + + public Metrics? GetAllMetrics() + { + return query.GetAllMetricsForNode(target); + } + + public MetricsSet GetMetric(string metricName) + { + return GetMetric(metricName, TimeSpan.FromSeconds(10)); + } + + public MetricsSet GetMetric(string metricName, TimeSpan timeout) + { + var start = DateTime.UtcNow; + + while (true) + { + var mostRecent = GetMostRecent(metricName); + if (mostRecent != null) return mostRecent; + if (DateTime.UtcNow - start > timeout) + { + throw new TimeoutException(); + } + + Time.Sleep(TimeSpan.FromSeconds(2)); + } + } + + private MetricsSet? GetMostRecent(string metricName) + { + var result = query.GetMostRecent(metricName, target); + if (result == null) return null; + return result.Sets.LastOrDefault(); + } + } +} diff --git a/DistTestCore/Metrics/MetricsDownloader.cs b/ProjectPlugins/MetricsPlugin/MetricsDownloader.cs similarity index 84% rename from DistTestCore/Metrics/MetricsDownloader.cs rename to ProjectPlugins/MetricsPlugin/MetricsDownloader.cs index d0d11cdc..1c642a04 100644 --- a/DistTestCore/Metrics/MetricsDownloader.cs +++ b/ProjectPlugins/MetricsPlugin/MetricsDownloader.cs @@ -1,29 +1,29 @@ using Logging; using System.Globalization; -namespace DistTestCore.Metrics +namespace MetricsPlugin { public class MetricsDownloader { - private readonly BaseLog log; + private readonly ILog log; - public MetricsDownloader(BaseLog log) + public MetricsDownloader(ILog log) { this.log = log; } - public void DownloadAllMetricsForNode(string nodeName, MetricsAccess access) + public LogFile? DownloadAllMetrics(string targetName, IMetricsAccess access) { var metrics = access.GetAllMetrics(); - if (metrics == null || metrics.Sets.Length == 0 || metrics.Sets.All(s => s.Values.Length == 0)) return; + if (metrics == null || metrics.Sets.Length == 0 || metrics.Sets.All(s => s.Values.Length == 0)) return null; var headers = new[] { "timestamp" }.Concat(metrics.Sets.Select(s => s.Name)).ToArray(); var map = CreateValueMap(metrics); - WriteToFile(nodeName, headers, map); + return WriteToFile(targetName, headers, map); } - private void WriteToFile(string nodeName, string[] headers, Dictionary> map) + private LogFile WriteToFile(string nodeName, string[] headers, Dictionary> map) { var file = log.CreateSubfile("csv"); log.Log($"Downloading metrics for {nodeName} to file {file.FullFilename}"); @@ -34,6 +34,8 @@ namespace DistTestCore.Metrics { file.WriteRaw(string.Join(",", new[] { FormatTimestamp(pair.Key) }.Concat(pair.Value))); } + + return file; } private Dictionary> CreateValueMap(Metrics metrics) diff --git a/ProjectPlugins/MetricsPlugin/MetricsPlugin.cs b/ProjectPlugins/MetricsPlugin/MetricsPlugin.cs new file mode 100644 index 00000000..4e39ffc8 --- /dev/null +++ b/ProjectPlugins/MetricsPlugin/MetricsPlugin.cs @@ -0,0 +1,51 @@ +using Core; +using KubernetesWorkflow; +using Logging; + +namespace MetricsPlugin +{ + public class MetricsPlugin : IProjectPlugin, IHasLogPrefix, IHasMetadata + { + private readonly IPluginTools tools; + private readonly PrometheusStarter starter; + + public MetricsPlugin(IPluginTools tools) + { + this.tools = tools; + starter = new PrometheusStarter(tools); + } + + public string LogPrefix => "(Metrics) "; + + public void Announce() + { + tools.GetLog().Log($"Prometheus plugin loaded with '{starter.GetPrometheusId()}'."); + } + + public void AddMetadata(IAddMetadata metadata) + { + metadata.Add("prometheusid", starter.GetPrometheusId()); + } + + public void Decommission() + { + } + + public RunningContainer DeployMetricsCollector(IMetricsScrapeTarget[] scrapeTargets) + { + return starter.CollectMetricsFor(scrapeTargets); + } + + public IMetricsAccess WrapMetricsCollectorDeployment(RunningContainer runningContainer, IMetricsScrapeTarget target) + { + runningContainer = SerializeGate.Gate(runningContainer); + return starter.CreateAccessForTarget(runningContainer, target); + } + + public LogFile? DownloadAllMetrics(IMetricsAccess metricsAccess, string targetName) + { + var downloader = new MetricsDownloader(tools.GetLog()); + return downloader.DownloadAllMetrics(targetName, metricsAccess); + } + } +} diff --git a/ProjectPlugins/MetricsPlugin/MetricsPlugin.csproj b/ProjectPlugins/MetricsPlugin/MetricsPlugin.csproj new file mode 100644 index 00000000..f9219345 --- /dev/null +++ b/ProjectPlugins/MetricsPlugin/MetricsPlugin.csproj @@ -0,0 +1,23 @@ + + + + net7.0 + enable + enable + + + + + + + + + Never + + + + + + + + diff --git a/DistTestCore/Metrics/MetricsQuery.cs b/ProjectPlugins/MetricsPlugin/MetricsQuery.cs similarity index 81% rename from DistTestCore/Metrics/MetricsQuery.cs rename to ProjectPlugins/MetricsPlugin/MetricsQuery.cs index baffe66f..7d634776 100644 --- a/DistTestCore/Metrics/MetricsQuery.cs +++ b/ProjectPlugins/MetricsPlugin/MetricsQuery.cs @@ -1,31 +1,24 @@ -using DistTestCore.Codex; +using Core; using KubernetesWorkflow; using System.Globalization; -namespace DistTestCore.Metrics +namespace MetricsPlugin { public class MetricsQuery { - private readonly Http http; + private readonly IHttp http; - public MetricsQuery(TestLifecycle lifecycle, RunningContainers runningContainers) + public MetricsQuery(IPluginTools tools, RunningContainer runningContainer) { - RunningContainers = runningContainers; - - var address = lifecycle.Configuration.GetAddress(runningContainers.Containers[0]); - - http = new Http( - lifecycle.Log, - lifecycle.TimeSet, - address, - "api/v1"); + RunningContainer = runningContainer; + http = tools.CreateHttp(RunningContainer.Address, "api/v1"); } - public RunningContainers RunningContainers { get; } + public RunningContainer RunningContainer { get; } - public Metrics? GetMostRecent(string metricName, RunningContainer node) + public Metrics? GetMostRecent(string metricName, IMetricsScrapeTarget target) { - var response = GetLastOverTime(metricName, GetInstanceStringForNode(node)); + var response = GetLastOverTime(metricName, GetInstanceStringForNode(target)); if (response == null) return null; return new Metrics @@ -48,9 +41,9 @@ namespace DistTestCore.Metrics return MapResponseToMetrics(response); } - public Metrics? GetAllMetricsForNode(RunningContainer node) + public Metrics? GetAllMetricsForNode(IMetricsScrapeTarget target) { - var response = http.HttpGetJson($"query?query={GetInstanceStringForNode(node)}{GetQueryTimeRange()}"); + var response = http.HttpGetJson($"query?query={GetInstanceStringForNode(target)}{GetQueryTimeRange()}"); if (response.status != "success") return null; return MapResponseToMetrics(response); } @@ -117,16 +110,14 @@ namespace DistTestCore.Metrics }; } - private string GetInstanceNameForNode(RunningContainer node) + private string GetInstanceNameForNode(IMetricsScrapeTarget target) { - var ip = node.Pod.PodInfo.Ip; - var port = node.Recipe.GetPortByTag(CodexContainerRecipe.MetricsPortTag).Number; - return $"{ip}:{port}"; + return $"{target.Ip}:{target.Port}"; } - private string GetInstanceStringForNode(RunningContainer node) + private string GetInstanceStringForNode(IMetricsScrapeTarget target) { - return "{instance=\"" + GetInstanceNameForNode(node) + "\"}"; + return "{instance=\"" + GetInstanceNameForNode(target) + "\"}"; } private string GetQueryTimeRange() diff --git a/ProjectPlugins/MetricsPlugin/MetricsScrapeTarget.cs b/ProjectPlugins/MetricsPlugin/MetricsScrapeTarget.cs new file mode 100644 index 00000000..2c7bda61 --- /dev/null +++ b/ProjectPlugins/MetricsPlugin/MetricsScrapeTarget.cs @@ -0,0 +1,45 @@ +using KubernetesWorkflow; + +namespace MetricsPlugin +{ + public interface IMetricsScrapeTarget + { + string Name { get; } + string Ip { get; } + int Port { get; } + } + + public interface IHasMetricsScrapeTarget + { + IMetricsScrapeTarget MetricsScrapeTarget { get; } + } + + public interface IHasManyMetricScrapeTargets + { + IMetricsScrapeTarget[] ScrapeTargets { get; } + } + + public class MetricsScrapeTarget : IMetricsScrapeTarget + { + public MetricsScrapeTarget(string ip, int port, string name) + { + Ip = ip; + Port = port; + Name = name; + } + + public MetricsScrapeTarget(RunningContainer container, int port) + : this(container.Pod.PodInfo.Ip, port, container.Name) + { + } + + public MetricsScrapeTarget(RunningContainer container, Port port) + : this(container, port.Number) + { + } + + public string Name { get; } + public string Ip { get; } + public int Port { get; } + } +} diff --git a/DistTestCore/Metrics/PrometheusContainerRecipe.cs b/ProjectPlugins/MetricsPlugin/PrometheusContainerRecipe.cs similarity index 69% rename from DistTestCore/Metrics/PrometheusContainerRecipe.cs rename to ProjectPlugins/MetricsPlugin/PrometheusContainerRecipe.cs index 62287121..26b9b48a 100644 --- a/DistTestCore/Metrics/PrometheusContainerRecipe.cs +++ b/ProjectPlugins/MetricsPlugin/PrometheusContainerRecipe.cs @@ -1,13 +1,13 @@ using KubernetesWorkflow; -namespace DistTestCore.Metrics +namespace MetricsPlugin { - public class PrometheusContainerRecipe : DefaultContainerRecipe + public class PrometheusContainerRecipe : ContainerRecipeFactory { public override string AppName => "prometheus"; public override string Image => "codexstorage/dist-tests-prometheus:latest"; - protected override void InitializeRecipe(StartupConfig startupConfig) + protected override void Initialize(StartupConfig startupConfig) { var config = startupConfig.Get(); diff --git a/ProjectPlugins/MetricsPlugin/PrometheusStarter.cs b/ProjectPlugins/MetricsPlugin/PrometheusStarter.cs new file mode 100644 index 00000000..b1b79a1d --- /dev/null +++ b/ProjectPlugins/MetricsPlugin/PrometheusStarter.cs @@ -0,0 +1,69 @@ +using Core; +using KubernetesWorkflow; +using System.Text; + +namespace MetricsPlugin +{ + public class PrometheusStarter + { + private readonly PrometheusContainerRecipe recipe = new PrometheusContainerRecipe(); + private readonly IPluginTools tools; + + public PrometheusStarter(IPluginTools tools) + { + this.tools = tools; + } + + public RunningContainer CollectMetricsFor(IMetricsScrapeTarget[] targets) + { + Log($"Starting metrics server for {targets.Length} targets..."); + var startupConfig = new StartupConfig(); + startupConfig.Add(new PrometheusStartupConfig(GeneratePrometheusConfig(targets))); + + var workflow = tools.CreateWorkflow(); + var runningContainers = workflow.Start(1, Location.Unspecified, recipe, startupConfig); + if (runningContainers.Containers.Length != 1) throw new InvalidOperationException("Expected only 1 Prometheus container to be created."); + + Log("Metrics server started."); + return runningContainers.Containers.Single(); + } + + public MetricsAccess CreateAccessForTarget(RunningContainer metricsContainer, IMetricsScrapeTarget target) + { + var metricsQuery = new MetricsQuery(tools, metricsContainer); + return new MetricsAccess(metricsQuery, target); + } + + public string GetPrometheusId() + { + return recipe.Image; + } + + private void Log(string msg) + { + tools.GetLog().Log(msg); + } + + private static string GeneratePrometheusConfig(IMetricsScrapeTarget[] targets) + { + var config = ""; + config += "global:\n"; + config += " scrape_interval: 10s\n"; + config += " scrape_timeout: 10s\n"; + config += "\n"; + config += "scrape_configs:\n"; + config += " - job_name: services\n"; + config += " metrics_path: /metrics\n"; + config += " static_configs:\n"; + config += " - targets:\n"; + + foreach (var target in targets) + { + config += $" - '{target.Ip}:{target.Port}'\n"; + } + + var bytes = Encoding.ASCII.GetBytes(config); + return Convert.ToBase64String(bytes); + } + } +} diff --git a/DistTestCore/Metrics/PrometheusStartupConfig.cs b/ProjectPlugins/MetricsPlugin/PrometheusStartupConfig.cs similarity index 88% rename from DistTestCore/Metrics/PrometheusStartupConfig.cs rename to ProjectPlugins/MetricsPlugin/PrometheusStartupConfig.cs index 7bf7fe62..a4904205 100644 --- a/DistTestCore/Metrics/PrometheusStartupConfig.cs +++ b/ProjectPlugins/MetricsPlugin/PrometheusStartupConfig.cs @@ -1,4 +1,4 @@ -namespace DistTestCore.Metrics +namespace MetricsPlugin { public class PrometheusStartupConfig { diff --git a/DistTestCore/Metrics/dashboard.json b/ProjectPlugins/MetricsPlugin/dashboard.json similarity index 100% rename from DistTestCore/Metrics/dashboard.json rename to ProjectPlugins/MetricsPlugin/dashboard.json diff --git a/README.md b/README.md index 5141e9cb..ee26ed46 100644 --- a/README.md +++ b/README.md @@ -1,37 +1,30 @@ # Distributed System Tests for Nim-Codex - Using a common dotnet unit-test framework and a few other libraries, this project allows you to write tests that use multiple Codex node instances in various configurations to test the distributed system in a controlled, reproducible environment. Nim-Codex: https://github.com/codex-storage/nim-codex -Dotnet: v6.0 +Dotnet: v7.0 Kubernetes: v1.25.4 Dotnet-kubernetes SDK: v10.1.4 https://github.com/kubernetes-client/csharp Nethereum: v4.14.0 -## Tests -Tests are devided into two assemblies: `/Tests` and `/LongTests`. -`/Tests` is to be used for tests that take several minutes to hours to execute. -`/LongTests` is to be used for tests that take hours to days to execute. +## Tests/CodexTests and Tests/CodexLongTests +These are test assemblies that use NUnit3 to perform tests against transient Codex nodes. -TODO: All tests will eventually be running as part of a dedicated CI pipeline and kubernetes cluster. Currently, we're developing these tests and the infra-code to support it by running the whole thing locally. +## Tests/ContinousTests +A console application that runs tests in an endless loop against a persistent deployment of Codex nodes. -## Configuration -Test executing can be configured using the following environment variables. -| Variable | Description | Default | -|----------------|------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------| -| KUBECONFIG | Optional path (abs or rel) to kubeconfig YAML file. When null, uses system default (docker-desktop) kubeconfig if available. | (null) | -| LOGPATH | Path (abs or rel) where log files will be saved. | "CodexTestLogs" | -| LOGDEBUG | When "true", enables additional test-runner debug log output. | "false" | -| DATAFILEPATH | Path (abs or rel) where temporary test data files will be saved. | "TestDataFiles" | -| LOGLEVEL | Codex log-level. (case-insensitive) | "Trace" | -| RUNNERLOCATION | Use "ExternalToCluster" when test app is running outside of the k8s cluster. Use "InternalToCluster" when tests are run from inside a pod/container. | "ExternalToCluster" | +## Tools/CodexNetDeployer +A console application that can deploy Codex nodes. ## Test logs Because tests potentially take a long time to run, logging is in place to help you investigate failures afterwards. Should a test fail, all Codex terminal output (as well as metrics if they have been enabled) will be downloaded and stored along with a detailed, step-by-step log of the test. If something's gone wrong and you're here to discover the details, head for the logs. +## How to contribute a plugin +If you want to add support for your project to the testing framework, follow the steps [HERE](/CONTRIBUTINGPLUGINS.MD) + ## How to contribute tests -An important goal of the test infra is to provide a simple, accessible way for developers to write their tests. If you want to contribute tests for Codex, please follow the steps [HERE](/CONTRIBUTINGTESTS.md). +If you want to contribute tests, please follow the steps [HERE](/CONTRIBUTINGTESTS.md). ## Run the tests on your machine Creating tests is much easier when you can debug them on your local system. This is possible, but requires some set-up. If you want to be able to run the tests on your local system, follow the steps [HERE](/docs/LOCALSETUP.md). Please note that tests which require explicit node locations cannot be executed locally. (Well, you could comment out the location statements and then it would probably work. But that might impact the validity/usefulness of the test.) diff --git a/Tests/BasicTests/ExampleTests.cs b/Tests/BasicTests/ExampleTests.cs deleted file mode 100644 index 1492ce1f..00000000 --- a/Tests/BasicTests/ExampleTests.cs +++ /dev/null @@ -1,86 +0,0 @@ -using DistTestCore; -using NUnit.Framework; -using Utils; - -namespace Tests.BasicTests -{ - [TestFixture] - public class ExampleTests : DistTest - { - [Test] - public void CodexLogExample() - { - var primary = SetupCodexNode(); - - primary.UploadFile(GenerateTestFile(5.MB())); - - var log = primary.DownloadLog(); - - log.AssertLogContains("Uploaded file"); - } - - [Test] - public void TwoMetricsExample() - { - var group = SetupCodexNodes(2, s => s.EnableMetrics()); - var group2 = SetupCodexNodes(2, s => s.EnableMetrics()); - - var primary = group[0]; - var secondary = group[1]; - var primary2 = group2[0]; - var secondary2 = group2[1]; - - primary.ConnectToPeer(secondary); - primary2.ConnectToPeer(secondary2); - - Thread.Sleep(TimeSpan.FromMinutes(2)); - - primary.Metrics.AssertThat("libp2p_peers", Is.EqualTo(1)); - primary2.Metrics.AssertThat("libp2p_peers", Is.EqualTo(1)); - } - - [Test] - public void MarketplaceExample() - { - var sellerInitialBalance = 234.TestTokens(); - var buyerInitialBalance = 1000.TestTokens(); - var fileSize = 10.MB(); - - var seller = SetupCodexNode(s => s - .WithStorageQuota(11.GB()) - .EnableMarketplace(sellerInitialBalance)); - - seller.Marketplace.AssertThatBalance(Is.EqualTo(sellerInitialBalance)); - seller.Marketplace.MakeStorageAvailable( - size: 10.GB(), - minPricePerBytePerSecond: 1.TestTokens(), - maxCollateral: 20.TestTokens(), - maxDuration: TimeSpan.FromMinutes(3)); - - var testFile = GenerateTestFile(fileSize); - - var buyer = SetupCodexNode(s => s - .WithBootstrapNode(seller) - .EnableMarketplace(buyerInitialBalance)); - - buyer.Marketplace.AssertThatBalance(Is.EqualTo(buyerInitialBalance)); - - var contentId = buyer.UploadFile(testFile); - var purchaseContract = buyer.Marketplace.RequestStorage(contentId, - pricePerSlotPerSecond: 2.TestTokens(), - requiredCollateral: 10.TestTokens(), - minRequiredNumberOfNodes: 1, - proofProbability: 5, - duration: TimeSpan.FromMinutes(1)); - - purchaseContract.WaitForStorageContractStarted(fileSize); - - seller.Marketplace.AssertThatBalance(Is.LessThan(sellerInitialBalance), "Collateral was not placed."); - - purchaseContract.WaitForStorageContractFinished(); - - seller.Marketplace.AssertThatBalance(Is.GreaterThan(sellerInitialBalance), "Seller was not paid for storage."); - buyer.Marketplace.AssertThatBalance(Is.LessThan(buyerInitialBalance), "Buyer was not charged for storage."); - } - } -} diff --git a/ContinuousTests/ContinuousTests.csproj b/Tests/CodexContinuousTests/CodexContinuousTests.csproj similarity index 64% rename from ContinuousTests/ContinuousTests.csproj rename to Tests/CodexContinuousTests/CodexContinuousTests.csproj index 5543d011..de07fe48 100644 --- a/ContinuousTests/ContinuousTests.csproj +++ b/Tests/CodexContinuousTests/CodexContinuousTests.csproj @@ -12,10 +12,10 @@ - + + + - - diff --git a/ContinuousTests/Configuration.cs b/Tests/CodexContinuousTests/Configuration.cs similarity index 91% rename from ContinuousTests/Configuration.cs rename to Tests/CodexContinuousTests/Configuration.cs index 04491a9b..50ef578d 100644 --- a/ContinuousTests/Configuration.cs +++ b/Tests/CodexContinuousTests/Configuration.cs @@ -1,6 +1,5 @@ using ArgsUniform; -using DistTestCore; -using DistTestCore.Codex; +using CodexPlugin; using Newtonsoft.Json; namespace ContinuousTests @@ -29,8 +28,6 @@ namespace ContinuousTests public bool DownloadContainerLogs { get; set; } = false; public CodexDeployment CodexDeployment { get; set; } = null!; - - public RunnerLocation RunnerLocation { get; set; } } public class ConfigLoader @@ -40,10 +37,7 @@ namespace ContinuousTests var uniformArgs = new ArgsUniform(PrintHelp, args); var result = uniformArgs.Parse(true); - result.CodexDeployment = ParseCodexDeploymentJson(result.CodexDeploymentJson); - result.RunnerLocation = RunnerLocationUtils.DetermineRunnerLocation(result.CodexDeployment.CodexContainers.First()); - return result; } diff --git a/Tests/CodexContinuousTests/ContinuousTest.cs b/Tests/CodexContinuousTests/ContinuousTest.cs new file mode 100644 index 00000000..2bef6dba --- /dev/null +++ b/Tests/CodexContinuousTests/ContinuousTest.cs @@ -0,0 +1,69 @@ +using CodexPlugin; +using Core; +using DistTestCore; +using FileUtils; +using Logging; + +namespace ContinuousTests +{ + public abstract class ContinuousTestLongTimeouts : ContinuousTest + { + public override ITimeSet TimeSet => new LongTimeSet(); + } + + public abstract class ContinuousTest + { + protected const int Zero = 0; + protected const int MinuteOne = 60; + protected const int MinuteFive = MinuteOne * 5; + protected const int HourOne = MinuteOne * 60; + protected const int HourThree = HourOne * 3; + protected const int DayOne = HourOne * 24; + protected const int DayThree = DayOne * 3; + + public void Initialize(ICodexNode[] nodes, ILog log, IFileManager fileManager, Configuration configuration, CancellationToken cancelToken) + { + Nodes = nodes; + Log = log; + FileManager = fileManager; + Configuration = configuration; + CancelToken = cancelToken; + + if (nodes != null) + { + NodeRunner = new NodeRunner(Nodes, configuration, Log, CustomK8sNamespace); + } + else + { + NodeRunner = null!; + } + } + + public ICodexNode[] Nodes { get; private set; } = null!; + public ILog Log { get; private set; } = null!; + public IFileManager FileManager { get; private set; } = null!; + public Configuration Configuration { get; private set; } = null!; + public virtual ITimeSet TimeSet { get { return new DefaultTimeSet(); } } + public CancellationToken CancelToken { get; private set; } = new CancellationToken(); + public NodeRunner NodeRunner { get; private set; } = null!; + + public abstract int RequiredNumberOfNodes { get; } + public abstract TimeSpan RunTestEvery { get; } + public abstract TestFailMode TestFailMode { get; } + public virtual string CustomK8sNamespace { get { return string.Empty; } } + + public string Name + { + get + { + return GetType().Name; + } + } + } + + public enum TestFailMode + { + StopAfterFirstFailure, + AlwaysRunAllMoments + } +} diff --git a/ContinuousTests/ContinuousTestRunner.cs b/Tests/CodexContinuousTests/ContinuousTestRunner.cs similarity index 57% rename from ContinuousTests/ContinuousTestRunner.cs rename to Tests/CodexContinuousTests/ContinuousTestRunner.cs index fb69e051..a0589b26 100644 --- a/ContinuousTests/ContinuousTestRunner.cs +++ b/Tests/CodexContinuousTests/ContinuousTestRunner.cs @@ -1,36 +1,47 @@ -using DistTestCore; +using DistTestCore.Logs; using Logging; namespace ContinuousTests { public class ContinuousTestRunner { - private readonly K8sFactory k8SFactory = new K8sFactory(); + private readonly EntryPointFactory entryPointFactory = new EntryPointFactory(); private readonly ConfigLoader configLoader = new ConfigLoader(); private readonly TestFactory testFactory = new TestFactory(); private readonly Configuration config; - private readonly StartupChecker startupChecker; private readonly CancellationToken cancelToken; public ContinuousTestRunner(string[] args, CancellationToken cancelToken) { config = configLoader.Load(args); - startupChecker = new StartupChecker(config, cancelToken); this.cancelToken = cancelToken; } public void Run() { + var overviewLog = new LogSplitter( + new FixtureLog(new LogConfig(config.LogPath, false), DateTime.UtcNow, "Overview"), + new ConsoleLog() + ); + + overviewLog.Log("Initializing..."); + + var entryPoint = entryPointFactory.CreateEntryPoint(config.KubeConfigFile, config.DataPath, config.CodexDeployment.Metadata.KubeNamespace, overviewLog); + entryPoint.Announce(); + + overviewLog.Log("Initialized. Performing startup checks..."); + + var startupChecker = new StartupChecker(entryPoint, config, cancelToken); startupChecker.Check(); var taskFactory = new TaskFactory(); - var overviewLog = new FixtureLog(new LogConfig(config.LogPath, false), DateTime.UtcNow, "Overview"); - overviewLog.Log("Continuous tests starting..."); + overviewLog.Log("Startup checks passed. Continuous tests starting..."); + overviewLog.Log(""); var allTests = testFactory.CreateTests(); ClearAllCustomNamespaces(allTests, overviewLog); - var testLoops = allTests.Select(t => new TestLoop(taskFactory, config, overviewLog, t.GetType(), t.RunTestEvery, startupChecker, cancelToken)).ToArray(); + var testLoops = allTests.Select(t => new TestLoop(entryPointFactory, taskFactory, config, overviewLog, t.GetType(), t.RunTestEvery, startupChecker, cancelToken)).ToArray(); foreach (var testLoop in testLoops) { @@ -48,18 +59,19 @@ namespace ContinuousTests overviewLog.Log("All tasks cancelled."); } - private void ClearAllCustomNamespaces(ContinuousTest[] allTests, FixtureLog log) + private void ClearAllCustomNamespaces(ContinuousTest[] allTests, ILog log) { foreach (var test in allTests) ClearAllCustomNamespaces(test, log); } - private void ClearAllCustomNamespaces(ContinuousTest test, FixtureLog log) + private void ClearAllCustomNamespaces(ContinuousTest test, ILog log) { if (string.IsNullOrEmpty(test.CustomK8sNamespace)) return; log.Log($"Clearing namespace '{test.CustomK8sNamespace}'..."); - var lifecycle = k8SFactory.CreateTestLifecycle(config.KubeConfigFile, config.LogPath, config.DataPath, test.CustomK8sNamespace, new DefaultTimeSet(), log); - lifecycle.WorkflowCreator.CreateWorkflow().DeleteTestResources(); + + var entryPoint = entryPointFactory.CreateEntryPoint(config.KubeConfigFile, config.DataPath, test.CustomK8sNamespace, log); + entryPoint.Tools.CreateWorkflow().DeleteNamespacesStartingWith(test.CustomK8sNamespace); } } } diff --git a/Tests/CodexContinuousTests/EntryPointFactory.cs b/Tests/CodexContinuousTests/EntryPointFactory.cs new file mode 100644 index 00000000..90b92b43 --- /dev/null +++ b/Tests/CodexContinuousTests/EntryPointFactory.cs @@ -0,0 +1,37 @@ +using Logging; +using Core; + +namespace ContinuousTests +{ + public class EntryPointFactory + { + public EntryPointFactory() + { + ProjectPlugin.Load(); + ProjectPlugin.Load(); + ProjectPlugin.Load(); + ProjectPlugin.Load(); + } + + public EntryPoint CreateEntryPoint(string kubeConfigFile, string dataFilePath, string customNamespace, ILog log) + { + var kubeConfig = GetKubeConfig(kubeConfigFile); + var lifecycleConfig = new KubernetesWorkflow.Configuration + ( + kubeConfigFile: kubeConfig, + operationTimeout: TimeSpan.FromSeconds(30), + retryDelay: TimeSpan.FromSeconds(10), + kubernetesNamespace: customNamespace + ); + + return new EntryPoint(log, lifecycleConfig, dataFilePath); + //DefaultContainerRecipe.TestsType = "continuous-tests"; + } + + private static string? GetKubeConfig(string kubeConfigFile) + { + if (string.IsNullOrEmpty(kubeConfigFile) || kubeConfigFile.ToLowerInvariant() == "null") return null; + return kubeConfigFile; + } + } +} diff --git a/Tests/CodexContinuousTests/NodeRunner.cs b/Tests/CodexContinuousTests/NodeRunner.cs new file mode 100644 index 00000000..85f01b29 --- /dev/null +++ b/Tests/CodexContinuousTests/NodeRunner.cs @@ -0,0 +1,73 @@ +using KubernetesWorkflow; +using NUnit.Framework; +using Logging; +using Utils; +using Core; +using CodexPlugin; + +namespace ContinuousTests +{ + public class NodeRunner + { + private readonly EntryPointFactory entryPointFactory = new EntryPointFactory(); + private readonly ICodexNode[] nodes; + private readonly Configuration config; + private readonly ILog log; + private readonly string customNamespace; + + public NodeRunner(ICodexNode[] nodes, Configuration config, ILog log, string customNamespace) + { + this.nodes = nodes; + this.config = config; + this.log = log; + this.customNamespace = customNamespace; + } + + public IDownloadedLog DownloadLog(RunningContainer container, int? tailLines = null) + { + var entryPoint = CreateEntryPoint(); + return entryPoint.CreateInterface().DownloadLog(container, tailLines); + } + + public void RunNode(Action setup, Action operation) + { + RunNode(nodes.ToList().PickOneRandom(), setup, operation); + } + + public void RunNode(ICodexNode bootstrapNode, Action setup, Action operation) + { + var entryPoint = CreateEntryPoint(); + + try + { + var debugInfo = bootstrapNode.GetDebugInfo(); + Assert.That(!string.IsNullOrEmpty(debugInfo.spr)); + + var node = entryPoint.CreateInterface().StartCodexNode(s => + { + setup(s); + s.WithBootstrapNode(bootstrapNode); + }); + + try + { + operation(node); + } + catch + { + DownloadLog(node.Container); + throw; + } + } + finally + { + entryPoint.Tools.CreateWorkflow().DeleteNamespace(); + } + } + + private EntryPoint CreateEntryPoint() + { + return entryPointFactory.CreateEntryPoint(config.KubeConfigFile, config.DataPath, customNamespace, log); + } + } +} diff --git a/ContinuousTests/Program.cs b/Tests/CodexContinuousTests/Program.cs similarity index 68% rename from ContinuousTests/Program.cs rename to Tests/CodexContinuousTests/Program.cs index 273f13a1..3db3b15e 100644 --- a/ContinuousTests/Program.cs +++ b/Tests/CodexContinuousTests/Program.cs @@ -5,7 +5,6 @@ public class Program public static void Main(string[] args) { Console.WriteLine("Codex Continous-Test-Runner."); - Console.WriteLine("Running..."); var runner = new ContinuousTestRunner(args, Cancellation.Cts.Token); @@ -20,14 +19,14 @@ public class Program runner.Run(); Console.WriteLine("Done."); } - - public static class Cancellation - { - static Cancellation() - { - Cts = new CancellationTokenSource(); - } - - public static CancellationTokenSource Cts { get; } - } +} + +public static class Cancellation +{ + static Cancellation() + { + Cts = new CancellationTokenSource(); + } + + public static CancellationTokenSource Cts { get; } } diff --git a/ContinuousTests/SingleTestRun.cs b/Tests/CodexContinuousTests/SingleTestRun.cs similarity index 82% rename from ContinuousTests/SingleTestRun.cs rename to Tests/CodexContinuousTests/SingleTestRun.cs index 1e2b3a44..7a233db4 100644 --- a/ContinuousTests/SingleTestRun.cs +++ b/Tests/CodexContinuousTests/SingleTestRun.cs @@ -1,31 +1,29 @@ -using DistTestCore.Codex; -using DistTestCore; -using Logging; +using Logging; using Utils; using KubernetesWorkflow; using NUnit.Framework.Internal; using System.Reflection; -using static Program; +using CodexPlugin; +using DistTestCore.Logs; +using Core; namespace ContinuousTests { public class SingleTestRun { - private readonly CodexAccessFactory codexNodeFactory = new CodexAccessFactory(); private readonly List exceptions = new List(); + private readonly EntryPoint entryPoint; private readonly TaskFactory taskFactory; private readonly Configuration config; - private readonly BaseLog overviewLog; + private readonly ILog overviewLog; private readonly TestHandle handle; private readonly CancellationToken cancelToken; - private readonly CodexAccess[] nodes; - private readonly FileManager fileManager; + private readonly ICodexNode[] nodes; private readonly FixtureLog fixtureLog; private readonly string testName; - private readonly string dataFolder; private static int failureCount = 0; - public SingleTestRun(TaskFactory taskFactory, Configuration config, BaseLog overviewLog, TestHandle handle, StartupChecker startupChecker, CancellationToken cancelToken) + public SingleTestRun(EntryPointFactory entryPointFactory, TaskFactory taskFactory, Configuration config, ILog overviewLog, TestHandle handle, StartupChecker startupChecker, CancellationToken cancelToken) { this.taskFactory = taskFactory; this.config = config; @@ -34,11 +32,10 @@ namespace ContinuousTests this.cancelToken = cancelToken; testName = handle.Test.GetType().Name; fixtureLog = new FixtureLog(new LogConfig(config.LogPath, true), DateTime.UtcNow, testName); + entryPoint = entryPointFactory.CreateEntryPoint(config.KubeConfigFile, config.DataPath, config.CodexDeployment.Metadata.KubeNamespace, fixtureLog); ApplyLogReplacements(fixtureLog, startupChecker); nodes = CreateRandomNodes(); - dataFolder = config.DataPath + "-" + Guid.NewGuid(); - fileManager = new FileManager(fixtureLog, CreateFileManagerConfiguration()); } public void Run(EventWaitHandle runFinishedHandle) @@ -48,8 +45,11 @@ namespace ContinuousTests try { RunTest(); - fileManager.DeleteAllTestFiles(); - Directory.Delete(dataFolder, true); + + entryPoint.Decommission( + deleteKubernetesResources: false, // This would delete the continuous test net. + deleteTrackedFiles: true + ); runFinishedHandle.Set(); } catch (Exception ex) @@ -141,14 +141,14 @@ namespace ContinuousTests private void DownloadClusterLogs() { - var k8sFactory = new K8sFactory(); + var entryPointFactory = new EntryPointFactory(); var log = new NullLog(); log.FullFilename = Path.Combine(config.LogPath, "NODE"); - var lifecycle = k8sFactory.CreateTestLifecycle(config.KubeConfigFile, config.LogPath, "dataPath", config.CodexDeployment.Metadata.KubeNamespace, new DefaultTimeSet(), log); + var entryPoint = entryPointFactory.CreateEntryPoint(config.KubeConfigFile, config.DataPath, config.CodexDeployment.Metadata.KubeNamespace, log); foreach (var container in config.CodexDeployment.CodexContainers) { - lifecycle.DownloadLog(container); + entryPoint.CreateInterface().DownloadLog(container); } } @@ -196,7 +196,7 @@ namespace ContinuousTests private void InitializeTest(string name) { Log($" > Running TestMoment '{name}'"); - handle.Test.Initialize(nodes, fixtureLog, fileManager, config, cancelToken); + handle.Test.Initialize(nodes, fixtureLog, entryPoint.Tools.GetFileManager(), config, cancelToken); } private void DecommissionTest() @@ -222,11 +222,11 @@ namespace ContinuousTests return $"({string.Join(",", nodes.Select(n => n.Container.Name))})"; } - private CodexAccess[] CreateRandomNodes() + private ICodexNode[] CreateRandomNodes() { var containers = SelectRandomContainers(); fixtureLog.Log("Selected nodes: " + string.Join(",", containers.Select(c => c.Name))); - return codexNodeFactory.Create(config, containers, fixtureLog, handle.Test.TimeSet); + return entryPoint.CreateInterface().WrapCodexContainers(containers).ToArray(); } private RunningContainer[] SelectRandomContainers() @@ -242,11 +242,5 @@ namespace ContinuousTests } return result; } - - private DistTestCore.Configuration CreateFileManagerConfiguration() - { - return new DistTestCore.Configuration(null, string.Empty, false, dataFolder, - CodexLogLevel.Error, string.Empty); - } } } diff --git a/ContinuousTests/StartupChecker.cs b/Tests/CodexContinuousTests/StartupChecker.cs similarity index 76% rename from ContinuousTests/StartupChecker.cs rename to Tests/CodexContinuousTests/StartupChecker.cs index 2fee5ebc..e90f93d8 100644 --- a/ContinuousTests/StartupChecker.cs +++ b/Tests/CodexContinuousTests/StartupChecker.cs @@ -1,5 +1,6 @@ -using DistTestCore.Codex; -using DistTestCore; +using CodexPlugin; +using Core; +using DistTestCore.Logs; using Logging; namespace ContinuousTests @@ -7,12 +8,13 @@ namespace ContinuousTests public class StartupChecker { private readonly TestFactory testFactory = new TestFactory(); - private readonly CodexAccessFactory codexNodeFactory = new CodexAccessFactory(); + private readonly EntryPoint entryPoint; private readonly Configuration config; private readonly CancellationToken cancelToken; - public StartupChecker(Configuration config, CancellationToken cancelToken) + public StartupChecker(EntryPoint entryPoint, Configuration config, CancellationToken cancelToken) { + this.entryPoint = entryPoint; this.config = config; this.cancelToken = cancelToken; LogReplacements = new List(); @@ -61,13 +63,13 @@ namespace ContinuousTests private void CheckCodexNodes(BaseLog log, Configuration config) { - var nodes = codexNodeFactory.Create(config, config.CodexDeployment.CodexContainers, log, new DefaultTimeSet()); + var nodes = entryPoint.CreateInterface().WrapCodexContainers(config.CodexDeployment.CodexContainers); var pass = true; foreach (var n in nodes) { cancelToken.ThrowIfCancellationRequested(); - log.Log($"Checking {n.Container.Name} @ '{n.Address.Host}:{n.Address.Port}'..."); + log.Log($"Checking {n.Container.Name} @ '{n.Container.Address.Host}:{n.Container.Address.Port}'..."); if (EnsureOnline(log, n)) { @@ -75,7 +77,7 @@ namespace ContinuousTests } else { - log.Error($"No response from '{n.Address.Host}'."); + log.Error($"No response from '{n.Container.Address.Host}'."); pass = false; } } @@ -85,7 +87,7 @@ namespace ContinuousTests } } - private bool EnsureOnline(BaseLog log, CodexAccess n) + private bool EnsureOnline(BaseLog log, ICodexNode n) { try { @@ -107,30 +109,9 @@ namespace ContinuousTests var errors = new List(); CheckRequiredNumberOfNodes(tests, errors); CheckCustomNamespaceClashes(tests, errors); - CheckEthereumIndexClashes(tests, errors); return errors; } - private void CheckEthereumIndexClashes(ContinuousTest[] tests, List errors) - { - var offLimits = config.CodexDeployment.CodexContainers.Length; - foreach (var test in tests) - { - if (test.EthereumAccountIndex != -1) - { - if (test.EthereumAccountIndex <= offLimits) - { - errors.Add($"Test '{test.Name}' has selected 'EthereumAccountIndex' = {test.EthereumAccountIndex}. All accounts up to and including {offLimits} are being used by the targetted Codex net. Select a different 'EthereumAccountIndex'."); - } - } - } - - DuplicatesCheck(tests, errors, - considerCondition: t => t.EthereumAccountIndex != -1, - getValue: t => t.EthereumAccountIndex, - propertyName: nameof(ContinuousTest.EthereumAccountIndex)); - } - private void CheckCustomNamespaceClashes(ContinuousTest[] tests, List errors) { DuplicatesCheck(tests, errors, diff --git a/ContinuousTests/TaskFactory.cs b/Tests/CodexContinuousTests/TaskFactory.cs similarity index 100% rename from ContinuousTests/TaskFactory.cs rename to Tests/CodexContinuousTests/TaskFactory.cs diff --git a/ContinuousTests/TestFactory.cs b/Tests/CodexContinuousTests/TestFactory.cs similarity index 100% rename from ContinuousTests/TestFactory.cs rename to Tests/CodexContinuousTests/TestFactory.cs diff --git a/ContinuousTests/TestHandle.cs b/Tests/CodexContinuousTests/TestHandle.cs similarity index 100% rename from ContinuousTests/TestHandle.cs rename to Tests/CodexContinuousTests/TestHandle.cs diff --git a/ContinuousTests/TestLoop.cs b/Tests/CodexContinuousTests/TestLoop.cs similarity index 79% rename from ContinuousTests/TestLoop.cs rename to Tests/CodexContinuousTests/TestLoop.cs index 79ae999a..48f66cae 100644 --- a/ContinuousTests/TestLoop.cs +++ b/Tests/CodexContinuousTests/TestLoop.cs @@ -4,17 +4,19 @@ namespace ContinuousTests { public class TestLoop { + private readonly EntryPointFactory entryPointFactory; private readonly TaskFactory taskFactory; private readonly Configuration config; - private readonly BaseLog overviewLog; + private readonly ILog overviewLog; private readonly Type testType; private readonly TimeSpan runsEvery; private readonly StartupChecker startupChecker; private readonly CancellationToken cancelToken; private readonly EventWaitHandle runFinishedHandle = new EventWaitHandle(true, EventResetMode.ManualReset); - public TestLoop(TaskFactory taskFactory, Configuration config, BaseLog overviewLog, Type testType, TimeSpan runsEvery, StartupChecker startupChecker, CancellationToken cancelToken) + public TestLoop(EntryPointFactory entryPointFactory, TaskFactory taskFactory, Configuration config, ILog overviewLog, Type testType, TimeSpan runsEvery, StartupChecker startupChecker, CancellationToken cancelToken) { + this.entryPointFactory = entryPointFactory; this.taskFactory = taskFactory; this.config = config; this.overviewLog = overviewLog; @@ -60,7 +62,7 @@ namespace ContinuousTests { var test = (ContinuousTest)Activator.CreateInstance(testType)!; var handle = new TestHandle(test); - var run = new SingleTestRun(taskFactory, config, overviewLog, handle, startupChecker, cancelToken); + var run = new SingleTestRun(entryPointFactory, taskFactory, config, overviewLog, handle, startupChecker, cancelToken); runFinishedHandle.Reset(); run.Run(runFinishedHandle); diff --git a/ContinuousTests/TestMomentAttribute.cs b/Tests/CodexContinuousTests/TestMomentAttribute.cs similarity index 100% rename from ContinuousTests/TestMomentAttribute.cs rename to Tests/CodexContinuousTests/TestMomentAttribute.cs diff --git a/ContinuousTests/Tests/HoldMyBeerTest.cs b/Tests/CodexContinuousTests/Tests/HoldMyBeerTest.cs similarity index 71% rename from ContinuousTests/Tests/HoldMyBeerTest.cs rename to Tests/CodexContinuousTests/Tests/HoldMyBeerTest.cs index 0ec268f2..76457e2e 100644 --- a/ContinuousTests/Tests/HoldMyBeerTest.cs +++ b/Tests/CodexContinuousTests/Tests/HoldMyBeerTest.cs @@ -1,4 +1,5 @@ -using DistTestCore; +using CodexPlugin; +using FileUtils; using NUnit.Framework; using Utils; @@ -11,19 +12,19 @@ namespace ContinuousTests.Tests public override TestFailMode TestFailMode => TestFailMode.StopAfterFirstFailure; private ContentId? cid; - private TestFile file = null!; + private TrackedFile file = null!; [TestMoment(t: Zero)] public void UploadTestFile() { var filesize = 80.MB(); - file = FileManager.GenerateTestFile(filesize); + file = FileManager.GenerateFile(filesize); - cid = UploadFile(Nodes[0], file); + cid = Nodes[0].UploadFile(file); Assert.That(cid, Is.Not.Null); - var dl = DownloadFile(Nodes[0], cid!); + var dl = Nodes[0].DownloadContent(cid); file.AssertIsEqual(dl); } } diff --git a/ContinuousTests/Tests/MarketplaceTest.cs b/Tests/CodexContinuousTests/Tests/MarketplaceTest.cs similarity index 100% rename from ContinuousTests/Tests/MarketplaceTest.cs rename to Tests/CodexContinuousTests/Tests/MarketplaceTest.cs diff --git a/ContinuousTests/Tests/PeersTest.cs b/Tests/CodexContinuousTests/Tests/PeersTest.cs similarity index 95% rename from ContinuousTests/Tests/PeersTest.cs rename to Tests/CodexContinuousTests/Tests/PeersTest.cs index 1cf68021..8d5d08b3 100644 --- a/ContinuousTests/Tests/PeersTest.cs +++ b/Tests/CodexContinuousTests/Tests/PeersTest.cs @@ -1,4 +1,4 @@ -using DistTestCore.Codex; +using CodexPlugin; using DistTestCore.Helpers; using NUnit.Framework; @@ -37,7 +37,7 @@ namespace ContinuousTests.Tests } } - private string AreAllPresent(CodexAccess n, string[] allIds) + private string AreAllPresent(ICodexNode n, string[] allIds) { var info = n.GetDebugInfo(); var known = info.table.nodes.Select(n => n.nodeId).ToArray(); diff --git a/ContinuousTests/Tests/PerformanceTests.cs b/Tests/CodexContinuousTests/Tests/PerformanceTests.cs similarity index 100% rename from ContinuousTests/Tests/PerformanceTests.cs rename to Tests/CodexContinuousTests/Tests/PerformanceTests.cs diff --git a/ContinuousTests/Tests/TransientNodeTest.cs b/Tests/CodexContinuousTests/Tests/TransientNodeTest.cs similarity index 100% rename from ContinuousTests/Tests/TransientNodeTest.cs rename to Tests/CodexContinuousTests/Tests/TransientNodeTest.cs diff --git a/ContinuousTests/Tests/TwoClientTest.cs b/Tests/CodexContinuousTests/Tests/TwoClientTest.cs similarity index 73% rename from ContinuousTests/Tests/TwoClientTest.cs rename to Tests/CodexContinuousTests/Tests/TwoClientTest.cs index e1799443..53cff4b6 100644 --- a/ContinuousTests/Tests/TwoClientTest.cs +++ b/Tests/CodexContinuousTests/Tests/TwoClientTest.cs @@ -1,4 +1,5 @@ -using DistTestCore; +using CodexPlugin; +using FileUtils; using NUnit.Framework; using Utils; @@ -11,21 +12,21 @@ namespace ContinuousTests.Tests public override TestFailMode TestFailMode => TestFailMode.StopAfterFirstFailure; private ContentId? cid; - private TestFile file = null!; + private TrackedFile file = null!; [TestMoment(t: Zero)] public void UploadTestFile() { - file = FileManager.GenerateTestFile(80.MB()); + file = FileManager.GenerateFile(80.MB()); - cid = UploadFile(Nodes[0], file); + cid = Nodes[0].UploadFile(file); Assert.That(cid, Is.Not.Null); } [TestMoment(t: 10)] public void DownloadTestFile() { - var dl = DownloadFile(Nodes[1], cid!); + var dl = Nodes[1].DownloadContent(cid!); file.AssertIsEqual(dl); } diff --git a/ContinuousTests/reports/CodexTestNetReport-August2023.md b/Tests/CodexContinuousTests/reports/CodexTestNetReport-August2023.md similarity index 100% rename from ContinuousTests/reports/CodexTestNetReport-August2023.md rename to Tests/CodexContinuousTests/reports/CodexTestNetReport-August2023.md diff --git a/ContinuousTests/reports/CodexTestNetReport-July2023.md b/Tests/CodexContinuousTests/reports/CodexTestNetReport-July2023.md similarity index 100% rename from ContinuousTests/reports/CodexTestNetReport-July2023.md rename to Tests/CodexContinuousTests/reports/CodexTestNetReport-July2023.md diff --git a/ContinuousTests/run.sh b/Tests/CodexContinuousTests/run.sh similarity index 100% rename from ContinuousTests/run.sh rename to Tests/CodexContinuousTests/run.sh diff --git a/LongTests/BasicTests/DownloadTests.cs b/Tests/CodexLongTests/BasicTests/DownloadTests.cs similarity index 76% rename from LongTests/BasicTests/DownloadTests.cs rename to Tests/CodexLongTests/BasicTests/DownloadTests.cs index 5e01e3c6..68de3b19 100644 --- a/LongTests/BasicTests/DownloadTests.cs +++ b/Tests/CodexLongTests/BasicTests/DownloadTests.cs @@ -1,11 +1,13 @@ using DistTestCore; +using FileUtils; using NUnit.Framework; +using Tests; using Utils; -namespace TestsLong.BasicTests +namespace CodexLongTests.BasicTests { [TestFixture] - public class DownloadTests : DistTest + public class DownloadTests : CodexDistTest { [TestCase(3, 500)] [TestCase(5, 100)] @@ -13,8 +15,8 @@ namespace TestsLong.BasicTests [UseLongTimeouts] public void ParallelDownload(int numberOfNodes, int filesizeMb) { - var group = SetupCodexNodes(numberOfNodes); - var host = SetupCodexNode(); + var group = AddCodex(numberOfNodes); + var host = AddCodex(); foreach (var node in group) { @@ -23,7 +25,7 @@ namespace TestsLong.BasicTests var testFile = GenerateTestFile(filesizeMb.MB()); var contentId = host.UploadFile(testFile); - var list = new List>(); + var list = new List>(); foreach (var node in group) { diff --git a/LongTests/BasicTests/LargeFileTests.cs b/Tests/CodexLongTests/BasicTests/LargeFileTests.cs similarity index 90% rename from LongTests/BasicTests/LargeFileTests.cs rename to Tests/CodexLongTests/BasicTests/LargeFileTests.cs index 2d834e55..68eb1a7c 100644 --- a/LongTests/BasicTests/LargeFileTests.cs +++ b/Tests/CodexLongTests/BasicTests/LargeFileTests.cs @@ -1,13 +1,14 @@ -using DistTestCore; -using DistTestCore.Codex; +using CodexPlugin; +using DistTestCore; using NUnit.Framework; using NUnit.Framework.Interfaces; +using Tests; using Utils; -namespace TestsLong.BasicTests +namespace CodexLongTests.BasicTests { [TestFixture] - public class LargeFileTests : DistTest + public class LargeFileTests : CodexDistTest { #region Abort test run after first failure @@ -47,7 +48,7 @@ namespace TestsLong.BasicTests var expectedFile = GenerateTestFile(sizeMB); - var node = SetupCodexNode(s => s.WithStorageQuota((size + 10).MB())); + var node = AddCodex(s => s.WithStorageQuota((size + 10).MB())); var uploadStart = DateTime.UtcNow; var cid = node.UploadFile(expectedFile); diff --git a/LongTests/BasicTests/TestInfraTests.cs b/Tests/CodexLongTests/BasicTests/TestInfraTests.cs similarity index 73% rename from LongTests/BasicTests/TestInfraTests.cs rename to Tests/CodexLongTests/BasicTests/TestInfraTests.cs index a6f68e35..caad5701 100644 --- a/LongTests/BasicTests/TestInfraTests.cs +++ b/Tests/CodexLongTests/BasicTests/TestInfraTests.cs @@ -1,14 +1,15 @@ using DistTestCore; using NUnit.Framework; +using Tests; -namespace TestsLong.BasicTests +namespace CodexLongTests.BasicTests { - public class TestInfraTests : DistTest + public class TestInfraTests : CodexDistTest { [Test, UseLongTimeouts] public void TestInfraShouldHave1000AddressSpacesPerPod() { - var group = SetupCodexNodes(1000, s => s.EnableMetrics()); // Increases use of port address space per node. + var group = AddCodex(1000, s => s.EnableMetrics()); var nodeIds = group.Select(n => n.GetDebugInfo().id).ToArray(); @@ -21,7 +22,7 @@ namespace TestsLong.BasicTests { for (var i = 0; i < 20; i++) { - var n = SetupCodexNode(); + var n = AddCodex(); Assert.That(!string.IsNullOrEmpty(n.GetDebugInfo().id)); } diff --git a/LongTests/BasicTests/UploadTests.cs b/Tests/CodexLongTests/BasicTests/UploadTests.cs similarity index 77% rename from LongTests/BasicTests/UploadTests.cs rename to Tests/CodexLongTests/BasicTests/UploadTests.cs index 69823eba..b2cadc04 100644 --- a/LongTests/BasicTests/UploadTests.cs +++ b/Tests/CodexLongTests/BasicTests/UploadTests.cs @@ -1,11 +1,14 @@ +using CodexPlugin; using DistTestCore; +using FileUtils; using NUnit.Framework; +using Tests; using Utils; -namespace TestsLong.BasicTests +namespace CodexLongTests.BasicTests { [TestFixture] - public class UploadTests : DistTest + public class UploadTests : CodexDistTest { [TestCase(3, 50)] [TestCase(5, 75)] @@ -13,15 +16,15 @@ namespace TestsLong.BasicTests [UseLongTimeouts] public void ParallelUpload(int numberOfNodes, int filesizeMb) { - var group = SetupCodexNodes(numberOfNodes); - var host = SetupCodexNode(); + var group = AddCodex(numberOfNodes); + var host = AddCodex(); foreach (var node in group) { host.ConnectToPeer(node); } - var testfiles = new List(); + var testfiles = new List(); var contentIds = new List>(); for (int i = 0; i < group.Count(); i++) @@ -30,7 +33,7 @@ namespace TestsLong.BasicTests var n = i; contentIds.Add(Task.Run(() => { return host.UploadFile(testfiles[n]); })); } - var downloads = new List>(); + var downloads = new List>(); for (int i = 0; i < group.Count(); i++) { var n = i; diff --git a/Tests/Tests.csproj b/Tests/CodexLongTests/CodexTestsLong.csproj similarity index 79% rename from Tests/Tests.csproj rename to Tests/CodexLongTests/CodexTestsLong.csproj index ab5558b2..71c67564 100644 --- a/Tests/Tests.csproj +++ b/Tests/CodexLongTests/CodexTestsLong.csproj @@ -13,7 +13,8 @@ - + + diff --git a/LongTests/DownloadConnectivityTests/LongFullyConnectedDownloadTests.cs b/Tests/CodexLongTests/DownloadConnectivityTests/LongFullyConnectedDownloadTests.cs similarity index 80% rename from LongTests/DownloadConnectivityTests/LongFullyConnectedDownloadTests.cs rename to Tests/CodexLongTests/DownloadConnectivityTests/LongFullyConnectedDownloadTests.cs index 6de5c381..5858d53a 100644 --- a/LongTests/DownloadConnectivityTests/LongFullyConnectedDownloadTests.cs +++ b/Tests/CodexLongTests/DownloadConnectivityTests/LongFullyConnectedDownloadTests.cs @@ -1,8 +1,9 @@ using DistTestCore; using NUnit.Framework; +using Tests; using Utils; -namespace TestsLong.DownloadConnectivityTests +namespace CodexLongTests.DownloadConnectivityTests { [TestFixture] public class LongFullyConnectedDownloadTests : AutoBootstrapDistTest @@ -14,7 +15,7 @@ namespace TestsLong.DownloadConnectivityTests [Values(10, 15, 20)] int numberOfNodes, [Values(10, 100)] int sizeMBs) { - for (var i = 0; i < numberOfNodes; i++) SetupCodexNode(); + for (var i = 0; i < numberOfNodes; i++) AddCodex(); CreatePeerDownloadTestHelpers().AssertFullDownloadInterconnectivity(GetAllOnlineCodexNodes(), sizeMBs.MB()); } diff --git a/Tests/Parallelism.cs b/Tests/CodexLongTests/Parallelism.cs similarity index 72% rename from Tests/Parallelism.cs rename to Tests/CodexLongTests/Parallelism.cs index f45d8f2e..51709b8d 100644 --- a/Tests/Parallelism.cs +++ b/Tests/CodexLongTests/Parallelism.cs @@ -1,6 +1,6 @@ using NUnit.Framework; [assembly: LevelOfParallelism(1)] -namespace Tests +namespace CodexLongTests { } diff --git a/Tests/CodexTests/AutoBootstrapDistTest.cs b/Tests/CodexTests/AutoBootstrapDistTest.cs new file mode 100644 index 00000000..046ff0a5 --- /dev/null +++ b/Tests/CodexTests/AutoBootstrapDistTest.cs @@ -0,0 +1,24 @@ +using CodexPlugin; +using NUnit.Framework; + +namespace Tests +{ + public class AutoBootstrapDistTest : CodexDistTest + { + private readonly List onlineCodexNodes = new List(); + + [SetUp] + public void SetUpBootstrapNode() + { + BootstrapNode = AddCodex(s => s.WithName("BOOTSTRAP")); + onlineCodexNodes.Add(BootstrapNode); + } + + protected override void OnCodexSetup(ICodexSetup setup) + { + if (BootstrapNode != null) setup.WithBootstrapNode(BootstrapNode); + } + + protected ICodexNode? BootstrapNode { get; private set; } + } +} diff --git a/Tests/BasicTests/ContinuousSubstitute.cs b/Tests/CodexTests/BasicTests/ContinuousSubstitute.cs similarity index 85% rename from Tests/BasicTests/ContinuousSubstitute.cs rename to Tests/CodexTests/BasicTests/ContinuousSubstitute.cs index 6f4ba163..5a4b32d0 100644 --- a/Tests/BasicTests/ContinuousSubstitute.cs +++ b/Tests/CodexTests/BasicTests/ContinuousSubstitute.cs @@ -1,4 +1,5 @@ -using DistTestCore; +using CodexPlugin; +using DistTestCore; using NUnit.Framework; using Utils; @@ -11,24 +12,24 @@ namespace Tests.BasicTests [Test] public void ContinuousTestSubstitute() { - var group = SetupCodexNodes(5, o => o - .EnableMetrics() - .EnableMarketplace(100000.TestTokens(), 0.Eth(), isValidator: true) + var group = AddCodex(5, o => o + //.EnableMetrics() + //.EnableMarketplace(100000.TestTokens(), 0.Eth(), isValidator: true) .WithBlockTTL(TimeSpan.FromMinutes(2)) .WithBlockMaintenanceInterval(TimeSpan.FromMinutes(2)) .WithBlockMaintenanceNumber(10000) .WithBlockTTL(TimeSpan.FromMinutes(2)) .WithStorageQuota(1.GB())); - var nodes = group.Cast().ToArray(); + var nodes = group.Cast().ToArray(); foreach (var node in nodes) { - node.Marketplace.MakeStorageAvailable( - size: 500.MB(), - minPricePerBytePerSecond: 1.TestTokens(), - maxCollateral: 1024.TestTokens(), - maxDuration: TimeSpan.FromMinutes(5)); + //node.Marketplace.MakeStorageAvailable( + //size: 500.MB(), + //minPricePerBytePerSecond: 1.TestTokens(), + //maxCollateral: 1024.TestTokens(), + //maxDuration: TimeSpan.FromMinutes(5)); } var endTime = DateTime.UtcNow + TimeSpan.FromHours(10); @@ -48,23 +49,23 @@ namespace Tests.BasicTests [Test] public void PeerTest() { - var group = SetupCodexNodes(5, o => o - .EnableMetrics() - .EnableMarketplace(100000.TestTokens(), 0.Eth(), isValidator: true) + var group = AddCodex(5, o => o + //.EnableMetrics() + //.EnableMarketplace(100000.TestTokens(), 0.Eth(), isValidator: true) .WithBlockTTL(TimeSpan.FromMinutes(2)) .WithBlockMaintenanceInterval(TimeSpan.FromMinutes(2)) .WithBlockMaintenanceNumber(10000) .WithBlockTTL(TimeSpan.FromMinutes(2)) .WithStorageQuota(1.GB())); - var nodes = group.Cast().ToArray(); + var nodes = group.Cast().ToArray(); var checkTime = DateTime.UtcNow + TimeSpan.FromMinutes(1); var endTime = DateTime.UtcNow + TimeSpan.FromHours(10); while (DateTime.UtcNow < endTime) { - CreatePeerConnectionTestHelpers().AssertFullyConnected(GetAllOnlineCodexNodes()); - CheckRoutingTables(GetAllOnlineCodexNodes()); + //CreatePeerConnectionTestHelpers().AssertFullyConnected(GetAllOnlineCodexNodes()); + //CheckRoutingTables(GetAllOnlineCodexNodes()); var node = RandomUtils.PickOneRandom(nodes.ToList()); var file = GenerateTestFile(50.MB()); @@ -74,7 +75,7 @@ namespace Tests.BasicTests } } - private void CheckRoutingTables(IEnumerable nodes) + private void CheckRoutingTables(IEnumerable nodes) { var all = nodes.ToArray(); var allIds = all.Select(n => n.GetDebugInfo().table.localNode.nodeId).ToArray(); @@ -87,7 +88,7 @@ namespace Tests.BasicTests } } - private string AreAllPresent(IOnlineCodexNode n, string[] allIds) + private string AreAllPresent(ICodexNode n, string[] allIds) { var info = n.GetDebugInfo(); var known = info.table.nodes.Select(n => n.nodeId).ToArray(); @@ -103,7 +104,7 @@ namespace Tests.BasicTests private ByteSize fileSize = 80.MB(); - private void PerformTest(IOnlineCodexNode primary, IOnlineCodexNode secondary) + private void PerformTest(ICodexNode primary, ICodexNode secondary) { ScopedTestFiles(() => { @@ -121,14 +122,14 @@ namespace Tests.BasicTests public void HoldMyBeerTest() { var blockExpirationTime = TimeSpan.FromMinutes(3); - var group = SetupCodexNodes(3, o => o - .EnableMetrics() + var group = AddCodex(3, o => o + //.EnableMetrics() .WithBlockTTL(blockExpirationTime) .WithBlockMaintenanceInterval(TimeSpan.FromMinutes(2)) .WithBlockMaintenanceNumber(10000) .WithStorageQuota(2000.MB())); - var nodes = group.Cast().ToArray(); + var nodes = group.Cast().ToArray(); var endTime = DateTime.UtcNow + TimeSpan.FromHours(24); @@ -159,7 +160,7 @@ namespace Tests.BasicTests var cidTag = cid.Id.Substring(cid.Id.Length - 6); Measure("upload-log-asserts", () => { - var uploadLog = node.DownloadLog(tailLines: 50000); + var uploadLog = Ci.DownloadLog(node, tailLines: 50000); var storeLines = uploadLog.FindLinesThatContain("Stored data", "topics=\"codex node\""); uploadLog.DeleteFile(); @@ -180,7 +181,7 @@ namespace Tests.BasicTests Measure("download-log-asserts", () => { - var downloadLog = node.DownloadLog(tailLines: 50000); + var downloadLog = Ci.DownloadLog(node, tailLines: 50000); var sentLines = downloadLog.FindLinesThatContain("Sent bytes", "topics=\"codex restapi\""); downloadLog.DeleteFile(); diff --git a/Tests/CodexTests/BasicTests/ExampleTests.cs b/Tests/CodexTests/BasicTests/ExampleTests.cs new file mode 100644 index 00000000..f35be566 --- /dev/null +++ b/Tests/CodexTests/BasicTests/ExampleTests.cs @@ -0,0 +1,100 @@ +using CodexContractsPlugin; +using DistTestCore; +using GethPlugin; +using MetricsPlugin; +using NUnit.Framework; +using Utils; + +namespace Tests.BasicTests +{ + [TestFixture] + public class ExampleTests : CodexDistTest + { + [Test] + public void CodexLogExample() + { + var primary = AddCodex(); + + primary.UploadFile(GenerateTestFile(5.MB())); + + var log = Ci.DownloadLog(primary); + + log.AssertLogContains("Uploaded file"); + } + + [Test] + public void TwoMetricsExample() + { + var group = AddCodex(2, s => s.EnableMetrics()); + var group2 = AddCodex(2, s => s.EnableMetrics()); + + var primary = group[0]; + var secondary = group[1]; + var primary2 = group2[0]; + var secondary2 = group2[1]; + + var metrics = Ci.GetMetricsFor(primary, primary2); + + primary.ConnectToPeer(secondary); + primary2.ConnectToPeer(secondary2); + + Thread.Sleep(TimeSpan.FromMinutes(2)); + + metrics[0].AssertThat("libp2p_peers", Is.EqualTo(1)); + metrics[1].AssertThat("libp2p_peers", Is.EqualTo(1)); + } + + [Test] + [Combinatorial] + public void MarketplaceExample( + [Values(true, false)] bool isValidator, + [Values(true, false)] bool simulateProofFailure) + { + var sellerInitialBalance = 234.TestTokens(); + var buyerInitialBalance = 1000.TestTokens(); + var fileSize = 10.MB(); + + var geth = Ci.StartGethNode(s => s.IsMiner().WithName("disttest-geth")); + var contracts = Ci.StartCodexContracts(geth); + + var seller = AddCodex(s => + { + s.WithStorageQuota(11.GB()); + s.EnableMarketplace(geth, contracts, initialEth: 10.Eth(), initialTokens: sellerInitialBalance, isValidator); + if (simulateProofFailure) s.WithSimulateProofFailures(3); + }); + + AssertBalance(geth, contracts, seller, Is.EqualTo(sellerInitialBalance)); + seller.Marketplace.MakeStorageAvailable( + size: 10.GB(), + minPriceForTotalSpace: 1.TestTokens(), + maxCollateral: 20.TestTokens(), + maxDuration: TimeSpan.FromMinutes(3)); + + var testFile = GenerateTestFile(fileSize); + + var buyer = AddCodex(s => s + .WithBootstrapNode(seller) + .EnableMarketplace(geth, contracts, initialEth: 10.Eth(), initialTokens: buyerInitialBalance)); + + AssertBalance(geth, contracts, buyer, Is.EqualTo(buyerInitialBalance)); + + var contentId = buyer.UploadFile(testFile); + var purchaseContract = buyer.Marketplace.RequestStorage(contentId, + pricePerSlotPerSecond: 2.TestTokens(), + requiredCollateral: 10.TestTokens(), + minRequiredNumberOfNodes: 1, + proofProbability: 5, + duration: TimeSpan.FromMinutes(1)); + + purchaseContract.WaitForStorageContractStarted(fileSize); + + AssertBalance(geth, contracts, seller, Is.LessThan(sellerInitialBalance), "Collateral was not placed."); + + purchaseContract.WaitForStorageContractFinished(); + + AssertBalance(geth, contracts, seller, Is.GreaterThan(sellerInitialBalance), "Seller was not paid for storage."); + AssertBalance(geth, contracts, buyer, Is.LessThan(buyerInitialBalance), "Buyer was not charged for storage."); + } + } +} diff --git a/Tests/BasicTests/NetworkIsolationTest.cs b/Tests/CodexTests/BasicTests/NetworkIsolationTest.cs similarity index 87% rename from Tests/BasicTests/NetworkIsolationTest.cs rename to Tests/CodexTests/BasicTests/NetworkIsolationTest.cs index e1c6e7b3..7388e4ae 100644 --- a/Tests/BasicTests/NetworkIsolationTest.cs +++ b/Tests/CodexTests/BasicTests/NetworkIsolationTest.cs @@ -1,4 +1,5 @@ -using DistTestCore; +using CodexPlugin; +using DistTestCore; using NUnit.Framework; using Utils; @@ -11,12 +12,12 @@ namespace Tests.BasicTests [Ignore("Disabled until a solution is implemented.")] public class NetworkIsolationTest : DistTest { - private IOnlineCodexNode? node = null; + private ICodexNode? node = null; [Test] public void SetUpANodeAndWait() { - node = SetupCodexNode(); + node = Ci.StartCodexNode(); Time.WaitUntil(() => node == null, TimeSpan.FromMinutes(5), TimeSpan.FromSeconds(5)); } @@ -24,7 +25,7 @@ namespace Tests.BasicTests [Test] public void ForeignNodeConnects() { - var myNode = SetupCodexNode(); + var myNode = Ci.StartCodexNode(); Time.WaitUntil(() => node != null, TimeSpan.FromMinutes(1), TimeSpan.FromSeconds(5)); diff --git a/Tests/BasicTests/OneClientTests.cs b/Tests/CodexTests/BasicTests/OneClientTests.cs similarity index 69% rename from Tests/BasicTests/OneClientTests.cs rename to Tests/CodexTests/BasicTests/OneClientTests.cs index a31be3ce..22fe0dd3 100644 --- a/Tests/BasicTests/OneClientTests.cs +++ b/Tests/CodexTests/BasicTests/OneClientTests.cs @@ -1,4 +1,5 @@ -using DistTestCore; +using CodexPlugin; +using DistTestCore; using NUnit.Framework; using Utils; @@ -10,7 +11,7 @@ namespace Tests.BasicTests [Test] public void OneClientTest() { - var primary = SetupCodexNode(); + var primary = Ci.StartCodexNode(); PerformOneClientTest(primary); } @@ -18,16 +19,16 @@ namespace Tests.BasicTests [Test] public void RestartTest() { - var primary = SetupCodexNode(); + var primary = Ci.StartCodexNode(); - var setup = primary.BringOffline(); + primary.Stop(); - primary = BringOnline(setup)[0]; + primary = Ci.StartCodexNode(); PerformOneClientTest(primary); } - private void PerformOneClientTest(IOnlineCodexNode primary) + private void PerformOneClientTest(ICodexNode primary) { var testFile = GenerateTestFile(1.MB()); diff --git a/Tests/BasicTests/ThreeClientTest.cs b/Tests/CodexTests/BasicTests/ThreeClientTest.cs similarity index 84% rename from Tests/BasicTests/ThreeClientTest.cs rename to Tests/CodexTests/BasicTests/ThreeClientTest.cs index c857e35f..5e44c975 100644 --- a/Tests/BasicTests/ThreeClientTest.cs +++ b/Tests/CodexTests/BasicTests/ThreeClientTest.cs @@ -10,8 +10,8 @@ namespace Tests.BasicTests [Test] public void ThreeClient() { - var primary = SetupCodexNode(); - var secondary = SetupCodexNode(); + var primary = AddCodex(); + var secondary = AddCodex(); var testFile = GenerateTestFile(10.MB()); diff --git a/Tests/BasicTests/TwoClientTests.cs b/Tests/CodexTests/BasicTests/TwoClientTests.cs similarity index 68% rename from Tests/BasicTests/TwoClientTests.cs rename to Tests/CodexTests/BasicTests/TwoClientTests.cs index 4bebc20a..69084949 100644 --- a/Tests/BasicTests/TwoClientTests.cs +++ b/Tests/CodexTests/BasicTests/TwoClientTests.cs @@ -1,4 +1,5 @@ -using DistTestCore; +using CodexPlugin; +using DistTestCore; using KubernetesWorkflow; using NUnit.Framework; using Utils; @@ -11,7 +12,7 @@ namespace Tests.BasicTests [Test] public void TwoClientTest() { - var group = SetupCodexNodes(2); + var group = Ci.StartCodexNodes(2); var primary = group[0]; var secondary = group[1]; @@ -22,18 +23,18 @@ namespace Tests.BasicTests [Test] public void TwoClientsTwoLocationsTest() { - var primary = SetupCodexNode(s => s.At(Location.One)); - var secondary = SetupCodexNode(s => s.At(Location.Two)); + var primary = Ci.StartCodexNode(s => s.At(Location.One)); + var secondary = Ci.StartCodexNode(s => s.At(Location.Two)); PerformTwoClientTest(primary, secondary); } - private void PerformTwoClientTest(IOnlineCodexNode primary, IOnlineCodexNode secondary) + private void PerformTwoClientTest(ICodexNode primary, ICodexNode secondary) { PerformTwoClientTest(primary, secondary, 1.MB()); } - private void PerformTwoClientTest(IOnlineCodexNode primary, IOnlineCodexNode secondary, ByteSize size) + private void PerformTwoClientTest(ICodexNode primary, ICodexNode secondary, ByteSize size) { primary.ConnectToPeer(secondary); diff --git a/Tests/CodexTests/CodexDistTest.cs b/Tests/CodexTests/CodexDistTest.cs new file mode 100644 index 00000000..32c6ec26 --- /dev/null +++ b/Tests/CodexTests/CodexDistTest.cs @@ -0,0 +1,73 @@ +using CodexContractsPlugin; +using CodexPlugin; +using Core; +using DistTestCore; +using DistTestCore.Helpers; +using GethPlugin; +using NUnit.Framework.Constraints; + +namespace Tests +{ + public class CodexDistTest : DistTest + { + private readonly List onlineCodexNodes = new List(); + + public CodexDistTest() + { + ProjectPlugin.Load(); + ProjectPlugin.Load(); + ProjectPlugin.Load(); + ProjectPlugin.Load(); + } + + public ICodexNode AddCodex() + { + return AddCodex(s => { }); + } + + public ICodexNode AddCodex(Action setup) + { + return AddCodex(1, setup)[0]; + } + + public ICodexNodeGroup AddCodex(int numberOfNodes) + { + return AddCodex(numberOfNodes, s => { }); + } + + public ICodexNodeGroup AddCodex(int numberOfNodes, Action setup) + { + var group = Ci.StartCodexNodes(numberOfNodes, s => + { + setup(s); + OnCodexSetup(s); + }); + onlineCodexNodes.AddRange(group); + return group; + } + + public PeerConnectionTestHelpers CreatePeerConnectionTestHelpers() + { + return new PeerConnectionTestHelpers(GetTestLog()); + } + + public PeerDownloadTestHelpers CreatePeerDownloadTestHelpers() + { + return new PeerDownloadTestHelpers(GetTestLog(), Get().GetFileManager()); + } + + public IEnumerable GetAllOnlineCodexNodes() + { + return onlineCodexNodes; + } + + public void AssertBalance(IGethNode gethNode, ICodexContracts contracts, ICodexNode codexNode, Constraint constraint, string msg = "") + { + AssertHelpers.RetryAssert(constraint, () => contracts.GetTestTokenBalance(gethNode, codexNode), nameof(AssertBalance) + msg); + } + + protected virtual void OnCodexSetup(ICodexSetup setup) + { + } + } +} diff --git a/LongTests/TestsLong.csproj b/Tests/CodexTests/CodexTests.csproj similarity index 59% rename from LongTests/TestsLong.csproj rename to Tests/CodexTests/CodexTests.csproj index 90f1cd6f..2c04d03a 100644 --- a/LongTests/TestsLong.csproj +++ b/Tests/CodexTests/CodexTests.csproj @@ -13,6 +13,10 @@ + + + + diff --git a/Tests/DownloadConnectivityTests/FullyConnectedDownloadTests.cs b/Tests/CodexTests/DownloadConnectivityTests/FullyConnectedDownloadTests.cs similarity index 78% rename from Tests/DownloadConnectivityTests/FullyConnectedDownloadTests.cs rename to Tests/CodexTests/DownloadConnectivityTests/FullyConnectedDownloadTests.cs index 9fb5630a..ed662408 100644 --- a/Tests/DownloadConnectivityTests/FullyConnectedDownloadTests.cs +++ b/Tests/CodexTests/DownloadConnectivityTests/FullyConnectedDownloadTests.cs @@ -1,5 +1,4 @@ -using DistTestCore; -using NUnit.Framework; +using NUnit.Framework; using Utils; namespace Tests.DownloadConnectivityTests @@ -10,7 +9,7 @@ namespace Tests.DownloadConnectivityTests [Test] public void MetricsDoesNotInterfereWithPeerDownload() { - SetupCodexNodes(2, s => s.EnableMetrics()); + AddCodex(2, s => s.EnableMetrics()); AssertAllNodesConnected(); } @@ -18,7 +17,7 @@ namespace Tests.DownloadConnectivityTests [Test] public void MarketplaceDoesNotInterfereWithPeerDownload() { - SetupCodexNodes(2, s => s.EnableMetrics().EnableMarketplace(1000.TestTokens())); + //AddCodex(2, s => s.EnableMetrics().EnableMarketplace(1000.TestTokens())); AssertAllNodesConnected(); } @@ -29,7 +28,7 @@ namespace Tests.DownloadConnectivityTests [Values(3, 5)] int numberOfNodes, [Values(10, 80)] int sizeMBs) { - SetupCodexNodes(numberOfNodes); + AddCodex(numberOfNodes); AssertAllNodesConnected(sizeMBs); } diff --git a/DistTestCore/Helpers/FullConnectivityHelper.cs b/Tests/CodexTests/Helpers/FullConnectivityHelper.cs similarity index 93% rename from DistTestCore/Helpers/FullConnectivityHelper.cs rename to Tests/CodexTests/Helpers/FullConnectivityHelper.cs index b69af0c0..e0ef335a 100644 --- a/DistTestCore/Helpers/FullConnectivityHelper.cs +++ b/Tests/CodexTests/Helpers/FullConnectivityHelper.cs @@ -1,4 +1,4 @@ -using DistTestCore.Codex; +using CodexPlugin; using Logging; using NUnit.Framework; @@ -14,21 +14,21 @@ namespace DistTestCore.Helpers public class FullConnectivityHelper { private static string Nl = Environment.NewLine; - private readonly BaseLog log; + private readonly ILog log; private readonly IFullConnectivityImplementation implementation; - public FullConnectivityHelper(BaseLog log, IFullConnectivityImplementation implementation) + public FullConnectivityHelper(ILog log, IFullConnectivityImplementation implementation) { this.log = log; this.implementation = implementation; } - public void AssertFullyConnected(IEnumerable nodes) + public void AssertFullyConnected(IEnumerable nodes) { AssertFullyConnected(nodes.ToArray()); } - private void AssertFullyConnected(CodexAccess[] nodes) + private void AssertFullyConnected(ICodexNode[] nodes) { Log($"Asserting '{implementation.Description()}' for nodes: '{string.Join(",", nodes.Select(n => n.GetName()))}'..."); var entries = CreateEntries(nodes); @@ -67,7 +67,7 @@ namespace DistTestCore.Helpers Log($"Connections successful:{Nl}{string.Join(Nl, results)}"); } - private Entry[] CreateEntries(CodexAccess[] nodes) + private Entry[] CreateEntries(ICodexNode[] nodes) { var entries = nodes.Select(n => new Entry(n)).ToArray(); @@ -107,13 +107,13 @@ namespace DistTestCore.Helpers public class Entry { - public Entry(CodexAccess node) + public Entry(ICodexNode node) { Node = node; Response = node.GetDebugInfo(); } - public CodexAccess Node { get; } + public ICodexNode Node { get; } public CodexDebugResponse Response { get; } public override string ToString() diff --git a/DistTestCore/Helpers/PeerConnectionTestHelpers.cs b/Tests/CodexTests/Helpers/PeerConnectionTestHelpers.cs similarity index 79% rename from DistTestCore/Helpers/PeerConnectionTestHelpers.cs rename to Tests/CodexTests/Helpers/PeerConnectionTestHelpers.cs index 65bb4ce0..bde5aeb9 100644 --- a/DistTestCore/Helpers/PeerConnectionTestHelpers.cs +++ b/Tests/CodexTests/Helpers/PeerConnectionTestHelpers.cs @@ -1,4 +1,4 @@ -using DistTestCore.Codex; +using CodexPlugin; using Logging; using static DistTestCore.Helpers.FullConnectivityHelper; @@ -8,17 +8,12 @@ namespace DistTestCore.Helpers { private readonly FullConnectivityHelper helper; - public PeerConnectionTestHelpers(BaseLog log) + public PeerConnectionTestHelpers(ILog log) { helper = new FullConnectivityHelper(log, this); } - public void AssertFullyConnected(IEnumerable nodes) - { - AssertFullyConnected(nodes.Select(n => ((OnlineCodexNode)n).CodexAccess)); - } - - public void AssertFullyConnected(IEnumerable nodes) + public void AssertFullyConnected(IEnumerable nodes) { helper.AssertFullyConnected(nodes); } @@ -63,8 +58,9 @@ namespace DistTestCore.Helpers var peer = allEntries.SingleOrDefault(e => e.Response.table.localNode.peerId == node.peerId); if (peer == null) return $"peerId: {node.peerId} is not known."; - var ip = peer.Node.Container.Pod.PodInfo.Ip; - var discPort = peer.Node.Container.Recipe.GetPortByTag(CodexContainerRecipe.DiscoveryPortTag); + var container = peer.Node.Container; + var ip = container.Pod.PodInfo.Ip; + var discPort = container.Recipe.GetPortByTag(CodexContainerRecipe.DiscoveryPortTag)!; return $"{ip}:{discPort.Number}"; } } diff --git a/DistTestCore/Helpers/PeerDownloadTestHelpers.cs b/Tests/CodexTests/Helpers/PeerDownloadTestHelpers.cs similarity index 62% rename from DistTestCore/Helpers/PeerDownloadTestHelpers.cs rename to Tests/CodexTests/Helpers/PeerDownloadTestHelpers.cs index c2af4159..6ad178f2 100644 --- a/DistTestCore/Helpers/PeerDownloadTestHelpers.cs +++ b/Tests/CodexTests/Helpers/PeerDownloadTestHelpers.cs @@ -1,4 +1,5 @@ -using DistTestCore.Codex; +using CodexPlugin; +using FileUtils; using Logging; using Utils; using static DistTestCore.Helpers.FullConnectivityHelper; @@ -8,11 +9,11 @@ namespace DistTestCore.Helpers public class PeerDownloadTestHelpers : IFullConnectivityImplementation { private readonly FullConnectivityHelper helper; - private readonly BaseLog log; - private readonly FileManager fileManager; + private readonly ILog log; + private readonly IFileManager fileManager; private ByteSize testFileSize; - public PeerDownloadTestHelpers(BaseLog log, FileManager fileManager) + public PeerDownloadTestHelpers(ILog log, IFileManager fileManager) { helper = new FullConnectivityHelper(log, this); testFileSize = 1.MB(); @@ -20,12 +21,7 @@ namespace DistTestCore.Helpers this.fileManager = fileManager; } - public void AssertFullDownloadInterconnectivity(IEnumerable nodes, ByteSize testFileSize) - { - AssertFullDownloadInterconnectivity(nodes.Select(n => ((OnlineCodexNode)n).CodexAccess), testFileSize); - } - - public void AssertFullDownloadInterconnectivity(IEnumerable nodes, ByteSize testFileSize) + public void AssertFullDownloadInterconnectivity(IEnumerable nodes, ByteSize testFileSize) { this.testFileSize = testFileSize; helper.AssertFullyConnected(nodes); @@ -43,11 +39,13 @@ namespace DistTestCore.Helpers public PeerConnectionState Check(Entry from, Entry to) { - fileManager.PushFileSet(); - var expectedFile = GenerateTestFile(from.Node, to.Node); + return fileManager.ScopedFiles(() => CheckConnectivity(from, to)); + } - using var uploadStream = File.OpenRead(expectedFile.Filename); - var contentId = Stopwatch.Measure(log, "Upload", () => from.Node.UploadFile(uploadStream)); + private PeerConnectionState CheckConnectivity(Entry from, Entry to) + { + var expectedFile = GenerateTestFile(from.Node, to.Node); + var contentId = Stopwatch.Measure(log, "Upload", () => from.Node.UploadFile(expectedFile)); try { @@ -61,29 +59,20 @@ namespace DistTestCore.Helpers // We consider that as no-connection for the purpose of this test. return PeerConnectionState.NoConnection; } - finally - { - fileManager.PopFileSet(); - } - // Should an exception occur during upload, then this try is inconclusive and we try again next loop. } - private TestFile DownloadFile(CodexAccess node, string contentId, string label) + private TrackedFile? DownloadFile(ICodexNode node, ContentId contentId, string label) { - var downloadedFile = fileManager.CreateEmptyTestFile(label); - using var downloadStream = File.OpenWrite(downloadedFile.Filename); - using var stream = node.DownloadFile(contentId); - stream.CopyTo(downloadStream); - return downloadedFile; + return node.DownloadContent(contentId, label); } - private TestFile GenerateTestFile(CodexAccess uploader, CodexAccess downloader) + private TrackedFile GenerateTestFile(ICodexNode uploader, ICodexNode downloader) { var up = uploader.GetName().Replace("<", "").Replace(">", ""); var down = downloader.GetName().Replace("<", "").Replace(">", ""); var label = $"~from:{up}-to:{down}~"; - return fileManager.GenerateTestFile(testFileSize, label); + return fileManager.GenerateFile(testFileSize, label); } } } diff --git a/Tests/CodexTests/MetricsAccessExtensions.cs b/Tests/CodexTests/MetricsAccessExtensions.cs new file mode 100644 index 00000000..aa5da83b --- /dev/null +++ b/Tests/CodexTests/MetricsAccessExtensions.cs @@ -0,0 +1,22 @@ +using DistTestCore.Helpers; +using Logging; +using MetricsPlugin; +using NUnit.Framework.Constraints; + +namespace Tests +{ + public static class MetricsAccessExtensions + { + public static void AssertThat(this IMetricsAccess access, string metricName, IResolveConstraint constraint, ILog? log = null, string message = "") + { + AssertHelpers.RetryAssert(constraint, () => + { + var metricSet = access.GetMetric(metricName); + var metricValue = metricSet.Values[0].Value; + + if (log != null) log.Log($"{access.TargetName} metric '{metricName}' = {metricValue}"); + return metricValue; + }, message); + } + } +} diff --git a/LongTests/Parallelism.cs b/Tests/CodexTests/Parallelism.cs similarity index 100% rename from LongTests/Parallelism.cs rename to Tests/CodexTests/Parallelism.cs diff --git a/Tests/CodexTests/PeerDiscoveryTests/LayeredDiscoveryTests.cs b/Tests/CodexTests/PeerDiscoveryTests/LayeredDiscoveryTests.cs new file mode 100644 index 00000000..97675af5 --- /dev/null +++ b/Tests/CodexTests/PeerDiscoveryTests/LayeredDiscoveryTests.cs @@ -0,0 +1,52 @@ +using CodexPlugin; +using NUnit.Framework; + +namespace Tests.PeerDiscoveryTests +{ + [TestFixture] + public class LayeredDiscoveryTests : CodexDistTest + { + [Test] + public void TwoLayersTest() + { + var root = Ci.StartCodexNode(); + var l1Source = Ci.StartCodexNode(s => s.WithBootstrapNode(root)); + var l1Node = Ci.StartCodexNode(s => s.WithBootstrapNode(root)); + var l2Target = Ci.StartCodexNode(s => s.WithBootstrapNode(l1Node)); + + AssertAllNodesConnected(); + } + + [Test] + public void ThreeLayersTest() + { + var root = Ci.StartCodexNode(); + var l1Source = Ci.StartCodexNode(s => s.WithBootstrapNode(root)); + var l1Node = Ci.StartCodexNode(s => s.WithBootstrapNode(root)); + var l2Node = Ci.StartCodexNode(s => s.WithBootstrapNode(l1Node)); + var l3Target = Ci.StartCodexNode(s => s.WithBootstrapNode(l2Node)); + + AssertAllNodesConnected(); + } + + [TestCase(3)] + [TestCase(5)] + [TestCase(10)] + [TestCase(20)] + public void NodeChainTest(int chainLength) + { + var node = Ci.StartCodexNode(); + for (var i = 1; i < chainLength; i++) + { + node = Ci.StartCodexNode(s => s.WithBootstrapNode(node)); + } + + AssertAllNodesConnected(); + } + + private void AssertAllNodesConnected() + { + CreatePeerConnectionTestHelpers().AssertFullyConnected(GetAllOnlineCodexNodes()); + } + } +} diff --git a/Tests/PeerDiscoveryTests/PeerDiscoveryTests.cs b/Tests/CodexTests/PeerDiscoveryTests/PeerDiscoveryTests.cs similarity index 80% rename from Tests/PeerDiscoveryTests/PeerDiscoveryTests.cs rename to Tests/CodexTests/PeerDiscoveryTests/PeerDiscoveryTests.cs index fcb44fdc..4141925e 100644 --- a/Tests/PeerDiscoveryTests/PeerDiscoveryTests.cs +++ b/Tests/CodexTests/PeerDiscoveryTests/PeerDiscoveryTests.cs @@ -1,5 +1,4 @@ -using DistTestCore; -using NUnit.Framework; +using NUnit.Framework; namespace Tests.PeerDiscoveryTests { @@ -10,7 +9,7 @@ namespace Tests.PeerDiscoveryTests public void CanReportUnknownPeerId() { var unknownId = "16Uiu2HAkv2CHWpff3dj5iuVNERAp8AGKGNgpGjPexJZHSqUstfsK"; - var node = SetupCodexNode(); + var node = AddCodex(); var result = node.GetDebugPeer(unknownId); Assert.That(result.IsPeerFound, Is.False); @@ -19,7 +18,7 @@ namespace Tests.PeerDiscoveryTests [Test] public void MetricsDoesNotInterfereWithPeerDiscovery() { - SetupCodexNodes(2, s => s.EnableMetrics()); + AddCodex(2, s => s.EnableMetrics()); AssertAllNodesConnected(); } @@ -27,7 +26,7 @@ namespace Tests.PeerDiscoveryTests [Test] public void MarketplaceDoesNotInterfereWithPeerDiscovery() { - SetupCodexNodes(2, s => s.EnableMarketplace(1000.TestTokens())); + //AddCodex(2, s => s.EnableMarketplace(1000.TestTokens())); AssertAllNodesConnected(); } @@ -38,7 +37,7 @@ namespace Tests.PeerDiscoveryTests [TestCase(20)] public void VariableNodes(int number) { - SetupCodexNodes(number); + AddCodex(number); AssertAllNodesConnected(); } diff --git a/Tests/DistTestCore/Configuration.cs b/Tests/DistTestCore/Configuration.cs new file mode 100644 index 00000000..35b98235 --- /dev/null +++ b/Tests/DistTestCore/Configuration.cs @@ -0,0 +1,73 @@ +using Core; +using KubernetesWorkflow; + +namespace DistTestCore +{ + public class Configuration + { + private readonly string? kubeConfigFile; + private readonly string logPath; + private readonly bool logDebug; + private readonly string dataFilesPath; + + public Configuration() + { + kubeConfigFile = GetNullableEnvVarOrDefault("KUBECONFIG", null); + logPath = GetEnvVarOrDefault("LOGPATH", "CodexTestLogs"); + logDebug = GetEnvVarOrDefault("LOGDEBUG", "false").ToLowerInvariant() == "true"; + dataFilesPath = GetEnvVarOrDefault("DATAFILEPATH", "TestDataFiles"); + } + + public Configuration(string? kubeConfigFile, string logPath, bool logDebug, string dataFilesPath) + { + this.kubeConfigFile = kubeConfigFile; + this.logPath = logPath; + this.logDebug = logDebug; + this.dataFilesPath = dataFilesPath; + } + + public KubernetesWorkflow.Configuration GetK8sConfiguration(ITimeSet timeSet, string k8sNamespace) + { + return GetK8sConfiguration(timeSet, new DoNothingK8sHooks(), k8sNamespace); + } + + public KubernetesWorkflow.Configuration GetK8sConfiguration(ITimeSet timeSet, IK8sHooks hooks, string k8sNamespace) + { + var config = new KubernetesWorkflow.Configuration( + kubeConfigFile: kubeConfigFile, + operationTimeout: timeSet.K8sOperationTimeout(), + retryDelay: timeSet.WaitForK8sServiceDelay(), + kubernetesNamespace: k8sNamespace + ); + + config.AllowNamespaceOverride = false; + config.Hooks = hooks; + + return config; + } + + public Logging.LogConfig GetLogConfig() + { + return new Logging.LogConfig(logPath, debugEnabled: logDebug); + } + + public string GetFileManagerFolder() + { + return dataFilesPath; + } + + private static string GetEnvVarOrDefault(string varName, string defaultValue) + { + var v = Environment.GetEnvironmentVariable(varName); + if (v == null) return defaultValue; + return v; + } + + private static string? GetNullableEnvVarOrDefault(string varName, string? defaultValue) + { + var v = Environment.GetEnvironmentVariable(varName); + if (v == null) return defaultValue; + return v; + } + } +} diff --git a/DistTestCore/DistTest.cs b/Tests/DistTestCore/DistTest.cs similarity index 56% rename from DistTestCore/DistTest.cs rename to Tests/DistTestCore/DistTest.cs index 155a24f0..8e9932a8 100644 --- a/DistTestCore/DistTest.cs +++ b/Tests/DistTestCore/DistTest.cs @@ -1,27 +1,26 @@ -using DistTestCore.Codex; -using DistTestCore.Helpers; +using Core; using DistTestCore.Logs; -using DistTestCore.Marketplace; -using DistTestCore.Metrics; -using KubernetesWorkflow; +using FileUtils; using Logging; using NUnit.Framework; using System.Reflection; using Utils; +using Assert = NUnit.Framework.Assert; namespace DistTestCore { [Parallelizable(ParallelScope.All)] public abstract class DistTest { - private const string TestsType = "dist-tests"; + private const string TestNamespacePrefix = "ct-"; private readonly Configuration configuration = new Configuration(); private readonly Assembly[] testAssemblies; private readonly FixtureLog fixtureLog; private readonly StatusLog statusLog; private readonly object lifecycleLock = new object(); + private readonly EntryPoint globalEntryPoint; private readonly Dictionary lifecycles = new Dictionary(); - + public DistTest() { var assemblies = AppDomain.CurrentDomain.GetAssemblies(); @@ -31,16 +30,15 @@ namespace DistTestCore var startTime = DateTime.UtcNow; fixtureLog = new FixtureLog(logConfig, startTime); statusLog = new StatusLog(logConfig, startTime); + + globalEntryPoint = new EntryPoint(fixtureLog, configuration.GetK8sConfiguration(new DefaultTimeSet(), TestNamespacePrefix), configuration.GetFileManagerFolder()); } [OneTimeSetUp] public void GlobalSetup() { - fixtureLog.Log($"Codex Distributed Tests are starting..."); - fixtureLog.Log($"Codex image: '{new CodexContainerRecipe().Image}'"); - fixtureLog.Log($"CodexContracts image: '{new CodexContractsContainerRecipe().Image}'"); - fixtureLog.Log($"Prometheus image: '{new PrometheusContainerRecipe().Image}'"); - fixtureLog.Log($"Geth image: '{new GethContainerRecipe().Image}'"); + fixtureLog.Log($"Distributed Tests are starting..."); + globalEntryPoint.Announce(); // Previous test run may have been interrupted. // Begin by cleaning everything up. @@ -48,8 +46,7 @@ namespace DistTestCore { Stopwatch.Measure(fixtureLog, "Global setup", () => { - var wc = new WorkflowCreator(fixtureLog, configuration.GetK8sConfiguration(GetTimeSet()), string.Empty); - wc.CreateWorkflow().DeleteAllResources(); + globalEntryPoint.Tools.CreateWorkflow().DeleteNamespacesStartingWith(TestNamespacePrefix); }); } catch (Exception ex) @@ -62,6 +59,16 @@ namespace DistTestCore fixtureLog.Log("Global setup cleanup successful"); } + [OneTimeTearDown] + public void GlobalTearDown() + { + globalEntryPoint.Decommission( + // There shouldn't be any of either, but clean everything up regardless. + deleteKubernetesResources: true, + deleteTrackedFiles: true + ); + } + [SetUp] public void SetUpDistTest() { @@ -89,9 +96,17 @@ namespace DistTestCore } } - public TestFile GenerateTestFile(ByteSize size, string label = "") + public CoreInterface Ci { - return Get().FileManager.GenerateTestFile(size, label); + get + { + return Get().CoreInterface; + } + } + + public TrackedFile GenerateTestFile(ByteSize size, string label = "") + { + return Get().GenerateTestFile(size, label); } /// @@ -100,60 +115,10 @@ namespace DistTestCore /// public void ScopedTestFiles(Action action) { - Get().FileManager.PushFileSet(); - action(); - Get().FileManager.PopFileSet(); + Get().GetFileManager().ScopedFiles(action); } - public IOnlineCodexNode SetupCodexBootstrapNode() - { - return SetupCodexBootstrapNode(s => { }); - } - - public virtual IOnlineCodexNode SetupCodexBootstrapNode(Action setup) - { - return SetupCodexNode(s => - { - setup(s); - s.WithName("Bootstrap"); - }); - } - - public IOnlineCodexNode SetupCodexNode() - { - return SetupCodexNode(s => { }); - } - - public IOnlineCodexNode SetupCodexNode(Action setup) - { - return SetupCodexNodes(1, setup)[0]; - } - - public ICodexNodeGroup SetupCodexNodes(int numberOfNodes) - { - return SetupCodexNodes(numberOfNodes, s => { }); - } - - public virtual ICodexNodeGroup SetupCodexNodes(int numberOfNodes, Action setup) - { - var codexSetup = CreateCodexSetup(numberOfNodes); - - setup(codexSetup); - - return BringOnline(codexSetup); - } - - public ICodexNodeGroup BringOnline(ICodexSetup codexSetup) - { - return Get().CodexStarter.BringOnline((CodexSetup)codexSetup); - } - - public IEnumerable GetAllOnlineCodexNodes() - { - return Get().CodexStarter.RunningGroups.SelectMany(g => g.Nodes); - } - - public BaseLog GetTestLog() + public ILog GetTestLog() { return Get().Log; } @@ -170,27 +135,12 @@ namespace DistTestCore GetTestLog().Debug(msg); } - public PeerConnectionTestHelpers CreatePeerConnectionTestHelpers() - { - return new PeerConnectionTestHelpers(GetTestLog()); - } - - public PeerDownloadTestHelpers CreatePeerDownloadTestHelpers() - { - return new PeerDownloadTestHelpers(GetTestLog(), Get().FileManager); - } - public void Measure(string name, Action action) { Stopwatch.Measure(Get().Log, name, action); } - protected CodexSetup CreateCodexSetup(int numberOfNodes) - { - return new CodexSetup(numberOfNodes, configuration.GetCodexLogLevel()); - } - - private TestLifecycle Get() + protected TestLifecycle Get() { lock (lifecycleLock) { @@ -206,11 +156,9 @@ namespace DistTestCore { lock (lifecycleLock) { - var testNamespace = Guid.NewGuid().ToString(); + var testNamespace = TestNamespacePrefix + Guid.NewGuid().ToString(); var lifecycle = new TestLifecycle(fixtureLog.CreateTestLog(), configuration, GetTimeSet(), testNamespace); lifecycles.Add(testName, lifecycle); - DefaultContainerRecipe.TestsType = TestsType; - DefaultContainerRecipe.ApplicationIds = lifecycle.GetApplicationIds(); } }); } @@ -221,16 +169,34 @@ namespace DistTestCore var testResult = GetTestResult(); var testDuration = lifecycle.GetTestDuration(); fixtureLog.Log($"{GetCurrentTestName()} = {testResult} ({testDuration})"); - statusLog.ConcludeTest(testResult, testDuration, lifecycle.GetApplicationIds()); + statusLog.ConcludeTest(testResult, testDuration, lifecycle.GetPluginMetadata()); Stopwatch.Measure(fixtureLog, $"Teardown for {GetCurrentTestName()}", () => { - lifecycle.Log.EndTest(); - IncludeLogsAndMetricsOnTestFailure(lifecycle); + WriteEndTestLog(lifecycle.Log); + + IncludeLogsOnTestFailure(lifecycle); lifecycle.DeleteAllResources(); lifecycle = null!; }); } + private void WriteEndTestLog(TestLog log) + { + var result = TestContext.CurrentContext.Result; + + Log($"*** Finished: {GetCurrentTestName()} = {result.Outcome.Status}"); + if (!string.IsNullOrEmpty(result.Message)) + { + Log(result.Message); + Log($"{result.StackTrace}"); + } + + if (result.Outcome.Status == NUnit.Framework.Interfaces.TestStatus.Failed) + { + log.MarkAsFailed(); + } + } + private ITimeSet GetTimeSet() { if (ShouldUseLongTimeouts()) return new LongTimeSet(); @@ -252,57 +218,25 @@ namespace DistTestCore return testMethods.Any(m => m.GetCustomAttribute() != null); } - private void IncludeLogsAndMetricsOnTestFailure(TestLifecycle lifecycle) + private void IncludeLogsOnTestFailure(TestLifecycle lifecycle) { var result = TestContext.CurrentContext.Result; if (result.Outcome.Status == NUnit.Framework.Interfaces.TestStatus.Failed) { fixtureLog.MarkAsFailed(); - if (IsDownloadingLogsAndMetricsEnabled()) + if (IsDownloadingLogsEnabled()) { - lifecycle.Log.Log("Downloading all CodexNode logs and metrics because of test failure..."); - DownloadAllLogs(lifecycle); - DownloadAllMetrics(lifecycle); + lifecycle.Log.Log("Downloading all container logs because of test failure..."); + lifecycle.DownloadAllLogs(); } else { - lifecycle.Log.Log("Skipping download of all CodexNode logs and metrics due to [DontDownloadLogsAndMetricsOnFailure] attribute."); + lifecycle.Log.Log("Skipping download of all container logs due to [DontDownloadLogsOnFailure] attribute."); } } } - private void DownloadAllLogs(TestLifecycle lifecycle) - { - OnEachCodexNode(lifecycle, node => - { - lifecycle.DownloadLog(node.CodexAccess.Container); - }); - } - - private void DownloadAllMetrics(TestLifecycle lifecycle) - { - var metricsDownloader = new MetricsDownloader(lifecycle.Log); - - OnEachCodexNode(lifecycle, node => - { - var m = node.Metrics as MetricsAccess; - if (m != null) - { - metricsDownloader.DownloadAllMetricsForNode(node.GetName(), m); - } - }); - } - - private void OnEachCodexNode(TestLifecycle lifecycle, Action action) - { - var allNodes = lifecycle.CodexStarter.RunningGroups.SelectMany(g => g.Nodes); - foreach (var node in allNodes) - { - action(node); - } - } - private string GetCurrentTestName() { return $"[{TestContext.CurrentContext.Test.Name}]"; @@ -313,10 +247,10 @@ namespace DistTestCore return TestContext.CurrentContext.Result.Outcome.Status.ToString(); } - private bool IsDownloadingLogsAndMetricsEnabled() + private bool IsDownloadingLogsEnabled() { var testProperties = TestContext.CurrentContext.Test.Properties; - return !testProperties.ContainsKey(DontDownloadLogsAndMetricsOnFailureAttribute.DontDownloadKey); + return !testProperties.ContainsKey(DontDownloadLogsOnFailureAttribute.DontDownloadKey); } } diff --git a/DistTestCore/DistTestCore.csproj b/Tests/DistTestCore/DistTestCore.csproj similarity index 55% rename from DistTestCore/DistTestCore.csproj rename to Tests/DistTestCore/DistTestCore.csproj index 89d4b1f9..91d1fd09 100644 --- a/DistTestCore/DistTestCore.csproj +++ b/Tests/DistTestCore/DistTestCore.csproj @@ -6,14 +6,6 @@ enable enable - - - - - - Never - - @@ -23,8 +15,7 @@ - - - + + diff --git a/Tests/DistTestCore/DontDownloadLogsOnFailureAttribute.cs b/Tests/DistTestCore/DontDownloadLogsOnFailureAttribute.cs new file mode 100644 index 00000000..800b35b8 --- /dev/null +++ b/Tests/DistTestCore/DontDownloadLogsOnFailureAttribute.cs @@ -0,0 +1,15 @@ +using NUnit.Framework; + +namespace DistTestCore +{ + [AttributeUsage(AttributeTargets.Method, AllowMultiple = false)] + public class DontDownloadLogsOnFailureAttribute : PropertyAttribute + { + public const string DontDownloadKey = "DontDownloadLogs"; + + public DontDownloadLogsOnFailureAttribute() + : base(DontDownloadKey) + { + } + } +} diff --git a/Tests/DistTestCore/DownloadedLogExtensions.cs b/Tests/DistTestCore/DownloadedLogExtensions.cs new file mode 100644 index 00000000..59b020a5 --- /dev/null +++ b/Tests/DistTestCore/DownloadedLogExtensions.cs @@ -0,0 +1,13 @@ +using Core; +using NUnit.Framework; + +namespace DistTestCore +{ + public static class DownloadedLogExtensions + { + public static void AssertLogContains(this IDownloadedLog log, string expectedString) + { + Assert.That(log.DoesLogContain(expectedString), $"Did not find '{expectedString}' in log."); + } + } +} diff --git a/DistTestCore/Helpers/AssertHelpers.cs b/Tests/DistTestCore/Helpers/AssertHelpers.cs similarity index 100% rename from DistTestCore/Helpers/AssertHelpers.cs rename to Tests/DistTestCore/Helpers/AssertHelpers.cs diff --git a/Tests/DistTestCore/Logs/BaseTestLog.cs b/Tests/DistTestCore/Logs/BaseTestLog.cs new file mode 100644 index 00000000..99cd0f14 --- /dev/null +++ b/Tests/DistTestCore/Logs/BaseTestLog.cs @@ -0,0 +1,29 @@ +using Logging; + +namespace DistTestCore.Logs +{ + public abstract class BaseTestLog : BaseLog + { + private bool hasFailed; + + public BaseTestLog(bool debug) + : base(debug) + { + } + + public void WriteLogTag() + { + var runId = NameUtils.GetRunId(); + var category = NameUtils.GetCategoryName(); + var name = NameUtils.GetTestMethodName(); + LogFile.WriteRaw($"{runId} {category} {name}"); + } + + public void MarkAsFailed() + { + if (hasFailed) return; + hasFailed = true; + LogFile.ConcatToFilename("_FAILED"); + } + } +} diff --git a/Logging/FixtureLog.cs b/Tests/DistTestCore/Logs/FixtureLog.cs similarity index 88% rename from Logging/FixtureLog.cs rename to Tests/DistTestCore/Logs/FixtureLog.cs index 809ff388..c32ce87d 100644 --- a/Logging/FixtureLog.cs +++ b/Tests/DistTestCore/Logs/FixtureLog.cs @@ -1,6 +1,8 @@ -namespace Logging +using Logging; + +namespace DistTestCore.Logs { - public class FixtureLog : BaseLog + public class FixtureLog : BaseTestLog { private readonly string fullName; private readonly LogConfig config; diff --git a/Tests/DistTestCore/Logs/StatusLog.cs b/Tests/DistTestCore/Logs/StatusLog.cs new file mode 100644 index 00000000..b7ce8913 --- /dev/null +++ b/Tests/DistTestCore/Logs/StatusLog.cs @@ -0,0 +1,46 @@ +using Logging; +using Newtonsoft.Json; + +namespace DistTestCore.Logs +{ + public class StatusLog + { + private readonly object fileLock = new object(); + private readonly string fullName; + private readonly string fixtureName; + + public StatusLog(LogConfig config, DateTime start, string name = "") + { + fullName = NameUtils.GetFixtureFullName(config, start, name) + "_STATUS.log"; + fixtureName = NameUtils.GetRawFixtureName(); + } + + public void ConcludeTest(string resultStatus, string testDuration, Dictionary data) + { + data.Add("timestamp", DateTime.UtcNow.ToString("o")); + data.Add("runid", NameUtils.GetRunId()); + data.Add("status", resultStatus); + data.Add("testid", NameUtils.GetTestId()); + data.Add("category", NameUtils.GetCategoryName()); + data.Add("fixturename", fixtureName); + data.Add("testname", NameUtils.GetTestMethodName()); + data.Add("testduration", testDuration); + Write(data); + } + + private void Write(Dictionary data) + { + try + { + lock (fileLock) + { + File.AppendAllLines(fullName, new[] { JsonConvert.SerializeObject(data) }); + } + } + catch (Exception ex) + { + Console.WriteLine("Unable to write to status log: " + ex); + } + } + } +} diff --git a/Tests/DistTestCore/Logs/TestLog.cs b/Tests/DistTestCore/Logs/TestLog.cs new file mode 100644 index 00000000..0f8831b6 --- /dev/null +++ b/Tests/DistTestCore/Logs/TestLog.cs @@ -0,0 +1,22 @@ +namespace DistTestCore.Logs +{ + public class TestLog : BaseTestLog + { + private readonly string methodName; + private readonly string fullName; + + public TestLog(string folder, bool debug, string name = "") + : base(debug) + { + methodName = NameUtils.GetTestMethodName(name); + fullName = Path.Combine(folder, methodName); + + Log($"*** Begin: {methodName}"); + } + + protected override string GetFullName() + { + return fullName; + } + } +} diff --git a/Tests/DistTestCore/LongTimeSet.cs b/Tests/DistTestCore/LongTimeSet.cs new file mode 100644 index 00000000..cc441ccd --- /dev/null +++ b/Tests/DistTestCore/LongTimeSet.cs @@ -0,0 +1,32 @@ +using Core; + +namespace DistTestCore +{ + public class LongTimeSet : ITimeSet + { + public TimeSpan HttpCallTimeout() + { + return TimeSpan.FromHours(2); + } + + public TimeSpan HttpCallRetryTime() + { + return TimeSpan.FromHours(5); + } + + public TimeSpan HttpCallRetryDelay() + { + return TimeSpan.FromSeconds(2); + } + + public TimeSpan WaitForK8sServiceDelay() + { + return TimeSpan.FromSeconds(10); + } + + public TimeSpan K8sOperationTimeout() + { + return TimeSpan.FromMinutes(15); + } + } +} diff --git a/Tests/DistTestCore/LongTimeoutsTestAttribute.cs b/Tests/DistTestCore/LongTimeoutsTestAttribute.cs new file mode 100644 index 00000000..5bbc612f --- /dev/null +++ b/Tests/DistTestCore/LongTimeoutsTestAttribute.cs @@ -0,0 +1,9 @@ +using NUnit.Framework; + +namespace DistTestCore +{ + [AttributeUsage(AttributeTargets.Method, AllowMultiple = false)] + public class UseLongTimeoutsAttribute : PropertyAttribute + { + } +} diff --git a/Logging/NameUtils.cs b/Tests/DistTestCore/NameUtils.cs similarity index 97% rename from Logging/NameUtils.cs rename to Tests/DistTestCore/NameUtils.cs index 2ca47e20..fea549d2 100644 --- a/Logging/NameUtils.cs +++ b/Tests/DistTestCore/NameUtils.cs @@ -1,6 +1,7 @@ -using NUnit.Framework; +using Logging; +using NUnit.Framework; -namespace Logging +namespace DistTestCore { public static class NameUtils { diff --git a/Tests/DistTestCore/TestLifecycle.cs b/Tests/DistTestCore/TestLifecycle.cs new file mode 100644 index 00000000..c0904c11 --- /dev/null +++ b/Tests/DistTestCore/TestLifecycle.cs @@ -0,0 +1,94 @@ +using Core; +using DistTestCore.Logs; +using FileUtils; +using KubernetesWorkflow; +using Utils; + +namespace DistTestCore +{ + public class TestLifecycle : IK8sHooks + { + private const string TestsType = "dist-tests"; + private readonly DateTime testStart; + private readonly EntryPoint entryPoint; + private readonly List runningContainers = new List(); + + public TestLifecycle(TestLog log, Configuration configuration, ITimeSet timeSet, string testNamespace) + { + Log = log; + Configuration = configuration; + TimeSet = timeSet; + testStart = DateTime.UtcNow; + + entryPoint = new EntryPoint(log, configuration.GetK8sConfiguration(timeSet, this, testNamespace), configuration.GetFileManagerFolder(), timeSet); + CoreInterface = entryPoint.CreateInterface(); + + log.WriteLogTag(); + } + + public TestLog Log { get; } + public Configuration Configuration { get; } + public ITimeSet TimeSet { get; } + public CoreInterface CoreInterface { get; } + + public void DeleteAllResources() + { + entryPoint.Decommission( + deleteKubernetesResources: true, + deleteTrackedFiles: true + ); + } + + public TrackedFile GenerateTestFile(ByteSize size, string label = "") + { + return entryPoint.Tools.GetFileManager().GenerateFile(size, label); + } + + public IFileManager GetFileManager() + { + return entryPoint.Tools.GetFileManager(); + } + + public Dictionary GetPluginMetadata() + { + return entryPoint.GetPluginMetadata(); + } + + public string GetTestDuration() + { + var testDuration = DateTime.UtcNow - testStart; + return Time.FormatDuration(testDuration); + } + + public void OnContainersStarted(RunningContainers rc) + { + runningContainers.Add(rc); + } + + public void OnContainersStopped(RunningContainers rc) + { + runningContainers.Remove(rc); + } + + public void OnContainerRecipeCreated(ContainerRecipe recipe) + { + recipe.PodLabels.Add("tests-type", TestsType); + recipe.PodLabels.Add("runid", NameUtils.GetRunId()); + recipe.PodLabels.Add("testid", NameUtils.GetTestId()); + recipe.PodLabels.Add("category", NameUtils.GetCategoryName()); + recipe.PodLabels.Add("fixturename", NameUtils.GetRawFixtureName()); + recipe.PodLabels.Add("testname", NameUtils.GetTestMethodName()); + } + + public void DownloadAllLogs() + { + foreach (var rc in runningContainers) + { + foreach (var c in rc.Containers) + { + CoreInterface.DownloadLog(c); + } + } + } + } +} diff --git a/Tests/PeerDiscoveryTests/LayeredDiscoveryTests.cs b/Tests/PeerDiscoveryTests/LayeredDiscoveryTests.cs deleted file mode 100644 index 8cd32a2d..00000000 --- a/Tests/PeerDiscoveryTests/LayeredDiscoveryTests.cs +++ /dev/null @@ -1,52 +0,0 @@ -using DistTestCore; -using NUnit.Framework; - -namespace Tests.PeerDiscoveryTests -{ - [TestFixture] - public class LayeredDiscoveryTests : DistTest - { - [Test] - public void TwoLayersTest() - { - var root = SetupCodexNode(); - var l1Source = SetupCodexNode(s => s.WithBootstrapNode(root)); - var l1Node = SetupCodexNode(s => s.WithBootstrapNode(root)); - var l2Target = SetupCodexNode(s => s.WithBootstrapNode(l1Node)); - - AssertAllNodesConnected(); - } - - [Test] - public void ThreeLayersTest() - { - var root = SetupCodexNode(); - var l1Source = SetupCodexNode(s => s.WithBootstrapNode(root)); - var l1Node = SetupCodexNode(s => s.WithBootstrapNode(root)); - var l2Node = SetupCodexNode(s => s.WithBootstrapNode(l1Node)); - var l3Target = SetupCodexNode(s => s.WithBootstrapNode(l2Node)); - - AssertAllNodesConnected(); - } - - [TestCase(3)] - [TestCase(5)] - [TestCase(10)] - [TestCase(20)] - public void NodeChainTest(int chainLength) - { - var node = SetupCodexNode(); - for (var i = 1; i < chainLength; i++) - { - node = SetupCodexNode(s => s.WithBootstrapNode(node)); - } - - AssertAllNodesConnected(); - } - - private void AssertAllNodesConnected() - { - CreatePeerConnectionTestHelpers().AssertFullyConnected(GetAllOnlineCodexNodes()); - } - } -} diff --git a/Tools/CodexNetDeployer/CodexNetDeployer.csproj b/Tools/CodexNetDeployer/CodexNetDeployer.csproj new file mode 100644 index 00000000..9168ae7d --- /dev/null +++ b/Tools/CodexNetDeployer/CodexNetDeployer.csproj @@ -0,0 +1,17 @@ + + + + Exe + net7.0 + enable + enable + + + + + + + + + + diff --git a/Tools/CodexNetDeployer/CodexNodeStarter.cs b/Tools/CodexNetDeployer/CodexNodeStarter.cs new file mode 100644 index 00000000..f1cfed48 --- /dev/null +++ b/Tools/CodexNetDeployer/CodexNodeStarter.cs @@ -0,0 +1,102 @@ +using CodexContractsPlugin; +using CodexPlugin; +using Core; +using GethPlugin; +using Utils; + +namespace CodexNetDeployer +{ + public class CodexNodeStarter + { + private readonly Configuration config; + private readonly CoreInterface ci; + private readonly IGethNode gethNode; + private readonly ICodexContracts contracts; + private ICodexNode? bootstrapNode = null; + private int validatorsLeft; + + public CodexNodeStarter(Configuration config, CoreInterface ci, IGethNode gethNode, ICodexContracts contracts, int numberOfValidators) + { + this.config = config; + this.ci = ci; + this.gethNode = gethNode; + this.contracts = contracts; + validatorsLeft = numberOfValidators; + } + + public CodexNodeStartResult? Start(int i) + { + var name = GetCodexContainerName(i); + Console.Write($" - {i} ({name})"); + Console.CursorLeft = 30; + + ICodexNode? codexNode = null; + try + { + codexNode = ci.StartCodexNode(s => + { + s.WithName(name); + s.WithLogLevel(config.CodexLogLevel); + s.WithStorageQuota(config.StorageQuota!.Value.MB()); + s.EnableMarketplace(gethNode, contracts, 100.Eth(), config.InitialTestTokens.TestTokens(), validatorsLeft > 0); + s.EnableMetrics(); + + if (bootstrapNode != null) s.WithBootstrapNode(bootstrapNode); + if (config.BlockTTL != Configuration.SecondsIn1Day) s.WithBlockTTL(TimeSpan.FromSeconds(config.BlockTTL)); + if (config.BlockMI != Configuration.TenMinutes) s.WithBlockMaintenanceInterval(TimeSpan.FromSeconds(config.BlockMI)); + if (config.BlockMN != 1000) s.WithBlockMaintenanceNumber(config.BlockMN); + }); + + var debugInfo = codexNode.GetDebugInfo(); + if (!string.IsNullOrWhiteSpace(debugInfo.spr)) + { + Console.Write("Online\t"); + + var response = codexNode.Marketplace.MakeStorageAvailable( + size: config.StorageSell!.Value.MB(), + minPriceForTotalSpace: config.MinPrice.TestTokens(), + maxCollateral: config.MaxCollateral.TestTokens(), + maxDuration: TimeSpan.FromSeconds(config.MaxDuration)); + + if (!string.IsNullOrEmpty(response)) + { + Console.Write("Storage available\tOK" + Environment.NewLine); + + validatorsLeft--; + if (bootstrapNode == null) bootstrapNode = codexNode; + return new CodexNodeStartResult(codexNode); + } + } + } + catch (Exception ex) + { + Console.WriteLine("Exception:" + ex.ToString()); + } + + Console.WriteLine("Unknown failure."); + if (codexNode != null) + { + Console.WriteLine("Downloading container log."); + ci.DownloadLog(codexNode); + } + + return null; + } + + private string GetCodexContainerName(int i) + { + if (i == 0) return "BOOTSTRAP"; + return "CODEX" + i; + } + } + + public class CodexNodeStartResult + { + public CodexNodeStartResult(ICodexNode codexNode) + { + CodexNode = codexNode; + } + + public ICodexNode CodexNode { get; } + } +} diff --git a/CodexNetDeployer/Configuration.cs b/Tools/CodexNetDeployer/Configuration.cs similarity index 93% rename from CodexNetDeployer/Configuration.cs rename to Tools/CodexNetDeployer/Configuration.cs index 7ad241a3..3ff1020e 100644 --- a/CodexNetDeployer/Configuration.cs +++ b/Tools/CodexNetDeployer/Configuration.cs @@ -1,6 +1,5 @@ using ArgsUniform; -using DistTestCore.Codex; -using DistTestCore.Metrics; +using CodexPlugin; namespace CodexNetDeployer { @@ -51,14 +50,14 @@ namespace CodexNetDeployer [Uniform("block-mn", "bmn", "BLOCKMN", false, "Number of blocks maintained per interval. Default is 1000 blocks.")] public int BlockMN { get; set; } = 1000; - [Uniform("metrics", "m", "METRICS", false, "[None*, Record, Dashboard]. Determines if metrics will be recorded and if a dashboard service will be created.")] - public MetricsMode Metrics { get; set; } = MetricsMode.None; + [Uniform("metrics", "m", "METRICS", false, "[true, false]. Determines if metrics will be recorded. Default is false.")] + public bool Metrics { get; set; } = false; [Uniform("teststype-podlabel", "ttpl", "TESTSTYPE-PODLABEL", false, "Each kubernetes pod will be created with a label 'teststype' with value 'continuous'. " + "set this option to override the label value.")] public string TestsTypePodLabel { get; set; } = "continuous-tests"; - [Uniform("check-connect", "cc", "CHECKCONNECT", false, "If true, deployer check ensure peer-connectivity between all deployed nodes after deployment.")] + [Uniform("check-connect", "cc", "CHECKCONNECT", false, "If true, deployer check ensure peer-connectivity between all deployed nodes after deployment. Default is false.")] public bool CheckPeerConnection { get; set; } = false; public List Validate() diff --git a/Tools/CodexNetDeployer/Deployer.cs b/Tools/CodexNetDeployer/Deployer.cs new file mode 100644 index 00000000..ec370b26 --- /dev/null +++ b/Tools/CodexNetDeployer/Deployer.cs @@ -0,0 +1,163 @@ +using CodexContractsPlugin; +using CodexPlugin; +using Core; +using GethPlugin; +using KubernetesWorkflow; +using Logging; +using MetricsPlugin; + +namespace CodexNetDeployer +{ + public class Deployer + { + private readonly Configuration config; + private readonly PeerConnectivityChecker peerConnectivityChecker; + private readonly EntryPoint entryPoint; + + public Deployer(Configuration config) + { + this.config = config; + peerConnectivityChecker = new PeerConnectivityChecker(); + + ProjectPlugin.Load(); + ProjectPlugin.Load(); + ProjectPlugin.Load(); + ProjectPlugin.Load(); + entryPoint = CreateEntryPoint(new NullLog()); + } + + public void AnnouncePlugins() + { + var ep = CreateEntryPoint(new ConsoleLog()); + + Log("Using plugins:" + Environment.NewLine); + var metadata = ep.GetPluginMetadata(); + var longestKey = metadata.Keys.Max(k => k.Length); + foreach (var entry in metadata) + { + Console.Write(entry.Key); + Console.CursorLeft = longestKey + 5; + Console.WriteLine($"= {entry.Value}"); + } + Log(""); + } + + public CodexDeployment Deploy() + { + Log("Initializing..."); + var ci = entryPoint.CreateInterface(); + + Log("Deploying Geth instance..."); + var gethDeployment = ci.DeployGeth(s => s.IsMiner()); + var gethNode = ci.WrapGethDeployment(gethDeployment); + + Log("Geth started. Deploying Codex contracts..."); + var contractsDeployment = ci.DeployCodexContracts(gethNode); + var contracts = ci.WrapCodexContractsDeployment(contractsDeployment); + Log("Codex contracts deployed."); + + Log("Starting Codex nodes..."); + var codexStarter = new CodexNodeStarter(config, ci, gethNode, contracts, config.NumberOfValidators!.Value); + var startResults = new List(); + for (var i = 0; i < config.NumberOfCodexNodes; i++) + { + var result = codexStarter.Start(i); + if (result != null) startResults.Add(result); + } + + Log("Codex nodes started."); + var metricsService = StartMetricsService(ci, startResults); + + CheckPeerConnectivity(startResults); + CheckContainerRestarts(startResults); + + var codexContainers = startResults.Select(s => s.CodexNode.Container).ToArray(); + return new CodexDeployment(codexContainers, gethDeployment, metricsService, CreateMetadata()); + } + + private EntryPoint CreateEntryPoint(ILog log) + { + var kubeConfig = GetKubeConfig(config.KubeConfigFile); + + var configuration = new KubernetesWorkflow.Configuration( + kubeConfig, + operationTimeout: TimeSpan.FromSeconds(30), + retryDelay: TimeSpan.FromSeconds(10), + kubernetesNamespace: config.KubeNamespace); + + return new EntryPoint(log, configuration, string.Empty); + } + + private RunningContainer? StartMetricsService(CoreInterface ci, List startResults) + { + if (!config.Metrics) return null; + + Log("Starting metrics service..."); + + var runningContainer = ci.DeployMetricsCollector(startResults.Select(r => r.CodexNode).ToArray()); + + Log("Metrics service started."); + + return runningContainer; + } + + private string? GetKubeConfig(string kubeConfigFile) + { + if (string.IsNullOrEmpty(kubeConfigFile) || kubeConfigFile.ToLowerInvariant() == "null") return null; + return kubeConfigFile; + } + + private void CheckPeerConnectivity(List codexContainers) + { + if (!config.CheckPeerConnection) return; + + Log("Starting peer connectivity check for deployed nodes..."); + peerConnectivityChecker.CheckConnectivity(codexContainers); + Log("Check passed."); + } + + private void CheckContainerRestarts(List startResults) + { + var crashes = new List(); + Log("Starting container crash check..."); + foreach (var startResult in startResults) + { + var watcher = startResult.CodexNode.CrashWatcher; + if (watcher == null) throw new Exception("Expected each CodexNode container to be created with a crash-watcher."); + if (watcher.HasContainerCrashed()) crashes.Add(startResult.CodexNode.Container); + } + + if (!crashes.Any()) + { + Log("Check passed."); + } + else + { + Log($"Check failed. The following containers have crashed: {string.Join(",", crashes.Select(c => c.Name))}"); + throw new Exception("Deployment failed: One or more containers crashed."); + } + } + + private DeploymentMetadata CreateMetadata() + { + return new DeploymentMetadata( + kubeNamespace: config.KubeNamespace, + numberOfCodexNodes: config.NumberOfCodexNodes!.Value, + numberOfValidators: config.NumberOfValidators!.Value, + storageQuotaMB: config.StorageQuota!.Value, + codexLogLevel: config.CodexLogLevel, + initialTestTokens: config.InitialTestTokens, + minPrice: config.MinPrice, + maxCollateral: config.MaxCollateral, + maxDuration: config.MaxDuration, + blockTTL: config.BlockTTL, + blockMI: config.BlockMI, + blockMN: config.BlockMN); + } + + private void Log(string msg) + { + Console.WriteLine(msg); + } + } +} diff --git a/Tools/CodexNetDeployer/PeerConnectivityChecker.cs b/Tools/CodexNetDeployer/PeerConnectivityChecker.cs new file mode 100644 index 00000000..87bdc316 --- /dev/null +++ b/Tools/CodexNetDeployer/PeerConnectivityChecker.cs @@ -0,0 +1,17 @@ +using DistTestCore.Helpers; +using Logging; + +namespace CodexNetDeployer +{ + public class PeerConnectivityChecker + { + public void CheckConnectivity(List startResults) + { + var log = new ConsoleLog(); + var checker = new PeerConnectionTestHelpers(log); + var nodes = startResults.Select(r => r.CodexNode); + + checker.AssertFullyConnected(nodes); + } + } +} diff --git a/CodexNetDeployer/Program.cs b/Tools/CodexNetDeployer/Program.cs similarity index 77% rename from CodexNetDeployer/Program.cs rename to Tools/CodexNetDeployer/Program.cs index 8ddf527b..73fef7fe 100644 --- a/CodexNetDeployer/Program.cs +++ b/Tools/CodexNetDeployer/Program.cs @@ -1,8 +1,5 @@ using ArgsUniform; using CodexNetDeployer; -using DistTestCore.Codex; -using DistTestCore.Marketplace; -using DistTestCore.Metrics; using Newtonsoft.Json; using Configuration = CodexNetDeployer.Configuration; @@ -26,12 +23,8 @@ public class Program return; } - Console.WriteLine("Using images:" + nl + - $"\tCodex image: '{new CodexContainerRecipe().Image}'" + nl + - $"\tCodexContracts image: '{new CodexContractsContainerRecipe().Image}'" + nl + - $"\tPrometheus image: '{new PrometheusContainerRecipe().Image}'" + nl + - $"\tGeth image: '{new GethContainerRecipe().Image}'" + nl + - $"\tGrafana image: '{new GrafanaContainerRecipe().Image}'" + nl); + var deployer = new Deployer(config); + deployer.AnnouncePlugins(); if (!args.Any(a => a == "-y")) { @@ -40,7 +33,6 @@ public class Program Console.WriteLine("I think so too."); } - var deployer = new Deployer(config); var deployment = deployer.Deploy(); Console.WriteLine("Writing codex-deployment.json..."); diff --git a/CodexNetDeployer/deploy-continuous-testnet.sh b/Tools/CodexNetDeployer/deploy-continuous-testnet.sh similarity index 93% rename from CodexNetDeployer/deploy-continuous-testnet.sh rename to Tools/CodexNetDeployer/deploy-continuous-testnet.sh index afcfc448..6a3bbe81 100644 --- a/CodexNetDeployer/deploy-continuous-testnet.sh +++ b/Tools/CodexNetDeployer/deploy-continuous-testnet.sh @@ -12,5 +12,5 @@ dotnet run \ --block-ttl=180 \ --block-mi=120 \ --block-mn=10000 \ - --metrics=Dashboard \ + --metrics=1 \ --check-connect=1 diff --git a/cs-codex-dist-testing.sln b/cs-codex-dist-testing.sln index 8e696b3d..7d3dda94 100644 --- a/cs-codex-dist-testing.sln +++ b/cs-codex-dist-testing.sln @@ -3,27 +3,45 @@ Microsoft Visual Studio Solution File, Format Version 12.00 # Visual Studio Version 17 VisualStudioVersion = 17.4.33213.308 MinimumVisualStudioVersion = 10.0.40219.1 -Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Tests", "Tests\Tests.csproj", "{57F57B85-A537-4D3A-B7AE-B72C66B74AAB}" +Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Tests", "Tests", "{88C2A621-8A98-4D07-8625-7900FC8EF89E}" EndProject -Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "TestsLong", "LongTests\TestsLong.csproj", "{AFCE270E-F844-4A7C-9006-69AE622BB1F4}" +Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Tools", "Tools", "{7591C5B3-D86E-4AE4-8ED2-B272D17FE7E3}" EndProject -Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "DistTestCore", "DistTestCore\DistTestCore.csproj", "{47F31305-6E68-4827-8E39-7B41DAA1CE7A}" +Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Framework", "Framework", "{81AE04BC-CBFA-4E6F-B039-8208E9AFAAE7}" EndProject -Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "KubernetesWorkflow", "KubernetesWorkflow\KubernetesWorkflow.csproj", "{359123AA-3D9B-4442-80F4-19E32E3EC9EA}" +Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "ProjectPlugins", "ProjectPlugins", "{8F1F1C2A-E313-4E0C-BE40-58FB0BA91124}" EndProject -Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Utils", "Utils\Utils.csproj", "{957DE3B8-9571-450A-8609-B267DCA8727C}" +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "ArgsUniform", "Framework\ArgsUniform\ArgsUniform.csproj", "{9922732F-01B3-4DBB-ADEC-E5451AB90CEE}" EndProject -Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Logging", "Logging\Logging.csproj", "{8481A4A6-4BDD-41B0-A3EB-EF53F7BD40D1}" +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Core", "Framework\Core\Core.csproj", "{D5E952BA-DCB1-4EA8-A038-6E2E0FCD64D9}" EndProject -Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "NethereumWorkflow", "Nethereum\NethereumWorkflow.csproj", "{D6C3555E-D52D-4993-A87B-71AB650398FD}" +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FileUtils", "Framework\FileUtils\FileUtils.csproj", "{D10125E6-FF03-4292-A22C-9D622B2ACEDE}" EndProject -Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "ContinuousTests", "ContinuousTests\ContinuousTests.csproj", "{025B7074-0A09-4FCC-9BB9-03AE2A961EA1}" +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "KubernetesWorkflow", "Framework\KubernetesWorkflow\KubernetesWorkflow.csproj", "{98198410-71F9-4498-8550-E6F08B1FC4FA}" EndProject -Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "CodexNetDeployer", "CodexNetDeployer\CodexNetDeployer.csproj", "{871CAF12-14BE-4509-BC6E-20FDF0B1083A}" +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Logging", "Framework\Logging\Logging.csproj", "{4FB7FC96-CB01-4905-9E40-3768692EDC0A}" EndProject -Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "ArgsUniform", "ArgsUniform\ArgsUniform.csproj", "{634324B1-E359-42B4-A269-BDC429936B3C}" +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "NethereumWorkflow", "Framework\NethereumWorkflow\NethereumWorkflow.csproj", "{70CFFF7A-FA63-48DB-B304-8C859998F339}" EndProject -Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "CodexNetDownloader", "CodexNetDownloader\CodexNetDownloader.csproj", "{6CDF35D2-906A-4285-8E1F-4794588B948B}" +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Utils", "Framework\Utils\Utils.csproj", "{8D264872-5361-4AC5-8A99-908137E13A22}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "CodexContractsPlugin", "ProjectPlugins\CodexContractsPlugin\CodexContractsPlugin.csproj", "{65D97CC1-E566-423E-9BD8-A1FA936CAAFA}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "CodexPlugin", "ProjectPlugins\CodexPlugin\CodexPlugin.csproj", "{F36DFCB1-C33F-426B-851B-FB1DEE7F028E}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "GethPlugin", "ProjectPlugins\GethPlugin\GethPlugin.csproj", "{8B39F251-F948-40AE-8922-3D8C4E529A86}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "MetricsPlugin", "ProjectPlugins\MetricsPlugin\MetricsPlugin.csproj", "{8DE8FF65-23CB-4FB3-8BE5-6C0BEC4BAA97}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "CodexContinuousTests", "Tests\CodexContinuousTests\CodexContinuousTests.csproj", "{ADEC06CF-6F3A-44C5-AA57-EAB94124AC82}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "CodexTestsLong", "Tests\CodexLongTests\CodexTestsLong.csproj", "{0C2D067F-053C-45A8-AE0D-4EB388E77C89}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "CodexTests", "Tests\CodexTests\CodexTests.csproj", "{562EC700-6984-4C9A-83BF-3BF4E3EB1A64}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "DistTestCore", "Tests\DistTestCore\DistTestCore.csproj", "{E849B7BA-FDCC-4CFF-998F-845ED2F1BF40}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "CodexNetDeployer", "Tools\CodexNetDeployer\CodexNetDeployer.csproj", "{3417D508-E2F4-4974-8988-BB124046D9E2}" EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution @@ -31,54 +49,92 @@ Global Release|Any CPU = Release|Any CPU EndGlobalSection GlobalSection(ProjectConfigurationPlatforms) = postSolution - {57F57B85-A537-4D3A-B7AE-B72C66B74AAB}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {57F57B85-A537-4D3A-B7AE-B72C66B74AAB}.Debug|Any CPU.Build.0 = Debug|Any CPU - {57F57B85-A537-4D3A-B7AE-B72C66B74AAB}.Release|Any CPU.ActiveCfg = Release|Any CPU - {57F57B85-A537-4D3A-B7AE-B72C66B74AAB}.Release|Any CPU.Build.0 = Release|Any CPU - {AFCE270E-F844-4A7C-9006-69AE622BB1F4}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {AFCE270E-F844-4A7C-9006-69AE622BB1F4}.Debug|Any CPU.Build.0 = Debug|Any CPU - {AFCE270E-F844-4A7C-9006-69AE622BB1F4}.Release|Any CPU.ActiveCfg = Release|Any CPU - {AFCE270E-F844-4A7C-9006-69AE622BB1F4}.Release|Any CPU.Build.0 = Release|Any CPU - {47F31305-6E68-4827-8E39-7B41DAA1CE7A}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {47F31305-6E68-4827-8E39-7B41DAA1CE7A}.Debug|Any CPU.Build.0 = Debug|Any CPU - {47F31305-6E68-4827-8E39-7B41DAA1CE7A}.Release|Any CPU.ActiveCfg = Release|Any CPU - {47F31305-6E68-4827-8E39-7B41DAA1CE7A}.Release|Any CPU.Build.0 = Release|Any CPU - {359123AA-3D9B-4442-80F4-19E32E3EC9EA}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {359123AA-3D9B-4442-80F4-19E32E3EC9EA}.Debug|Any CPU.Build.0 = Debug|Any CPU - {359123AA-3D9B-4442-80F4-19E32E3EC9EA}.Release|Any CPU.ActiveCfg = Release|Any CPU - {359123AA-3D9B-4442-80F4-19E32E3EC9EA}.Release|Any CPU.Build.0 = Release|Any CPU - {957DE3B8-9571-450A-8609-B267DCA8727C}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {957DE3B8-9571-450A-8609-B267DCA8727C}.Debug|Any CPU.Build.0 = Debug|Any CPU - {957DE3B8-9571-450A-8609-B267DCA8727C}.Release|Any CPU.ActiveCfg = Release|Any CPU - {957DE3B8-9571-450A-8609-B267DCA8727C}.Release|Any CPU.Build.0 = Release|Any CPU - {8481A4A6-4BDD-41B0-A3EB-EF53F7BD40D1}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {8481A4A6-4BDD-41B0-A3EB-EF53F7BD40D1}.Debug|Any CPU.Build.0 = Debug|Any CPU - {8481A4A6-4BDD-41B0-A3EB-EF53F7BD40D1}.Release|Any CPU.ActiveCfg = Release|Any CPU - {8481A4A6-4BDD-41B0-A3EB-EF53F7BD40D1}.Release|Any CPU.Build.0 = Release|Any CPU - {D6C3555E-D52D-4993-A87B-71AB650398FD}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {D6C3555E-D52D-4993-A87B-71AB650398FD}.Debug|Any CPU.Build.0 = Debug|Any CPU - {D6C3555E-D52D-4993-A87B-71AB650398FD}.Release|Any CPU.ActiveCfg = Release|Any CPU - {D6C3555E-D52D-4993-A87B-71AB650398FD}.Release|Any CPU.Build.0 = Release|Any CPU - {025B7074-0A09-4FCC-9BB9-03AE2A961EA1}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {025B7074-0A09-4FCC-9BB9-03AE2A961EA1}.Debug|Any CPU.Build.0 = Debug|Any CPU - {025B7074-0A09-4FCC-9BB9-03AE2A961EA1}.Release|Any CPU.ActiveCfg = Release|Any CPU - {025B7074-0A09-4FCC-9BB9-03AE2A961EA1}.Release|Any CPU.Build.0 = Release|Any CPU - {871CAF12-14BE-4509-BC6E-20FDF0B1083A}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {871CAF12-14BE-4509-BC6E-20FDF0B1083A}.Debug|Any CPU.Build.0 = Debug|Any CPU - {871CAF12-14BE-4509-BC6E-20FDF0B1083A}.Release|Any CPU.ActiveCfg = Release|Any CPU - {871CAF12-14BE-4509-BC6E-20FDF0B1083A}.Release|Any CPU.Build.0 = Release|Any CPU - {634324B1-E359-42B4-A269-BDC429936B3C}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {634324B1-E359-42B4-A269-BDC429936B3C}.Debug|Any CPU.Build.0 = Debug|Any CPU - {634324B1-E359-42B4-A269-BDC429936B3C}.Release|Any CPU.ActiveCfg = Release|Any CPU - {634324B1-E359-42B4-A269-BDC429936B3C}.Release|Any CPU.Build.0 = Release|Any CPU - {6CDF35D2-906A-4285-8E1F-4794588B948B}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {6CDF35D2-906A-4285-8E1F-4794588B948B}.Debug|Any CPU.Build.0 = Debug|Any CPU - {6CDF35D2-906A-4285-8E1F-4794588B948B}.Release|Any CPU.ActiveCfg = Release|Any CPU - {6CDF35D2-906A-4285-8E1F-4794588B948B}.Release|Any CPU.Build.0 = Release|Any CPU + {9922732F-01B3-4DBB-ADEC-E5451AB90CEE}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {9922732F-01B3-4DBB-ADEC-E5451AB90CEE}.Debug|Any CPU.Build.0 = Debug|Any CPU + {9922732F-01B3-4DBB-ADEC-E5451AB90CEE}.Release|Any CPU.ActiveCfg = Release|Any CPU + {9922732F-01B3-4DBB-ADEC-E5451AB90CEE}.Release|Any CPU.Build.0 = Release|Any CPU + {D5E952BA-DCB1-4EA8-A038-6E2E0FCD64D9}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {D5E952BA-DCB1-4EA8-A038-6E2E0FCD64D9}.Debug|Any CPU.Build.0 = Debug|Any CPU + {D5E952BA-DCB1-4EA8-A038-6E2E0FCD64D9}.Release|Any CPU.ActiveCfg = Release|Any CPU + {D5E952BA-DCB1-4EA8-A038-6E2E0FCD64D9}.Release|Any CPU.Build.0 = Release|Any CPU + {D10125E6-FF03-4292-A22C-9D622B2ACEDE}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {D10125E6-FF03-4292-A22C-9D622B2ACEDE}.Debug|Any CPU.Build.0 = Debug|Any CPU + {D10125E6-FF03-4292-A22C-9D622B2ACEDE}.Release|Any CPU.ActiveCfg = Release|Any CPU + {D10125E6-FF03-4292-A22C-9D622B2ACEDE}.Release|Any CPU.Build.0 = Release|Any CPU + {98198410-71F9-4498-8550-E6F08B1FC4FA}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {98198410-71F9-4498-8550-E6F08B1FC4FA}.Debug|Any CPU.Build.0 = Debug|Any CPU + {98198410-71F9-4498-8550-E6F08B1FC4FA}.Release|Any CPU.ActiveCfg = Release|Any CPU + {98198410-71F9-4498-8550-E6F08B1FC4FA}.Release|Any CPU.Build.0 = Release|Any CPU + {4FB7FC96-CB01-4905-9E40-3768692EDC0A}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {4FB7FC96-CB01-4905-9E40-3768692EDC0A}.Debug|Any CPU.Build.0 = Debug|Any CPU + {4FB7FC96-CB01-4905-9E40-3768692EDC0A}.Release|Any CPU.ActiveCfg = Release|Any CPU + {4FB7FC96-CB01-4905-9E40-3768692EDC0A}.Release|Any CPU.Build.0 = Release|Any CPU + {70CFFF7A-FA63-48DB-B304-8C859998F339}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {70CFFF7A-FA63-48DB-B304-8C859998F339}.Debug|Any CPU.Build.0 = Debug|Any CPU + {70CFFF7A-FA63-48DB-B304-8C859998F339}.Release|Any CPU.ActiveCfg = Release|Any CPU + {70CFFF7A-FA63-48DB-B304-8C859998F339}.Release|Any CPU.Build.0 = Release|Any CPU + {8D264872-5361-4AC5-8A99-908137E13A22}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {8D264872-5361-4AC5-8A99-908137E13A22}.Debug|Any CPU.Build.0 = Debug|Any CPU + {8D264872-5361-4AC5-8A99-908137E13A22}.Release|Any CPU.ActiveCfg = Release|Any CPU + {8D264872-5361-4AC5-8A99-908137E13A22}.Release|Any CPU.Build.0 = Release|Any CPU + {65D97CC1-E566-423E-9BD8-A1FA936CAAFA}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {65D97CC1-E566-423E-9BD8-A1FA936CAAFA}.Debug|Any CPU.Build.0 = Debug|Any CPU + {65D97CC1-E566-423E-9BD8-A1FA936CAAFA}.Release|Any CPU.ActiveCfg = Release|Any CPU + {65D97CC1-E566-423E-9BD8-A1FA936CAAFA}.Release|Any CPU.Build.0 = Release|Any CPU + {F36DFCB1-C33F-426B-851B-FB1DEE7F028E}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {F36DFCB1-C33F-426B-851B-FB1DEE7F028E}.Debug|Any CPU.Build.0 = Debug|Any CPU + {F36DFCB1-C33F-426B-851B-FB1DEE7F028E}.Release|Any CPU.ActiveCfg = Release|Any CPU + {F36DFCB1-C33F-426B-851B-FB1DEE7F028E}.Release|Any CPU.Build.0 = Release|Any CPU + {8B39F251-F948-40AE-8922-3D8C4E529A86}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {8B39F251-F948-40AE-8922-3D8C4E529A86}.Debug|Any CPU.Build.0 = Debug|Any CPU + {8B39F251-F948-40AE-8922-3D8C4E529A86}.Release|Any CPU.ActiveCfg = Release|Any CPU + {8B39F251-F948-40AE-8922-3D8C4E529A86}.Release|Any CPU.Build.0 = Release|Any CPU + {8DE8FF65-23CB-4FB3-8BE5-6C0BEC4BAA97}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {8DE8FF65-23CB-4FB3-8BE5-6C0BEC4BAA97}.Debug|Any CPU.Build.0 = Debug|Any CPU + {8DE8FF65-23CB-4FB3-8BE5-6C0BEC4BAA97}.Release|Any CPU.ActiveCfg = Release|Any CPU + {8DE8FF65-23CB-4FB3-8BE5-6C0BEC4BAA97}.Release|Any CPU.Build.0 = Release|Any CPU + {ADEC06CF-6F3A-44C5-AA57-EAB94124AC82}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {ADEC06CF-6F3A-44C5-AA57-EAB94124AC82}.Debug|Any CPU.Build.0 = Debug|Any CPU + {ADEC06CF-6F3A-44C5-AA57-EAB94124AC82}.Release|Any CPU.ActiveCfg = Release|Any CPU + {ADEC06CF-6F3A-44C5-AA57-EAB94124AC82}.Release|Any CPU.Build.0 = Release|Any CPU + {0C2D067F-053C-45A8-AE0D-4EB388E77C89}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {0C2D067F-053C-45A8-AE0D-4EB388E77C89}.Debug|Any CPU.Build.0 = Debug|Any CPU + {0C2D067F-053C-45A8-AE0D-4EB388E77C89}.Release|Any CPU.ActiveCfg = Release|Any CPU + {0C2D067F-053C-45A8-AE0D-4EB388E77C89}.Release|Any CPU.Build.0 = Release|Any CPU + {562EC700-6984-4C9A-83BF-3BF4E3EB1A64}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {562EC700-6984-4C9A-83BF-3BF4E3EB1A64}.Debug|Any CPU.Build.0 = Debug|Any CPU + {562EC700-6984-4C9A-83BF-3BF4E3EB1A64}.Release|Any CPU.ActiveCfg = Release|Any CPU + {562EC700-6984-4C9A-83BF-3BF4E3EB1A64}.Release|Any CPU.Build.0 = Release|Any CPU + {E849B7BA-FDCC-4CFF-998F-845ED2F1BF40}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {E849B7BA-FDCC-4CFF-998F-845ED2F1BF40}.Debug|Any CPU.Build.0 = Debug|Any CPU + {E849B7BA-FDCC-4CFF-998F-845ED2F1BF40}.Release|Any CPU.ActiveCfg = Release|Any CPU + {E849B7BA-FDCC-4CFF-998F-845ED2F1BF40}.Release|Any CPU.Build.0 = Release|Any CPU + {3417D508-E2F4-4974-8988-BB124046D9E2}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {3417D508-E2F4-4974-8988-BB124046D9E2}.Debug|Any CPU.Build.0 = Debug|Any CPU + {3417D508-E2F4-4974-8988-BB124046D9E2}.Release|Any CPU.ActiveCfg = Release|Any CPU + {3417D508-E2F4-4974-8988-BB124046D9E2}.Release|Any CPU.Build.0 = Release|Any CPU EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE EndGlobalSection + GlobalSection(NestedProjects) = preSolution + {9922732F-01B3-4DBB-ADEC-E5451AB90CEE} = {81AE04BC-CBFA-4E6F-B039-8208E9AFAAE7} + {D5E952BA-DCB1-4EA8-A038-6E2E0FCD64D9} = {81AE04BC-CBFA-4E6F-B039-8208E9AFAAE7} + {D10125E6-FF03-4292-A22C-9D622B2ACEDE} = {81AE04BC-CBFA-4E6F-B039-8208E9AFAAE7} + {98198410-71F9-4498-8550-E6F08B1FC4FA} = {81AE04BC-CBFA-4E6F-B039-8208E9AFAAE7} + {4FB7FC96-CB01-4905-9E40-3768692EDC0A} = {81AE04BC-CBFA-4E6F-B039-8208E9AFAAE7} + {70CFFF7A-FA63-48DB-B304-8C859998F339} = {81AE04BC-CBFA-4E6F-B039-8208E9AFAAE7} + {8D264872-5361-4AC5-8A99-908137E13A22} = {81AE04BC-CBFA-4E6F-B039-8208E9AFAAE7} + {65D97CC1-E566-423E-9BD8-A1FA936CAAFA} = {8F1F1C2A-E313-4E0C-BE40-58FB0BA91124} + {F36DFCB1-C33F-426B-851B-FB1DEE7F028E} = {8F1F1C2A-E313-4E0C-BE40-58FB0BA91124} + {8B39F251-F948-40AE-8922-3D8C4E529A86} = {8F1F1C2A-E313-4E0C-BE40-58FB0BA91124} + {8DE8FF65-23CB-4FB3-8BE5-6C0BEC4BAA97} = {8F1F1C2A-E313-4E0C-BE40-58FB0BA91124} + {ADEC06CF-6F3A-44C5-AA57-EAB94124AC82} = {88C2A621-8A98-4D07-8625-7900FC8EF89E} + {0C2D067F-053C-45A8-AE0D-4EB388E77C89} = {88C2A621-8A98-4D07-8625-7900FC8EF89E} + {562EC700-6984-4C9A-83BF-3BF4E3EB1A64} = {88C2A621-8A98-4D07-8625-7900FC8EF89E} + {E849B7BA-FDCC-4CFF-998F-845ED2F1BF40} = {88C2A621-8A98-4D07-8625-7900FC8EF89E} + {3417D508-E2F4-4974-8988-BB124046D9E2} = {7591C5B3-D86E-4AE4-8ED2-B272D17FE7E3} + EndGlobalSection GlobalSection(ExtensibilityGlobals) = postSolution SolutionGuid = {237BF0AA-9EC4-4659-AD9A-65DEB974250C} EndGlobalSection diff --git a/docs/Architecture.png b/docs/CodexTestNetArchitecture.png similarity index 100% rename from docs/Architecture.png rename to docs/CodexTestNetArchitecture.png diff --git a/docs/FrameworkArchitecture.png b/docs/FrameworkArchitecture.png new file mode 100644 index 00000000..f3bcb21a Binary files /dev/null and b/docs/FrameworkArchitecture.png differ diff --git a/docs/LOCALSETUP.md b/docs/LOCALSETUP.md index 862995c6..ad65c0e5 100644 --- a/docs/LOCALSETUP.md +++ b/docs/LOCALSETUP.md @@ -4,18 +4,10 @@ These steps will help you set up everything you need to run and debug the tests on your local system. ### Installing the requirements. -1. Install dotnet v6.0 or newer. (If you install a newer version, consider updating the .csproj files by replacing all mention of `net6.0` with your version.) +1. Install dotnet v7.0 or newer. (If you install a newer version, consider updating the .csproj files by replacing all mention of `net7.0` with your version.) 1. Set up a nice C# IDE or plugin for your current IDE. 1. Install docker desktop. 1. In the docker-desktop settings, enable kubernetes. (This might take a few minutes.) -### Configure to taste. -The tests should run as-is. You can change the configuration. The items below explain the what and how. -1. Open `DistTestCore/Configuration.cs`. -1. `k8sNamespace` defines the Kubernetes namespace the tests will use. All Kubernetes resources used during the test will be created in it. At the beginning of a test run and at the end of each test, the namespace and all resources in it will be deleted. -1. `kubeConfigFile`. If you are using the Kubernetes cluster created in docker desktop, this field should be set to null. If you wish to use a different cluster, set this field to the path (absolute or relative) of your KubeConfig file. -1. `LogConfig(path, debugEnabled)`. Path defines the path (absolute or relative) where the tests logs will be saved. The debug flag allows you to enable additional logging. This is mostly useful when something's wrong with the test infra. -1. `FileManagerFolder` defines the path (absolute or relative) where the test infra will generate and store test data files. This folder will be deleted at the end of every test run. - ### Running the tests -Most IDEs will let you run individual tests or test fixtures straight from the code file. If you want to run all the tests, you can use `dotnet test`. You can control which tests to run by specifying which folder of tests to run. `dotnet test Tests` will run only the tests in `/Tests` and exclude the long tests. +Most IDEs will let you run individual tests or test fixtures straight from the code file. If you want to run all the tests, you can use `dotnet test`. You can control which tests to run by specifying which folder of tests to run. `dotnet test Tests/CodexTests` will run only the tests in `/Tests/CodexTests` and exclude the long tests.