logos-storage/logos-storage-nim-cs-dist-tests
1
0
Fork 0
mirror of https://github.com/logos-storage/logos-storage-nim-cs-dist-tests.git synced 2026-05-25 18:49:25 +00:00
Code Issues Packages Projects Releases Wiki Activity

Rename and remove all instances of Codex

Browse Source
This commit is contained in:
E M 2026-04-15 20:34:49 +10:00
parent 42eabf67b6
commit 32bba67bef
No known key found for this signature in database
176 changed files with 1661 additions and 1417 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.github/workflows/docker-runner.yml vendored
View File

@ -20,6 +20,6 @@ jobs:
uses: logos-storage/github-actions/.github/workflows/docker-reusable.yml@master
with:
docker_file: docker/Dockerfile
dockerhub_repo: logosstorage/cs-codex-dist-tests
dockerhub_repo: logosstorage/logos-storage-dist-tests
tag_latest: ${{ github.ref_name == github.event.repository.default_branch }}
secrets: inherit

2
.github/workflows/run-continuous-tests.yaml vendored
View File

@ -44,7 +44,7 @@ on:
required: false
type: string
storagedockerimage:
description: Codex Docker image (logosstorage/logos-storage-nim:latest-dist-tests)
description: Logos Storage Docker image (logosstorage/logos-storage-nim:latest-dist-tests)
required: false
type: string
nameprefix:

4
.github/workflows/run-dist-tests.yaml vendored
View File

@ -21,7 +21,7 @@ on:
required: false
type: string
command:
description: Command (dotnet test Tests/CodexTests)
description: Command (dotnet test Tests/LogosStorageTests)
required: false
type: string
@ -31,7 +31,7 @@ env:
BRANCH: ${{ github.ref_name }}
NAMEPREFIX: d-tests-runner
NAMESPACE: default
COMMAND: dotnet test Tests/CodexTests
COMMAND: dotnet test Tests/LogosStorageTests
JOB_MANIFEST: docker/job-dist-tests.yaml
KUBE_CONFIG: ${{ secrets.KUBE_CONFIG }}
KUBE_VERSION: v1.28.2

2
.github/workflows/run-release-tests.yaml vendored
View File

@ -36,7 +36,7 @@ env:
NAMEPREFIX: r-tests
NAMESPACE: default
JOB_MANIFEST: docker/job-release-tests.yaml
COMMAND: dotnet test Tests/CodexReleaseTests
COMMAND: dotnet test Tests/LogosStorageReleaseTests
KUBE_CONFIG: ${{ secrets.KUBE_CONFIG }}
KUBE_VERSION: v1.30.5

14
CONTRIBUTINGPLUGINS.MD
View File

@ -1,7 +1,7 @@
# Distributed System Tests for Nim-Codex
# Distributed System Tests for Logos Storage
## Contributing plugins
The testing framework was created for testing Codex. However, it's been designed such that other containerized projects can 'easily' be added.
The testing framework was created for testing Logos Storage. However, it's been designed such that other containerized projects can 'easily' be added.
In this file, you'll see 'users' (in quote) mentioned once or twice. This refers to code/projects/tests which end up making use of your plugin. 'Users' come in many shapes and sizes and tend to have many differen use-cases in mind. Please consider this when reading this document and writing your plugin.
@ -117,7 +117,7 @@ public static class CoreInterfaceExtensions
Should your deploy methods not return framework-types like RunningContainers, please make sure that your custom times are serializable. (Decorate them with the `[Serializable]` attribute.) Tools have been built using this framework which rely on the ability to serialize and store deployment information for later use. Please don't break this possibility. (Consider using the `SerializeGate` type to help ensure compatibility.)
The primary reason to decouple deploying and wrapping functionalities is that some use cases require these steps to be performed by separate applications, and different moments in time. For this reason, whatever is returned by the deploy methods should be serializable. After deserialization at some later time, it should then be valid input for the wrap method. The Codex continuous tests system is a clear example of this use case: The `CodexNetDeployer` tool uses deploy methods to create Codex nodes. Then it writes the returned objects to a JSON file. Some time later, the `CodexContinuousTests` application uses this JSON file to reconstruct the objects created by the deploy methods. It then uses the wrap methods to create accessors and interactors, which are used for testing.
The primary reason to decouple deploying and wrapping functionalities is that some use cases require these steps to be performed by separate applications, and different moments in time. For this reason, whatever is returned by the deploy methods should be serializable. After deserialization at some later time, it should then be valid input for the wrap method.
## Container Recipes
In order to run a container of your application, the framework needs to know how to create that container. Think of a container recipe as being similar to a docker-compose.yaml file: You specify the docker image, ports, environment variables, persistent volumes, and secrets. However, container recipes are code. This allows you to add conditional behaviour to how your container is constructed. For example: The 'user' of your plugin specifies in their call input that they want to run your application in a certain mode. This would cause your container recipe to set certain environment variables, which cause the application to behave in the requested way.
@ -141,18 +141,18 @@ Example:
```C#
{
var location = Ci.GetKnownLocations().Get("kbnode_euwest_paris1");
var codex = Ci.StartCodexNode(s => s.At(location));
var node = Ci.StartStorageNode(s => s.At(location));
}
```
In this example, 'Ci' is an instance of the core interface. The CodexPlugin exposes a function 'StartCodexNode', which allows its user to specify a location. This location is then passed to the `workflow` tool when the Codex plugin starts its container.
In this example, 'Ci' is an instance of the core interface. The StoragePlugin exposes a function 'StartStorageNode', which allows its user to specify a location. This location is then passed to the `workflow` tool when the Logos Storage plugin starts its container.
The available locations array guarantees that each entry corresponds to a different kubernetes host.
```C#
{
var knownLocations = Ci.GetKnownLocations();
// I don't care where exactly, as long as they are different locations.
var codexAtZero = Ci.StartCodexNode(s => s.At(knownLocations.Get(0)));
var codexAtOne = Ci.StartCodexNode(s => s.At(knownLocations.Get(1)));
var storageAtZero = Ci.StartStorageNode(s => s.At(knownLocations.Get(0)));
var storageAtOne = Ci.StartStorageNode(s => s.At(knownLocations.Get(1)));
}
```

32
CONTRIBUTINGTESTS.MD
View File

@ -1,4 +1,4 @@
# Distributed System Tests for Nim-Codex
# Distributed System Tests for Logos Storage
## Contributing tests
Do you want to write some tests using this distributed test setup? Great! Here's what you do.
@ -9,32 +9,32 @@ Do you want to write some tests using this distributed test setup? Great! Here's
1. Add Project references to `Core`, as well as any project plugin you'll be using.
1. Write tests! Use existing tests for inspiration.
## Tips for writing tests for Codex
## Tips for writing tests for Logos Storage
### Transient tests
1. Add new code files to `Tests/CodexTests`
1. Inherrit from `CodexDistTest` or `AutoBootstrapDistTest`.
1. When using `CodexDistTest`:
1. You must start your own Codex bootstrap node. You can use `AddCodex(...)` for this.
1. When you start other Codex nodes with `AddCodex(...)` you can pass the bootstrap node by adding the `.WithBootstrapNode(...)` option.
1. Add new code files to `Tests/LogosStorageTests`
1. Inherrit from `LogosStorageDistTest` or `AutoBootstrapDistTest`.
1. When using `LogosStorageDistTest`:
1. You must start your own Codex bootstrap node. You can use `AddLogosStorage(...)` for this.
1. When you start other Logos Storage nodes with `AddLogosStorage(...)` you can pass the bootstrap node by adding the `.WithBootstrapNode(...)` option.
1. When using `AutoBootstrapDistTest`:
1. The test-infra creates the bootstrap node for you, and automatically passes it to each Codex node you create in your tests. Handy for keeping your tests clean and to-the-point.
1. When using the auto-bootstrap, you have no control over the bootstrap node from your tests. You can't (for example) shut it down during the course of the test. If you need this level of control for your scenario, use the `CodexDistTest` instead.
1. The test-infra creates the bootstrap node for you, and automatically passes it to each Logos Storage node you create in your tests. Handy for keeping your tests clean and to-the-point.
1. When using the auto-bootstrap, you have no control over the bootstrap node from your tests. You can't (for example) shut it down during the course of the test. If you need this level of control for your scenario, use the `LogosStorageDistTest` instead.
1. If your test needs a long time to run, add the `[UseLongTimeouts]` function attribute. This will greatly increase maximum time-out values for operations like for example uploading and downloading files.
### Continuous tests
1. Add new code files to `Tests/CodexContinuousTests/Tests`
1. Add new code files to `Tests/LogosStorageContinuousTests/Tests`
1. Inherrit from `ContinuousTest`
1. Define one or more methods and decorate them with the `[TestMoment(...)]` attribute.
1. The TestMoment takes a number of seconds as argument. Each moment will be executed by the continuous test runner applying the given seconds as delay. (Non-cumulative. So two moments at T:10 will be executed one after another without delay, in this case the order of execution should not be depended upon.)
1. Continuous tests automatically receive access to the Codex nodes that the tests are being run against.
1. Additionally, Continuous tests can start their own transient Codex nodes and bootstrap them against the persistent nodes.
1. Continuous tests automatically receive access to the Logos Storage nodes that the tests are being run against.
1. Additionally, Continuous tests can start their own transient Logos Storage nodes and bootstrap them against the persistent nodes.
### Tips for either type of test
1. You can generate files of random test data by calling `GenerateTestFile(...)`.
1. You can enable access to the Codex node metrics by adding the option `.EnableMetrics()`. Enabling metrics will make the test-infra download and save all Codex metrics in case of a test failure. (The metrics are stored as CSV, in the same location as the test log file.)
1. You can enable access to the Logos Storage node metrics by adding the option `.EnableMetrics()`. Enabling metrics will make the test-infra download and save all Logos Storage metrics in case of a test failure. (The metrics are stored as CSV, in the same location as the test log file.)
1. You can enable access to the blockchain marketplace by adding the option `.EnableMarketplace(...)`.
1. Enabling metrics and/or enabling the marketplace takes extra resources from the test-infra and increases the time needed during Codex node setup. Please don't enable these features unless your tests need them.
1. Tip: Codex nodes can be named. Use the option `WithName(...)` and make reading your test logs a little nicer!
1. Enabling metrics and/or enabling the marketplace takes extra resources from the test-infra and increases the time needed during Logos Storage node setup. Please don't enable these features unless your tests need them.
1. Tip: Logos Storage nodes can be named. Use the option `WithName(...)` and make reading your test logs a little nicer!
1. Tip: Commit often.
## Don't forget
1. Once you're happy with your tests, please create a pull-request and ask a Codex core contributor to review your changes.
1. Once you're happy with your tests, please create a pull-request and ask a Logos Storage core contributor to review your changes.

10
Framework/Logging/ApplicationIds.cs
View File

@ -2,19 +2,19 @@
{
public class ApplicationIds
{
public ApplicationIds(string codexId, string gethId, string prometheusId, string codexContractsId, string grafanaId)
public ApplicationIds(string logosStorageId, string gethId, string prometheusId, string logosStorageContractsId, string grafanaId)
{
CodexId = codexId;
LogosStorageId = logosStorageId;
GethId = gethId;
PrometheusId = prometheusId;
CodexContractsId = codexContractsId;
LogosStorageContractsId = logosStorageContractsId;
GrafanaId = grafanaId;
}
public string CodexId { get; }
public string LogosStorageId { get; }
public string GethId { get; }
public string PrometheusId { get; }
public string CodexContractsId { get; }
public string LogosStorageContractsId { get; }
public string GrafanaId { get; }
}
}

46
ProjectPlugins/CodexClient/CodexNodeFactory.cs
View File

@ -1,46 +0,0 @@
using CodexClient.Hooks;
using FileUtils;
using Logging;
using WebUtils;
namespace CodexClient
{
public class CodexNodeFactory
{
private readonly ILog log;
private readonly IFileManager fileManager;
private readonly CodexHooksFactory hooksFactory;
private readonly IHttpFactory httpFactory;
private readonly IProcessControlFactory processControlFactory;
public CodexNodeFactory(ILog log, IFileManager fileManager, CodexHooksFactory hooksFactory, IHttpFactory httpFactory, IProcessControlFactory processControlFactory)
{
this.log = log;
this.fileManager = fileManager;
this.hooksFactory = hooksFactory;
this.httpFactory = httpFactory;
this.processControlFactory = processControlFactory;
}
public CodexNodeFactory(ILog log, HttpFactory httpFactory, string dataDir)
: this(log, new FileManager(log, dataDir), new CodexHooksFactory(), httpFactory, new DoNothingProcessControlFactory())
{
}
public CodexNodeFactory(ILog log, string dataDir)
: this(log, new HttpFactory(log), dataDir)
{
}
public ICodexNode CreateCodexNode(ICodexInstance instance)
{
var processControl = processControlFactory.CreateProcessControl(instance);
var access = new CodexAccess(log, httpFactory, processControl, instance);
var hooks = hooksFactory.CreateHooks(access.GetName());
var node = new CodexNode(log, access, fileManager, hooks);
node.Initialize();
return node;
}
}
}

61
ProjectPlugins/CodexClient/Hooks/CodexHooksFactory.cs
View File

@ -1,61 +0,0 @@
using Utils;
namespace CodexClient.Hooks
{
public interface ICodexHooksProvider
{
ICodexNodeHooks CreateHooks(string nodeName);
}
public class CodexHooksFactory
{
public List<ICodexHooksProvider> Providers { get; } = new List<ICodexHooksProvider>();
public ICodexNodeHooks CreateHooks(string nodeName)
{
if (Providers.Count == 0) return new DoNothingCodexHooks();
var hooks = Providers.Select(p => p.CreateHooks(nodeName)).ToArray();
return new MuxingCodexNodeHooks(hooks);
}
}
public class DoNothingHooksProvider : ICodexHooksProvider
{
public ICodexNodeHooks CreateHooks(string nodeName)
{
return new DoNothingCodexHooks();
}
}
public class DoNothingCodexHooks : ICodexNodeHooks
{
public void OnFileDownloaded(ByteSize size, ContentId cid)
{
}
public void OnFileDownloading(ContentId cid)
{
}
public void OnFileUploaded(string uid, ByteSize size, ContentId cid)
{
}
public void OnFileUploading(string uid, ByteSize size)
{
}
public void OnNodeStarted(ICodexNode node, string peerId, string nodeId)
{
}
public void OnNodeStarting(DateTime startUtc, string image)
{
}
public void OnNodeStopping()
{
}
}
}

96
ProjectPlugins/CodexPlugin/CodexPlugin.cs
View File

@ -1,96 +0,0 @@
using CodexClient;
using CodexClient.Hooks;
using Core;
namespace CodexPlugin
{
public class CodexPlugin : IProjectPlugin, IHasLogPrefix, IHasMetadata
{
private const bool UseContainers = true;
private readonly ICodexStarter codexStarter;
private readonly IPluginTools tools;
private readonly CodexLogLevel defaultLogLevel = CodexLogLevel.Trace;
private readonly CodexHooksFactory hooksFactory = new CodexHooksFactory();
private readonly ProcessControlMap processControlMap = new ProcessControlMap();
private readonly CodexDockerImage codexDockerImage = new CodexDockerImage();
private readonly CodexContainerRecipe recipe;
private readonly CodexWrapper codexWrapper;
public CodexPlugin(IPluginTools tools)
{
this.tools = tools;
recipe = new CodexContainerRecipe(codexDockerImage);
codexStarter = CreateCodexStarter();
codexWrapper = new CodexWrapper(tools, processControlMap, hooksFactory);
}
private ICodexStarter CreateCodexStarter()
{
if (UseContainers)
{
Log("Using Containerized Codex instances");
return new ContainerCodexStarter(tools, recipe, processControlMap);
}
Log("Using Binary Codex instances");
return new BinaryCodexStarter(tools, processControlMap);
}
public string LogPrefix => "(Codex) ";
public void Awake(IPluginAccess access)
{
}
public void Announce()
{
// give codex docker image to contracts plugin.
Log($"Loaded with Codex ID: '{codexWrapper.GetCodexId()}' - Revision: {codexWrapper.GetCodexRevision()}");
}
public void AddMetadata(IAddMetadata metadata)
{
metadata.Add("codexid", codexWrapper.GetCodexId());
metadata.Add("codexrevision", codexWrapper.GetCodexRevision());
}
public void Decommission()
{
codexStarter.Decommission();
}
public ICodexInstance[] DeployCodexNodes(int numberOfNodes, Action<ICodexSetup> setup)
{
var codexSetup = GetSetup(numberOfNodes, setup);
return codexStarter.BringOnline(codexSetup);
}
public ICodexNodeGroup WrapCodexContainers(ICodexInstance[] instances)
{
instances = instances.Select(c => SerializeGate.Gate(c as CodexInstance)).ToArray();
return codexWrapper.WrapCodexInstances(instances);
}
public void AddCodexHooksProvider(ICodexHooksProvider hooksProvider)
{
if (hooksFactory.Providers.Contains(hooksProvider)) return;
hooksFactory.Providers.Add(hooksProvider);
}
private CodexSetup GetSetup(int numberOfNodes, Action<ICodexSetup> setup)
{
var codexSetup = new CodexSetup(numberOfNodes);
codexSetup.LogLevel = defaultLogLevel;
setup(codexSetup);
return codexSetup;
}
private void Log(string msg)
{
tools.GetLog().Log(msg);
}
}
}

135
ProjectPlugins/CodexPlugin/CodexSetup.cs
View File

@ -1,135 +0,0 @@
using CodexClient;
using KubernetesWorkflow;
using Utils;
namespace CodexPlugin
{
public interface ICodexSetup
{
ICodexSetup WithName(string name);
ICodexSetup At(ILocation location);
ICodexSetup WithBootstrapNode(ICodexNode node);
ICodexSetup WithLogLevel(CodexLogLevel level);
ICodexSetup WithLogLevel(CodexLogLevel level, CodexLogCustomTopics customTopics);
ICodexSetup WithStorageQuota(ByteSize storageQuota);
ICodexSetup WithBlockTTL(TimeSpan duration);
ICodexSetup WithBlockMaintenanceInterval(TimeSpan duration);
ICodexSetup WithBlockMaintenanceNumber(int numberOfBlocks);
ICodexSetup EnableMetrics();
ICodexSetup AsPublicTestNet(CodexTestNetConfig testNetConfig);
}
public class CodexLogCustomTopics
{
public CodexLogCustomTopics(CodexLogLevel discV5, CodexLogLevel libp2p, CodexLogLevel blockExchange)
{
DiscV5 = discV5;
Libp2p = libp2p;
BlockExchange = blockExchange;
}
public CodexLogCustomTopics(CodexLogLevel discV5, CodexLogLevel libp2p)
{
DiscV5 = discV5;
Libp2p = libp2p;
}
public CodexLogLevel DiscV5 { get; set; }
public CodexLogLevel Libp2p { get; set; }
public CodexLogLevel ContractClock { get; set; } = CodexLogLevel.Warn;
public CodexLogLevel? BlockExchange { get; }
public CodexLogLevel JsonSerialize { get; set; } = CodexLogLevel.Warn;
public CodexLogLevel MarketplaceInfra { get; set; } = CodexLogLevel.Warn;
}
public class CodexSetup : CodexStartupConfig, ICodexSetup
{
public int NumberOfNodes { get; }
public CodexSetup(int numberOfNodes)
{
NumberOfNodes = numberOfNodes;
}
public ICodexSetup WithName(string name)
{
NameOverride = name;
return this;
}
public ICodexSetup At(ILocation location)
{
Location = location;
return this;
}
public ICodexSetup WithBootstrapNode(ICodexNode node)
{
BootstrapSpr = node.GetDebugInfo().Spr;
return this;
}
public ICodexSetup WithLogLevel(CodexLogLevel level)
{
LogLevel = level;
return this;
}
public ICodexSetup WithLogLevel(CodexLogLevel level, CodexLogCustomTopics customTopics)
{
LogLevel = level;
CustomTopics = customTopics;
return this;
}
public ICodexSetup WithStorageQuota(ByteSize storageQuota)
{
StorageQuota = storageQuota;
return this;
}
public ICodexSetup WithBlockTTL(TimeSpan duration)
{
BlockTTL = Convert.ToInt32(duration.TotalSeconds);
return this;
}
public ICodexSetup WithBlockMaintenanceInterval(TimeSpan duration)
{
BlockMaintenanceInterval = duration;
return this;
}
public ICodexSetup WithBlockMaintenanceNumber(int numberOfBlocks)
{
BlockMaintenanceNumber = numberOfBlocks;
return this;
}
public ICodexSetup EnableMetrics()
{
MetricsEnabled = true;
return this;
}
public ICodexSetup AsPublicTestNet(CodexTestNetConfig testNetConfig)
{
PublicTestNet = testNetConfig;
return this;
}
public string Describe()
{
var args = string.Join(',', DescribeArgs());
return $"({NumberOfNodes} CodexNodes with args:[{args}])";
}
private IEnumerable<string> DescribeArgs()
{
if (PublicTestNet != null) yield return $"<!>Public TestNet with listenPort: {PublicTestNet.PublicListenPort}<!>";
yield return $"LogLevel={LogLevelWithTopics()}";
if (BootstrapSpr != null) yield return $"BootstrapNode={BootstrapSpr}";
if (StorageQuota != null) yield return $"StorageQuota={StorageQuota}";
}
}
}

51
ProjectPlugins/CodexPlugin/CoreInterfaceExtensions.cs
View File

@ -1,51 +0,0 @@
using CodexClient;
using CodexClient.Hooks;
using Core;
namespace CodexPlugin
{
public static class CoreInterfaceExtensions
{
public static ICodexInstance[] DeployCodexNodes(this CoreInterface ci, int number, Action<ICodexSetup> setup)
{
return Plugin(ci).DeployCodexNodes(number, setup);
}
public static ICodexNodeGroup WrapCodexContainers(this CoreInterface ci, ICodexInstance[] instances)
{
return Plugin(ci).WrapCodexContainers(instances);
}
public static ICodexNode StartCodexNode(this CoreInterface ci)
{
return ci.StartCodexNodes(1)[0];
}
public static ICodexNode StartCodexNode(this CoreInterface ci, Action<ICodexSetup> setup)
{
return ci.StartCodexNodes(1, setup)[0];
}
public static ICodexNodeGroup StartCodexNodes(this CoreInterface ci, int number, Action<ICodexSetup> setup)
{
var rc = ci.DeployCodexNodes(number, setup);
var result = ci.WrapCodexContainers(rc);
return result;
}
public static ICodexNodeGroup StartCodexNodes(this CoreInterface ci, int number)
{
return ci.StartCodexNodes(number, s => { });
}
public static void AddCodexHooksProvider(this CoreInterface ci, ICodexHooksProvider hooksProvider)
{
Plugin(ci).AddCodexHooksProvider(hooksProvider);
}
private static CodexPlugin Plugin(CoreInterface ci)
{
return ci.GetPlugin<CodexPlugin>();
}
}
}

10
ProjectPlugins/CodexPlugin/ICodexStarter.cs
View File

@ -1,10 +0,0 @@
using CodexClient;
namespace CodexPlugin
{
public interface ICodexStarter
{
ICodexInstance[] BringOnline(CodexSetup codexSetup);
void Decommission();
}
}

16
ProjectPlugins/DeployAndRunPlugin/DeployAndRunContainerRecipe.cs
View File

@ -12,9 +12,9 @@ namespace DeployAndRunPlugin
{
var setup = config.Get<RunConfig>();
if (setup.CodexImageOverride != null)
if (setup.LogosStorageImageOverride != null)
{
AddEnvVar("CODEXDOCKERIMAGE", setup.CodexImageOverride);
AddEnvVar("STORAGEDOCKERIMAGE", setup.LogosStorageImageOverride);
}
AddEnvVar("DNR_REP", setup.Replications.ToString());
@ -23,28 +23,28 @@ namespace DeployAndRunPlugin
AddEnvVar("DNR_DURATION", setup.Duration.TotalSeconds.ToString());
AddEnvVar("KUBECONFIG", "/opt/kubeconfig.yaml");
AddEnvVar("LOGPATH", "/var/log/codex-continuous-tests");
AddEnvVar("LOGPATH", "/var/log/storage-continuous-tests");
AddVolume(name: "kubeconfig", mountPath: "/opt/kubeconfig.yaml", subPath: "kubeconfig.yaml", secret: "codex-dist-tests-app-kubeconfig");
AddVolume(name: "logs", mountPath: "/var/log/codex-continuous-tests", hostPath: "/var/log/codex-continuous-tests");
AddVolume(name: "kubeconfig", mountPath: "/opt/kubeconfig.yaml", subPath: "kubeconfig.yaml", secret: "storage-dist-tests-app-kubeconfig");
AddVolume(name: "logs", mountPath: "/var/log/storage-continuous-tests", hostPath: "/var/log/storage-continuous-tests");
}
}
public class RunConfig
{
public RunConfig(string name, string filter, TimeSpan duration, int replications, string? codexImageOverride = null)
public RunConfig(string name, string filter, TimeSpan duration, int replications, string? logosStorageImageOverride = null)
{
Name = name;
Filter = filter;
Duration = duration;
Replications = replications;
CodexImageOverride = codexImageOverride;
LogosStorageImageOverride = logosStorageImageOverride;
}
public string Name { get; }
public string Filter { get; }
public TimeSpan Duration { get; }
public int Replications { get; }
public string? CodexImageOverride { get; }
public string? LogosStorageImageOverride { get; }
}
}

61
ProjectPlugins/LogosStorageClient/Hooks/LogosStorageHooksFactory.cs Normal file
View File

@ -0,0 +1,61 @@
using Utils;
namespace LogosStorageClient.Hooks
{
public interface ILogosStorageHooksProvider
{
IStorageNodeHooks CreateHooks(string nodeName);
}
public class LogosStorageHooksFactory
{
public List<ILogosStorageHooksProvider> Providers { get; } = new List<ILogosStorageHooksProvider>();
public IStorageNodeHooks CreateHooks(string nodeName)
{
if (Providers.Count == 0) return new DoNothingLogosStorageHooks();
var hooks = Providers.Select(p => p.CreateHooks(nodeName)).ToArray();
return new MuxingStorageNodeHooks(hooks);
}
}
public class DoNothingHooksProvider : ILogosStorageHooksProvider
{
public IStorageNodeHooks CreateHooks(string nodeName)
{
return new DoNothingLogosStorageHooks();
}
}
public class DoNothingLogosStorageHooks : IStorageNodeHooks
{
public void OnFileDownloaded(ByteSize size, ContentId cid)
{
}
public void OnFileDownloading(ContentId cid)
{
}
public void OnFileUploaded(string uid, ByteSize size, ContentId cid)
{
}
public void OnFileUploading(string uid, ByteSize size)
{
}
public void OnNodeStarted(IStorageNode node, string peerId, string nodeId)
{
}
public void OnNodeStarting(DateTime startUtc, string image)
{
}
public void OnNodeStopping()
{
}
}
}

14
ProjectPlugins/CodexClient/Hooks/CodexNodeHooks.cs → ProjectPlugins/LogosStorageClient/Hooks/StorageNodeHooks.cs
View File

@ -1,11 +1,11 @@
using Utils;
namespace CodexClient.Hooks
namespace LogosStorageClient.Hooks
{
public interface ICodexNodeHooks
public interface IStorageNodeHooks
{
void OnNodeStarting(DateTime startUtc, string image);
void OnNodeStarted(ICodexNode node, string peerId, string nodeId);
void OnNodeStarted(IStorageNode node, string peerId, string nodeId);
void OnNodeStopping();
void OnFileUploading(string uid, ByteSize size);
void OnFileUploaded(string uid, ByteSize size, ContentId cid);
@ -13,11 +13,11 @@ namespace CodexClient.Hooks
void OnFileDownloaded(ByteSize size, ContentId cid);
}
public class MuxingCodexNodeHooks : ICodexNodeHooks
public class MuxingStorageNodeHooks : IStorageNodeHooks
{
private readonly ICodexNodeHooks[] backingHooks;
private readonly IStorageNodeHooks[] backingHooks;
public MuxingCodexNodeHooks(ICodexNodeHooks[] backingHooks)
public MuxingStorageNodeHooks(IStorageNodeHooks[] backingHooks)
{
this.backingHooks = backingHooks;
}
@ -42,7 +42,7 @@ namespace CodexClient.Hooks
foreach (var h in backingHooks) h.OnFileUploading(uid, size);
}
public void OnNodeStarted(ICodexNode node, string peerId, string nodeId)
public void OnNodeStarted(IStorageNode node, string peerId, string nodeId)
{
foreach (var h in backingHooks) h.OnNodeStarted(node, peerId, nodeId);
}

44
ProjectPlugins/CodexClient/CodexAccess.cs → ProjectPlugins/LogosStorageClient/LogosStorageAccess.cs
View File

@ -1,23 +1,23 @@
using System.Threading;
using CodexOpenApi;
using StorageOpenApi;
using Logging;
using Newtonsoft.Json;
using Utils;
using WebUtils;
namespace CodexClient
namespace LogosStorageClient
{
public class CodexAccess
public class LogosStorageAccess
{
public const string API_BASE_URL = "/api/storage/v1";
private readonly ILog log;
private readonly IHttpFactory httpFactory;
private readonly IProcessControl processControl;
private readonly ICodexInstance instance;
private readonly ILogosStorageInstance instance;
private readonly Mapper mapper = new Mapper();
public CodexAccess(ILog log, IHttpFactory httpFactory, IProcessControl processControl, ICodexInstance instance)
public LogosStorageAccess(ILog log, IHttpFactory httpFactory, IProcessControl processControl, ILogosStorageInstance instance)
{
this.log = log;
this.httpFactory = httpFactory;
@ -49,14 +49,14 @@ namespace CodexClient
public DebugInfo GetDebugInfo()
{
return mapper.Map(OnCodex(api => api.GetDebugInfoAsync()));
return mapper.Map(OnLogosStorage(api => api.GetDebugInfoAsync()));
}
public void SetLogLevel(string logLevel)
{
try
{
OnCodex(async api =>
OnLogosStorage(async api =>
{
await api.SetDebugLogLevelAsync(logLevel);
return string.Empty;
@ -108,7 +108,7 @@ namespace CodexClient
public void ConnectToPeer(string peerId, string[] peerMultiAddresses)
{
OnCodex(api =>
OnLogosStorage(api =>
{
Time.Wait(api.ConnectPeerAsync(peerId, peerMultiAddresses));
return Task.FromResult(string.Empty);
@ -117,36 +117,36 @@ namespace CodexClient
public string UploadFile(UploadInput uploadInput)
{
return OnCodex(api => api.UploadAsync(uploadInput.ContentType, uploadInput.ContentDisposition, uploadInput.FileStream));
return OnLogosStorage(api => api.UploadAsync(uploadInput.ContentType, uploadInput.ContentDisposition, uploadInput.FileStream));
}
public Stream DownloadFile(string contentId)
{
var fileResponse = OnCodexNoRetry(api => api.DownloadNetworkStreamAsync(contentId));
var fileResponse = OnLogosStorageNoRetry(api => api.DownloadNetworkStreamAsync(contentId));
if (fileResponse.StatusCode != 200) throw new Exception("Download failed with StatusCode: " + fileResponse.StatusCode);
return fileResponse.Stream;
}
public LocalDataset DownloadStreamless(ContentId cid)
{
var response = OnCodex(api => api.DownloadNetworkAsync(cid.Id));
var response = OnLogosStorage(api => api.DownloadNetworkAsync(cid.Id));
return mapper.Map(response);
}
public LocalDataset DownloadManifestOnly(ContentId cid)
{
var response = OnCodex(api => api.DownloadNetworkManifestAsync(cid.Id));
var response = OnLogosStorage(api => api.DownloadNetworkManifestAsync(cid.Id));
return mapper.Map(response);
}
public LocalDatasetList LocalFiles()
{
return mapper.Map(OnCodex(api => api.ListDataAsync()));
return mapper.Map(OnLogosStorage(api => api.ListDataAsync()));
}
public CodexSpace Space()
public LogosStorageSpace Space()
{
var space = OnCodex(api => api.SpaceAsync());
var space = OnLogosStorage(api => api.SpaceAsync());
return mapper.Map(space);
}
@ -185,29 +185,29 @@ namespace CodexClient
processControl.DeleteDataDirFolder();
}
private T OnCodexNoRetry<T>(Func<CodexApiClient, Task<T>> action)
private T OnLogosStorageNoRetry<T>(Func<StorageApiClient, Task<T>> action)
{
var timeSet = httpFactory.WebCallTimeSet;
var noRetry = new Retry(nameof(OnCodexNoRetry),
var noRetry = new Retry(nameof(OnLogosStorageNoRetry),
maxTimeout: TimeSpan.FromSeconds(1.0),
sleepAfterFail: TimeSpan.FromSeconds(2.0),
onFail: f => { },
failFast: true);
var result = httpFactory.CreateHttp(GetHttpId(), h => CheckContainerCrashed()).OnClient(client => CallCodex(client, action), noRetry);
var result = httpFactory.CreateHttp(GetHttpId(), h => CheckContainerCrashed()).OnClient(client => CallLogosStorage(client, action), noRetry);
return result;
}
private T OnCodex<T>(Func<CodexApiClient, Task<T>> action)
private T OnLogosStorage<T>(Func<StorageApiClient, Task<T>> action)
{
var result = httpFactory.CreateHttp(GetHttpId(), h => CheckContainerCrashed()).OnClient(client => CallCodex(client, action));
var result = httpFactory.CreateHttp(GetHttpId(), h => CheckContainerCrashed()).OnClient(client => CallLogosStorage(client, action));
return result;
}
private T CallCodex<T>(HttpClient client, Func<CodexApiClient, Task<T>> action)
private T CallLogosStorage<T>(HttpClient client, Func<StorageApiClient, Task<T>> action)
{
var address = GetAddress();
var api = new CodexApiClient(client);
var api = new StorageApiClient(client);
api.BaseUrl = $"{address.Host}:{address.Port}{API_BASE_URL}";
return CrashCheck(() => Time.Wait(action(api)));
}

2
ProjectPlugins/CodexClient/CodexClient.csproj → ProjectPlugins/LogosStorageClient/LogosStorageClient.csproj
View File

@ -11,7 +11,7 @@
</ItemGroup>
<ItemGroup>
<OpenApiReference Include="openapi.yaml" CodeGenerator="NSwagCSharp" Namespace="CodexOpenApi" ClassName="CodexApiClient" />
<OpenApiReference Include="openapi.yaml" CodeGenerator="NSwagCSharp" Namespace="StorageOpenApi" ClassName="StorageApiClient" />
</ItemGroup>
<ItemGroup>

12
ProjectPlugins/CodexClient/CodexInstance.cs → ProjectPlugins/LogosStorageClient/LogosStorageInstance.cs
View File

@ -1,8 +1,8 @@
using Utils;
namespace CodexClient
namespace LogosStorageClient
{
public interface ICodexInstance
public interface ILogosStorageInstance
{
string Name { get; }
string ImageName { get; }
@ -14,9 +14,9 @@ namespace CodexClient
Address? MetricsEndpoint { get; }
}
public class CodexInstance : ICodexInstance
public class LogosStorageInstance : ILogosStorageInstance
{
public CodexInstance(string name, string imageName, DateTime startUtc, Address discoveryEndpoint, Address apiEndpoint, Address listenEndpoint, EthAccount? ethAccount, Address? metricsEndpoint)
public LogosStorageInstance(string name, string imageName, DateTime startUtc, Address discoveryEndpoint, Address apiEndpoint, Address listenEndpoint, EthAccount? ethAccount, Address? metricsEndpoint)
{
Name = name;
ImageName = imageName;
@ -37,9 +37,9 @@ namespace CodexClient
public EthAccount? EthAccount { get; }
public Address? MetricsEndpoint { get; }
public static ICodexInstance CreateFromApiEndpoint(string name, Address apiEndpoint, EthAccount? ethAccount = null)
public static ILogosStorageInstance CreateFromApiEndpoint(string name, Address apiEndpoint, EthAccount? ethAccount = null)
{
return new CodexInstance(
return new LogosStorageInstance(
name,
imageName: "-",
startUtc: DateTime.UtcNow,

4
ProjectPlugins/CodexClient/CodexLogLevel.cs → ProjectPlugins/LogosStorageClient/LogosStorageLogLevel.cs
View File

@ -1,6 +1,6 @@
namespace CodexClient
namespace LogosStorageClient
{
public enum CodexLogLevel
public enum LogosStorageLogLevel
{
Trace,
Debug,

8
ProjectPlugins/CodexClient/CodexLogLine.cs → ProjectPlugins/LogosStorageClient/LogosStorageLogLine.cs
View File

@ -1,10 +1,10 @@
using System.Globalization;
namespace CodexClient
namespace LogosStorageClient
{
public class CodexLogLine
public class LogosStorageLogLine
{
public static CodexLogLine? Parse(string line)
public static LogosStorageLogLine? Parse(string line)
{
try
{
@ -29,7 +29,7 @@ namespace CodexClient
var format = "yyyy-MM-dd HH:mm:ss.fff";
var dt = DateTime.ParseExact(dtLine, format, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal).ToUniversalTime();
return new CodexLogLine()
return new LogosStorageLogLine()
{
LogLevel = level,
TimestampUtc = dt,

4
ProjectPlugins/CodexClient/CodexTypes.cs → ProjectPlugins/LogosStorageClient/LogosStorageTypes.cs
View File

@ -1,7 +1,7 @@
using Newtonsoft.Json;
using Utils;
namespace CodexClient
namespace LogosStorageClient
{
public class DebugInfo
{
@ -110,7 +110,7 @@ namespace CodexClient
}
}
public class CodexSpace
public class LogosStorageSpace
{
public long TotalBlocks { get; set; }
public long QuotaMaxBytes { get; set; }

4
ProjectPlugins/CodexClient/CodexUtils.cs → ProjectPlugins/LogosStorageClient/LogosStorageUtils.cs
View File

@ -1,6 +1,6 @@
namespace CodexClient
namespace LogosStorageClient
{
public static class CodexUtils
public static class LogosStorageUtils
{
public static string ToShortId(string id)
{

22
ProjectPlugins/CodexClient/Mapper.cs → ProjectPlugins/LogosStorageClient/Mapper.cs
View File

@ -2,11 +2,11 @@
using System.Numerics;
using Utils;
namespace CodexClient
namespace LogosStorageClient
{
public class Mapper
{
public DebugInfo Map(CodexOpenApi.DebugInfo debugInfo)
public DebugInfo Map(StorageOpenApi.DebugInfo debugInfo)
{
return new DebugInfo
{
@ -19,7 +19,7 @@ namespace CodexClient
};
}
public LocalDatasetList Map(CodexOpenApi.DataList dataList)
public LocalDatasetList Map(StorageOpenApi.DataList dataList)
{
return new LocalDatasetList
{
@ -27,7 +27,7 @@ namespace CodexClient
};
}
public LocalDataset Map(CodexOpenApi.DataItem dataItem)
public LocalDataset Map(StorageOpenApi.DataItem dataItem)
{
return new LocalDataset
{
@ -35,9 +35,9 @@ namespace CodexClient
Manifest = MapManifest(dataItem.Manifest)
};
}
public CodexSpace Map(CodexOpenApi.Space space)
public LogosStorageSpace Map(StorageOpenApi.Space space)
{
return new CodexSpace
return new LogosStorageSpace
{
QuotaMaxBytes = space.QuotaMaxBytes,
QuotaReservedBytes = space.QuotaReservedBytes,
@ -46,7 +46,7 @@ namespace CodexClient
};
}
private DebugInfoVersion Map(CodexOpenApi.StorageVersion obj)
private DebugInfoVersion Map(StorageOpenApi.StorageVersion obj)
{
return new DebugInfoVersion
{
@ -55,7 +55,7 @@ namespace CodexClient
};
}
private DebugInfoTable Map(CodexOpenApi.PeersTable obj)
private DebugInfoTable Map(StorageOpenApi.PeersTable obj)
{
return new DebugInfoTable
{
@ -64,7 +64,7 @@ namespace CodexClient
};
}
private DebugInfoTableNode Map(CodexOpenApi.Node? token)
private DebugInfoTableNode Map(StorageOpenApi.Node? token)
{
if (token == null) return new DebugInfoTableNode();
return new DebugInfoTableNode
@ -77,7 +77,7 @@ namespace CodexClient
};
}
private DebugInfoTableNode[] Map(ICollection<CodexOpenApi.Node> nodes)
private DebugInfoTableNode[] Map(ICollection<StorageOpenApi.Node> nodes)
{
if (nodes == null || nodes.Count == 0)
{
@ -87,7 +87,7 @@ namespace CodexClient
return nodes.Select(Map).ToArray();
}
private Manifest MapManifest(CodexOpenApi.ManifestItem manifest)
private Manifest MapManifest(StorageOpenApi.ManifestItem manifest)
{
return new Manifest
{

6
ProjectPlugins/CodexClient/ProcessControl.cs → ProjectPlugins/LogosStorageClient/ProcessControl.cs
View File

@ -1,10 +1,10 @@
using Logging;
namespace CodexClient
namespace LogosStorageClient
{
public interface IProcessControlFactory
{
IProcessControl CreateProcessControl(ICodexInstance instance);
IProcessControl CreateProcessControl(ILogosStorageInstance instance);
}
public interface IProcessControl
@ -17,7 +17,7 @@ namespace CodexClient
public class DoNothingProcessControlFactory : IProcessControlFactory
{
public IProcessControl CreateProcessControl(ICodexInstance instance)
public IProcessControl CreateProcessControl(ILogosStorageInstance instance)
{
return new DoNothingProcessControl();
}

90
ProjectPlugins/CodexClient/CodexNode.cs → ProjectPlugins/LogosStorageClient/StorageNode.cs
View File

@ -1,11 +1,11 @@
using CodexClient.Hooks;
using LogosStorageClient.Hooks;
using FileUtils;
using Logging;
using Utils;
namespace CodexClient
namespace LogosStorageClient
{
public partial interface ICodexNode : IHasMetricsScrapeTarget
public partial interface IStorageNode : IHasMetricsScrapeTarget
{
string GetName();
string GetImageName();
@ -26,8 +26,8 @@ namespace CodexClient
LocalDataset DownloadStreamlessWait(ContentId cid, ByteSize size);
LocalDataset DownloadManifestOnly(ContentId cid);
LocalDatasetList LocalFiles();
CodexSpace Space();
void ConnectToPeer(ICodexNode node);
LogosStorageSpace Space();
void ConnectToPeer(IStorageNode node);
DebugInfoVersion Version { get; }
ITransferSpeeds TransferSpeeds { get; }
@ -45,20 +45,20 @@ namespace CodexClient
bool HasCrashed();
}
public class CodexNode : ICodexNode
public class StorageNode : IStorageNode
{
private const string UploadFailedMessage = "Unable to store block";
private readonly ILog log;
private readonly ICodexNodeHooks hooks;
private readonly IStorageNodeHooks hooks;
private readonly TransferSpeeds transferSpeeds;
private string peerId = string.Empty;
private string nodeId = string.Empty;
private readonly CodexAccess codexAccess;
private readonly LogosStorageAccess logosStorageAccess;
private readonly IFileManager fileManager;
public CodexNode(ILog log, CodexAccess codexAccess, IFileManager fileManager, ICodexNodeHooks hooks)
public StorageNode(ILog log, LogosStorageAccess logosStorageAccess, IFileManager fileManager, IStorageNodeHooks hooks)
{
this.codexAccess = codexAccess;
this.logosStorageAccess = logosStorageAccess;
this.fileManager = fileManager;
this.hooks = hooks;
Version = new DebugInfoVersion();
@ -69,14 +69,14 @@ namespace CodexClient
public void Awake()
{
hooks.OnNodeStarting(codexAccess.GetStartUtc(), codexAccess.GetImageName());
hooks.OnNodeStarting(logosStorageAccess.GetStartUtc(), logosStorageAccess.GetImageName());
}
public void Initialize()
{
// This is the moment we first connect to a codex node. Sometimes, Kubernetes takes a while to spin up the
// container. So we'll adding a custom, generous retry here.
var kubeSpinupRetry = new Retry("CodexNode_Initialize",
var kubeSpinupRetry = new Retry("StorageNode_Initialize",
maxTimeout: TimeSpan.FromMinutes(10.0),
sleepAfterFail: TimeSpan.FromSeconds(10.0),
onFail: f => { },
@ -94,12 +94,12 @@ namespace CodexClient
public string GetName()
{
return codexAccess.GetName();
return logosStorageAccess.GetName();
}
public string GetImageName()
{
return codexAccess.GetImageName();
return logosStorageAccess.GetImageName();
}
public string GetPeerId()
@ -109,7 +109,7 @@ namespace CodexClient
public DebugInfo GetDebugInfo(bool log = false)
{
var debugInfo = codexAccess.GetDebugInfo();
var debugInfo = logosStorageAccess.GetDebugInfo();
if (log)
{
var known = string.Join(",", debugInfo.Table.Nodes.Select(n => n.PeerId));
@ -120,17 +120,17 @@ namespace CodexClient
public void SetLogLevel(string logLevel)
{
codexAccess.SetLogLevel(logLevel);
logosStorageAccess.SetLogLevel(logLevel);
}
public string GetSpr()
{
return codexAccess.GetSpr();
return logosStorageAccess.GetSpr();
}
public DebugPeer GetDebugPeer(string peerId)
{
return codexAccess.GetDebugPeer(peerId);
return logosStorageAccess.GetDebugPeer(peerId);
}
public ContentId UploadFile(TrackedFile file)
@ -150,7 +150,7 @@ namespace CodexClient
var logMessage = $"Uploading file {file.Describe()} with contentType: '{input.ContentType}' and disposition: '{input.ContentDisposition}'...";
var measurement = Stopwatch.Measure(log, logMessage, () =>
{
return codexAccess.UploadFile(input);
return logosStorageAccess.UploadFile(input);
});
var response = measurement.Value;
@ -190,7 +190,7 @@ namespace CodexClient
public LocalDataset DownloadStreamless(ContentId cid)
{
Log($"Downloading streamless '{cid}' (no-wait)");
return codexAccess.DownloadStreamless(cid);
return logosStorageAccess.DownloadStreamless(cid);
}
public LocalDataset DownloadStreamlessWait(ContentId cid, ByteSize size)
@ -200,7 +200,7 @@ namespace CodexClient
var sw = Stopwatch.Measure(log, nameof(DownloadStreamlessWait), () =>
{
var startSpace = Space();
var result = codexAccess.DownloadStreamless(cid);
var result = logosStorageAccess.DownloadStreamless(cid);
WaitUntilQuotaUsedIncreased(startSpace, size);
return result;
});
@ -211,85 +211,85 @@ namespace CodexClient
public LocalDataset DownloadManifestOnly(ContentId cid)
{
Log($"Downloading manifest-only '{cid}'");
return codexAccess.DownloadManifestOnly(cid);
return logosStorageAccess.DownloadManifestOnly(cid);
}
public LocalDatasetList LocalFiles()
{
return codexAccess.LocalFiles();
return logosStorageAccess.LocalFiles();
}
public CodexSpace Space()
public LogosStorageSpace Space()
{
return codexAccess.Space();
return logosStorageAccess.Space();
}
public void ConnectToPeer(ICodexNode node)
public void ConnectToPeer(IStorageNode node)
{
var peer = (CodexNode)node;
var peer = (StorageNode)node;
Log($"Connecting to peer {peer.GetName()}...");
var peerInfo = node.GetDebugInfo();
codexAccess.ConnectToPeer(peerInfo.Id, GetPeerMultiAddresses(peer, peerInfo));
logosStorageAccess.ConnectToPeer(peerInfo.Id, GetPeerMultiAddresses(peer, peerInfo));
Log($"Successfully connected to peer {peer.GetName()}.");
}
public void DeleteDataDirFolder()
{
codexAccess.DeleteDataDirFolder();
logosStorageAccess.DeleteDataDirFolder();
}
public void Stop(bool waitTillStopped)
{
Log("Stopping...");
hooks.OnNodeStopping();
codexAccess.Stop(waitTillStopped);
logosStorageAccess.Stop(waitTillStopped);
}
public IDownloadedLog DownloadLog(string additionalName = "")
{
return codexAccess.DownloadLog(additionalName);
return logosStorageAccess.DownloadLog(additionalName);
}
public Address GetDiscoveryEndpoint()
{
return codexAccess.GetDiscoveryEndpoint();
return logosStorageAccess.GetDiscoveryEndpoint();
}
public Address GetApiEndpoint()
{
return codexAccess.GetApiEndpoint();
return logosStorageAccess.GetApiEndpoint();
}
public Address GetListenEndpoint()
{
return codexAccess.GetListenEndpoint();
return logosStorageAccess.GetListenEndpoint();
}
public Address GetMetricsScrapeTarget()
{
var address = codexAccess.GetMetricsEndpoint();
var address = logosStorageAccess.GetMetricsEndpoint();
if (address == null) throw new Exception("Metrics ScrapeTarget accessed, but node was not started with EnableMetrics()");
return address;
}
public bool HasCrashed()
{
return codexAccess.HasCrashed();
return logosStorageAccess.HasCrashed();
}
public override string ToString()
{
return $"CodexNode:{GetName()}";
return $"StorageNode:{GetName()}";
}
private void InitializePeerNodeId()
{
var debugInfo = codexAccess.GetDebugInfo();
var debugInfo = logosStorageAccess.GetDebugInfo();
if (!debugInfo.Version.IsValid())
{
throw new Exception($"Invalid version information received from Codex node {GetName()}: {debugInfo.Version}");
throw new Exception($"Invalid version information received from Logos Storage node {GetName()}: {debugInfo.Version}");
}
peerId = debugInfo.Id;
@ -302,12 +302,12 @@ namespace CodexClient
var nodeName = GetName();
log.AddStringReplace(peerId, nodeName);
log.AddStringReplace(CodexUtils.ToShortId(peerId), nodeName);
log.AddStringReplace(LogosStorageUtils.ToShortId(peerId), nodeName);
log.AddStringReplace(nodeId, nodeName);
log.AddStringReplace(CodexUtils.ToShortId(nodeId), nodeName);
log.AddStringReplace(LogosStorageUtils.ToShortId(nodeId), nodeName);
}
private string[] GetPeerMultiAddresses(CodexNode peer, DebugInfo peerInfo)
private string[] GetPeerMultiAddresses(StorageNode peer, DebugInfo peerInfo)
{
var peerId = peer.GetDiscoveryEndpoint().Host
.Replace("http://", "")
@ -330,7 +330,7 @@ namespace CodexClient
var cts = new CancellationTokenSource();
var downloadTask = Task.Run(() =>
{
using var downloadStream = codexAccess.DownloadFile(contentId);
using var downloadStream = logosStorageAccess.DownloadFile(contentId);
downloadStream.CopyTo(fileStream);
}, cts.Token);
@ -351,13 +351,13 @@ namespace CodexClient
}
}
public void WaitUntilQuotaUsedIncreased(CodexSpace startSpace, ByteSize expectedIncreaseOfQuotaUsed)
public void WaitUntilQuotaUsedIncreased(LogosStorageSpace startSpace, ByteSize expectedIncreaseOfQuotaUsed)
{
WaitUntilQuotaUsedIncreased(startSpace, expectedIncreaseOfQuotaUsed, TimeSpan.FromMinutes(30));
}
public void WaitUntilQuotaUsedIncreased(
CodexSpace startSpace,
LogosStorageSpace startSpace,
ByteSize expectedIncreaseOfQuotaUsed,
TimeSpan maxTimeout)
{

46
ProjectPlugins/LogosStorageClient/StorageNodeFactory.cs Normal file
View File

@ -0,0 +1,46 @@
using LogosStorageClient.Hooks;
using FileUtils;
using Logging;
using WebUtils;
namespace LogosStorageClient
{
public class StorageNodeFactory
{
private readonly ILog log;
private readonly IFileManager fileManager;
private readonly LogosStorageHooksFactory hooksFactory;
private readonly IHttpFactory httpFactory;
private readonly IProcessControlFactory processControlFactory;
public StorageNodeFactory(ILog log, IFileManager fileManager, LogosStorageHooksFactory hooksFactory, IHttpFactory httpFactory, IProcessControlFactory processControlFactory)
{
this.log = log;
this.fileManager = fileManager;
this.hooksFactory = hooksFactory;
this.httpFactory = httpFactory;
this.processControlFactory = processControlFactory;
}
public StorageNodeFactory(ILog log, HttpFactory httpFactory, string dataDir)
: this(log, new FileManager(log, dataDir), new LogosStorageHooksFactory(), httpFactory, new DoNothingProcessControlFactory())
{
}
public StorageNodeFactory(ILog log, string dataDir)
: this(log, new HttpFactory(log), dataDir)
{
}
public IStorageNode CreateStorageNode(ILogosStorageInstance instance)
{
var processControl = processControlFactory.CreateProcessControl(instance);
var access = new LogosStorageAccess(log, httpFactory, processControl, instance);
var hooks = hooksFactory.CreateHooks(access.GetName());
var node = new StorageNode(log, access, fileManager, hooks);
node.Initialize();
return node;
}
}
}

2
ProjectPlugins/CodexClient/TransferSpeeds.cs → ProjectPlugins/LogosStorageClient/TransferSpeeds.cs
View File

@ -1,6 +1,6 @@
using Utils;
namespace CodexClient
namespace LogosStorageClient
{
public interface ITransferSpeeds
{

0
ProjectPlugins/CodexClient/openapi.yaml → ProjectPlugins/LogosStorageClient/openapi.yaml
View File

8
ProjectPlugins/CodexDiscordBotPlugin/CoreInterfaceExtensions.cs → ProjectPlugins/LogosStorageDiscordBotPlugin/CoreInterfaceExtensions.cs
View File

@ -1,11 +1,11 @@
using Core;
using KubernetesWorkflow.Types;
namespace CodexDiscordBotPlugin
namespace LogosStorageDiscordBotPlugin
{
public static class CoreInterfaceExtensions
{
public static RunningPod DeployCodexDiscordBot(this CoreInterface ci, DiscordBotStartupConfig config)
public static RunningPod DeployLogosStorageDiscordBot(this CoreInterface ci, DiscordBotStartupConfig config)
{
return Plugin(ci).Deploy(config);
}
@ -15,9 +15,9 @@ namespace CodexDiscordBotPlugin
return Plugin(ci).DeployRewarder(config);
}
private static CodexDiscordBotPlugin Plugin(CoreInterface ci)
private static LogosStorageDiscordBotPlugin Plugin(CoreInterface ci)
{
return ci.GetPlugin<CodexDiscordBotPlugin>();
return ci.GetPlugin<LogosStorageDiscordBotPlugin>();
}
}
}

4
ProjectPlugins/CodexDiscordBotPlugin/DiscordBotContainerRecipe.cs → ProjectPlugins/LogosStorageDiscordBotPlugin/DiscordBotContainerRecipe.cs
View File

@ -2,12 +2,12 @@
using KubernetesWorkflow.Recipe;
using Utils;
namespace CodexDiscordBotPlugin
namespace LogosStorageDiscordBotPlugin
{
public class DiscordBotContainerRecipe : ContainerRecipeFactory
{
public override string AppName => "discordbot-bibliotech";
public override string Image => "codexstorage/codex-discordbot:sha-8033da1";
public override string Image => "logosstorage/logos-storage-discordbot:sha-8033da1";
public static string RewardsPort = "bot_rewards_port";

2
ProjectPlugins/CodexDiscordBotPlugin/DiscordBotStartupConfig.cs → ProjectPlugins/LogosStorageDiscordBotPlugin/DiscordBotStartupConfig.cs
View File

@ -1,4 +1,4 @@
namespace CodexDiscordBotPlugin
namespace LogosStorageDiscordBotPlugin
{
public class DiscordBotStartupConfig
{

10
ProjectPlugins/CodexDiscordBotPlugin/CodexDiscordBotPlugin.cs → ProjectPlugins/LogosStorageDiscordBotPlugin/LogosStorageDiscordBotPlugin.cs
View File

@ -3,14 +3,14 @@ using KubernetesWorkflow;
using KubernetesWorkflow.Types;
using Utils;
namespace CodexDiscordBotPlugin
namespace LogosStorageDiscordBotPlugin
{
public class CodexDiscordBotPlugin : IProjectPlugin, IHasLogPrefix, IHasMetadata
public class LogosStorageDiscordBotPlugin : IProjectPlugin, IHasLogPrefix, IHasMetadata
{
private const string ExpectedStartupMessage = "Debug option is set. Discord connection disabled!";
private readonly IPluginTools tools;
public CodexDiscordBotPlugin(IPluginTools tools)
public LogosStorageDiscordBotPlugin(IPluginTools tools)
{
this.tools = tools;
}
@ -23,12 +23,12 @@ namespace CodexDiscordBotPlugin
public void Announce()
{
tools.GetLog().Log($"Codex DiscordBot (BiblioTech) loaded.");
tools.GetLog().Log($"Logos Storage DiscordBot (BiblioTech) loaded.");
}
public void AddMetadata(IAddMetadata metadata)
{
metadata.Add("codexdiscordbotid", new DiscordBotContainerRecipe().Image);
metadata.Add("logosstoragediscordbotid", new DiscordBotContainerRecipe().Image);
}
public void Decommission()

2
ProjectPlugins/CodexDiscordBotPlugin/CodexDiscordBotPlugin.csproj → ProjectPlugins/LogosStorageDiscordBotPlugin/LogosStorageDiscordBotPlugin.csproj
View File

@ -8,7 +8,7 @@
<ItemGroup>
<ProjectReference Include="..\..\Framework\Core\Core.csproj" />
<ProjectReference Include="..\CodexPlugin\CodexPlugin.csproj" />
<ProjectReference Include="..\StoragePlugin\StoragePlugin.csproj" />
</ItemGroup>
</Project>

4
ProjectPlugins/CodexDiscordBotPlugin/RewarderBotContainerRecipe.cs → ProjectPlugins/LogosStorageDiscordBotPlugin/RewarderBotContainerRecipe.cs
View File

@ -2,12 +2,12 @@
using KubernetesWorkflow;
using Utils;
namespace CodexDiscordBotPlugin
namespace LogosStorageDiscordBotPlugin
{
public class RewarderBotContainerRecipe : ContainerRecipeFactory
{
public override string AppName => "discordbot-rewarder";
public override string Image => "codexstorage/codex-rewarderbot:sha-8033da1";
public override string Image => "logosstorage/logos-storage-rewarderbot:sha-8033da1";
protected override void Initialize(StartupConfig startupConfig)
{

0
ProjectPlugins/CodexPluginPrebuild/CodexPluginPrebuild.csproj → ProjectPlugins/LogosStoragePluginPrebuild/LogosStoragePluginPrebuild.csproj
View File

22
ProjectPlugins/CodexPluginPrebuild/Program.cs → ProjectPlugins/LogosStoragePluginPrebuild/Program.cs
View File

@ -5,16 +5,16 @@ using Utils;
public static class Program
{
private const string Search = "<INSERT-OPENAPI-YAML-HASH>";
private const string CodexPluginFolderName = "CodexPlugin";
private const string StoragePluginFolderName = "StoragePlugin";
private const string ProjectPluginsFolderName = "ProjectPlugins";
public static void Main(string[] args)
{
Console.WriteLine("Injecting hash of 'openapi.yaml'...");
var pluginRoot = FindCodexPluginFolder();
var clientRoot = FindCodexClientFolder();
Console.WriteLine("Located CodexPlugin: " + pluginRoot);
var pluginRoot = FindStoragePluginFolder();
var clientRoot = FindLogosStorageClientFolder();
Console.WriteLine("Located StoragePlugin: " + pluginRoot);
var openApiFile = Path.Combine(clientRoot, "openapi.yaml");
var clientFile = Path.Combine(clientRoot, "obj", "openapiClient.cs");
var targetFile = Path.Combine(pluginRoot, "ApiChecker.cs");
@ -29,7 +29,7 @@ public static class Program
SearchAndInject(hash, targetFile);
// This program runs as the pre-build trigger for "CodexPlugin".
// This program runs as the pre-build trigger for "StoragePlugin".
// You might be wondering why this work isn't done by a shell script.
// This is because this project is being run on many different platforms.
// (Mac, Unix, Win, but also desktop/cloud containers.)
@ -40,17 +40,17 @@ public static class Program
Console.WriteLine("Done!");
}
private static string FindCodexPluginFolder()
private static string FindStoragePluginFolder()
{
var folder = Path.Combine(PluginPathUtils.ProjectPluginsDir, "CodexPlugin");
if (!Directory.Exists(folder)) throw new Exception("CodexPlugin folder not found. Expected: " + folder);
var folder = Path.Combine(PluginPathUtils.ProjectPluginsDir, "StoragePlugin");
if (!Directory.Exists(folder)) throw new Exception("StoragePlugin folder not found. Expected: " + folder);
return folder;
}
private static string FindCodexClientFolder()
private static string FindLogosStorageClientFolder()
{
var folder = Path.Combine(PluginPathUtils.ProjectPluginsDir, "CodexClient");
if (!Directory.Exists(folder)) throw new Exception("CodexClient folder not found. Expected: " + folder);
var folder = Path.Combine(PluginPathUtils.ProjectPluginsDir, "LogosStorageClient");
if (!Directory.Exists(folder)) throw new Exception("LogosStorageClient folder not found. Expected: " + folder);
return folder;
}

2
ProjectPlugins/MetricsPlugin/PrometheusContainerRecipe.cs
View File

@ -6,7 +6,7 @@ namespace MetricsPlugin
public class PrometheusContainerRecipe : ContainerRecipeFactory
{
public override string AppName => "prometheus";
public override string Image => "codexstorage/dist-tests-prometheus:latest";
public override string Image => "logosstorage/dist-tests-prometheus:latest";
public const string PortTag = "prometheus_port_tag";

2
ProjectPlugins/MetricsPlugin/dashboard.json
View File

@ -2057,7 +2057,7 @@
},
"timepicker": {},
"timezone": "",
"title": "Codex",
"title": "Logos Storage",
"uid": "ca93f344-dd41-48c4-95a7-876c340e3005",
"version": 6,
"weekStart": ""

24
ProjectPlugins/CodexPlugin/ApiChecker.cs → ProjectPlugins/StoragePlugin/ApiChecker.cs
View File

@ -5,27 +5,27 @@ using System.Security.Cryptography;
using System.Text;
using Utils;
namespace CodexPlugin
namespace StoragePlugin
{
public class ApiChecker
{
// <INSERT-OPENAPI-YAML-HASH>
private const string OpenApiYamlHash = "47-B8-22-44-3B-51-EB-82-CC-C1-DD-56-F0-7E-EC-7F-CD-E0-8F-5E-F6-3B-40-E0-02-E6-71-DB-B8-03-65-18";
private const string OpenApiFilePath = "/logosstorage/openapi.yaml";
private const string DisableEnvironmentVariable = "CODEXPLUGIN_DISABLE_APICHECK";
private const string DisableEnvironmentVariable = "StoragePlugin_DISABLE_APICHECK";
private const bool Disable = false;
private const string Warning =
"Warning: CodexPlugin was unable to find the openapi.yaml file in the Codex container. Are you running an old version of Codex? " +
"Warning: StoragePlugin was unable to find the openapi.yaml file in the Logos Storage container. Are you running an old version of Logos Storage? " +
"Plugin will continue as normal, but API compatibility is not guaranteed!";
private const string Failure =
"Codex API compatibility check failed! " +
"openapi.yaml used by CodexPlugin does not match openapi.yaml in Codex container. The openapi.yaml in " +
"'ProjectPlugins/CodexPlugin' has been overwritten with the container one. " +
"Logos Storage API compatibility check failed! " +
"openapi.yaml used by StoragePlugin does not match openapi.yaml in Logos Storage container. The openapi.yaml in " +
"'ProjectPlugins/StoragePlugin' has been overwritten with the container one. " +
"Please and rebuild this project. If you wish to disable API compatibility checking, please set " +
$"the environment variable '{DisableEnvironmentVariable}' or set the disable bool in 'ProjectPlugins/CodexPlugin/ApiChecker.cs'.";
$"the environment variable '{DisableEnvironmentVariable}' or set the disable bool in 'ProjectPlugins/StoragePlugin/ApiChecker.cs'.";
private static bool checkPassed = false;
@ -44,7 +44,7 @@ namespace CodexPlugin
{
if (checkPassed) return;
Log("CodexPlugin is checking API compatibility...");
Log("StoragePlugin is checking API compatibility...");
if (Disable || !string.IsNullOrEmpty(Environment.GetEnvironmentVariable(DisableEnvironmentVariable)))
{
@ -81,13 +81,13 @@ namespace CodexPlugin
private void OverwriteOpenApiYaml(string containerApi)
{
Log("API compatibility check failed. Updating CodexPlugin...");
var openApiFilePath = Path.Combine(PluginPathUtils.ProjectPluginsDir, "CodexClient", "openapi.yaml");
if (!File.Exists(openApiFilePath)) throw new Exception("Unable to locate CodexClient/openapi.yaml. Expected: " + openApiFilePath);
Log("API compatibility check failed. Updating StoragePlugin...");
var openApiFilePath = Path.Combine(PluginPathUtils.ProjectPluginsDir, "LogosStorageClient", "openapi.yaml");
if (!File.Exists(openApiFilePath)) throw new Exception("Unable to locate LogosStorageClient/openapi.yaml. Expected: " + openApiFilePath);
File.Delete(openApiFilePath);
File.WriteAllText(openApiFilePath, containerApi);
Log("CodexClient/openapi.yaml has been updated.");
Log("LogosStorageClient/openapi.yaml has been updated.");
}
private string Hash(string file)

8
ProjectPlugins/CodexPlugin/BinaryProcessControl.cs → ProjectPlugins/StoragePlugin/BinaryProcessControl.cs
View File

@ -1,20 +1,20 @@
using System.Diagnostics;
using CodexClient;
using LogosStorageClient;
using Logging;
namespace CodexPlugin
namespace StoragePlugin
{
public class BinaryProcessControl : IProcessControl
{
private readonly LogFile logFile;
private readonly Process process;
private readonly CodexProcessConfig config;
private readonly LogosStorageProcessConfig config;
private List<string> logBuffer = new List<string>();
private readonly object bufferLock = new object();
private readonly List<Task> streamTasks = new List<Task>();
private bool running;
public BinaryProcessControl(ILog log, Process process, CodexProcessConfig config)
public BinaryProcessControl(ILog log, Process process, LogosStorageProcessConfig config)
{
logFile = log.CreateSubfile(config.Name);

50
ProjectPlugins/CodexPlugin/BinaryCodexStarter.cs → ProjectPlugins/StoragePlugin/BinaryStorageStarter.cs
View File

@ -1,40 +1,40 @@
using CodexClient;
using LogosStorageClient;
using Core;
using Utils;
using System.Diagnostics;
namespace CodexPlugin
namespace StoragePlugin
{
public class BinaryCodexStarter : ICodexStarter
public class BinaryLogosStorageStarter : ILogosStorageStarter
{
private readonly IPluginTools pluginTools;
private readonly ProcessControlMap processControlMap;
private readonly static NumberSource numberSource = new NumberSource(1);
private readonly static FreePortFinder freePortFinder = new FreePortFinder();
private readonly static object _lock = new object();
private readonly static string dataParentDir = "codex_disttest_datadirs";
private readonly static CodexExePath codexExePath = new CodexExePath();
private readonly static string dataParentDir = "storage_disttest_datadirs";
private readonly static LogosStorageExePath logosStorageExePath = new LogosStorageExePath();
static BinaryCodexStarter()
static BinaryLogosStorageStarter()
{
StopAllCodexProcesses();
StopAllLogosStorageProcesses();
DeleteParentDataDir();
}
public BinaryCodexStarter(IPluginTools pluginTools, ProcessControlMap processControlMap)
public BinaryLogosStorageStarter(IPluginTools pluginTools, ProcessControlMap processControlMap)
{
this.pluginTools = pluginTools;
this.processControlMap = processControlMap;
}
public ICodexInstance[] BringOnline(CodexSetup codexSetup)
public ILogosStorageInstance[] BringOnline(LogosStorageSetup logosStorageSetup)
{
lock (_lock)
{
LogSeparator();
Log($"Starting {codexSetup.Describe()}...");
Log($"Starting {logosStorageSetup.Describe()}...");
return StartCodexBinaries(codexSetup, codexSetup.NumberOfNodes);
return StartLogosStorageBinaries(logosStorageSetup, logosStorageSetup.NumberOfNodes);
}
}
@ -46,9 +46,9 @@ namespace CodexPlugin
}
}
private ICodexInstance[] StartCodexBinaries(CodexStartupConfig startupConfig, int numberOfNodes)
private ILogosStorageInstance[] StartLogosStorageBinaries(LogosStorageStartupConfig startupConfig, int numberOfNodes)
{
var result = new List<ICodexInstance>();
var result = new List<ILogosStorageInstance>();
for (var i = 0; i < numberOfNodes; i++)
{
result.Add(StartBinary(startupConfig));
@ -57,14 +57,14 @@ namespace CodexPlugin
return result.ToArray();
}
private ICodexInstance StartBinary(CodexStartupConfig config)
private ILogosStorageInstance StartBinary(LogosStorageStartupConfig config)
{
var name = GetName(config);
var dataDir = Path.Combine(dataParentDir, $"datadir_{numberSource.GetNextNumber()}");
var pconfig = new CodexProcessConfig(name, freePortFinder, dataDir);
var pconfig = new LogosStorageProcessConfig(name, freePortFinder, dataDir);
Log(pconfig);
var factory = new CodexProcessRecipe(pconfig, codexExePath);
var factory = new LogosStorageProcessRecipe(pconfig, logosStorageExePath);
var recipe = factory.Initialize(config);
var startInfo = new ProcessStartInfo(
@ -82,7 +82,7 @@ namespace CodexPlugin
}
var local = "localhost";
var instance = new CodexInstance(
var instance = new LogosStorageInstance(
name: name,
imageName: "binary",
startUtc: DateTime.UtcNow,
@ -99,13 +99,13 @@ namespace CodexPlugin
return instance;
}
private string GetName(CodexStartupConfig config)
private string GetName(LogosStorageStartupConfig config)
{
if (!string.IsNullOrEmpty(config.NameOverride))
{
return config.NameOverride + "_" + numberSource.GetNextNumber();
}
return "codex_" + numberSource.GetNextNumber();
return "storage_" + numberSource.GetNextNumber();
}
private void LogSeparator()
@ -113,7 +113,7 @@ namespace CodexPlugin
Log("----------------------------------------------------------------------------");
}
private void Log(CodexProcessConfig pconfig)
private void Log(LogosStorageProcessConfig pconfig)
{
Log(
"NodeConfig:Name=" + pconfig.Name +
@ -137,16 +137,16 @@ namespace CodexPlugin
}
}
private static void StopAllCodexProcesses()
private static void StopAllLogosStorageProcesses()
{
var processes = Process.GetProcesses();
var codexes = processes.Where(p =>
p.ProcessName.ToLowerInvariant() == "codex" &&
var storageProcesses = processes.Where(p =>
p.ProcessName.ToLowerInvariant() == "storage" &&
p.MainModule != null &&
p.MainModule.FileName == codexExePath.Get()
p.MainModule.FileName == logosStorageExePath.Get()
).ToArray();
foreach (var c in codexes)
foreach (var c in storageProcesses)
{
c.Kill();
c.WaitForExit();

32
ProjectPlugins/CodexPlugin/ContainerCodexStarter.cs → ProjectPlugins/StoragePlugin/ContainerLogosStorageStarter.cs
View File

@ -1,19 +1,19 @@
using CodexClient;
using LogosStorageClient;
using Core;
using KubernetesWorkflow;
using KubernetesWorkflow.Types;
using Utils;
namespace CodexPlugin
namespace StoragePlugin
{
public class ContainerCodexStarter : ICodexStarter
public class ContainerLogosStorageStarter : ILogosStorageStarter
{
private readonly IPluginTools pluginTools;
private readonly ProcessControlMap processControlMap;
private readonly CodexContainerRecipe recipe;
private readonly LogosStorageContainerRecipe recipe;
private readonly ApiChecker apiChecker;
public ContainerCodexStarter(IPluginTools pluginTools, CodexContainerRecipe recipe, ProcessControlMap processControlMap)
public ContainerLogosStorageStarter(IPluginTools pluginTools, LogosStorageContainerRecipe recipe, ProcessControlMap processControlMap)
{
this.pluginTools = pluginTools;
this.recipe = recipe;
@ -21,14 +21,14 @@ namespace CodexPlugin
apiChecker = new ApiChecker(pluginTools);
}
public ICodexInstance[] BringOnline(CodexSetup codexSetup)
public ILogosStorageInstance[] BringOnline(LogosStorageSetup logosStorageSetup)
{
LogSeparator();
Log($"Starting {codexSetup.Describe()}...");
Log($"Starting {logosStorageSetup.Describe()}...");
var startupConfig = CreateStartupConfig(codexSetup);
var startupConfig = CreateStartupConfig(logosStorageSetup);
var containers = StartCodexContainers(startupConfig, codexSetup.NumberOfNodes, codexSetup.Location);
var containers = StartLogosStorageContainers(startupConfig, logosStorageSetup.NumberOfNodes, logosStorageSetup.Location);
apiChecker.CheckCompatibility(containers);
@ -48,15 +48,15 @@ namespace CodexPlugin
{
}
private StartupConfig CreateStartupConfig(CodexSetup codexSetup)
private StartupConfig CreateStartupConfig(LogosStorageSetup logosStorageSetup)
{
var startupConfig = new StartupConfig();
startupConfig.NameOverride = codexSetup.NameOverride;
startupConfig.Add(codexSetup);
startupConfig.NameOverride = logosStorageSetup.NameOverride;
startupConfig.Add(logosStorageSetup);
return startupConfig;
}
private RunningPod[] StartCodexContainers(StartupConfig startupConfig, int numberOfNodes, ILocation location)
private RunningPod[] StartLogosStorageContainers(StartupConfig startupConfig, int numberOfNodes, ILocation location)
{
var futureContainers = new List<FutureContainers>();
for (var i = 0; i < numberOfNodes; i++)
@ -76,10 +76,10 @@ namespace CodexPlugin
return workflow.GetPodInfo(rc);
}
private ICodexInstance CreateInstance(RunningPod pod)
private ILogosStorageInstance CreateInstance(RunningPod pod)
{
var instance = CodexInstanceContainerExtension.CreateFromPod(pod);
var processControl = new CodexContainerProcessControl(pluginTools, pod, onStop: () =>
var instance = LogosStorageInstanceContainerExtension.CreateFromPod(pod);
var processControl = new LogosStorageContainerProcessControl(pluginTools, pod, onStop: () =>
{
processControlMap.Remove(instance);
});

8
ProjectPlugins/CodexPlugin/CodexContainerProcessControl.cs → ProjectPlugins/StoragePlugin/ContainerProcessControl.cs
View File

@ -1,19 +1,19 @@
using CodexClient;
using LogosStorageClient;
using Core;
using KubernetesWorkflow;
using KubernetesWorkflow.Types;
using Logging;
namespace CodexPlugin
namespace StoragePlugin
{
public class CodexContainerProcessControl : IProcessControl
public class LogosStorageContainerProcessControl : IProcessControl
{
private readonly IPluginTools tools;
private readonly RunningPod pod;
private readonly Action onStop;
private readonly ContainerCrashWatcher crashWatcher;
public CodexContainerProcessControl(IPluginTools tools, RunningPod pod, Action onStop)
public LogosStorageContainerProcessControl(IPluginTools tools, RunningPod pod, Action onStop)
{
this.tools = tools;
this.pod = pod;

32
ProjectPlugins/CodexPlugin/CodexContainerRecipe.cs → ProjectPlugins/StoragePlugin/ContainerRecipe.cs
View File

@ -2,26 +2,26 @@ using KubernetesWorkflow;
using KubernetesWorkflow.Recipe;
using Utils;
namespace CodexPlugin
namespace StoragePlugin
{
public class CodexContainerRecipe : ContainerRecipeFactory
public class LogosStorageContainerRecipe : ContainerRecipeFactory
{
public const string ApiPortTag = "codex_api_port";
public const string ListenPortTag = "codex_listen_port";
public const string MetricsPortTag = "codex_metrics_port";
public const string DiscoveryPortTag = "codex_discovery_port";
public const string ApiPortTag = "storage_api_port";
public const string ListenPortTag = "storage_listen_port";
public const string MetricsPortTag = "storage_metrics_port";
public const string DiscoveryPortTag = "storage_discovery_port";
// Used by tests for time-constraint assertions.
public static readonly TimeSpan MaxUploadTimePerMegabyte = TimeSpan.FromSeconds(2.0);
public static readonly TimeSpan MaxDownloadTimePerMegabyte = TimeSpan.FromSeconds(2.0);
private readonly CodexDockerImage codexDockerImage;
private readonly LogosStorageDockerImage logosStorageDockerImage;
public override string AppName => "codex";
public override string Image => codexDockerImage.GetCodexDockerImage();
public override string AppName => "storage";
public override string Image => logosStorageDockerImage.GetLogosStorageDockerImage();
public CodexContainerRecipe(CodexDockerImage codexDockerImage)
public LogosStorageContainerRecipe(LogosStorageDockerImage logosStorageDockerImage)
{
this.codexDockerImage = codexDockerImage;
this.logosStorageDockerImage = logosStorageDockerImage;
}
protected override void Initialize(StartupConfig startupConfig)
@ -32,7 +32,7 @@ namespace CodexPlugin
SetSchedulingAffinity(notIn: "false");
SetSystemCriticalPriority();
var config = startupConfig.Get<CodexStartupConfig>();
var config = startupConfig.Get<LogosStorageStartupConfig>();
var apiPort = CreateApiPort(config, ApiPortTag);
AddEnvVar("STORAGE_API_PORT", apiPort);
@ -97,27 +97,27 @@ namespace CodexPlugin
}
}
private Port CreateApiPort(CodexStartupConfig config, string tag)
private Port CreateApiPort(LogosStorageStartupConfig config, string tag)
{
if (config.PublicTestNet == null) return AddExposedPort(tag);
return AddInternalPort(tag);
}
private Port CreateListenPort(CodexStartupConfig config)
private Port CreateListenPort(LogosStorageStartupConfig config)
{
if (config.PublicTestNet == null) return AddInternalPort(ListenPortTag);
return AddExposedPort(config.PublicTestNet.PublicListenPort, ListenPortTag);
}
private Port CreateDiscoveryPort(CodexStartupConfig config)
private Port CreateDiscoveryPort(LogosStorageStartupConfig config)
{
if (config.PublicTestNet == null) return AddInternalPort(DiscoveryPortTag, PortProtocol.UDP);
return AddExposedPort(config.PublicTestNet.PublicDiscoveryPort, DiscoveryPortTag, PortProtocol.UDP);
}
private ByteSize GetVolumeCapacity(CodexStartupConfig config)
private ByteSize GetVolumeCapacity(LogosStorageStartupConfig config)
{
if (config.StorageQuota != null) return config.StorageQuota.Multiply(1.2);
// Default Codex quota: 8 Gb, using +20% to be safe.

51
ProjectPlugins/StoragePlugin/CoreInterfaceExtensions.cs Normal file
View File

@ -0,0 +1,51 @@
using LogosStorageClient;
using LogosStorageClient.Hooks;
using Core;
namespace StoragePlugin
{
public static class CoreInterfaceExtensions
{
public static ILogosStorageInstance[] DeployLogosStorageNodes(this CoreInterface ci, int number, Action<ILogosStorageSetup> setup)
{
return Plugin(ci).DeployLogosStorageNodes(number, setup);
}
public static IStorageNodeGroup WrapLogosStorageContainers(this CoreInterface ci, ILogosStorageInstance[] instances)
{
return Plugin(ci).WrapLogosStorageContainers(instances);
}
public static IStorageNode StartStorageNode(this CoreInterface ci)
{
return ci.StartStorageNodes(1)[0];
}
public static IStorageNode StartStorageNode(this CoreInterface ci, Action<ILogosStorageSetup> setup)
{
return ci.StartStorageNodes(1, setup)[0];
}
public static IStorageNodeGroup StartStorageNodes(this CoreInterface ci, int number, Action<ILogosStorageSetup> setup)
{
var rc = ci.DeployLogosStorageNodes(number, setup);
var result = ci.WrapLogosStorageContainers(rc);
return result;
}
public static IStorageNodeGroup StartStorageNodes(this CoreInterface ci, int number)
{
return ci.StartStorageNodes(number, s => { });
}
public static void AddLogosStorageHooksProvider(this CoreInterface ci, ILogosStorageHooksProvider hooksProvider)
{
Plugin(ci).AddLogosStorageHooksProvider(hooksProvider);
}
private static StoragePlugin Plugin(CoreInterface ci)
{
return ci.GetPlugin<StoragePlugin>();
}
}
}

22
ProjectPlugins/CodexPlugin/CodexDeployment.cs → ProjectPlugins/StoragePlugin/Deployment.cs
View File

@ -1,24 +1,24 @@
using CodexClient;
using LogosStorageClient;
using KubernetesWorkflow.Types;
namespace CodexPlugin
namespace StoragePlugin
{
public class CodexDeployment
public class LogosStorageDeployment
{
public CodexDeployment(CodexInstance[] codexInstances,
public LogosStorageDeployment(LogosStorageInstance[] logosStorageInstances,
RunningPod? prometheusContainer,
RunningPod? discordBotContainer, DeploymentMetadata metadata,
string id)
{
Id = id;
CodexInstances = codexInstances;
LogosStorageInstances = logosStorageInstances;
PrometheusContainer = prometheusContainer;
DiscordBotContainer = discordBotContainer;
Metadata = metadata;
}
public string Id { get; }
public CodexInstance[] CodexInstances { get; }
public LogosStorageInstance[] LogosStorageInstances { get; }
public RunningPod? PrometheusContainer { get; }
public RunningPod? DiscordBotContainer { get; }
public DeploymentMetadata Metadata { get; }
@ -27,7 +27,7 @@ namespace CodexPlugin
public class DeploymentMetadata
{
public DeploymentMetadata(string name, DateTime startUtc, DateTime finishedUtc, string kubeNamespace,
int numberOfCodexNodes, int numberOfValidators, int storageQuotaMB, CodexLogLevel codexLogLevel,
int numberOfLogosStorageNodes, int numberOfValidators, int storageQuotaMB, LogosStorageLogLevel logosStorageLogLevel,
int initialTestTokens, int minPrice, int maxCollateral, int maxDuration, int blockTTL, int blockMI,
int blockMN)
{
@ -35,10 +35,10 @@ namespace CodexPlugin
StartUtc = startUtc;
FinishedUtc = finishedUtc;
KubeNamespace = kubeNamespace;
NumberOfCodexNodes = numberOfCodexNodes;
NumberOfLogosStorageNodes = numberOfLogosStorageNodes;
NumberOfValidators = numberOfValidators;
StorageQuotaMB = storageQuotaMB;
CodexLogLevel = codexLogLevel;
LogosStorageLogLevel = logosStorageLogLevel;
InitialTestTokens = initialTestTokens;
MinPrice = minPrice;
MaxCollateral = maxCollateral;
@ -52,10 +52,10 @@ namespace CodexPlugin
public DateTime StartUtc { get; }
public DateTime FinishedUtc { get; }
public string KubeNamespace { get; }
public int NumberOfCodexNodes { get; }
public int NumberOfLogosStorageNodes { get; }
public int NumberOfValidators { get; }
public int StorageQuotaMB { get; }
public CodexLogLevel CodexLogLevel { get; }
public LogosStorageLogLevel LogosStorageLogLevel { get; }
public int InitialTestTokens { get; }
public int MinPrice { get; }
public int MaxCollateral { get; }

8
ProjectPlugins/CodexPlugin/CodexDockerImage.cs → ProjectPlugins/StoragePlugin/DockerImage.cs
View File

@ -1,14 +1,14 @@
namespace CodexPlugin
namespace StoragePlugin
{
public class CodexDockerImage
public class LogosStorageDockerImage
{
private const string DefaultDockerImage = "logosstorage/logos-storage-nim:latest-dist-tests";
public static string Override { get; set; } = string.Empty;
public string GetCodexDockerImage()
public string GetLogosStorageDockerImage()
{
var image = Environment.GetEnvironmentVariable("CODEXDOCKERIMAGE");
var image = Environment.GetEnvironmentVariable("STORAGEDOCKERIMAGE");
if (!string.IsNullOrEmpty(image)) return image;
if (!string.IsNullOrEmpty(Override)) return Override;
return DefaultDockerImage;

2
ProjectPlugins/CodexPlugin/FreePortFinder.cs → ProjectPlugins/StoragePlugin/FreePortFinder.cs
View File

@ -1,6 +1,6 @@
using System.Net.NetworkInformation;
namespace CodexPlugin
namespace StoragePlugin
{
public class FreePortFinder
{

10
ProjectPlugins/StoragePlugin/ILogosStorageStarter.cs Normal file
View File

@ -0,0 +1,10 @@
using LogosStorageClient;
namespace StoragePlugin
{
public interface ILogosStorageStarter
{
ILogosStorageInstance[] BringOnline(LogosStorageSetup logosStorageSetup);
void Decommission();
}
}

18
ProjectPlugins/CodexPlugin/CodexInstanceContainerExtension.cs → ProjectPlugins/StoragePlugin/InstanceContainerExtension.cs
View File

@ -1,22 +1,22 @@
using CodexClient;
using LogosStorageClient;
using KubernetesWorkflow.Types;
using Utils;
namespace CodexPlugin
namespace StoragePlugin
{
public static class CodexInstanceContainerExtension
public static class LogosStorageInstanceContainerExtension
{
public static ICodexInstance CreateFromPod(RunningPod pod)
public static ILogosStorageInstance CreateFromPod(RunningPod pod)
{
var container = pod.Containers.Single();
return new CodexInstance(
return new LogosStorageInstance(
name: container.Name,
imageName: container.Recipe.Image,
startUtc: container.Recipe.RecipeCreatedUtc,
discoveryEndpoint: SetClusterInternalIpAddress(pod, container.GetInternalAddress(CodexContainerRecipe.DiscoveryPortTag)),
apiEndpoint: container.GetAddress(CodexContainerRecipe.ApiPortTag),
listenEndpoint: container.GetInternalAddress(CodexContainerRecipe.ListenPortTag),
discoveryEndpoint: SetClusterInternalIpAddress(pod, container.GetInternalAddress(LogosStorageContainerRecipe.DiscoveryPortTag)),
apiEndpoint: container.GetAddress(LogosStorageContainerRecipe.ApiPortTag),
listenEndpoint: container.GetInternalAddress(LogosStorageContainerRecipe.ListenPortTag),
ethAccount: container.Recipe.Additionals.Get<EthAccount>(),
metricsEndpoint: GetMetricsEndpoint(container)
);
@ -35,7 +35,7 @@ namespace CodexPlugin
{
try
{
return container.GetInternalAddress(CodexContainerRecipe.MetricsPortTag);
return container.GetInternalAddress(LogosStorageContainerRecipe.MetricsPortTag);
}
catch
{

34
ProjectPlugins/CodexPlugin/LocalCodexBuilder.cs → ProjectPlugins/StoragePlugin/LocalNodeBuilder.cs
View File

@ -1,29 +1,29 @@
using CodexPlugin;
using StoragePlugin;
using Logging;
using System.Diagnostics;
namespace CodexNetDeployer
namespace StoragePlugin
{
public class LocalCodexBuilder
public class LocalNodeBuilder
{
private readonly ILog log;
private readonly string? repoPath;
private readonly string? dockerUsername;
public LocalCodexBuilder(ILog log, string? repoPath, string? dockerUsername)
public LocalNodeBuilder(ILog log, string? repoPath, string? dockerUsername)
{
this.log = new LogPrefixer(log, "(LocalCodexBuilder) ");
this.log = new LogPrefixer(log, "(LocalNodeBuilder) ");
this.repoPath = repoPath;
this.dockerUsername = dockerUsername;
}
public LocalCodexBuilder(ILog log, string? repoPath)
public LocalNodeBuilder(ILog log, string? repoPath)
: this(log, repoPath, Environment.GetEnvironmentVariable("DOCKERUSERNAME"))
{
}
public LocalCodexBuilder(ILog log)
: this(log, Environment.GetEnvironmentVariable("CODEXREPOPATH"))
public LocalNodeBuilder(ILog log)
: this(log, Environment.GetEnvironmentVariable("STORAGEREPOPATH"))
{
}
@ -32,15 +32,15 @@ namespace CodexNetDeployer
if (!IsEnabled()) return;
if (string.IsNullOrEmpty(dockerUsername)) throw new Exception("Docker username required. (Pass to constructor or set 'DOCKERUSERNAME' environment variable.)");
if (string.IsNullOrEmpty(repoPath)) throw new Exception("Codex repo path required. (Pass to constructor or set 'CODEXREPOPATH' environment variable.)");
if (string.IsNullOrEmpty(repoPath)) throw new Exception("Logos Storage repo path required. (Pass to constructor or set 'CODEXREPOPATH' environment variable.)");
if (!Directory.Exists(repoPath)) throw new Exception($"Path '{repoPath}' does not exist.");
var files = Directory.GetFiles(repoPath);
if (!files.Any(f => f.ToLowerInvariant().EndsWith("codex.nim"))) throw new Exception($"Path '{repoPath}' does not appear to be the Codex repo root.");
Log($"Codex docker image will be built in path '{repoPath}'.");
Log("Please note this can take several minutes. If you're not trying to use a Codex image with local code changes,");
Log("Consider using the default test image or consider setting the 'CODEXDOCKERIMAGE' environment variable to use an already built image.");
CodexDockerImage.Override = $"Using docker image locally built in path '{repoPath}'.";
Log($"Logos Storage docker image will be built in path '{repoPath}'.");
Log("Please note this can take several minutes. If you're not trying to use a Logos Storage image with local code changes,");
Log("Consider using the default test image or consider setting the 'STORAGEDOCKERIMAGE' environment variable to use an already built image.");
LogosStorageDockerImage.Override = $"Using docker image locally built in path '{repoPath}'.";
}
public void Build()
@ -49,7 +49,7 @@ namespace CodexNetDeployer
Log("Docker login...");
DockerLogin();
Log($"Logged in. Building Codex image in path '{repoPath}'...");
Log($"Logged in. Building Logos Storage image in path '{repoPath}'...");
var customImage = GenerateImageName();
Docker($"build", "-t", customImage, "-f", "./codex.Dockerfile",
@ -62,7 +62,7 @@ namespace CodexNetDeployer
Docker("push", customImage);
CodexDockerImage.Override = customImage;
LogosStorageDockerImage.Override = customImage;
Log("Image pushed. Good to go!");
}
@ -96,8 +96,8 @@ namespace CodexNetDeployer
private string GenerateImageName()
{
var tag = Environment.GetEnvironmentVariable("DOCKERTAG");
if (string.IsNullOrEmpty(tag)) return $"{dockerUsername!}/nim-codex-autoimage:{Guid.NewGuid().ToString().ToLowerInvariant()}";
return $"{dockerUsername}/nim-codex-autoimage:{tag}";
if (string.IsNullOrEmpty(tag)) return $"{dockerUsername!}/logos-storage-nim-autoimage:{Guid.NewGuid().ToString().ToLowerInvariant()}";
return $"{dockerUsername}/logos-storage-nim-autoimage:{tag}";
}
private void Docker(params string[] args)

10
ProjectPlugins/CodexPlugin/CodexExePath.cs → ProjectPlugins/StoragePlugin/LogosStorageExePath.cs
View File

@ -1,15 +1,15 @@
namespace CodexPlugin
namespace StoragePlugin
{
public class CodexExePath
public class LogosStorageExePath
{
private readonly string[] paths = [
Path.Combine("d:", "Dev", "nim-codex", "build", "codex.exe"),
Path.Combine("c:", "Projects", "nim-codex", "build", "codex.exe")
Path.Combine("d:", "Dev", "logos-storage-nim", "build", "storage.exe"),
Path.Combine("c:", "Projects", "logos-storage-nim", "build", "storage.exe")
];
private string selectedPath = string.Empty;
public CodexExePath()
public LogosStorageExePath()
{
foreach (var p in paths)
{

20
ProjectPlugins/CodexPlugin/CodexProcessRecipe.cs → ProjectPlugins/StoragePlugin/LogosStorageProcessRecipe.cs
View File

@ -1,7 +1,7 @@
using System.Net.Sockets;
using System.Net;
namespace CodexPlugin
namespace StoragePlugin
{
public class ProcessRecipe
{
@ -15,9 +15,9 @@ namespace CodexPlugin
public string[] Args { get; }
}
public class CodexProcessConfig
public class LogosStorageProcessConfig
{
public CodexProcessConfig(string name, FreePortFinder freePortFinder, string dataDir)
public LogosStorageProcessConfig(string name, FreePortFinder freePortFinder, string dataDir)
{
ApiPort = freePortFinder.GetNextFreePort();
DiscPort = freePortFinder.GetNextFreePort();
@ -38,18 +38,18 @@ namespace CodexPlugin
public IPAddress LocalIpAddrs { get; }
}
public class CodexProcessRecipe
public class LogosStorageProcessRecipe
{
private readonly CodexProcessConfig pc;
private readonly CodexExePath codexExePath;
private readonly LogosStorageProcessConfig pc;
private readonly LogosStorageExePath logosStorageExePath;
public CodexProcessRecipe(CodexProcessConfig pc, CodexExePath codexExePath)
public LogosStorageProcessRecipe(LogosStorageProcessConfig pc, LogosStorageExePath logosStorageExePath)
{
this.pc = pc;
this.codexExePath = codexExePath;
this.logosStorageExePath = logosStorageExePath;
}
public ProcessRecipe Initialize(CodexStartupConfig config)
public ProcessRecipe Initialize(LogosStorageStartupConfig config)
{
args.Clear();
@ -104,7 +104,7 @@ namespace CodexPlugin
private ProcessRecipe Create()
{
return new ProcessRecipe(
cmd: codexExePath.Get(),
cmd: logosStorageExePath.Get(),
args: args.ToArray());
}

135
ProjectPlugins/StoragePlugin/LogosStorageSetup.cs Normal file
View File

@ -0,0 +1,135 @@
using LogosStorageClient;
using KubernetesWorkflow;
using Utils;
namespace StoragePlugin
{
public interface ILogosStorageSetup
{
ILogosStorageSetup WithName(string name);
ILogosStorageSetup At(ILocation location);
ILogosStorageSetup WithBootstrapNode(IStorageNode node);
ILogosStorageSetup WithLogLevel(LogosStorageLogLevel level);
ILogosStorageSetup WithLogLevel(LogosStorageLogLevel level, LogosStorageLogCustomTopics customTopics);
ILogosStorageSetup WithStorageQuota(ByteSize storageQuota);
ILogosStorageSetup WithBlockTTL(TimeSpan duration);
ILogosStorageSetup WithBlockMaintenanceInterval(TimeSpan duration);
ILogosStorageSetup WithBlockMaintenanceNumber(int numberOfBlocks);
ILogosStorageSetup EnableMetrics();
ILogosStorageSetup AsPublicTestNet(LogosStorageTestNetConfig testNetConfig);
}
public class LogosStorageLogCustomTopics
{
public LogosStorageLogCustomTopics(LogosStorageLogLevel discV5, LogosStorageLogLevel libp2p, LogosStorageLogLevel blockExchange)
{
DiscV5 = discV5;
Libp2p = libp2p;
BlockExchange = blockExchange;
}
public LogosStorageLogCustomTopics(LogosStorageLogLevel discV5, LogosStorageLogLevel libp2p)
{
DiscV5 = discV5;
Libp2p = libp2p;
}
public LogosStorageLogLevel DiscV5 { get; set; }
public LogosStorageLogLevel Libp2p { get; set; }
public LogosStorageLogLevel ContractClock { get; set; } = LogosStorageLogLevel.Warn;
public LogosStorageLogLevel? BlockExchange { get; }
public LogosStorageLogLevel JsonSerialize { get; set; } = LogosStorageLogLevel.Warn;
public LogosStorageLogLevel MarketplaceInfra { get; set; } = LogosStorageLogLevel.Warn;
}
public class LogosStorageSetup : LogosStorageStartupConfig, ILogosStorageSetup
{
public int NumberOfNodes { get; }
public LogosStorageSetup(int numberOfNodes)
{
NumberOfNodes = numberOfNodes;
}
public ILogosStorageSetup WithName(string name)
{
NameOverride = name;
return this;
}
public ILogosStorageSetup At(ILocation location)
{
Location = location;
return this;
}
public ILogosStorageSetup WithBootstrapNode(IStorageNode node)
{
BootstrapSpr = node.GetDebugInfo().Spr;
return this;
}
public ILogosStorageSetup WithLogLevel(LogosStorageLogLevel level)
{
LogLevel = level;
return this;
}
public ILogosStorageSetup WithLogLevel(LogosStorageLogLevel level, LogosStorageLogCustomTopics customTopics)
{
LogLevel = level;
CustomTopics = customTopics;
return this;
}
public ILogosStorageSetup WithStorageQuota(ByteSize storageQuota)
{
StorageQuota = storageQuota;
return this;
}
public ILogosStorageSetup WithBlockTTL(TimeSpan duration)
{
BlockTTL = Convert.ToInt32(duration.TotalSeconds);
return this;
}
public ILogosStorageSetup WithBlockMaintenanceInterval(TimeSpan duration)
{
BlockMaintenanceInterval = duration;
return this;
}
public ILogosStorageSetup WithBlockMaintenanceNumber(int numberOfBlocks)
{
BlockMaintenanceNumber = numberOfBlocks;
return this;
}
public ILogosStorageSetup EnableMetrics()
{
MetricsEnabled = true;
return this;
}
public ILogosStorageSetup AsPublicTestNet(LogosStorageTestNetConfig testNetConfig)
{
PublicTestNet = testNetConfig;
return this;
}
public string Describe()
{
var args = string.Join(',', DescribeArgs());
return $"({NumberOfNodes} Logos Storage Nodes with args:[{args}])";
}
private IEnumerable<string> DescribeArgs()
{
if (PublicTestNet != null) yield return $"<!>Public TestNet with listenPort: {PublicTestNet.PublicListenPort}<!>";
yield return $"LogLevel={LogLevelWithTopics()}";
if (BootstrapSpr != null) yield return $"BootstrapNode={BootstrapSpr}";
if (StorageQuota != null) yield return $"StorageQuota={StorageQuota}";
}
}
}

16
ProjectPlugins/CodexPlugin/CodexStartupConfig.cs → ProjectPlugins/StoragePlugin/LogosStorageStartupConfig.cs
View File

@ -1,15 +1,15 @@
using CodexClient;
using LogosStorageClient;
using KubernetesWorkflow;
using Utils;
namespace CodexPlugin
namespace StoragePlugin
{
public class CodexStartupConfig
public class LogosStorageStartupConfig
{
public string? NameOverride { get; set; }
public ILocation Location { get; set; } = KnownLocations.UnspecifiedLocation;
public CodexLogLevel LogLevel { get; set; }
public CodexLogCustomTopics? CustomTopics { get; set; } = new CodexLogCustomTopics(CodexLogLevel.Info, CodexLogLevel.Warn);
public LogosStorageLogLevel LogLevel { get; set; }
public LogosStorageLogCustomTopics? CustomTopics { get; set; } = new LogosStorageLogCustomTopics(LogosStorageLogLevel.Info, LogosStorageLogLevel.Warn);
public ByteSize? StorageQuota { get; set; }
public bool MetricsEnabled { get; set; }
public string? BootstrapSpr { get; set; }
@ -17,7 +17,7 @@ namespace CodexPlugin
public bool? EnableValidator { get; set; }
public TimeSpan? BlockMaintenanceInterval { get; set; }
public int? BlockMaintenanceNumber { get; set; }
public CodexTestNetConfig? PublicTestNet { get; set; }
public LogosStorageTestNetConfig? PublicTestNet { get; set; }
public string LogLevelWithTopics()
{
@ -84,7 +84,7 @@ namespace CodexPlugin
$"{CustomTopics.DiscV5.ToString()!.ToLowerInvariant()}:{string.Join(",", discV5Topics)};" +
$"{CustomTopics.Libp2p.ToString()!.ToLowerInvariant()}:{string.Join(",", libp2pTopics)};" +
$"{CustomTopics.JsonSerialize.ToString().ToLowerInvariant()}:{string.Join(",", jsonSerializeTopics)};" +
$"{CodexLogLevel.Error.ToString()}:{string.Join(",", alwaysIgnoreTopics)}";
$"{LogosStorageLogLevel.Error.ToString()}:{string.Join(",", alwaysIgnoreTopics)}";
if (CustomTopics.BlockExchange != null)
{
@ -95,7 +95,7 @@ namespace CodexPlugin
}
}
public class CodexTestNetConfig
public class LogosStorageTestNetConfig
{
public int PublicDiscoveryPort { get; set; }
public int PublicListenPort { get; set; }

36
ProjectPlugins/CodexPlugin/CodexWrapper.cs → ProjectPlugins/StoragePlugin/LogosStorageWrapper.cs
View File

@ -1,57 +1,57 @@
using CodexClient;
using CodexClient.Hooks;
using LogosStorageClient;
using LogosStorageClient.Hooks;
using Core;
using Logging;
namespace CodexPlugin
namespace StoragePlugin
{
public class CodexWrapper
public class LogosStorageWrapper
{
private readonly IPluginTools pluginTools;
private readonly ProcessControlMap processControlMap;
private readonly CodexHooksFactory hooksFactory;
private readonly LogosStorageHooksFactory hooksFactory;
private DebugInfoVersion? versionResponse;
public CodexWrapper(IPluginTools pluginTools, ProcessControlMap processControlMap, CodexHooksFactory hooksFactory)
public LogosStorageWrapper(IPluginTools pluginTools, ProcessControlMap processControlMap, LogosStorageHooksFactory hooksFactory)
{
this.pluginTools = pluginTools;
this.processControlMap = processControlMap;
this.hooksFactory = hooksFactory;
}
public string GetCodexId()
public string GetLogosStorageId()
{
if (versionResponse != null) return versionResponse.Version;
return "unknown";
}
public string GetCodexRevision()
public string GetLogosStorageRevision()
{
if (versionResponse != null) return versionResponse.Revision;
return "unknown";
}
public ICodexNodeGroup WrapCodexInstances(ICodexInstance[] instances)
public IStorageNodeGroup WrapLogosStorageInstances(ILogosStorageInstance[] instances)
{
var codexNodeFactory = new CodexNodeFactory(
var storageNodeFactory = new StorageNodeFactory(
log: pluginTools.GetLog(),
fileManager: pluginTools.GetFileManager(),
hooksFactory: hooksFactory,
httpFactory: pluginTools,
processControlFactory: processControlMap);
var group = CreateCodexGroup(instances, codexNodeFactory);
var group = CreateStorageNodeGroup(instances, storageNodeFactory);
pluginTools.GetLog().Log($"Codex version: {group.Version}");
pluginTools.GetLog().Log($"Logos Storage version: {group.Version}");
versionResponse = group.Version;
return group;
}
private CodexNodeGroup CreateCodexGroup(ICodexInstance[] instances, CodexNodeFactory codexNodeFactory)
private StorageNodeGroup CreateStorageNodeGroup(ILogosStorageInstance[] instances, StorageNodeFactory storageNodeFactory)
{
var nodes = instances.Select(codexNodeFactory.CreateCodexNode).ToArray();
var group = new CodexNodeGroup(pluginTools, nodes);
var nodes = instances.Select(storageNodeFactory.CreateStorageNode).ToArray();
var group = new StorageNodeGroup(pluginTools, nodes);
try
{
@ -59,16 +59,16 @@ namespace CodexPlugin
}
catch
{
CodexNodesNotOnline(instances);
LogosStorageNodesNotOnline(instances);
throw;
}
return group;
}
private void CodexNodesNotOnline(ICodexInstance[] instances)
private void LogosStorageNodesNotOnline(ILogosStorageInstance[] instances)
{
pluginTools.GetLog().Log("Codex nodes failed to start");
pluginTools.GetLog().Log("Logos Storage nodes failed to start");
var log = pluginTools.GetLog();
foreach (var i in instances)
{

18
ProjectPlugins/CodexPlugin/OverwatchSupport/IdentityMap.cs → ProjectPlugins/StoragePlugin/OverwatchSupport/IdentityMap.cs
View File

@ -1,16 +1,16 @@
using CodexClient;
using LogosStorageClient;
namespace CodexPlugin.OverwatchSupport
namespace StoragePlugin.OverwatchSupport
{
public class IdentityMap
{
private readonly List<CodexNodeIdentity> nodes = new List<CodexNodeIdentity>();
private readonly List<StorageNodeIdentity> nodes = new List<StorageNodeIdentity>();
private readonly Dictionary<string, int> nameIndexMap = new Dictionary<string, int>();
private readonly Dictionary<string, string> shortToLong = new Dictionary<string, string>();
public void Add(string name, string peerId, string nodeId)
{
Add(new CodexNodeIdentity
Add(new StorageNodeIdentity
{
Name = name,
PeerId = peerId,
@ -20,7 +20,7 @@ namespace CodexPlugin.OverwatchSupport
nameIndexMap.Add(name, nameIndexMap.Count);
}
public void Add(CodexNodeIdentity identity)
public void Add(StorageNodeIdentity identity)
{
if (string.IsNullOrWhiteSpace(identity.Name)) throw new Exception("Name required");
if (string.IsNullOrWhiteSpace(identity.PeerId) || identity.PeerId.Length < 11) throw new Exception("PeerId invalid");
@ -28,11 +28,11 @@ namespace CodexPlugin.OverwatchSupport
nodes.Add(identity);
shortToLong.Add(CodexUtils.ToShortId(identity.PeerId), identity.PeerId);
shortToLong.Add(CodexUtils.ToNodeIdShortId(identity.NodeId), identity.NodeId);
shortToLong.Add(LogosStorageUtils.ToShortId(identity.PeerId), identity.PeerId);
shortToLong.Add(LogosStorageUtils.ToNodeIdShortId(identity.NodeId), identity.NodeId);
}
public CodexNodeIdentity[] Get()
public StorageNodeIdentity[] Get()
{
return nodes.ToArray();
}
@ -42,7 +42,7 @@ namespace CodexPlugin.OverwatchSupport
return nameIndexMap[name];
}
public CodexNodeIdentity GetId(string name)
public StorageNodeIdentity GetId(string name)
{
return nodes.Single(n => n.Name == name);
}

10
ProjectPlugins/CodexPlugin/OverwatchSupport/KademliaPositionFinder.cs → ProjectPlugins/StoragePlugin/OverwatchSupport/KademliaPositionFinder.cs
View File

@ -2,11 +2,11 @@
using Utils;
using YamlDotNet.Core.Tokens;
namespace CodexPlugin.OverwatchSupport
namespace StoragePlugin.OverwatchSupport
{
public class KademliaPositionFinder
{
public CodexNodeIdentity[] DeterminePositions(CodexNodeIdentity[] identities)
public StorageNodeIdentity[] DeterminePositions(StorageNodeIdentity[] identities)
{
var zero = identities.First();
var distances = CalculateDistances(zero, identities);
@ -17,9 +17,9 @@ namespace CodexPlugin.OverwatchSupport
return identities;
}
private Dictionary<CodexNodeIdentity, BigInteger> CalculateDistances(CodexNodeIdentity zero, CodexNodeIdentity[] identities)
private Dictionary<StorageNodeIdentity, BigInteger> CalculateDistances(StorageNodeIdentity zero, StorageNodeIdentity[] identities)
{
var result = new Dictionary<CodexNodeIdentity, BigInteger>();
var result = new Dictionary<StorageNodeIdentity, BigInteger>();
foreach (var id in identities.Skip(1))
{
result.Add(id, GetDistance(zero.NodeId, id.NodeId));
@ -51,7 +51,7 @@ namespace CodexPlugin.OverwatchSupport
return result;
}
private void CalculateNormalizedPositions(Dictionary<CodexNodeIdentity, BigInteger> distances, BigInteger maxDistance)
private void CalculateNormalizedPositions(Dictionary<StorageNodeIdentity, BigInteger> distances, BigInteger maxDistance)
{
foreach (var pair in distances)
{

6
ProjectPlugins/CodexPlugin/OverwatchSupport/LineConverters/BlockReceivedLineConverter.cs → ProjectPlugins/StoragePlugin/OverwatchSupport/LineConverters/BlockReceivedLineConverter.cs
View File

@ -1,12 +1,12 @@
using CodexClient;
using LogosStorageClient;
namespace CodexPlugin.OverwatchSupport.LineConverters
namespace StoragePlugin.OverwatchSupport.LineConverters
{
public class BlockReceivedLineConverter : ILineConverter
{
public string Interest => "Received blocks from peer";
public void Process(CodexLogLine line, Action<Action<OverwatchCodexEvent>> addEvent)
public void Process(LogosStorageLogLine line, Action<Action<OverwatchLogosStorageEvent>> addEvent)
{
var peer = line.Attributes["peer"];
var blockAddresses = line.Attributes["blocks"];

6
ProjectPlugins/CodexPlugin/OverwatchSupport/LineConverters/BootstrapLineConverter.cs → ProjectPlugins/StoragePlugin/OverwatchSupport/LineConverters/BootstrapLineConverter.cs
View File

@ -1,6 +1,6 @@
using CodexClient;
using LogosStorageClient;
namespace CodexPlugin.OverwatchSupport.LineConverters
namespace StoragePlugin.OverwatchSupport.LineConverters
{
public class BootstrapLineConverter : ILineConverter
{
@ -8,7 +8,7 @@ namespace CodexPlugin.OverwatchSupport.LineConverters
public string Interest => "Starting codex node";
public void Process(CodexLogLine line, Action<Action<OverwatchCodexEvent>> addEvent)
public void Process(LogosStorageLogLine line, Action<Action<OverwatchLogosStorageEvent>> addEvent)
{
// "(
// configFile: none(InputFile),

6
ProjectPlugins/CodexPlugin/OverwatchSupport/LineConverters/DialSuccessfulLineConverter.cs → ProjectPlugins/StoragePlugin/OverwatchSupport/LineConverters/DialSuccessfulLineConverter.cs
View File

@ -1,12 +1,12 @@
using CodexClient;
using LogosStorageClient;
namespace CodexPlugin.OverwatchSupport.LineConverters
namespace StoragePlugin.OverwatchSupport.LineConverters
{
public class DialSuccessfulLineConverter : ILineConverter
{
public string Interest => "Dial successful";
public void Process(CodexLogLine line, Action<Action<OverwatchCodexEvent>> addEvent)
public void Process(LogosStorageLogLine line, Action<Action<OverwatchLogosStorageEvent>> addEvent)
{
var peerId = line.Attributes["peerId"];

6
ProjectPlugins/CodexPlugin/OverwatchSupport/LineConverters/PeerDroppedLineConverter.cs → ProjectPlugins/StoragePlugin/OverwatchSupport/LineConverters/PeerDroppedLineConverter.cs
View File

@ -1,12 +1,12 @@
using CodexClient;
using LogosStorageClient;
namespace CodexPlugin.OverwatchSupport.LineConverters
namespace StoragePlugin.OverwatchSupport.LineConverters
{
public class PeerDroppedLineConverter : ILineConverter
{
public string Interest => "Dropping peer";
public void Process(CodexLogLine line, Action<Action<OverwatchCodexEvent>> addEvent)
public void Process(LogosStorageLogLine line, Action<Action<OverwatchLogosStorageEvent>> addEvent)
{
var peerId = line.Attributes["peer"];

38
ProjectPlugins/CodexPlugin/OverwatchSupport/CodexLogConverter.cs → ProjectPlugins/StoragePlugin/OverwatchSupport/LogosStorageLogConverter.cs
View File

@ -1,18 +1,18 @@
using CodexClient;
using CodexPlugin.OverwatchSupport.LineConverters;
using LogosStorageClient;
using StoragePlugin.OverwatchSupport.LineConverters;
using Logging;
using OverwatchTranscript;
using Utils;
namespace CodexPlugin.OverwatchSupport
namespace StoragePlugin.OverwatchSupport
{
public class CodexLogConverter
public class LogosStorageLogConverter
{
private readonly ITranscriptWriter writer;
private readonly CodexTranscriptWriterConfig config;
private readonly LogosStorageTranscriptWriterConfig config;
private readonly IdentityMap identityMap;
public CodexLogConverter(ITranscriptWriter writer, CodexTranscriptWriterConfig config, IdentityMap identityMap)
public LogosStorageLogConverter(ITranscriptWriter writer, LogosStorageTranscriptWriterConfig config, IdentityMap identityMap)
{
this.writer = writer;
this.config = config;
@ -43,7 +43,7 @@ namespace CodexPlugin.OverwatchSupport
private readonly int nodeIdentityIndex;
private readonly ILineConverter[] converters;
public ConversionRunner(ITranscriptWriter writer, CodexTranscriptWriterConfig config, IdentityMap nameIdMap, int nodeIdentityIndex)
public ConversionRunner(ITranscriptWriter writer, LogosStorageTranscriptWriterConfig config, IdentityMap nameIdMap, int nodeIdentityIndex)
{
this.nodeIdentityIndex = nodeIdentityIndex;
this.writer = writer;
@ -52,7 +52,7 @@ namespace CodexPlugin.OverwatchSupport
converters = CreateConverters(config).ToArray();
}
private IEnumerable<ILineConverter> CreateConverters(CodexTranscriptWriterConfig config)
private IEnumerable<ILineConverter> CreateConverters(LogosStorageTranscriptWriterConfig config)
{
if (config.IncludeBlockReceivedEvents)
{
@ -74,9 +74,9 @@ namespace CodexPlugin.OverwatchSupport
});
}
private void AddEvent(DateTime utc, Action<OverwatchCodexEvent> action)
private void AddEvent(DateTime utc, Action<OverwatchLogosStorageEvent> action)
{
var e = new OverwatchCodexEvent
var e = new OverwatchLogosStorageEvent
{
NodeIdentity = nodeIdentityIndex,
};
@ -89,18 +89,18 @@ namespace CodexPlugin.OverwatchSupport
{
if (!line.Contains(converter.Interest)) return;
var codexLine = CodexLogLine.Parse(line);
var logosStorageLogLine = LogosStorageLogLine.Parse(line);
if (codexLine == null) throw new Exception("Unable to parse required line");
EnsureFullIds(codexLine);
if (logosStorageLogLine == null) throw new Exception("Unable to parse required line");
EnsureFullIds(logosStorageLogLine);
converter.Process(codexLine, (action) =>
converter.Process(logosStorageLogLine, (action) =>
{
AddEvent(codexLine.TimestampUtc, action);
AddEvent(logosStorageLogLine.TimestampUtc, action);
});
}
private void EnsureFullIds(CodexLogLine codexLine)
private void EnsureFullIds(LogosStorageLogLine logosStorageLogLine)
{
// The issue is: node IDs occure both in full and short version.
// Downstream tools will assume that a node ID string-equals its own ID.
@ -111,11 +111,11 @@ namespace CodexPlugin.OverwatchSupport
// But sometimes, it is part of a larger string:
// "thing=abc:123*567890,def"
foreach (var pair in codexLine.Attributes)
foreach (var pair in logosStorageLogLine.Attributes)
{
if (pair.Value.Contains("*"))
{
codexLine.Attributes[pair.Key] = nameIdMap.ReplaceShortIds(pair.Value);
logosStorageLogLine.Attributes[pair.Key] = nameIdMap.ReplaceShortIds(pair.Value);
}
}
}
@ -124,6 +124,6 @@ namespace CodexPlugin.OverwatchSupport
public interface ILineConverter
{
string Interest { get; }
void Process(CodexLogLine line, Action<Action<OverwatchCodexEvent>> addEvent);
void Process(LogosStorageLogLine line, Action<Action<OverwatchLogosStorageEvent>> addEvent);
}
}

38
ProjectPlugins/CodexPlugin/OverwatchSupport/CodexTranscriptWriter.cs → ProjectPlugins/StoragePlugin/OverwatchSupport/LogosStorageTranscriptWriter.cs
View File

@ -1,33 +1,33 @@
using CodexClient.Hooks;
using LogosStorageClient.Hooks;
using Logging;
using OverwatchTranscript;
using Utils;
namespace CodexPlugin.OverwatchSupport
namespace StoragePlugin.OverwatchSupport
{
public class CodexTranscriptWriter : ICodexHooksProvider
public class LogosStorageTranscriptWriter : ILogosStorageHooksProvider
{
private const string CodexHeaderKey = "cdx_h";
private const string LogosStorageHeaderKey = "cdx_h";
private readonly ILog log;
private readonly CodexTranscriptWriterConfig config;
private readonly LogosStorageTranscriptWriterConfig config;
private readonly ITranscriptWriter writer;
private readonly CodexLogConverter converter;
private readonly LogosStorageLogConverter converter;
private readonly IdentityMap identityMap = new IdentityMap();
private readonly KademliaPositionFinder positionFinder = new KademliaPositionFinder();
public CodexTranscriptWriter(ILog log, CodexTranscriptWriterConfig config, ITranscriptWriter transcriptWriter)
public LogosStorageTranscriptWriter(ILog log, LogosStorageTranscriptWriterConfig config, ITranscriptWriter transcriptWriter)
{
this.log = log;
this.config = config;
writer = transcriptWriter;
converter = new CodexLogConverter(writer, config, identityMap);
converter = new LogosStorageLogConverter(writer, config, identityMap);
}
public void FinalizeWriter()
{
log.Log("Finalizing Codex transcript...");
log.Log("Finalizing Logos Storage transcript...");
writer.AddHeader(CodexHeaderKey, CreateCodexHeader());
writer.AddHeader(LogosStorageHeaderKey, CreateLogosStorageHeader());
writer.Write(GetOutputFullPath());
log.Log("Done");
@ -44,10 +44,10 @@ namespace CodexPlugin.OverwatchSupport
return outputFile;
}
public ICodexNodeHooks CreateHooks(string nodeName)
public IStorageNodeHooks CreateHooks(string nodeName)
{
nodeName = Str.Between(nodeName, "'", "'");
return new CodexNodeTranscriptWriter(writer, identityMap, nodeName);
return new StorageNodeTranscriptWriter(writer, identityMap, nodeName);
}
public void IncludeFile(string filepath)
@ -64,9 +64,9 @@ namespace CodexPlugin.OverwatchSupport
// Not all of these logs are necessarily Codex logs.
// Check, and process only the Codex ones.
if (IsCodexLog(l))
if (IsLogosStorageLog(l))
{
log.Log("Processing Codex log: " + l.GetFilepath());
log.Log("Processing Logos Storage log: " + l.GetFilepath());
converter.ProcessLog(l);
}
}
@ -74,7 +74,7 @@ namespace CodexPlugin.OverwatchSupport
public void AddResult(bool success, string result)
{
writer.Add(DateTime.UtcNow, new OverwatchCodexEvent
writer.Add(DateTime.UtcNow, new OverwatchLogosStorageEvent
{
NodeIdentity = -1,
ScenarioFinished = new ScenarioFinishedEvent
@ -85,17 +85,17 @@ namespace CodexPlugin.OverwatchSupport
});
}
private OverwatchCodexHeader CreateCodexHeader()
private OverwatchLogosStorageHeader CreateLogosStorageHeader()
{
return new OverwatchCodexHeader
return new OverwatchLogosStorageHeader
{
Nodes = positionFinder.DeterminePositions(identityMap.Get())
};
}
private bool IsCodexLog(IDownloadedLog log)
private bool IsLogosStorageLog(IDownloadedLog log)
{
return log.GetLinesContaining("Run Codex node").Any();
return log.GetLinesContaining("Run Logos Storage node").Any();
}
}
}

6
ProjectPlugins/CodexPlugin/OverwatchSupport/CodexTranscriptWriterConfig.cs → ProjectPlugins/StoragePlugin/OverwatchSupport/LogosStorageTranscriptWriterConfig.cs
View File

@ -1,8 +1,8 @@
namespace CodexPlugin.OverwatchSupport
namespace StoragePlugin.OverwatchSupport
{
public class CodexTranscriptWriterConfig
public class LogosStorageTranscriptWriterConfig
{
public CodexTranscriptWriterConfig(string outputPath, bool includeBlockReceivedEvents)
public LogosStorageTranscriptWriterConfig(string outputPath, bool includeBlockReceivedEvents)
{
OutputPath = outputPath;
IncludeBlockReceivedEvents = includeBlockReceivedEvents;

14
ProjectPlugins/CodexPlugin/OverwatchSupport/ModelExtensions.cs → ProjectPlugins/StoragePlugin/OverwatchSupport/ModelExtensions.cs
View File

@ -1,16 +1,16 @@
using CodexClient;
using LogosStorageClient;
using OverwatchTranscript;
namespace CodexPlugin.OverwatchSupport
namespace StoragePlugin.OverwatchSupport
{
[Serializable]
public class OverwatchCodexHeader
public class OverwatchLogosStorageHeader
{
public CodexNodeIdentity[] Nodes { get; set; } = Array.Empty<CodexNodeIdentity>();
public StorageNodeIdentity[] Nodes { get; set; } = Array.Empty<StorageNodeIdentity>();
}
[Serializable]
public class OverwatchCodexEvent
public class OverwatchLogosStorageEvent
{
public int NodeIdentity { get; set; } = -1;
public ScenarioFinishedEvent? ScenarioFinished { get; set; }
@ -48,7 +48,7 @@ namespace CodexPlugin.OverwatchSupport
}
[Serializable]
public class CodexNodeIdentity
public class StorageNodeIdentity
{
public string Name { get; set; } = string.Empty;
public string PeerId { get; set; } = string.Empty;
@ -117,7 +117,7 @@ namespace CodexPlugin.OverwatchSupport
#endregion
#region Codex Generated Events
#region Logos Storage Generated Events
[Serializable]
public class BlockReceivedEvent

38
ProjectPlugins/CodexPlugin/OverwatchSupport/CodexNodeTranscriptWriter.cs → ProjectPlugins/StoragePlugin/OverwatchSupport/StorageNodeTranscriptWriter.cs
View File

@ -1,19 +1,19 @@
using CodexClient;
using CodexClient.Hooks;
using LogosStorageClient;
using LogosStorageClient.Hooks;
using OverwatchTranscript;
using Utils;
namespace CodexPlugin.OverwatchSupport
namespace StoragePlugin.OverwatchSupport
{
public class CodexNodeTranscriptWriter : ICodexNodeHooks
public class StorageNodeTranscriptWriter : IStorageNodeHooks
{
private readonly ITranscriptWriter writer;
private readonly IdentityMap identityMap;
private readonly string name;
private int identityIndex = -1;
private readonly List<(DateTime, OverwatchCodexEvent)> pendingEvents = new List<(DateTime, OverwatchCodexEvent)>();
private readonly List<(DateTime, OverwatchLogosStorageEvent)> pendingEvents = new List<(DateTime, OverwatchLogosStorageEvent)>();
public CodexNodeTranscriptWriter(ITranscriptWriter writer, IdentityMap identityMap, string name)
public StorageNodeTranscriptWriter(ITranscriptWriter writer, IdentityMap identityMap, string name)
{
this.writer = writer;
this.identityMap = identityMap;
@ -22,7 +22,7 @@ namespace CodexPlugin.OverwatchSupport
public void OnNodeStarting(DateTime startUtc, string image)
{
WriteCodexEvent(startUtc, e =>
WriteLogosStorageEvent(startUtc, e =>
{
e.NodeStarting = new NodeStartingEvent
{
@ -31,7 +31,7 @@ namespace CodexPlugin.OverwatchSupport
});
}
public void OnNodeStarted(ICodexNode node, string peerId, string nodeId)
public void OnNodeStarted(IStorageNode node, string peerId, string nodeId)
{
if (string.IsNullOrEmpty(peerId) || string.IsNullOrEmpty(nodeId))
{
@ -41,7 +41,7 @@ namespace CodexPlugin.OverwatchSupport
identityMap.Add(name, peerId, nodeId);
identityIndex = identityMap.GetIndex(name);
WriteCodexEvent(e =>
WriteLogosStorageEvent(e =>
{
e.NodeStarted = new NodeStartedEvent
{
@ -51,7 +51,7 @@ namespace CodexPlugin.OverwatchSupport
public void OnNodeStopping()
{
WriteCodexEvent(e =>
WriteLogosStorageEvent(e =>
{
e.NodeStopping = new NodeStoppingEvent
{
@ -61,7 +61,7 @@ namespace CodexPlugin.OverwatchSupport
public void OnFileDownloading(ContentId cid)
{
WriteCodexEvent(e =>
WriteLogosStorageEvent(e =>
{
e.FileDownloading = new FileDownloadingEvent
{
@ -72,7 +72,7 @@ namespace CodexPlugin.OverwatchSupport
public void OnFileDownloaded(ByteSize size, ContentId cid)
{
WriteCodexEvent(e =>
WriteLogosStorageEvent(e =>
{
e.FileDownloaded = new FileDownloadedEvent
{
@ -84,7 +84,7 @@ namespace CodexPlugin.OverwatchSupport
public void OnFileUploading(string uid, ByteSize size)
{
WriteCodexEvent(e =>
WriteLogosStorageEvent(e =>
{
e.FileUploading = new FileUploadingEvent
{
@ -96,7 +96,7 @@ namespace CodexPlugin.OverwatchSupport
public void OnFileUploaded(string uid, ByteSize size, ContentId cid)
{
WriteCodexEvent(e =>
WriteLogosStorageEvent(e =>
{
e.FileUploaded = new FileUploadedEvent
{
@ -107,14 +107,14 @@ namespace CodexPlugin.OverwatchSupport
});
}
private void WriteCodexEvent(Action<OverwatchCodexEvent> action)
private void WriteLogosStorageEvent(Action<OverwatchLogosStorageEvent> action)
{
WriteCodexEvent(DateTime.UtcNow, action);
WriteLogosStorageEvent(DateTime.UtcNow, action);
}
private void WriteCodexEvent(DateTime utc, Action<OverwatchCodexEvent> action)
private void WriteLogosStorageEvent(DateTime utc, Action<OverwatchLogosStorageEvent> action)
{
var e = new OverwatchCodexEvent
var e = new OverwatchLogosStorageEvent
{
NodeIdentity = identityIndex
};
@ -135,7 +135,7 @@ namespace CodexPlugin.OverwatchSupport
}
}
private void AddToCache(DateTime utc, OverwatchCodexEvent e)
private void AddToCache(DateTime utc, OverwatchLogosStorageEvent e)
{
pendingEvents.Add((utc, e));
}

12
ProjectPlugins/CodexPlugin/ProcessControlMap.cs → ProjectPlugins/StoragePlugin/ProcessControlMap.cs
View File

@ -1,27 +1,27 @@
using CodexClient;
using LogosStorageClient;
namespace CodexPlugin
namespace StoragePlugin
{
public class ProcessControlMap : IProcessControlFactory
{
private readonly Dictionary<string, IProcessControl> processControlMap = new Dictionary<string, IProcessControl>();
public void Add(ICodexInstance instance, IProcessControl control)
public void Add(ILogosStorageInstance instance, IProcessControl control)
{
processControlMap.Add(instance.Name, control);
}
public void Remove(ICodexInstance instance)
public void Remove(ILogosStorageInstance instance)
{
processControlMap.Remove(instance.Name);
}
public IProcessControl CreateProcessControl(ICodexInstance instance)
public IProcessControl CreateProcessControl(ILogosStorageInstance instance)
{
return Get(instance);
}
public IProcessControl Get(ICodexInstance instance)
public IProcessControl Get(ILogosStorageInstance instance)
{
return processControlMap[instance.Name];
}

32
ProjectPlugins/CodexPlugin/CodexNodeGroup.cs → ProjectPlugins/StoragePlugin/StorageNodeGroup.cs
View File

@ -1,27 +1,27 @@
using CodexClient;
using LogosStorageClient;
using Core;
using System.Collections;
using Utils;
namespace CodexPlugin
namespace StoragePlugin
{
public interface ICodexNodeGroup : IEnumerable<ICodexNode>, IHasManyMetricScrapeTargets
public interface IStorageNodeGroup : IEnumerable<IStorageNode>, IHasManyMetricScrapeTargets
{
void Stop(bool waitTillStopped);
ICodexNode this[int index] { get; }
IStorageNode this[int index] { get; }
}
public class CodexNodeGroup : ICodexNodeGroup
public class StorageNodeGroup : IStorageNodeGroup
{
private readonly ICodexNode[] nodes;
private readonly IStorageNode[] nodes;
public CodexNodeGroup(IPluginTools tools, ICodexNode[] nodes)
public StorageNodeGroup(IPluginTools tools, IStorageNode[] nodes)
{
this.nodes = nodes;
Version = new DebugInfoVersion();
}
public ICodexNode this[int index]
public IStorageNode this[int index]
{
get
{
@ -34,12 +34,12 @@ namespace CodexPlugin
foreach (var node in Nodes) node.Stop(waitTillStopped);
}
public void Stop(CodexNode node, bool waitTillStopped)
public void Stop(StorageNode node, bool waitTillStopped)
{
node.Stop(waitTillStopped);
}
public ICodexNode[] Nodes => nodes;
public IStorageNode[] Nodes => nodes;
public DebugInfoVersion Version { get; private set; }
public Address[] GetMetricsScrapeTargets()
@ -47,9 +47,9 @@ namespace CodexPlugin
return Nodes.Select(n => n.GetMetricsScrapeTarget()).ToArray();
}
public IEnumerator<ICodexNode> GetEnumerator()
public IEnumerator<IStorageNode> GetEnumerator()
{
return Nodes.Cast<ICodexNode>().GetEnumerator();
return Nodes.Cast<IStorageNode>().GetEnumerator();
}
IEnumerator IEnumerable.GetEnumerator()
@ -74,7 +74,7 @@ namespace CodexPlugin
var first = versionResponses.First();
if (!versionResponses.All(v => v.Version == first.Version && v.Revision == first.Revision))
{
throw new Exception("Inconsistent version information received from one or more Codex nodes: " +
throw new Exception("Inconsistent version information received from one or more Logos Storage nodes: " +
string.Join(",", versionResponses.Select(v => v.ToString())));
}
@ -82,14 +82,14 @@ namespace CodexPlugin
}
}
public static class CodexNodeGroupExtensions
public static class StorageNodeGroupExtensions
{
public static string Names(this ICodexNode[] nodes)
public static string Names(this IStorageNode[] nodes)
{
return $"[{string.Join(",", nodes.Select(n => n.GetName()))}]";
}
public static string Names(this List<ICodexNode> nodes)
public static string Names(this List<IStorageNode> nodes)
{
return $"[{string.Join(",", nodes.Select(n => n.GetName()))}]";
}

96
ProjectPlugins/StoragePlugin/StoragePlugin.cs Normal file
View File

@ -0,0 +1,96 @@
using LogosStorageClient;
using LogosStorageClient.Hooks;
using Core;
namespace StoragePlugin
{
public class StoragePlugin : IProjectPlugin, IHasLogPrefix, IHasMetadata
{
private const bool UseContainers = true;
private readonly ILogosStorageStarter logosStorageStarter;
private readonly IPluginTools tools;
private readonly LogosStorageLogLevel defaultLogLevel = LogosStorageLogLevel.Trace;
private readonly LogosStorageHooksFactory hooksFactory = new LogosStorageHooksFactory();
private readonly ProcessControlMap processControlMap = new ProcessControlMap();
private readonly LogosStorageDockerImage logosStorageDockerImage = new LogosStorageDockerImage();
private readonly LogosStorageContainerRecipe recipe;
private readonly LogosStorageWrapper logosStorageWrapper;
public StoragePlugin(IPluginTools tools)
{
this.tools = tools;
recipe = new LogosStorageContainerRecipe(logosStorageDockerImage);
logosStorageStarter = CreateLogosStorageStarter();
logosStorageWrapper = new LogosStorageWrapper(tools, processControlMap, hooksFactory);
}
private ILogosStorageStarter CreateLogosStorageStarter()
{
if (UseContainers)
{
Log("Using Containerized Logos Storage instances");
return new ContainerLogosStorageStarter(tools, recipe, processControlMap);
}
Log("Using Binary Logos Storage instances");
return new BinaryLogosStorageStarter(tools, processControlMap);
}
public string LogPrefix => "(LogosStorage) ";
public void Awake(IPluginAccess access)
{
}
public void Announce()
{
// give codex docker image to contracts plugin.
Log($"Loaded with Logos Storage ID: '{logosStorageWrapper.GetLogosStorageId()}' - Revision: {logosStorageWrapper.GetLogosStorageRevision()}");
}
public void AddMetadata(IAddMetadata metadata)
{
metadata.Add("storageid", logosStorageWrapper.GetLogosStorageId());
metadata.Add("storagerevision", logosStorageWrapper.GetLogosStorageRevision());
}
public void Decommission()
{
logosStorageStarter.Decommission();
}
public ILogosStorageInstance[] DeployLogosStorageNodes(int numberOfNodes, Action<ILogosStorageSetup> setup)
{
var logosStorageSetup = GetSetup(numberOfNodes, setup);
return logosStorageStarter.BringOnline(logosStorageSetup);
}
public IStorageNodeGroup WrapLogosStorageContainers(ILogosStorageInstance[] instances)
{
instances = instances.Select(c => SerializeGate.Gate(c as LogosStorageInstance)).ToArray();
return logosStorageWrapper.WrapLogosStorageInstances(instances);
}
public void AddLogosStorageHooksProvider(ILogosStorageHooksProvider hooksProvider)
{
if (hooksFactory.Providers.Contains(hooksProvider)) return;
hooksFactory.Providers.Add(hooksProvider);
}
private LogosStorageSetup GetSetup(int numberOfNodes, Action<ILogosStorageSetup> setup)
{
var logosStorageSetup = new LogosStorageSetup(numberOfNodes);
logosStorageSetup.LogLevel = defaultLogLevel;
setup(logosStorageSetup);
return logosStorageSetup;
}
private void Log(string msg)
{
tools.GetLog().Log(msg);
}
}
}

4
ProjectPlugins/CodexPlugin/CodexPlugin.csproj → ProjectPlugins/StoragePlugin/StoragePlugin.csproj
View File

@ -22,12 +22,12 @@
<ProjectReference Include="..\..\Framework\Core\Core.csproj" />
<ProjectReference Include="..\..\Framework\KubernetesWorkflow\KubernetesWorkflow.csproj" />
<ProjectReference Include="..\..\Framework\OverwatchTranscript\OverwatchTranscript.csproj" />
<ProjectReference Include="..\CodexClient\CodexClient.csproj" />
<ProjectReference Include="..\LogosStorageClient\LogosStorageClient.csproj" />
<ProjectReference Include="..\MetricsPlugin\MetricsPlugin.csproj" />
</ItemGroup>
<Target Name="PreBuild" BeforeTargets="PreBuildEvent">
<Exec Command="dotnet run --project $(ProjectDir)\..\CodexPluginPrebuild" />
<Exec Command="dotnet run --project $(ProjectDir)\..\LogosStoragePluginPrebuild" />
</Target>
</Project>

12
README.md
View File

@ -7,7 +7,7 @@ Kubernetes: v1.34.1
Dotnet-kubernetes SDK: v18.0.3 https://github.com/kubernetes-client/csharp
Nethereum: v4.14.0
Currently, this project is mainly used for distributed testing of [Nim-Codex](https://github.com/codex-storage/nim-codex). However, its plugin-structure allows for other projects to be on-boarded (relatively) easily. (See 'contribute a plugin`.)
Currently, this project is mainly used for distributed testing of [Logos Storage](https://github.com/logos-storage/logos-storage-nim). However, its plugin-structure allows for other projects to be on-boarded (relatively) easily. (See 'contribute a plugin`.)
## Tests/DistTestCore
Library with generic distributed-testing functionality. Uses NUnit3. Reference this project to build unit-test style scenarios: setup, run test, teardown. The DistTestCore responds to the following env-vars:
@ -15,13 +15,13 @@ Library with generic distributed-testing functionality. Uses NUnit3. Reference t
- `DATAFILEPATH` = Path where (temporary) data files will be stored.
- `ALWAYS_LOGS` = When set, DistTestCore will always download all container logs at the end of a test run. By default, logs are only downloaded on test failure.
## Tests/CodexTests and Tests/CodexLongTests
These are test assemblies that use DistTestCore to perform tests against transient Codex nodes.
Read more [HERE](/Tests/CodexTests/README.md)
## Tests/LogosStorageTests and Tests/LogosStorageLongTests
These are test assemblies that use DistTestCore to perform tests against transient Logos Storage nodes.
Read more [HERE](/Tests/LogosStorageTests/README.md)
## Tests/ContinuousTests
A console application that runs tests in an endless loop against a persistent deployment of Codex nodes.
Read more [HERE](/Tests/CodexContinuousTests/README.md)
A console application that runs tests in an endless loop against a persistent deployment of Logos Storage nodes.
Read more [HERE](/Tests/LogosStorageContinuousTests/README.md)
## Framework architecture
The framework is designed to be extended by project-specific plugins. These plugins contribute functionality and abstractions to the framework. Users of the framework use these to perform tasks such as testing and deploying.

2
Tests/DistTestCore/Configuration.cs
View File

@ -12,7 +12,7 @@ namespace DistTestCore
public Configuration()
{
kubeConfigFile = GetNullableEnvVarOrDefault("KUBECONFIG", null);
logPath = GetEnvVarOrDefault("LOGPATH", "CodexTestLogs");
logPath = GetEnvVarOrDefault("LOGPATH", "LogosStorageTestLogs");
dataFilesPath = GetEnvVarOrDefault("DATAFILEPATH", "TestDataFiles");
AlwaysDownloadContainerLogs = !string.IsNullOrEmpty(GetEnvVarOrDefault("ALWAYS_LOGS", ""));
}

4
Tests/DistTestCore/DistTest.cs
View File

@ -181,7 +181,7 @@ namespace DistTestCore
{
if (IsRunningInCluster())
{
Log(" > Detected we're running in the cluster. Using long webCall timeset.");
fixtureLog.Log(" > Detected we're running in the cluster. Using long webCall timeset.");
return new LongWebCallTimeSet();
}
@ -193,7 +193,7 @@ namespace DistTestCore
{
if (IsRunningInCluster())
{
Log(" > Detected we're running in the cluster. Using long kubernetes timeset.");
fixtureLog.Log(" > Detected we're running in the cluster. Using long kubernetes timeset.");
return new LongK8sTimeSet();
}

14
Tests/ExperimentalTests/AutoBootstrapDistTest.cs
View File

@ -1,20 +1,20 @@
using CodexClient;
using CodexPlugin;
using LogosStorageClient;
using StoragePlugin;
using NUnit.Framework;
namespace CodexTests
namespace LogosStorageTests
{
public class AutoBootstrapDistTest : CodexDistTest
public class AutoBootstrapDistTest : LogosStorageDistTest
{
private bool isBooting = false;
public ICodexNode BootstrapNode { get; private set; } = null!;
public IStorageNode BootstrapNode { get; private set; } = null!;
[SetUp]
public void SetupBootstrapNode()
{
isBooting = true;
BootstrapNode = StartCodex(s => s.WithName("BOOTSTRAP_" + GetTestNamespace()));
BootstrapNode = StartLogosStorage(s => s.WithName("BOOTSTRAP_" + GetTestNamespace()));
isBooting = false;
}
@ -24,7 +24,7 @@ namespace CodexTests
BootstrapNode.Stop(waitTillStopped: false);
}
protected override void OnCodexSetup(ICodexSetup setup)
protected override void OnLogosStorageSetup(ILogosStorageSetup setup)
{
if (isBooting) return;

6
Tests/ExperimentalTests/BasicTests/AsyncProfiling.cs
View File

@ -1,17 +1,17 @@
using NUnit.Framework;
using MetricsPlugin;
using Utils;
using CodexTests;
using LogosStorageTests;
namespace ExperimentalTests.BasicTests
{
[TestFixture]
public class AsyncProfiling : CodexDistTest
public class AsyncProfiling : LogosStorageDistTest
{
[Test]
public void AsyncProfileMetricsPlz()
{
var node = StartCodex(s => s.EnableMetrics());
var node = StartLogosStorage(s => s.EnableMetrics());
var metrics = Ci.GetMetricsFor(scrapeInterval: TimeSpan.FromSeconds(3.0), node).Single();
var file = GenerateTestFile(100.MB());

16
Tests/ExperimentalTests/BasicTests/ExampleTests.cs
View File

@ -1,6 +1,6 @@
using CodexClient;
using CodexPlugin;
using CodexTests;
using LogosStorageClient;
using StoragePlugin;
using LogosStorageTests;
using DistTestCore;
using MetricsPlugin;
using NUnit.Framework;
@ -9,12 +9,12 @@ using Utils;
namespace ExperimentalTests.BasicTests
{
[TestFixture]
public class ExampleTests : CodexDistTest
public class ExampleTests : LogosStorageDistTest
{
[Test]
public void CodexLogExample()
public void LogosStorageLogExample()
{
var primary = StartCodex(s => s.WithLogLevel(CodexLogLevel.Trace, new CodexLogCustomTopics(CodexLogLevel.Warn, CodexLogLevel.Warn)));
var primary = StartLogosStorage(s => s.WithLogLevel(LogosStorageLogLevel.Trace, new LogosStorageLogCustomTopics(LogosStorageLogLevel.Warn, LogosStorageLogLevel.Warn)));
var cid = primary.UploadFile(GenerateTestFile(5.MB()));
@ -29,8 +29,8 @@ namespace ExperimentalTests.BasicTests
[Test]
public void TwoMetricsExample()
{
var group = StartCodex(2, s => s.EnableMetrics());
var group2 = StartCodex(2, s => s.EnableMetrics());
var group = StartLogosStorage(2, s => s.EnableMetrics());
var group2 = StartLogosStorage(2, s => s.EnableMetrics());
var primary = group[0];
var secondary = group[1];

20
Tests/ExperimentalTests/BasicTests/PyramidTests.cs
View File

@ -1,12 +1,12 @@
using CodexClient;
using CodexTests;
using LogosStorageClient;
using LogosStorageTests;
using NUnit.Framework;
using Utils;
namespace ExperimentalTests.BasicTests
{
[TestFixture]
public class PyramidTests : CodexDistTest
public class PyramidTests : LogosStorageDistTest
{
[Test]
[CreateTranscript(nameof(PyramidTest))]
@ -22,17 +22,17 @@ namespace ExperimentalTests.BasicTests
DownloadAllFilesFromEachNodeInLayer(bottomLayer, cids);
}
private List<ICodexNode> StartLayers(int numberOfLayers)
private List<IStorageNode> StartLayers(int numberOfLayers)
{
var layer = new List<ICodexNode>();
layer.Add(StartCodex(s => s.WithName("Top")));
var layer = new List<IStorageNode>();
layer.Add(StartLogosStorage(s => s.WithName("Top")));
for (var i = 0; i < numberOfLayers; i++)
{
var newLayer = new List<ICodexNode>();
var newLayer = new List<IStorageNode>();
foreach (var node in layer)
{
newLayer.AddRange(StartCodex(2, s => s.WithBootstrapNode(node).WithName("Layer[" + i + "]")));
newLayer.AddRange(StartLogosStorage(2, s => s.WithBootstrapNode(node).WithName("Layer[" + i + "]")));
}
layer.Clear();
@ -42,7 +42,7 @@ namespace ExperimentalTests.BasicTests
return layer;
}
private ContentId[] UploadFiles(List<ICodexNode> layer, ByteSize size)
private ContentId[] UploadFiles(List<IStorageNode> layer, ByteSize size)
{
var uploadTasks = new List<Task<ContentId>>();
foreach (var node in layer)
@ -63,7 +63,7 @@ namespace ExperimentalTests.BasicTests
return cids;
}
private void DownloadAllFilesFromEachNodeInLayer(List<ICodexNode> layer, ContentId[] cids)
private void DownloadAllFilesFromEachNodeInLayer(List<IStorageNode> layer, ContentId[] cids)
{
var downloadTasks = new List<Task>();
foreach (var node in layer)

16
Tests/ExperimentalTests/DownloadConnectivityTests/DatalayerReliabilityTests.cs
View File

@ -1,5 +1,5 @@
using CodexClient;
using CodexTests;
using LogosStorageClient;
using LogosStorageTests;
using NUnit.Framework;
using Utils;
@ -19,8 +19,8 @@ namespace ExperimentalTests.DownloadConnectivityTests
)
{
var file = GenerateTestFile(fileSizeMb.MB());
var uploader = StartCodex(n => n.WithName("uploader"));
var downloaders = StartCodex(numDownloaders, n => n.WithName("downloader"));
var uploader = StartLogosStorage(n => n.WithName("uploader"));
var downloaders = StartLogosStorage(numDownloaders, n => n.WithName("downloader"));
var cid = uploader.UploadFile(file);
@ -40,14 +40,14 @@ namespace ExperimentalTests.DownloadConnectivityTests
public class TransferPlan
{
public ICodexNode Uploader { get; set; } = null!;
public IStorageNode Uploader { get; set; } = null!;
public ContentId Cid { get; set; } = null!;
public List<DownloaderPlan> Downloaders { get; } = new List<DownloaderPlan>();
}
public class DownloaderPlan
{
public ICodexNode Node { get; set; } = null!;
public IStorageNode Node { get; set; } = null!;
public List<TransferPlan> TransferPlans { get; } = new List<TransferPlan>();
}
@ -102,7 +102,7 @@ namespace ExperimentalTests.DownloadConnectivityTests
)
{
var plans = new List<TransferPlan>();
var uploaders = StartCodex(numDatasets, n => n.WithName("uploader"));
var uploaders = StartLogosStorage(numDatasets, n => n.WithName("uploader"));
foreach (var n in uploaders)
{
plans.Add(new TransferPlan
@ -122,7 +122,7 @@ namespace ExperimentalTests.DownloadConnectivityTests
var allDownloaderPlans = available.GetAll();
Assert.That(allDownloaderPlans.Length, Is.LessThan(100));
Log($"Using {allDownloaderPlans.Length} downloaders...");
var nodes = StartCodex(allDownloaderPlans.Length, n => n.WithName("downloader"));
var nodes = StartLogosStorage(allDownloaderPlans.Length, n => n.WithName("downloader"));
for (var i = 0; i < allDownloaderPlans.Length; i++)
{
allDownloaderPlans[i].Node = nodes[i];

4
Tests/ExperimentalTests/DownloadConnectivityTests/DetectBlockRetransmitTest.cs
View File

@ -1,4 +1,4 @@
using CodexTests;
using LogosStorageTests;
using NUnit.Framework;
using Utils;
@ -15,7 +15,7 @@ namespace ExperimentalTests.DownloadConnectivityTests
[Values(3, 5, 10, 20)] int numNodes
)
{
var nodes = StartCodex(numNodes);
var nodes = StartLogosStorage(numNodes);
var file = GenerateTestFile(fileSize.MB());
var cid = nodes[0].UploadFile(file);

10
Tests/ExperimentalTests/DownloadConnectivityTests/FullyConnectedDownloadTests.cs
View File

@ -1,5 +1,5 @@
using CodexClient;
using CodexTests;
using LogosStorageClient;
using LogosStorageTests;
using NUnit.Framework;
using Utils;
@ -11,7 +11,7 @@ namespace ExperimentalTests.DownloadConnectivityTests
[Test]
public void MetricsDoesNotInterfereWithPeerDownload()
{
var nodes = StartCodex(2, s => s.EnableMetrics());
var nodes = StartLogosStorage(2, s => s.EnableMetrics());
AssertAllNodesConnected(nodes);
}
@ -22,12 +22,12 @@ namespace ExperimentalTests.DownloadConnectivityTests
[Values(2, 5)] int numberOfNodes,
[Values(1, 10)] int sizeMBs)
{
var nodes = StartCodex(numberOfNodes);
var nodes = StartLogosStorage(numberOfNodes);
AssertAllNodesConnected(nodes, sizeMBs);
}
private void AssertAllNodesConnected(IEnumerable<ICodexNode> nodes, int sizeMBs = 10)
private void AssertAllNodesConnected(IEnumerable<IStorageNode> nodes, int sizeMBs = 10)
{
CreatePeerDownloadTestHelpers().AssertFullDownloadInterconnectivity(nodes, sizeMBs.MB());
}

8
Tests/ExperimentalTests/DownloadConnectivityTests/MultiswarmTests.cs
View File

@ -1,5 +1,5 @@
using CodexClient;
using CodexTests;
using LogosStorageClient;
using LogosStorageTests;
using FileUtils;
using Logging;
using NUnit.Framework;
@ -30,7 +30,7 @@ namespace ExperimentalTests.DownloadConnectivityTests
private void RunThePlan(Plan plan, int fileSizeMb)
{
foreach (var filePlan in plan.FilePlans) filePlan.File = GenerateTestFile(fileSizeMb.MB());
var nodes = StartCodex(plan.NodePlans.Count);
var nodes = StartLogosStorage(plan.NodePlans.Count);
for (int i = 0; i < plan.NodePlans.Count; i++) plan.NodePlans[i].Node = nodes[i];
// Upload all files to their nodes.
@ -112,7 +112,7 @@ namespace ExperimentalTests.DownloadConnectivityTests
}
public int Number { get; }
public ICodexNode? Node { get; set; }
public IStorageNode? Node { get; set; }
public List<FilePlan> Uploads { get; } = new List<FilePlan>();
public List<FilePlan> Downloads { get; } = new List<FilePlan>();

4
Tests/ExperimentalTests/ExperimentalTests.csproj
View File

@ -13,8 +13,8 @@
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\ProjectPlugins\CodexDiscordBotPlugin\CodexDiscordBotPlugin.csproj" />
<ProjectReference Include="..\..\ProjectPlugins\CodexPlugin\CodexPlugin.csproj" />
<ProjectReference Include="..\..\ProjectPlugins\LogosStorageDiscordBotPlugin\LogosStorageDiscordBotPlugin.csproj" />
<ProjectReference Include="..\..\ProjectPlugins\StoragePlugin\StoragePlugin.csproj" />
<ProjectReference Include="..\..\ProjectPlugins\MetricsPlugin\MetricsPlugin.csproj" />
<ProjectReference Include="..\DistTestCore\DistTestCore.csproj" />
</ItemGroup>

16
Tests/ExperimentalTests/Helpers/FullConnectivityHelper.cs
View File

@ -1,9 +1,9 @@
using CodexClient;
using CodexPlugin;
using LogosStorageClient;
using StoragePlugin;
using Logging;
using NUnit.Framework;
namespace CodexTests.Helpers
namespace LogosStorageTests.Helpers
{
public interface IFullConnectivityImplementation
{
@ -24,12 +24,12 @@ namespace CodexTests.Helpers
this.implementation = implementation;
}
public void AssertFullyConnected(IEnumerable<ICodexNode> nodes)
public void AssertFullyConnected(IEnumerable<IStorageNode> nodes)
{
AssertFullyConnected(nodes.ToArray());
}
private void AssertFullyConnected(ICodexNode[] nodes)
private void AssertFullyConnected(IStorageNode[] nodes)
{
Log($"Asserting '{implementation.Description()}' for nodes: '{nodes.Names()}'...");
Assert.That(nodes.Length, Is.GreaterThan(1));
@ -70,7 +70,7 @@ namespace CodexTests.Helpers
Log($"Connections successful:{Nl}{string.Join(Nl, results)}");
}
private Entry[] CreateEntries(ICodexNode[] nodes)
private Entry[] CreateEntries(IStorageNode[] nodes)
{
var entries = nodes.Select(n => new Entry(n)).ToArray();
@ -110,13 +110,13 @@ namespace CodexTests.Helpers
public class Entry
{
public Entry(ICodexNode node)
public Entry(IStorageNode node)
{
Node = node;
Response = node.GetDebugInfo();
}
public ICodexNode Node { get; }
public IStorageNode Node { get; }
public DebugInfo Response { get; }
public override string ToString()

8
Tests/ExperimentalTests/Helpers/PeerConnectionTestHelpers.cs
View File

@ -1,8 +1,8 @@
using CodexClient;
using LogosStorageClient;
using Logging;
using static CodexTests.Helpers.FullConnectivityHelper;
using static LogosStorageTests.Helpers.FullConnectivityHelper;
namespace CodexTests.Helpers
namespace LogosStorageTests.Helpers
{
public class PeerConnectionTestHelpers : IFullConnectivityImplementation
{
@ -13,7 +13,7 @@ namespace CodexTests.Helpers
helper = new FullConnectivityHelper(log, this);
}
public void AssertFullyConnected(IEnumerable<ICodexNode> nodes)
public void AssertFullyConnected(IEnumerable<IStorageNode> nodes)
{
helper.AssertFullyConnected(nodes);
}

12
Tests/ExperimentalTests/Helpers/PeerDownloadTestHelpers.cs
View File

@ -1,10 +1,10 @@
using CodexClient;
using LogosStorageClient;
using FileUtils;
using Logging;
using Utils;
using static CodexTests.Helpers.FullConnectivityHelper;
using static LogosStorageTests.Helpers.FullConnectivityHelper;
namespace CodexTests.Helpers
namespace LogosStorageTests.Helpers
{
public class PeerDownloadTestHelpers : IFullConnectivityImplementation
{
@ -19,7 +19,7 @@ namespace CodexTests.Helpers
this.fileManager = fileManager;
}
public void AssertFullDownloadInterconnectivity(IEnumerable<ICodexNode> nodes, ByteSize testFileSize)
public void AssertFullDownloadInterconnectivity(IEnumerable<IStorageNode> nodes, ByteSize testFileSize)
{
this.testFileSize = testFileSize;
helper.AssertFullyConnected(nodes);
@ -60,12 +60,12 @@ namespace CodexTests.Helpers
// Should an exception occur during upload, then this try is inconclusive and we try again next loop.
}
private TrackedFile? DownloadFile(ICodexNode node, ContentId contentId, string label)
private TrackedFile? DownloadFile(IStorageNode node, ContentId contentId, string label)
{
return node.DownloadContent(contentId, label);
}
private TrackedFile GenerateTestFile(ICodexNode uploader, ICodexNode downloader)
private TrackedFile GenerateTestFile(IStorageNode uploader, IStorageNode downloader)
{
var up = uploader.GetName().Replace("<", "").Replace(">", "");
var down = downloader.GetName().Replace("<", "").Replace(">", "");

75
Tests/ExperimentalTests/CodexDistTest.cs → Tests/ExperimentalTests/LogosStorageDistTest.cs
View File

@ -1,8 +1,7 @@
using CodexClient;
using CodexNetDeployer;
using CodexPlugin;
using CodexPlugin.OverwatchSupport;
using CodexTests.Helpers;
using LogosStorageClient;
using StoragePlugin;
using StoragePlugin.OverwatchSupport;
using LogosStorageTests.Helpers;
using Core;
using DistTestCore;
using DistTestCore.Logs;
@ -13,61 +12,61 @@ using NUnit.Framework;
using OverwatchTranscript;
using Utils;
namespace CodexTests
namespace LogosStorageTests
{
public class CodexDistTest : DistTest
public class LogosStorageDistTest : DistTest
{
private readonly List<ICodexNode> nodes = new List<ICodexNode>();
private CodexTranscriptWriter? writer;
private readonly List<IStorageNode> nodes = new List<IStorageNode>();
private LogosStorageTranscriptWriter? writer;
public CodexDistTest()
public LogosStorageDistTest()
{
ProjectPlugin.Load<CodexPlugin.CodexPlugin>();
ProjectPlugin.Load<StoragePlugin.StoragePlugin>();
ProjectPlugin.Load<MetricsPlugin.MetricsPlugin>();
}
[SetUp]
public void SetupCodexDistTest()
public void SetupLogosStorageDistTest()
{
writer = SetupTranscript();
}
[TearDown]
public void TearDownCodexDistTest()
public void TearDownLogosStorageDistTest()
{
TeardownTranscript();
}
protected override void Initialize(FixtureLog fixtureLog)
{
var localBuilder = new LocalCodexBuilder(fixtureLog);
var localBuilder = new LocalNodeBuilder(fixtureLog);
localBuilder.Intialize();
localBuilder.Build();
Ci.AddCodexHooksProvider(new CodexLogTrackerProvider(nodes.Add));
Ci.AddLogosStorageHooksProvider(new LogosStorageLogTrackerProvider(nodes.Add));
}
public ICodexNode StartCodex()
public IStorageNode StartLogosStorage()
{
return StartCodex(s => { });
return StartLogosStorage(s => { });
}
public ICodexNode StartCodex(Action<ICodexSetup> setup)
public IStorageNode StartLogosStorage(Action<ILogosStorageSetup> setup)
{
return StartCodex(1, setup)[0];
return StartLogosStorage(1, setup)[0];
}
public ICodexNodeGroup StartCodex(int numberOfNodes)
public IStorageNodeGroup StartLogosStorage(int numberOfNodes)
{
return StartCodex(numberOfNodes, s => { });
return StartLogosStorage(numberOfNodes, s => { });
}
public ICodexNodeGroup StartCodex(int numberOfNodes, Action<ICodexSetup> setup)
public IStorageNodeGroup StartLogosStorage(int numberOfNodes, Action<ILogosStorageSetup> setup)
{
var group = Ci.StartCodexNodes(numberOfNodes, s =>
var group = Ci.StartStorageNodes(numberOfNodes, s =>
{
setup(s);
OnCodexSetup(s);
OnLogosStorageSetup(s);
});
return group;
@ -83,12 +82,12 @@ namespace CodexTests
return new PeerDownloadTestHelpers(GetTestLog(), GetFileManager());
}
public void CheckLogForErrors(params ICodexNode[] nodes)
public void CheckLogForErrors(params IStorageNode[] nodes)
{
foreach (var node in nodes) CheckLogForErrors(node);
}
public void CheckLogForErrors(ICodexNode node)
public void CheckLogForErrors(IStorageNode node)
{
Log($"Checking {node.GetName()} log for errors.");
var log = node.DownloadLog();
@ -97,23 +96,23 @@ namespace CodexTests
log.AssertLogDoesNotContainLinesStartingWith("ERR ");
}
public void LogNodeStatus(ICodexNode node, IMetricsAccess? metrics = null)
public void LogNodeStatus(IStorageNode node, IMetricsAccess? metrics = null)
{
Log("Status for " + node.GetName() + Environment.NewLine +
GetBasicNodeStatus(node));
}
public void WaitAndCheckNodesStaysAlive(TimeSpan duration, ICodexNodeGroup nodes)
public void WaitAndCheckNodesStaysAlive(TimeSpan duration, IStorageNodeGroup nodes)
{
WaitAndCheckNodesStaysAlive(duration, nodes.ToArray());
}
public void WaitAndCheckNodesStaysAlive(TimeSpan duration, List<ICodexNode> nodes)
public void WaitAndCheckNodesStaysAlive(TimeSpan duration, List<IStorageNode> nodes)
{
WaitAndCheckNodesStaysAlive(duration, nodes.ToArray());
}
public void WaitAndCheckNodesStaysAlive(TimeSpan duration, params ICodexNode[] nodes)
public void WaitAndCheckNodesStaysAlive(TimeSpan duration, params IStorageNode[] nodes)
{
Log($"{nameof(WaitAndCheckNodesStaysAlive)} {Time.FormatDuration(duration)}...");
@ -136,12 +135,12 @@ namespace CodexTests
Log($"{nameof(WaitAndCheckNodesStaysAlive)} OK");
}
public void AssertNodesContainFile(ContentId cid, ICodexNodeGroup nodes)
public void AssertNodesContainFile(ContentId cid, IStorageNodeGroup nodes)
{
AssertNodesContainFile(cid, nodes.ToArray());
}
public void AssertNodesContainFile(ContentId cid, params ICodexNode[] nodes)
public void AssertNodesContainFile(ContentId cid, params IStorageNode[] nodes)
{
Log($"{nameof(AssertNodesContainFile)} {nodes.Names()} {cid}...");
@ -154,13 +153,13 @@ namespace CodexTests
Log($"{nameof(AssertNodesContainFile)} OK");
}
private string GetBasicNodeStatus(ICodexNode node)
private string GetBasicNodeStatus(IStorageNode node)
{
return JsonConvert.SerializeObject(node.GetDebugInfo(), Formatting.Indented) + Environment.NewLine +
node.Space().ToString() + Environment.NewLine;
}
protected virtual void OnCodexSetup(ICodexSetup setup)
protected virtual void OnLogosStorageSetup(ILogosStorageSetup setup)
{
}
@ -171,19 +170,19 @@ namespace CodexTests
return null;
}
private CodexTranscriptWriter? SetupTranscript()
private LogosStorageTranscriptWriter? SetupTranscript()
{
var attr = GetTranscriptAttributeOfCurrentTest();
if (attr == null) return null;
var config = new CodexTranscriptWriterConfig(
var config = new LogosStorageTranscriptWriterConfig(
attr.OutputFilename,
attr.IncludeBlockReceivedEvents
);
var log = new LogPrefixer(GetTestLog(), "(Transcript) ");
var writer = new CodexTranscriptWriter(log, config, Transcript.NewWriter(log));
Ci.AddCodexHooksProvider(writer);
var writer = new LogosStorageTranscriptWriter(log, config, Transcript.NewWriter(log));
Ci.AddLogosStorageHooksProvider(writer);
return writer;
}

24
Tests/ExperimentalTests/CodexLogTrackerProvider.cs → Tests/ExperimentalTests/LogosStorageLogTrackerProvider.cs
View File

@ -1,29 +1,29 @@
using CodexClient;
using CodexClient.Hooks;
using LogosStorageClient;
using LogosStorageClient.Hooks;
using Utils;
namespace CodexTests
namespace LogosStorageTests
{
public class CodexLogTrackerProvider : ICodexHooksProvider
public class LogosStorageLogTrackerProvider : ILogosStorageHooksProvider
{
private readonly Action<ICodexNode> addNode;
private readonly Action<IStorageNode> addNode;
public CodexLogTrackerProvider(Action<ICodexNode> addNode)
public LogosStorageLogTrackerProvider(Action<IStorageNode> addNode)
{
this.addNode = addNode;
}
// See TestLifecycle.cs DownloadAllLogs()
public ICodexNodeHooks CreateHooks(string nodeName)
public IStorageNodeHooks CreateHooks(string nodeName)
{
return new CodexLogTracker(addNode);
return new LogosStorageLogTracker(addNode);
}
public class CodexLogTracker : ICodexNodeHooks
public class LogosStorageLogTracker : IStorageNodeHooks
{
private readonly Action<ICodexNode> addNode;
private readonly Action<IStorageNode> addNode;
public CodexLogTracker(Action<ICodexNode> addNode)
public LogosStorageLogTracker(Action<IStorageNode> addNode)
{
this.addNode = addNode;
}
@ -44,7 +44,7 @@ namespace CodexTests
{
}
public void OnNodeStarted(ICodexNode node, string peerId, string nodeId)
public void OnNodeStarted(IStorageNode node, string peerId, string nodeId)
{
addNode(node);
}

2
Tests/ExperimentalTests/MetricsAccessExtensions.cs
View File

@ -3,7 +3,7 @@ using Logging;
using MetricsPlugin;
using NUnit.Framework.Constraints;
namespace CodexTests
namespace LogosStorageTests
{
public static class MetricsAccessExtensions
{

2
Tests/ExperimentalTests/Parallelism.cs
View File

@ -1,6 +1,6 @@
using NUnit.Framework;
[assembly: LevelOfParallelism(1)]
namespace CodexTests
namespace LogosStorageTests
{
}

32
Tests/ExperimentalTests/PeerDiscoveryTests/LayeredDiscoveryTests.cs
View File

@ -1,19 +1,19 @@
using CodexClient;
using CodexTests;
using LogosStorageClient;
using LogosStorageTests;
using NUnit.Framework;
namespace ExperimentalTests.PeerDiscoveryTests
{
[TestFixture]
public class LayeredDiscoveryTests : CodexDistTest
public class LayeredDiscoveryTests : LogosStorageDistTest
{
[Test]
public void TwoLayersTest()
{
var root = StartCodex();
var l1Source = StartCodex(s => s.WithBootstrapNode(root));
var l1Node = StartCodex(s => s.WithBootstrapNode(root));
var l2Target = StartCodex(s => s.WithBootstrapNode(l1Node));
var root = StartLogosStorage();
var l1Source = StartLogosStorage(s => s.WithBootstrapNode(root));
var l1Node = StartLogosStorage(s => s.WithBootstrapNode(root));
var l2Target = StartLogosStorage(s => s.WithBootstrapNode(l1Node));
AssertAllNodesConnected(root, l1Source, l1Node, l2Target);
}
@ -21,11 +21,11 @@ namespace ExperimentalTests.PeerDiscoveryTests
[Test]
public void ThreeLayersTest()
{
var root = StartCodex();
var l1Source = StartCodex(s => s.WithBootstrapNode(root));
var l1Node = StartCodex(s => s.WithBootstrapNode(root));
var l2Node = StartCodex(s => s.WithBootstrapNode(l1Node));
var l3Target = StartCodex(s => s.WithBootstrapNode(l2Node));
var root = StartLogosStorage();
var l1Source = StartLogosStorage(s => s.WithBootstrapNode(root));
var l1Node = StartLogosStorage(s => s.WithBootstrapNode(root));
var l2Node = StartLogosStorage(s => s.WithBootstrapNode(l1Node));
var l3Target = StartLogosStorage(s => s.WithBootstrapNode(l2Node));
AssertAllNodesConnected(root, l1Source, l1Node, l2Node, l3Target);
}
@ -35,20 +35,20 @@ namespace ExperimentalTests.PeerDiscoveryTests
[TestCase(10)]
public void NodeChainTest(int chainLength)
{
var nodes = new List<ICodexNode>();
var node = StartCodex();
var nodes = new List<IStorageNode>();
var node = StartLogosStorage();
nodes.Add(node);
for (var i = 1; i < chainLength; i++)
{
node = StartCodex(s => s.WithBootstrapNode(node));
node = StartLogosStorage(s => s.WithBootstrapNode(node));
nodes.Add(node);
}
AssertAllNodesConnected(nodes.ToArray());
}
private void AssertAllNodesConnected(params ICodexNode[] nodes)
private void AssertAllNodesConnected(params IStorageNode[] nodes)
{
CreatePeerConnectionTestHelpers().AssertFullyConnected(nodes);
}

14
Tests/ExperimentalTests/PeerDiscoveryTests/PeerDiscoveryTests.cs
View File

@ -1,7 +1,7 @@
using CodexClient;
using LogosStorageClient;
using NUnit.Framework;
using Utils;
using CodexTests;
using LogosStorageTests;
namespace ExperimentalTests.PeerDiscoveryTests
{
@ -12,7 +12,7 @@ namespace ExperimentalTests.PeerDiscoveryTests
public void CanReportUnknownPeerId()
{
var unknownId = "16Uiu2HAkv2CHWpff3dj5iuVNERAp8AGKGNgpGjPexJZHSqUstfsK";
var node = StartCodex();
var node = StartLogosStorage();
var result = node.GetDebugPeer(unknownId);
Assert.That(result.IsPeerFound, Is.False);
@ -21,7 +21,7 @@ namespace ExperimentalTests.PeerDiscoveryTests
[Test]
public void MetricsDoesNotInterfereWithPeerDiscovery()
{
var nodes = StartCodex(2, s => s.EnableMetrics());
var nodes = StartLogosStorage(2, s => s.EnableMetrics());
AssertAllNodesConnected(nodes);
}
@ -31,12 +31,12 @@ namespace ExperimentalTests.PeerDiscoveryTests
[TestCase(10)]
public void VariableNodes(int number)
{
var nodes = StartCodex(number);
var nodes = StartLogosStorage(number);
AssertAllNodesConnected(nodes);
}
private void AssertAllNodesConnected(IEnumerable<ICodexNode> nodes)
private void AssertAllNodesConnected(IEnumerable<IStorageNode> nodes)
{
nodes = nodes.Concat(new[] { BootstrapNode }).ToArray()!;
@ -44,7 +44,7 @@ namespace ExperimentalTests.PeerDiscoveryTests
CheckRoutingTable(nodes);
}
private void CheckRoutingTable(IEnumerable<ICodexNode> allNodes)
private void CheckRoutingTable(IEnumerable<IStorageNode> allNodes)
{
var allResponses = allNodes.Select(n => n.GetDebugInfo()).ToArray();

14
Tests/ExperimentalTests/README.md
View File

@ -1,11 +1,11 @@
# Codex Tests
# Logos Storage Tests
This is an NUnit test assembly that can be used with the standard dotnet test runner. For all its CLI options, run `dotnet test --help`.
## Example tests
Running all the tests in the assembly can take a while. In order to check basic viability of your setup as well as the Codex image you're using, consider running only the example tests using the filter option: `dotnet test --filter=Example`.
Running all the tests in the assembly can take a while. In order to check basic viability of your setup as well as the Logos Storage image you're using, consider running only the example tests using the filter option: `dotnet test --filter=Example`.
## Output
The test runner will produce a folder named `CodexTestLogs` with all the test logs. They are sorted by timestamp and reflect the names of the test fixtures and individual tests. When a test fails, the log file for that specific test will be postfixed with `_FAILED`. The same applies to the fixture log file. The `STATUS` files contain the test results in JSON, for easy machine reading.
The test runner will produce a folder named `LogosStorageTestLogs` with all the test logs. They are sorted by timestamp and reflect the names of the test fixtures and individual tests. When a test fails, the log file for that specific test will be postfixed with `_FAILED`. The same applies to the fixture log file. The `STATUS` files contain the test results in JSON, for easy machine reading.
## Overrides
The following environment variables allow you to override specific aspects of the behaviour of the tests.
@ -14,14 +14,14 @@ The following environment variables allow you to override specific aspects of th
|------------------|----------------------------------------------------------------------------------------------------------------|
| DEPLOYID | A pod-label 'deployid' is added to each pod created during the tests. Use this to set the value of that label. |
| TESTID | Similar to RUNID, except the label is 'testid'. |
| CODEXDOCKERIMAGE | If set, this will be used instead of the default Codex docker image. |
| STORAGEDOCKERIMAGE | If set, this will be used instead of the default Logos Storage docker image. |
## Using a local Codex repository
If you have a clone of the Codex git repository, and you want to run the tests using your local modifications, the following environment variable options are for you. Please note that any changes made in Codex's 'vendor' directory will be discarded during the build process.
## Using a local Logos Storage repository
If you have a clone of the Logos Storage git repository, and you want to run the tests using your local modifications, the following environment variable options are for you. Please note that any changes made in Logos Storage's 'vendor' directory will be discarded during the build process.
| Variable | Description |
|----------------|--------------------------------------------------------------------------------------------------------------------------|
| CODEXREPOPATH | Path to the Codex repository. |
| STORAGEREPOPATH | Path to the Logos Storage repository. |
| DOCKERUSERNAME | Username of your Dockerhub account. |
| DOCKERPASSWORD | Password OR access-token for your Dockerhub account. You can omit this variable to use your system-default account. |
| DOCKERTAG | Optional. Tag used for docker image that will be built and pushed to the Dockerhub account. Random ID used when not set. |

12
Tests/ExperimentalTests/UtilityTests/LogHelperTests.cs
View File

@ -1,5 +1,5 @@
using CodexClient;
using CodexTests;
using LogosStorageClient;
using LogosStorageTests;
using NUnit.Framework;
using Utils;
@ -12,8 +12,8 @@ namespace ExperimentalTests.UtilityTests
[Ignore("Used to find the most common log messages.")]
public void FindMostCommonLogMessages()
{
var uploader = StartCodex(s => s.WithName("uploader").WithLogLevel(CodexLogLevel.Trace));
var downloader = StartCodex(s => s.WithName("downloader").WithLogLevel(CodexLogLevel.Trace));
var uploader = StartLogosStorage(s => s.WithName("uploader").WithLogLevel(LogosStorageLogLevel.Trace));
var downloader = StartLogosStorage(s => s.WithName("downloader").WithLogLevel(LogosStorageLogLevel.Trace));
var cid = uploader.UploadFile(GenerateTestFile(100.MB()));
@ -34,13 +34,13 @@ namespace ExperimentalTests.UtilityTests
}
}
private Dictionary<string, int> GetLogMap(ICodexNode node, DateTime? startUtc = null)
private Dictionary<string, int> GetLogMap(IStorageNode node, DateTime? startUtc = null)
{
var log = node.DownloadLog();
var map = new Dictionary<string, int>();
log.IterateLines(line =>
{
var log = CodexLogLine.Parse(line);
var log = LogosStorageLogLine.Parse(line);
if (log == null) return;
if (startUtc.HasValue)

12
Tests/ExperimentalTests/UtilityTests/NetworkIsolationTest.cs
View File

@ -1,5 +1,5 @@
using CodexClient;
using CodexPlugin;
using LogosStorageClient;
using StoragePlugin;
using DistTestCore;
using NUnit.Framework;
using Utils;
@ -13,12 +13,12 @@ namespace ExperimentalTests.UtilityTests
[Ignore("Disabled until a solution is implemented.")]
public class NetworkIsolationTest : DistTest
{
private ICodexNode? node = null;
private IStorageNode? node = null;
[Test]
public void SetUpANodeAndWait()
{
node = Ci.StartCodexNode();
node = Ci.StartStorageNode();
Time.WaitUntil(() => node == null, TimeSpan.FromMinutes(5), TimeSpan.FromSeconds(5), nameof(SetUpANodeAndWait));
}
@ -26,7 +26,7 @@ namespace ExperimentalTests.UtilityTests
[Test]
public void ForeignNodeConnects()
{
var myNode = Ci.StartCodexNode();
var myNode = Ci.StartStorageNode();
Time.WaitUntil(() => node != null, TimeSpan.FromMinutes(1), TimeSpan.FromSeconds(5), nameof(ForeignNodeConnects));
@ -41,7 +41,7 @@ namespace ExperimentalTests.UtilityTests
return;
}
Assert.Fail("Connection could be established between two Codex nodes running in different namespaces. " +
Assert.Fail("Connection could be established between two Logos Storage nodes running in different namespaces. " +
"This may cause cross-test interference. Network isolation policy should be applied. Test infra failure.");
}
}

2
Tests/FrameworkTests/FrameworkTests.csproj
View File

@ -16,7 +16,7 @@
<ItemGroup>
<ProjectReference Include="..\..\Framework\OverwatchTranscript\OverwatchTranscript.csproj" />
<ProjectReference Include="..\..\Framework\Utils\Utils.csproj" />
<ProjectReference Include="..\..\ProjectPlugins\CodexClient\CodexClient.csproj" />
<ProjectReference Include="..\..\ProjectPlugins\LogosStorageClient\LogosStorageClient.csproj" />
</ItemGroup>
</Project>

4
Tests/FrameworkTests/CodexClient/ContentIdTests.cs → Tests/FrameworkTests/LogosStorageClient/ContentIdTests.cs
View File

@ -1,4 +1,4 @@
using CodexClient;
using LogosStorageClient;
using NUnit.Framework;
using System;
using System.Collections.Generic;
@ -6,7 +6,7 @@ using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace FrameworkTests.CodexClient
namespace FrameworkTests.LogosStorageClient
{
[TestFixture]
public class ContentIdTests

16
Tests/CodexContinuousTests/Configuration.cs → Tests/LogosStorageContinuousTests/Configuration.cs
View File

@ -1,5 +1,5 @@
using ArgsUniform;
using CodexPlugin;
using StoragePlugin;
using Newtonsoft.Json;
namespace ContinuousTests
@ -12,8 +12,8 @@ namespace ContinuousTests
[Uniform("data-path", "d", "DATAPATH", true, "Path where temporary data files will be written.")]
public string DataPath { get; set; } = "data";
[Uniform("codex-deployment", "c", "CODEXDEPLOYMENT", true, "Path to codex-deployment JSON file.")]
public string CodexDeploymentJson { get; set; } = string.Empty;
[Uniform("storage-deployment", "c", "CODEXDEPLOYMENT", true, "Path to storage-deployment JSON file.")]
public string LogosStorageDeploymentJson { get; set; } = string.Empty;
[Uniform("keep", "k", "KEEP", false, "Set to 1 or 'true' to retain logs of successful tests.")]
public bool KeepPassedTestLogs { get; set; } = false;
@ -37,7 +37,7 @@ namespace ContinuousTests
" the timestamp of the start of the network deployment. Otherwise, logs will start from the test start timestamp.")]
public bool FullContainerLogs { get; set; } = false;
public CodexDeployment CodexDeployment { get; set; } = null!;
public LogosStorageDeployment LogosStorageDeployment { get; set; } = null!;
}
public class ConfigLoader
@ -47,13 +47,13 @@ namespace ContinuousTests
var uniformArgs = new ArgsUniform<Configuration>(PrintHelp, args);
var result = uniformArgs.Parse(true);
result.CodexDeployment = ParseCodexDeploymentJson(result.CodexDeploymentJson);
result.LogosStorageDeployment = ParseLogosStorageDeploymentJson(result.LogosStorageDeploymentJson);
return result;
}
private CodexDeployment ParseCodexDeploymentJson(string filename)
private LogosStorageDeployment ParseLogosStorageDeploymentJson(string filename)
{
var d = JsonConvert.DeserializeObject<CodexDeployment>(File.ReadAllText(filename))!;
var d = JsonConvert.DeserializeObject<LogosStorageDeployment>(File.ReadAllText(filename))!;
if (d == null) throw new Exception("Unable to parse " + filename);
return d;
}
@ -61,7 +61,7 @@ namespace ContinuousTests
private static void PrintHelp()
{
var nl = Environment.NewLine;
Console.WriteLine("ContinuousTests will run a set of tests against a codex deployment given a codex-deployment.json file." + nl +
Console.WriteLine("ContinuousTests will run a set of tests against a logos-storage deployment given a storage-deployment.json file." + nl +
"The tests will run in an endless loop unless otherwise specified." + nl);
}
}

12
Tests/CodexContinuousTests/ContinuousTest.cs → Tests/LogosStorageContinuousTests/ContinuousTest.cs
View File

@ -1,4 +1,4 @@
using CodexClient;
using LogosStorageClient;
using FileUtils;
using Logging;
using MetricsPlugin;
@ -20,7 +20,7 @@ namespace ContinuousTests
protected const int DayOne = HourOne * 24;
protected const int DayThree = DayOne * 3;
public void Initialize(ICodexNode[] nodes, ILog log, IFileManager fileManager, Configuration configuration, CancellationToken cancelToken)
public void Initialize(IStorageNode[] nodes, ILog log, IFileManager fileManager, Configuration configuration, CancellationToken cancelToken)
{
Nodes = nodes;
Log = log;
@ -38,7 +38,7 @@ namespace ContinuousTests
}
}
public ICodexNode[] Nodes { get; private set; } = null!;
public IStorageNode[] Nodes { get; private set; } = null!;
public ILog Log { get; private set; } = null!;
public IFileManager FileManager { get; private set; } = null!;
public Configuration Configuration { get; private set; } = null!;
@ -52,11 +52,11 @@ namespace ContinuousTests
public IMetricsAccess CreateMetricsAccess(Address target)
{
if (Configuration.CodexDeployment.PrometheusContainer == null) throw new Exception("Expected prometheus to be part of Codex deployment.");
if (Configuration.LogosStorageDeployment.PrometheusContainer == null) throw new Exception("Expected prometheus to be part of Logos Storage deployment.");
var entryPointFactory = new EntryPointFactory();
var entryPoint = entryPointFactory.CreateEntryPoint(Configuration.KubeConfigFile, Configuration.DataPath, Configuration.CodexDeployment.Metadata.KubeNamespace, Log);
return entryPoint.CreateInterface().WrapMetricsCollector(Configuration.CodexDeployment.PrometheusContainer, target);
var entryPoint = entryPointFactory.CreateEntryPoint(Configuration.KubeConfigFile, Configuration.DataPath, Configuration.LogosStorageDeployment.Metadata.KubeNamespace, Log);
return entryPoint.CreateInterface().WrapMetricsCollector(Configuration.LogosStorageDeployment.PrometheusContainer, target);
}
public abstract int RequiredNumberOfNodes { get; }

Some files were not shown because too many files have changed in this diff Show More