logos-storage-research/analysis/failure_model.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Codex",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LzbZW-vg0z9S"
      },
      "outputs": [],
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Model Codex Platform Structure\n",
        "\n",
        "This work attemps to simulate as many details as possible of the Codex platform. The details about the aspects we want to model can be found in the following notes: https://hackmd.io/yTRlMInIRi6fG-nk984ULQ\n",
        "\n",
        "First, we want to build a model of what we expect the Codex platform to look like, in terms of number of nodes, storage used and available, file size distribution, among others."
      ],
      "metadata": {
        "id": "BT9ssm0WZkZL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import random as random\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.rcParams['figure.figsize'] = [15, 8]\n",
        "\n",
        "\n",
        "#@title Platform parameters\n",
        "\n",
        "#@markdown Number of peers in the network\n",
        "NetworkSize = 5000 #@param {type:\"slider\", min:1, max:10000, step:1}\n",
        "#@markdown Total storage provided in the system (Petabytes)\n",
        "StorageProvided = 5 #@param {type:\"slider\", min:1, max:100, step:1}\n",
        "#@markdown Total storage in use (Petabytes)\n",
        "StorageUsed = 2 #@param {type:\"slider\", min:1, max:100, step:1}\n",
        "#@markdown Variance in storage offered by nodes (%)\n",
        "StorageVariance = 40 #@param {type:\"slider\", min:1, max:50, step:1}\n",
        "#@markdown Mean file size (Kilobytes)\n",
        "MeanFileSize = 4096 #@param {type:\"slider\", min:1, max:1048576, step:128}\n",
        "#@markdown Block size for erasure coding (Kilobytes)\n",
        "BlockSize = 4 #@param {type:\"slider\", min:1, max:32, step:1}\n",
        "\n",
        "totalUsed = 0\n",
        "totalFree = 0\n",
        "nodesProvided = [0]*NetworkSize\n",
        "nodesUsed = [0]*NetworkSize\n",
        "nodesFree = [0]*NetworkSize\n",
        "for i in range(NetworkSize):\n",
        "  sign = random.randint(-1,1)\n",
        "  variance = random.random() * StorageVariance / 100\n",
        "  nodesProvided[i] = (StorageProvided*1024/NetworkSize)+(sign*variance*(StorageProvided*1024/NetworkSize))\n",
        "  nodesUsed[i] = (StorageUsed*1024/NetworkSize)+(sign*variance*(StorageUsed*1024/NetworkSize))\n",
        "  nodesFree[i] = nodesProvided[i] - nodesUsed[i]\n",
        "  totalUsed += nodesUsed[i]\n",
        "  totalFree += nodesFree[i]\n",
        "\n",
        "def plotDist(nodesUsed, nodesFree):\n",
        "  x = np.arange(len(nodesFree))\n",
        "  plt.rcParams['figure.figsize'] = [15, 8]\n",
        "  plt.subplot(211)\n",
        "  plt.xlabel(\"Nodes\")\n",
        "  plt.ylabel(\"Storage Provided (Terabytes)\")\n",
        "  plt.bar(x-1, nodesUsed, label=\"Used\")\n",
        "  plt.bar(x-1, nodesFree, label=\"Free\", bottom=nodesUsed)\n",
        "  plt.legend(loc=\"upper left\")\n",
        "\n",
        " \n",
        "print(\"Total storage used: %f Terabytes\" % totalUsed)\n",
        "print(\"Total storage free: %f Terabytes\" % totalFree)\n",
        "print(\"Total number of blocks in the system: %i\" % (totalUsed*1024*1024/BlockSize)) \n",
        "print(\"Total number of files in the system: %i\" % (totalUsed*1024*1024/MeanFileSize)) \n",
        "plotDist(nodesUsed, nodesFree)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 329
        },
        "id": "1mfYz2oeaWZS",
        "outputId": "5617edd4-99a7-4822-c916-2292d965793d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total storage used: 2054.255442 Terabytes\n",
            "Total storage free: 3081.383163 Terabytes\n",
            "Total number of blocks in the system: 538510738\n",
            "Total number of files in the system: 525889\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1080x576 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3gAAADzCAYAAADU8bTIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de7gkdXng8e/LMDooN4Uhiww4sxEJExwBhxFXV0EU0RjQDYZBvASJoAHFaLLCRmFEXXDVJELQQCIRbyABEydIFBbZYLwyI5PhtshERhgkO8hlAAUBefePrgOHwznddc7p6q6u+n6ep59zqrq66v1dq39dt8hMJEmSJEmjb7NhByBJkiRJ6g8HeJIkSZLUEA7wJEmSJKkhHOBJkiRJUkM4wJMkSZKkhnCAJ0mSJEkNsfmwA5iu7bffPhcuXDjsMCRJkiRpKFavXv3zzJw/2XsjN8BbuHAhq1atGnYYkiRJkjQUEfHTqd7zFE1JkiRJaggHeJIkSZLUEA7wJEmSJKkhRu4avMk8/PDDbNiwgQcffHDYoQzEvHnzWLBgAXPnzh12KJIkSZJqpBEDvA0bNrDVVluxcOFCImLY4VQqM7nzzjvZsGEDixYtGnY4kiRJkmqkEadoPvjgg2y33XaNH9wBRATbbbdda45WSpIkSSqvEQM8oBWDuzFtSqskSZKk8hozwBu29evXs8ceezxh3ooVK/jEJz4xq/Xut99+PvdPkiRJUimVXYMXEecArwU2ZuYeXZbbB/gesDwzL+zHthee8PV+rOYx60/7nb6ur1VWbAMrNg07CpVleT3RqOfHqMc/W21Pv6Rq2Leo5qo8gvc54KBuC0TEHOBjwKUVxjF0p59+OosXL2bJkiUsX74cgF/84he87W1vY9myZey111587WtfA+CBBx5g+fLl7L777rz+9a/ngQceGGbokiR1vtBKVbKOSX1T2RG8zLwyIhb2WOxdwEXAPlXFUQennXYaN998M0996lO55557APjoRz/Ky1/+cs455xzuueceli1bxite8QrOOussnva0p3HDDTewdu1a9t577yFHPwD+EiZJUv+4X5VabWjX4EXETsDrgc8MK4Z+murGJxHBkiVLOOKII/jiF7/I5pt3xtSXXnopp512GnvuuSf77bcfDz74ILfccgtXXnklb3rTmwBYsmQJS5YsGVgaNED+UqkqWb+mzzzTdFlnVAXrlfpgmDdZ+Uvg/Zn5aK8FI+LoiFgVEavuuOOOAYQ2fdtttx133313Z+JnVwNw1113sf322/P1r3+dY489lh/96Efss88+PPLII2QmF110EWvWrGHNmjXccsst7L777kNMgSphR90MlqOGwXonSZqBYQ7wlgLnR8R64FDg0xHxuskWzMyzM3NpZi6dP3/+IGMsbcstt2THHXfkW9/6FtAZ3H3jG9/gJS95Cbfeeiv7778/H/vYx9i0aRP3338/r3rVqzjjjDPITACuvrozKHzpS1/Kl7/8ZQCuvfZa1q5dO5wESdJsOUBRP1mfJKmUyq7B6yUzF439HxGfAy7OzH8cVjz98PnPf55jjz2W9278GczdgpNPPplddtmF/fffn02bNpGZvPvd72bbbbflgx/8IO95z3tYsmQJjz76KIsWLeLiiy/mne98J0ceeSS77747u+++Oy94wQuGnSyNGq+9mD7zTJIkNUSVj0k4D9gP2D4iNgAnA3MBMvOvq9ouDO+xBosXL+aKK67onKL5rL0em/+v//qvT1p2iy224Kyzzpp0/vnnn19pnBqAQQ8Yem3PAUw9WA6SJKliVd5F8/BpLPsHVcUhtd4onNY0rIFP3QdcdY9PkiTVzjCvwZOmNgqDEkkaFfapktQaDvDUPH6REQy/Hgx7+1I/WI/VL9al/jI/1YUDPA2HHZOkmbDvkJrBtixVxgGeJEmSpHrxR4AZc4AnzYadT7tZ/pLaZib9XpP7ymGmrcn5qllxgNcnc+bMYc899+y8Xrmc9evXDzukatiZ1IPlIA2O7U3SKLLvaq2hPei8Uv2u0CVuU77FFluwZs2azsTProZnLXzsvcwkM9lsM8fTkhrMxzqoTkalPlYV56ikX1LfOeKoyPr169ltt914y1vewh577MGtt97Kxz/+cfbZZx+WLFnCySef/NiyX/ziF1m2bBl77rknxxxzDL/+9a+HGHnN+WtU/VRdJoMuc+uYpKaxX1OVytQv6+BAOcDrkwceeOCxUzRff9T7ALjpppv4oz/6I6677jpuvPFGbrrpJn74wx+yZs0aVq9ezZVXXskNN9zAV77yFb7zne+wZs0a5syZw5e+9KX+BWaDkiRNxv2DJDVSz1M0I2IH4MXAs4AHgGuBVZn5aMWxjZSJp2iufwie/exns++++wJw6aWXcumll7LXXnsBcP/993PTTTexdu1aVq9ezT777AN0Boo77LDDUNIgSVKreVpjtcxfaSCmHOBFxP7ACcAzgauBjcA84HXAb0bEhcAnM/PeQQQ6ip7+9Kc/9n9mcuKJJ3LMMcc8YZkzzjiDt771rZx66qmDDk+SZscva1K92UalVup2iuZrgLdn5j6ZeXRmfiAz/yQzDwaeT2fQ98qBRNkAr3rVqzjnnHO4//77AbjtttvYuHEjBxxwABdeeCEbN24E4K677uKnP/3p4x/82dXDCHd0eIqR1G72AZop647ULLbpx0w5wMvMP83MW6Z475HM/MfMvKi60JrlwAMP5I1vfCMvetGLeN7znsehhx7Kfffdx+LFi/nIRz7CgQceyJIlS3jlK1/J7bffPuxw1RT96OzsMAfPPJc0yuzDpKEqcw3e8cDfAfcBfwvsBZyQmZdWHNvMDeF0hLEjc2MWLlzItdde+4R5xx9/PMcff/yTPnvYYYdx2GGHVRqfNHSeKlQd81ZtN+w2MOztqzqWbXM1uGzL3EXzbcV1dgcCzwDeDJxWaVSqlr+sqYms16PLspM0G/Yhmo0G1p8yA7wo/r4G+EJmXjdunqrk9Xca08DOp3bMY0mql6r6Zft7NVyZAd7qiLiUzgDvmxGxFeAjEiS1g18E+s88faJh5odlUX+9ysgylDRBmQHeUXQel7BPZv4SeApwZK8PRcQ5EbExIq6d4v0jImJtRFwTEd+NiOdPK/IJMnM2Hx8pbUqr1Bd+AZIkSS1RZoCXwGLg3cX00+k8D6+XzwEHdXn/ZuBlmfk84MPA2SXWOal58+Zx5513tmLgk5nceeedzJtXpggabNBf2B0gSJLqxP3S6LMM66chZdLzLprAp+mckvly4BQ6d9O8CNin24cy88qIWNjl/e+Om/w+sKBELJNasGABGzZs4I477pjpKvrrno2w6Yb+rAeetK558+axYMGMs0tS2zT4TmF90ZAdumrC9iZpyMoM8F6YmXtHxNUAmXl3RDylz3EcBfzzTD88d+5cFi1a1MdwZmnFvv3p3FfsW/x1RyEBfnGSJKkMf7hqtTKnaD4cEXPonKpJRMynjzdZiYj96Qzw3t9lmaMjYlVErKrNUbq2saOQqmHbUpWsX+WYTxpVbay7bUzzNJUZ4J0O/AOwQ0R8FPhX4NR+bDwiltB5ePohmXnnVMtl5tmZuTQzl86fP78fm1abNbVjaGq6msZyKsd8UlNYl6X6aEl77HmKZmZ+KSJWAwfQef7d6zJz1heYRcQuwFeBN2fmj2e7PkmSpMbyFHVJJfU8ghcRX8jM/5uZZ2bmX2XmDRHxhRKfOw/4HrBbRGyIiKMi4h0R8Y5ikZOA7YBPR8SaiFg1q5TocS35daIr80BqLtu3pDHT7Q/sP4bL/B+IMqdo/vb4ieJ6vBf0+lBmHp6ZO2bm3MxckJmfzcy/zsy/Lt7/w8x8RmbuWbyWziwJUkPY6amOrJf1ZxlJo802rD6bcoAXESdGxH3Akoi4t3jdB2wEvjawCDW6+tVh2fFJkurOfVV1zNsO80ElTTnAy8xTM3Mr4OOZuXXx2iozt8vMEwcYYzvNthE7uJIklWE/L6kM+4qRUeYUzd0i4jURUWZZqdnq1LnVKRb1h2U6WOa3JKmBy
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Failure Model\n",
        "\n",
        "Here we try to capture the most common type of failures and their impact in the system."
      ],
      "metadata": {
        "id": "0BdiiYaXNpnR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@title Failure model parameters\n",
        "\n",
        "#@markdown Annual Failure Rate (% of network nodes down per year)\n",
        "AnnualFailureRate = 3 #@param {type:\"slider\", min:1, max:10, step:1}\n",
        "#@markdown Number of correlated failures event per year\n",
        "CorrelatedFailureEvents = 1 #@param {type:\"slider\", min:0, max:10, step:1}\n",
        "#@markdown Number of nodes affected by correlated failures \n",
        "CorrelatedFailureImpact = 100 #@param {type:\"slider\", min:2, max:1000, step:1}\n",
        "\n",
        "AnnualNodesFailed = NetworkSize*AnnualFailureRate/100\n",
        "MeanTimeBetweenFailures = 365*24/AnnualNodesFailed\n",
        "\n",
        "print(\"Number of nodes failed per year: %i\" % AnnualNodesFailed)\n",
        "print(\"Mean time between failures: %i hours\" % MeanTimeBetweenFailures)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "75TtMGepOxl7",
        "outputId": "72c29478-8c72-4eb9-fc1a-bb5cc637f553"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of nodes failed per year: 150\n",
            "Mean time between failures: 58 hours\n"
          ]
        }
      ]
    }
  ]
}