180 lines
6.8 KiB
Plaintext
180 lines
6.8 KiB
Plaintext
|
{
|
||
|
|
"cells": [
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"# Pubsub Test Runner Notebook\n",
|
||
|
|
"\n",
|
||
|
|
"This notebook helps you configure a pubsub test plan and run it on testground.\n",
|
||
|
|
"\n",
|
||
|
|
"**Important**: when you first run the notebook after checking out from git, you'll probably need to restart the Jupyter kernel and run all cells in the notebook. Use the `Kernel > Restart and Run All` Juypter menu command.\n",
|
||
|
|
"\n",
|
||
|
|
"## Configuring the Test\n",
|
||
|
|
"\n",
|
||
|
|
"Use the UI widgets to dial in a configuration for the test run.\n",
|
||
|
|
"\n",
|
||
|
|
"### Saving and loading the config\n",
|
||
|
|
"\n",
|
||
|
|
"You can save a snapshot of the current configuration by pressing the `Save Config` button below, which will dump the current value of the UI to a json file (`configs/snapshot.json` by default).\n",
|
||
|
|
"\n",
|
||
|
|
"Loading a snapshot will set the UI to the values from the snapshot file.\n",
|
||
|
|
"\n",
|
||
|
|
"This is useful if you need to restart the notebook kernel but don't want to lose your current config. You could also use it to save a baseline config that you want to make tweaks to in a future run.\n",
|
||
|
|
"\n",
|
||
|
|
"The default `config/snapshot.json` file\n",
|
||
|
|
"is ignored by git, so it can be used to quickly stash a config when reloading the notebook, etc. If you want to keep\n",
|
||
|
|
"the config and check it into the repo, just save it with a different name.\n",
|
||
|
|
"\n",
|
||
|
|
"**NOTE**: The `Test Execution` output directory param is ignored when loading saved snapshots, to avoid overwriting the output of a prior run.\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"## Running the test\n",
|
||
|
|
"\n",
|
||
|
|
"You'll need to be running the testground daemon somewhere accessible. By default, we'll attempt to use the daemon running at `localhost:8080`, but you can set the daemon endpoint param in the UI to target a different endpoint.\n",
|
||
|
|
"\n",
|
||
|
|
"Run the test by clicking the `Run Test` button below.\n",
|
||
|
|
"This will invoke testground and print the testground client output below the cell.\n",
|
||
|
|
"\n",
|
||
|
|
"You can halt the test by interrupting the Jupyter kernel, either with the square Stop button in the Jupyter toolbar or with the `Kernel > Interrupt` menu item.\n",
|
||
|
|
"\n",
|
||
|
|
"To re-run a test, you can re-run the Jupyter cell that contains the `Run Test` button, which will clear any existing printed output and re-create the run button, but won't change any of the configuration values. Note that this will overwrite any test outputs from a previous run, unless you change the output dir in the `Test Execution` config before running again.\n",
|
||
|
|
"\n",
|
||
|
|
"### Test outputs\n",
|
||
|
|
"\n",
|
||
|
|
"At the moment, test ouputs are stored only on the machine where the notebook is running. The output directory is configurable in the `Test Execution` config UI, and defaults to a timestamped subdirectory of `./output`.\n",
|
||
|
|
"\n",
|
||
|
|
"Inside that directory will be:\n",
|
||
|
|
"\n",
|
||
|
|
"- a generated `composition.toml` file that was sent to testground to execute\n",
|
||
|
|
"- a `template-params.toml` file that was used to parameterize the composition.\n",
|
||
|
|
"- a `test-output.tgz` (or `.zip`) file containing the collected testground output from all instances\n",
|
||
|
|
"- an `analysis` directory containing outputs from running the analysis script on the `test-output.tgz` archive\n",
|
||
|
|
"\n",
|
||
|
|
"The `anaylsis` directory will contain another Jupyter notebook called `Analysis.ipynb`. This contains interesting charts derived from the test output, and can be explored interactively. The path to the analysis notebook will be printed when the test completes successfully.\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"## Troubleshooting\n",
|
||
|
|
"\n",
|
||
|
|
"Expand the sections below if you're having issues running.\n",
|
||
|
|
"\n",
|
||
|
|
"<br/>\n",
|
||
|
|
"<details>\n",
|
||
|
|
" <summary>\n",
|
||
|
|
" <b>The UI Widgets aren't showing up</b>\n",
|
||
|
|
" </summary>\n",
|
||
|
|
"\n",
|
||
|
|
"If you don't see any UI widgets, select `Restart & Run All` from the `Kernel` menu. If you're viewing this notebook in JupyterLab, make sure to [install the JupyterLab widgets extension](https://ipywidgets.readthedocs.io/en/latest/user_install.html#installing-the-jupyterlab-extension).\n",
|
||
|
|
"</details>\n",
|
||
|
|
"\n",
|
||
|
|
"<br/>\n",
|
||
|
|
"\n",
|
||
|
|
"<details> \n",
|
||
|
|
" <summary>\n",
|
||
|
|
" <b>The test fails immediately</b>\n",
|
||
|
|
" </summary>\n",
|
||
|
|
" \n",
|
||
|
|
"#### can't reach the daemon\n",
|
||
|
|
"\n",
|
||
|
|
"If the output has a line like\n",
|
||
|
|
"\n",
|
||
|
|
"```\n",
|
||
|
|
"fatal error from daemon: Post http://localhost:8080/build: dial tcp [::1]:8080: connect: connection refused\n",
|
||
|
|
"```\n",
|
||
|
|
"\n",
|
||
|
|
"it means that the daemon isn't running, or we can't connect to it on the given `host:port`. Make sure we can reach the daemon, for example by running `./testground list` manually in a shell on the same machine that's running the notebook.\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"#### aws config not setup\n",
|
||
|
|
"\n",
|
||
|
|
"If you see something like:\n",
|
||
|
|
"\n",
|
||
|
|
"```\n",
|
||
|
|
"WARN\tengine build error: MissingRegion: could not find region configuration\t{\"ruid\": \"1255d41c\"}\n",
|
||
|
|
"```\n",
|
||
|
|
"\n",
|
||
|
|
"It probably means you're trying to run on kubernetes (which is the default), but your daemon isn't configured to run jobs on AWS. Consult the [infra docs](https://github.com/testground/infra) to make sure your cluster environment is\n",
|
||
|
|
"set up correctly.\n",
|
||
|
|
"\n",
|
||
|
|
"</details> \n",
|
||
|
|
"<br/>\n",
|
||
|
|
"\n",
|
||
|
|
"<details>\n",
|
||
|
|
" <summary>\n",
|
||
|
|
" <b>The test completes, but it prints errors about missing programs</b>\n",
|
||
|
|
" </summary>\n",
|
||
|
|
" \n",
|
||
|
|
"The analysis script expects `go` to be on the `PATH`. Make sure that `go` is accessible and restart the Jupyter server.\n",
|
||
|
|
" \n",
|
||
|
|
"</details>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {
|
||
|
|
"init_cell": true
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"import ui\n",
|
||
|
|
"%autosave 0"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {
|
||
|
|
"init_cell": true,
|
||
|
|
"scrolled": false
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"config = ui.ConfigPanel()\n",
|
||
|
|
"config.ui()"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {
|
||
|
|
"init_cell": true,
|
||
|
|
"scrolled": true
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"b = ui.RunButton(config)\n",
|
||
|
|
"b.wait()"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": null,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"source": []
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"metadata": {
|
||
|
|
"kernelspec": {
|
||
|
|
"display_name": "Python 3",
|
||
|
|
"language": "python",
|
||
|
|
"name": "python3"
|
||
|
|
},
|
||
|
|
"language_info": {
|
||
|
|
"codemirror_mode": {
|
||
|
|
"name": "ipython",
|
||
|
|
"version": 3
|
||
|
|
},
|
||
|
|
"file_extension": ".py",
|
||
|
|
"mimetype": "text/x-python",
|
||
|
|
"name": "python",
|
||
|
|
"nbconvert_exporter": "python",
|
||
|
|
"pygments_lexer": "ipython3",
|
||
|
|
"version": "3.7.7"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"nbformat": 4,
|
||
|
|
"nbformat_minor": 4
|
||
|
|
}
|