Update README.md
This commit is contained in:
parent
2787fea5fe
commit
772a3800e6
|
@ -1,23 +1,22 @@
|
|||
# Eth2.0 Test Generators
|
||||
# Eth 2.0 Test Generators
|
||||
|
||||
This directory contains all the generators for YAML tests, consumed by Eth 2.0 client implementations.
|
||||
|
||||
Any issues with the generators and/or generated tests should be filed
|
||||
in the repository that hosts the generator outputs, here: [ethereum/eth2.0-tests](https://github.com/ethereum/eth2.0-tests/).
|
||||
Any issues with the generators and/or generated tests should be filed in the repository that hosts the generator outputs, here: [ethereum/eth2.0-tests](https://github.com/ethereum/eth2.0-tests).
|
||||
|
||||
Whenever a release is made, the new tests are automatically built and
|
||||
Whenever a release is made, the new tests are automatically built, and
|
||||
[eth2TestGenBot](https://github.com/eth2TestGenBot) commits the changes to the test repository.
|
||||
|
||||
## How to run generators
|
||||
|
||||
pre-requisites:
|
||||
Prerequisites:
|
||||
- Python 3 installed
|
||||
- PIP 3
|
||||
- GNU make
|
||||
|
||||
### Cleaning
|
||||
|
||||
This removes the existing virtual environments (`/test_generators/<generator>/venv`), and generated tests (`/yaml_tests/`).
|
||||
This removes the existing virtual environments (`/test_generators/<generator>/venv`) and generated tests (`/yaml_tests/`).
|
||||
|
||||
```bash
|
||||
make clean
|
||||
|
@ -25,7 +24,7 @@ make clean
|
|||
|
||||
### Running all test generators
|
||||
|
||||
This runs all the generators.
|
||||
This runs all of the generators.
|
||||
|
||||
```bash
|
||||
make -j 4 gen_yaml_tests
|
||||
|
@ -36,8 +35,7 @@ The `-j N` flag makes the generators run in parallel, with `N` being the amount
|
|||
|
||||
### Running a single generator
|
||||
|
||||
The make file auto-detects generators in the `test_generators/` directory,
|
||||
and provides a tests-gen target for each generator, see example.
|
||||
The makefile auto-detects generators in the `test_generators/` directory and provides a tests-gen target for each generator. See example:
|
||||
|
||||
```bash
|
||||
make ./yaml_tests/shuffling/
|
||||
|
@ -45,7 +43,7 @@ make ./yaml_tests/shuffling/
|
|||
|
||||
## Developing a generator
|
||||
|
||||
Simply open up the generator (not all at once) of choice in your favorite IDE/editor, and run:
|
||||
Simply open up the generator (not all at once) of choice in your favorite IDE/editor and run:
|
||||
|
||||
```bash
|
||||
# From the root of the generator directory:
|
||||
|
@ -65,10 +63,10 @@ eth-utils==1.4.1
|
|||
../../test_libs/config_helpers
|
||||
../../test_libs/pyspec
|
||||
```
|
||||
The config helper and pyspec is optional, but preferred. We encourage generators to derive tests from the spec itself, to prevent code duplication and outdated tests.
|
||||
Applying configurations to the spec is simple, and enables you to create test suites with different contexts.
|
||||
The config helper and pyspec is optional, but preferred. We encourage generators to derive tests from the spec itself in order to prevent code duplication and outdated tests.
|
||||
Applying configurations to the spec is simple and enables you to create test suites with different contexts.
|
||||
|
||||
Note: make sure to run `make pyspec` from the root of the specs repository, to build the pyspec requirement.
|
||||
Note: make sure to run `make pyspec` from the root of the specs repository in order to build the pyspec requirement.
|
||||
|
||||
Install all the necessary requirements (re-run when you add more):
|
||||
```bash
|
||||
|
@ -77,7 +75,7 @@ pip3 install -e .[pyspec]
|
|||
|
||||
And write your initial test generator, extending the base generator:
|
||||
|
||||
Write a `main.py` file, here's an example:
|
||||
Write a `main.py` file. See example:
|
||||
|
||||
```python
|
||||
from gen_base import gen_runner, gen_suite, gen_typing
|
||||
|
@ -134,26 +132,26 @@ if __name__ == "__main__":
|
|||
```
|
||||
|
||||
Recommendations:
|
||||
- you can have more than just 1 suite creator, e.g. ` gen_runner.run_generator("foo", [bar_test_suite, abc_test_suite, example_test_suite])`
|
||||
- you can concatenate lists of test cases, if you don't want to split it up in suites, however make sure they could be run with one handler.
|
||||
- you can split your suite creators into different python files/packages, good for code organization.
|
||||
- use config "minimal" for performance. But also implement a suite with the default config where necessary.
|
||||
- you may be able to write your test suite creator in a way where it does not make assumptions on constants.
|
||||
- You can have more than just one suite creator, e.g. ` gen_runner.run_generator("foo", [bar_test_suite, abc_test_suite, example_test_suite])`.
|
||||
- You can concatenate lists of test cases if you don't want to split it up in suites, however, make sure they can be run with one handler.
|
||||
- You can split your suite creators into different python files/packages; this is good for code organization.
|
||||
- Use config "minimal" for performance, but also implement a suite with the default config where necessary.
|
||||
- You may be able to write your test suite creator in a way where it does not make assumptions on constants.
|
||||
If so, you can generate test suites with different configurations for the same scenario (see example).
|
||||
- the test-generator accepts `--output` and `--force` (overwrite output)
|
||||
- The test-generator accepts `--output` and `--force` (overwrite output).
|
||||
|
||||
## How to add a new test generator
|
||||
|
||||
In order to add a new test generator that builds `New Tests`:
|
||||
To add a new test generator that builds `New Tests`:
|
||||
|
||||
1. Create a new directory `new_tests`, within the `test_generators` directory.
|
||||
1. Create a new directory `new_tests` within the `test_generators` directory.
|
||||
Note that `new_tests` is also the name of the directory in which the tests will appear in the tests repository later.
|
||||
2. Your generator is assumed to have a `requirements.txt` file,
|
||||
with any dependencies it may need. Leave it empty if your generator has none.
|
||||
3. Your generator is assumed to have a `main.py` file in its root.
|
||||
By adding the base generator to your requirements, you can make a generator really easily. See docs below.
|
||||
4. Your generator is called with `-o some/file/path/for_testing/can/be_anything -c some/other/path/to_configs/`.
|
||||
The base generator helps you handle this; you only have to define suite headers,
|
||||
The base generator helps you handle this; you only have to define suite headers
|
||||
and a list of tests for each suite you generate.
|
||||
5. Finally, add any linting or testing commands to the
|
||||
[circleci config file](https://github.com/ethereum/eth2.0-test-generators/blob/master/.circleci/config.yml)
|
||||
|
@ -168,6 +166,6 @@ Do note that generators should be easy to maintain, lean, and based on the spec.
|
|||
|
||||
If a test generator is not needed anymore, undo the steps described above and make a new release:
|
||||
|
||||
1. remove the generator directory
|
||||
2. remove the generated tests in the `eth2.0-tests` repository by opening a PR there.
|
||||
3. make a new release
|
||||
1. Remove the generator directory.
|
||||
2. Remove the generated tests in the [`eth2.0-tests`](https://github.com/ethereum/eth2.0-tests) repository by opening a PR there.
|
||||
3. Make a new release.
|
||||
|
|
Loading…
Reference in New Issue