status-im
/
eth2.0-specs
mirror of https://github.com/status-im/eth2.0-specs.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge pull request #1006 from JSON/patch-1 

Tidying up the test_gen readme
This commit is contained in:
Danny Ryan 2019-04-29 11:58:19 -06:00 committed by GitHub
parent 90cacf5bfc c2555f7ce9
commit 2348595f6d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 25 additions and 27 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
specs/core/0_fork-choice.md
View File

@ -86,7 +86,7 @@ def lmd_ghost(store: Store, start_state: BeaconState, start_block: BeaconBlock)
# made for optimized implementations that precompute and save data # made for optimized implementations that precompute and save data
def get_vote_count(block: BeaconBlock) -> int: def get_vote_count(block: BeaconBlock) -> int:
return sum( return sum(
start_state.validator_registry[validator_index].high_balance start_state.validator_registry[validator_index].effective_balance
for validator_index, target in attestation_targets for validator_index, target in attestation_targets
if get_ancestor(store, target, block.slot) == block if get_ancestor(store, target, block.slot) == block
) )

48
test_generators/README.md
View File

@ -2,22 +2,21 @@
This directory contains all the generators for YAML tests, consumed by Eth 2.0 client implementations. This directory contains all the generators for YAML tests, consumed by Eth 2.0 client implementations.
Any issues with the generators and/or generated tests should be filed Any issues with the generators and/or generated tests should be filed in the repository that hosts the generator outputs, here: [ethereum/eth2.0-tests](https://github.com/ethereum/eth2.0-tests).
in the repository that hosts the generator outputs, here: [ethereum/eth2.0-tests](https://github.com/ethereum/eth2.0-tests/).
Whenever a release is made, the new tests are automatically built and Whenever a release is made, the new tests are automatically built, and
[eth2TestGenBot](https://github.com/eth2TestGenBot) commits the changes to the test repository. [eth2TestGenBot](https://github.com/eth2TestGenBot) commits the changes to the test repository.
## How to run generators ## How to run generators
pre-requisites: Prerequisites:
- Python 3 installed - Python 3 installed
- PIP 3 - PIP 3
- GNU make - GNU make
### Cleaning ### Cleaning
This removes the existing virtual environments (`/test_generators/<generator>/venv`), and generated tests (`/yaml_tests/`). This removes the existing virtual environments (`/test_generators/<generator>/venv`) and generated tests (`/yaml_tests/`).
```bash ```bash
make clean make clean
@ -25,7 +24,7 @@ make clean
### Running all test generators ### Running all test generators
This runs all the generators. This runs all of the generators.
```bash ```bash
make -j 4 gen_yaml_tests make -j 4 gen_yaml_tests
@ -36,8 +35,7 @@ The `-j N` flag makes the generators run in parallel, with `N` being the amount
### Running a single generator ### Running a single generator
The make file auto-detects generators in the `test_generators/` directory, The makefile auto-detects generators in the `test_generators` directory and provides a tests-gen target for each generator. See example:
and provides a tests-gen target for each generator, see example.
```bash ```bash
make ./yaml_tests/shuffling/ make ./yaml_tests/shuffling/
@ -45,7 +43,7 @@ make ./yaml_tests/shuffling/
## Developing a generator ## Developing a generator
Simply open up the generator (not all at once) of choice in your favorite IDE/editor, and run: Simply open up the generator (not all at once) of choice in your favorite IDE/editor and run:
```bash ```bash
# From the root of the generator directory: # From the root of the generator directory:
@ -65,10 +63,10 @@ eth-utils==1.4.1
../../test_libs/config_helpers ../../test_libs/config_helpers
../../test_libs/pyspec ../../test_libs/pyspec
``` ```
The config helper and pyspec is optional, but preferred. We encourage generators to derive tests from the spec itself, to prevent code duplication and outdated tests. The config helper and pyspec is optional, but preferred. We encourage generators to derive tests from the spec itself in order to prevent code duplication and outdated tests.
Applying configurations to the spec is simple, and enables you to create test suites with different contexts. Applying configurations to the spec is simple and enables you to create test suites with different contexts.
Note: make sure to run `make pyspec` from the root of the specs repository, to build the pyspec requirement. Note: make sure to run `make pyspec` from the root of the specs repository in order to build the pyspec requirement.
Install all the necessary requirements (re-run when you add more): Install all the necessary requirements (re-run when you add more):
```bash ```bash
@ -77,7 +75,7 @@ pip3 install -e .[pyspec]
And write your initial test generator, extending the base generator: And write your initial test generator, extending the base generator:
Write a `main.py` file, here's an example: Write a `main.py` file. See example:
```python ```python
from gen_base import gen_runner, gen_suite, gen_typing from gen_base import gen_runner, gen_suite, gen_typing
@ -134,26 +132,26 @@ if __name__ == "__main__":
``` ```
Recommendations: Recommendations:
- you can have more than just 1 suite creator, e.g. ` gen_runner.run_generator("foo", [bar_test_suite, abc_test_suite, example_test_suite])` - You can have more than just one suite creator, e.g. ` gen_runner.run_generator("foo", [bar_test_suite, abc_test_suite, example_test_suite])`.
- you can concatenate lists of test cases, if you don't want to split it up in suites, however make sure they could be run with one handler. - You can concatenate lists of test cases if you don't want to split it up in suites, however, make sure they can be run with one handler.
- you can split your suite creators into different python files/packages, good for code organization. - You can split your suite creators into different python files/packages; this is good for code organization.
- use config "minimal" for performance. But also implement a suite with the default config where necessary. - Use config "minimal" for performance, but also implement a suite with the default config where necessary.
- you may be able to write your test suite creator in a way where it does not make assumptions on constants. - You may be able to write your test suite creator in a way where it does not make assumptions on constants.
If so, you can generate test suites with different configurations for the same scenario (see example). If so, you can generate test suites with different configurations for the same scenario (see example).
- the test-generator accepts `--output` and `--force` (overwrite output) - The test-generator accepts `--output` and `--force` (overwrite output).
## How to add a new test generator ## How to add a new test generator
In order to add a new test generator that builds `New Tests`: To add a new test generator that builds `New Tests`:
1. Create a new directory `new_tests`, within the `test_generators` directory. 1. Create a new directory `new_tests` within the `test_generators` directory.
Note that `new_tests` is also the name of the directory in which the tests will appear in the tests repository later. Note that `new_tests` is also the name of the directory in which the tests will appear in the tests repository later.
2. Your generator is assumed to have a `requirements.txt` file, 2. Your generator is assumed to have a `requirements.txt` file,
with any dependencies it may need. Leave it empty if your generator has none. with any dependencies it may need. Leave it empty if your generator has none.
3. Your generator is assumed to have a `main.py` file in its root. 3. Your generator is assumed to have a `main.py` file in its root.
By adding the base generator to your requirements, you can make a generator really easily. See docs below. By adding the base generator to your requirements, you can make a generator really easily. See docs below.
4. Your generator is called with `-o some/file/path/for_testing/can/be_anything -c some/other/path/to_configs/`. 4. Your generator is called with `-o some/file/path/for_testing/can/be_anything -c some/other/path/to_configs/`.
The base generator helps you handle this; you only have to define suite headers, The base generator helps you handle this; you only have to define suite headers
and a list of tests for each suite you generate. and a list of tests for each suite you generate.
5. Finally, add any linting or testing commands to the 5. Finally, add any linting or testing commands to the
[circleci config file](https://github.com/ethereum/eth2.0-test-generators/blob/master/.circleci/config.yml) [circleci config file](https://github.com/ethereum/eth2.0-test-generators/blob/master/.circleci/config.yml)
@ -168,6 +166,6 @@ Do note that generators should be easy to maintain, lean, and based on the spec.
If a test generator is not needed anymore, undo the steps described above and make a new release: If a test generator is not needed anymore, undo the steps described above and make a new release:
1. remove the generator directory 1. Remove the generator directory.
2. remove the generated tests in the `eth2.0-tests` repository by opening a PR there. 2. Remove the generated tests in the [`eth2.0-tests`](https://github.com/ethereum/eth2.0-tests) repository by opening a PR there.
3. make a new release 3. Make a new release.