A new more complete test format for ETH 2.0 testing
This commit is contained in:
protolambda
parent
f5c5c166af
commit
80067721e2
|
|
@ -1,71 +0,0 @@
|
|||
# General test format [WIP]
|
||||
|
||||
This document defines the general YAML format to which all tests should conform. Testing specifications in Eth2.0 are still a work in progress. _Expect breaking changes_
|
||||
|
||||
## ToC
|
||||
|
||||
* [About](#about)
|
||||
* [YAML Fields](#yaml-fields)
|
||||
* [Example test suite](#example-test-suite)
|
||||
|
||||
## About
|
||||
Ethereum 2.0 uses YAML as the format for all cross client tests. This document describes at a high level the general format to which all test files should conform.
|
||||
|
||||
The particular formats of specific types of tests (test suites) are defined in separate documents.
|
||||
|
||||
## YAML fields
|
||||
`title` _(required)_
|
||||
|
||||
`summary` _(optional)_
|
||||
|
||||
`test_suite` _(required)_ string defining the test suite to which the test cases conform
|
||||
|
||||
`fork` _(required)_ production release versioning
|
||||
|
||||
`version` _(required)_ version for particular test document
|
||||
|
||||
`test_cases` _(required)_ list of test cases each of which is formatted to conform to the `test_case` standard defined by `test_suite`. All test cases have optional `name` and `description` string fields.
|
||||
|
||||
## Example test suite
|
||||
`shuffle` is a test suite that defines test cases for the `shuffle()` helper function defined in the `beacon-chain` spec.
|
||||
|
||||
Test cases that conform to the `shuffle` test suite have the following fields:
|
||||
|
||||
* `input` _(required)_ the list of items passed into `shuffle()`
|
||||
* `output` _(required)_ the expected list returned by `shuffle()`
|
||||
* `seed` _(required)_ the seed of entropy passed into `shuffle()`
|
||||
|
||||
As for all test cases, `name` and `description` are optional string fields.
|
||||
|
||||
The following is a sample YAML document for the `shuffle` test suite:
|
||||
|
||||
```yaml
|
||||
title: Shuffling Algorithm Tests
|
||||
summary: Test vectors for shuffling a list based upon a seed using `shuffle`
|
||||
test_suite: shuffle
|
||||
fork: tchaikovsky
|
||||
version: 1.0
|
||||
|
||||
test_cases:
|
||||
- input: []
|
||||
output: []
|
||||
seed: !!binary ""
|
||||
- name: boring_list
|
||||
description: List with a single element, 0
|
||||
input: [0]
|
||||
output: [0]
|
||||
seed: !!binary ""
|
||||
- input: [255]
|
||||
output: [255]
|
||||
seed: !!binary ""
|
||||
- input: [4, 6, 2, 6, 1, 4, 6, 2, 1, 5]
|
||||
output: [1, 6, 4, 1, 6, 6, 2, 2, 4, 5]
|
||||
seed: !!binary ""
|
||||
- input: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]
|
||||
output: [4, 7, 10, 13, 3, 1, 2, 9, 12, 6, 11, 8, 5]
|
||||
seed: !!binary ""
|
||||
- input: [65, 6, 2, 6, 1, 4, 6, 2, 1, 5]
|
||||
output: [6, 65, 2, 5, 4, 2, 6, 6, 1, 1]
|
||||
seed: !!binary |
|
||||
JlAYJ5H2j8g7PLiPHZI/rTS1uAvKiieOrifPN6Moso0=
|
||||
```
|
||||
|
|
@ -0,0 +1,168 @@
|
|||
# General test format
|
||||
|
||||
This document defines the YAML format and structure used for ETH 2.0 testing.
|
||||
|
||||
## ToC
|
||||
|
||||
* [About](#about)
|
||||
* [Glossary](#glossary)
|
||||
* [Test format philosophy](#test-format-philosophy)
|
||||
* [Test Suite](#yaml-suite)
|
||||
* [Config](#config)
|
||||
* [Fork-timeline](#fork-timeline)
|
||||
* [Config sourcing](#config-sourcing)
|
||||
* [Test structure](#test-structure)
|
||||
|
||||
## About
|
||||
|
||||
Ethereum 2.0 uses YAML as the format for all cross client tests. This document describes at a high level the general format to which all test files should conform.
|
||||
|
||||
The particular formats of specific types of tests (test suites) are defined in separate documents.
|
||||
|
||||
## Glossary
|
||||
|
||||
- `generator`: a program that outputs one or more `suite` files.
|
||||
- `type`: the specialization of one single `generator`.
|
||||
- `suite`: a YAML file with:
|
||||
- a header: describes the `suite`, and defines what the `suite` is for
|
||||
- a list of test cases
|
||||
- `runner`: where a generator is a "producer", this is the 1-to-1 "consumer". A `runner` focuses on one `type`.
|
||||
- `handler`: a `runner` may be too limited sometimes, you may have a `suite` with a specific focus that requires a different format.
|
||||
To facilitate this, you specify a `handler`: the runner can deal with the format by using the specified handler.
|
||||
Using a `handler` in a `runner` is optional.
|
||||
- `case`: a test case, an entry in the `test_cases` list of a `suite`. A case can be anything in general,
|
||||
but its format should be well-defined in the documentation corresponding to the `type` (and `handler`).\
|
||||
A test has the same exact configuration and fork context as the other entries in the `case` list of its `suite`.
|
||||
- `forks_timeline`: a fork timeline definition, a YAML file containing a key for each fork-name, and a slot number as value.
|
||||
|
||||
## Test format philosophy
|
||||
|
||||
### Config design
|
||||
|
||||
After long discussion, the following types of configured constants were identified:
|
||||
- Never changing: genesis data
|
||||
- Changing, but reliant on old value: e.g. a slot time may change, but if you want to do the conversion
|
||||
`(genesis data, timestamp) -> slot number` you end up needing both constants.
|
||||
- Changing, but kept around during fork transition: finalization may take a while,
|
||||
e.g. an executable has to deal with new deposits and old deposits at the same time. Another example may be economic constants.
|
||||
- Additional, back-wards compatible: new constants are introduced for later phases
|
||||
- Changing: there is a very small chance some constant may really be *replaced*.
|
||||
In this off-chance, it is likely better to include it as an additional variable,
|
||||
and some clients may simply stop supporting the old one, if they do not want to sync from genesis.
|
||||
|
||||
Based on these types of changes, we model the config as a list of key value pairs,
|
||||
that only grows with every fork (they may change in development versions of forks however, git manages this).
|
||||
With this approach, configurations are backwards compatible (older clients ignore unknown variables), and easy to maintain.
|
||||
|
||||
### Fork config design
|
||||
|
||||
There are two types of fork-data:
|
||||
1) timeline: when does a fork take place?
|
||||
2) coverage: what forks are covered by a test?
|
||||
|
||||
The first is neat to have as a separate form: we prevent duplication, and can run with different presets
|
||||
(e.g. fork timeline for a minimal local test, for a public testnet, or for main-net)
|
||||
|
||||
The second is still somewhat ambiguous: some tests may want cover multiple forks, and can do so in different ways:
|
||||
- run one test, transitioning from one to the other
|
||||
- run the same test for both
|
||||
- run a test for every transition from one fork to the other
|
||||
- more
|
||||
|
||||
There is a common factor here however: the options are exclusive, and give a clear idea on what test suites need to be ran to cover testing for a specific fork.
|
||||
The way this list of forks is interpreted, is up to the test-runner:
|
||||
State-transition test suites may want to just declare forks that are being covered in the test suite,
|
||||
whereas shuffling test suites may want to declare a list of forks to test the shuffling algorithm for individually.
|
||||
|
||||
### Test completeness
|
||||
|
||||
We want tests to be independent from any sync-data. If one wants to run a test, the input data should be available from the YAML.
|
||||
The aim is to provide clients with a well-defined scope of work to run a particular set of test-suites.
|
||||
|
||||
- Clients that are complete are expected to contribute to testing, seeking for better resources to get conformance with the spec, and other clients.
|
||||
- Clients that are not complete in functionality can choose to ignore suites that use certain test-runners, or specific handlers of these test-runners.
|
||||
- Clients that are on older versions can test there work based on older releases of the generated tests, and catch up with newer releases when possible.
|
||||
|
||||
## Test Suite
|
||||
|
||||
```
|
||||
title: <required, string, short, one line> -- Display name for the test suite
|
||||
summary: <required, string, average, 1-3 lines> -- Summarizes the test suite
|
||||
forks_timeline: <required, string, reference to a fork definition file, without extension> -- Used to determine the forking timeline
|
||||
forks: <required, list of strings> -- Runner decides what to do: run for each fork, or run for all at once, each fork transition, etc.
|
||||
- ... <required, string, first the phase name, then the spec version>
|
||||
config: <required, string, reference to a config file, without extension> -- Used to determine which set of constants to run (possibly compile time) with
|
||||
runner: <required, string, no spaces, python-like naming format> *MUST be consistent with folder structure*
|
||||
handler: <optional, string, no spaces, python-like naming format> *MUST be consistent with folder structure*
|
||||
|
||||
test_cases: <list, values being maps defining a test case each>
|
||||
...
|
||||
|
||||
```
|
||||
|
||||
## Config
|
||||
|
||||
A configuration is a separate YAML file.
|
||||
Separation of configuration and tests aims to:
|
||||
- prevent duplication of a minimal set of tests
|
||||
- make all tests easy to upgrade when a new config constant is introduced.
|
||||
- clearly define which constants to use
|
||||
- share-able between clients, for cross-client short or long lived test-nets
|
||||
- minimize the amounts of different constants permutations to compile as a client. \**
|
||||
|
||||
\**: Some clients prefer compile-time constants and optimizations.
|
||||
They should compile for each configuration once, and run the corresponding tests per build target.
|
||||
|
||||
|
||||
## Fork-timeline
|
||||
|
||||
A fork timeline is (preferably) loaded in as a configuration object into a client, as opposed to the constants configuration:
|
||||
- we do not allocate or optimize any code based on slot numbers
|
||||
- when we transition from one fork to the other, it is preferred to stay online.
|
||||
- we may decide on a slot number for a fork based on external events (e.g. Eth1 log event),
|
||||
a client should be able to activate a fork dynamically.
|
||||
|
||||
Note that phases are considered to be "super forks",
|
||||
i.e. they will just have a fork name, and be more heavy on changes.
|
||||
|
||||
## Config sourcing
|
||||
|
||||
The constants configurations are located in:
|
||||
|
||||
```
|
||||
<specs repo root>/configs/contants/<config name>.yaml
|
||||
```
|
||||
|
||||
And copied by CI for testing purposes to:
|
||||
|
||||
```
|
||||
<tests repo root>/configs/contants/<config name>.yaml
|
||||
```
|
||||
|
||||
|
||||
The fork timelines are located in:
|
||||
|
||||
```
|
||||
<specs repo root>/configs/fork_timelines/<timeline name>.yaml
|
||||
```
|
||||
|
||||
And copied by CI for testing purposes to:
|
||||
|
||||
```
|
||||
<tests repo root>/configs/fork_timelines/<timeline name>.yaml
|
||||
```
|
||||
|
||||
## Test structure
|
||||
|
||||
To prevent parsing of hundreds of different YAML files to test a specific test type,
|
||||
or even more specific, just a handler, tests should be structured in the following nested form:
|
||||
|
||||
```
|
||||
. <--- root of eth2.0 tests repository
|
||||
├── bls <--- collection of handler for a specific test-runner, example runner: "bls"
|
||||
│ ├── signing <--- collection of test suites for a specific handler, example handler: "signing". If no handler, use a dummy folder "main"
|
||||
│ │ ├── sign_msg.yml <--- an entry list of test suites
|
||||
│ │ ... <--- more suite files (optional)
|
||||
│ ... <--- more handlers
|
||||
... <--- more test types
|
||||
```
|
||||
Loading…
Reference in New Issue