A new more complete test format for ETH 2.0 testing
This commit is contained in:
protolambda
parent
f5c5c166af
commit
80067721e2
|
|
@ -1,71 +0,0 @@
|
||||||
# General test format [WIP]
|
|
||||||
|
|
||||||
This document defines the general YAML format to which all tests should conform. Testing specifications in Eth2.0 are still a work in progress. _Expect breaking changes_
|
|
||||||
|
|
||||||
## ToC
|
|
||||||
|
|
||||||
* [About](#about)
|
|
||||||
* [YAML Fields](#yaml-fields)
|
|
||||||
* [Example test suite](#example-test-suite)
|
|
||||||
|
|
||||||
## About
|
|
||||||
Ethereum 2.0 uses YAML as the format for all cross client tests. This document describes at a high level the general format to which all test files should conform.
|
|
||||||
|
|
||||||
The particular formats of specific types of tests (test suites) are defined in separate documents.
|
|
||||||
|
|
||||||
## YAML fields
|
|
||||||
`title` _(required)_
|
|
||||||
|
|
||||||
`summary` _(optional)_
|
|
||||||
|
|
||||||
`test_suite` _(required)_ string defining the test suite to which the test cases conform
|
|
||||||
|
|
||||||
`fork` _(required)_ production release versioning
|
|
||||||
|
|
||||||
`version` _(required)_ version for particular test document
|
|
||||||
|
|
||||||
`test_cases` _(required)_ list of test cases each of which is formatted to conform to the `test_case` standard defined by `test_suite`. All test cases have optional `name` and `description` string fields.
|
|
||||||
|
|
||||||
## Example test suite
|
|
||||||
`shuffle` is a test suite that defines test cases for the `shuffle()` helper function defined in the `beacon-chain` spec.
|
|
||||||
|
|
||||||
Test cases that conform to the `shuffle` test suite have the following fields:
|
|
||||||
|
|
||||||
* `input` _(required)_ the list of items passed into `shuffle()`
|
|
||||||
* `output` _(required)_ the expected list returned by `shuffle()`
|
|
||||||
* `seed` _(required)_ the seed of entropy passed into `shuffle()`
|
|
||||||
|
|
||||||
As for all test cases, `name` and `description` are optional string fields.
|
|
||||||
|
|
||||||
The following is a sample YAML document for the `shuffle` test suite:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
title: Shuffling Algorithm Tests
|
|
||||||
summary: Test vectors for shuffling a list based upon a seed using `shuffle`
|
|
||||||
test_suite: shuffle
|
|
||||||
fork: tchaikovsky
|
|
||||||
version: 1.0
|
|
||||||
|
|
||||||
test_cases:
|
|
||||||
- input: []
|
|
||||||
output: []
|
|
||||||
seed: !!binary ""
|
|
||||||
- name: boring_list
|
|
||||||
description: List with a single element, 0
|
|
||||||
input: [0]
|
|
||||||
output: [0]
|
|
||||||
seed: !!binary ""
|
|
||||||
- input: [255]
|
|
||||||
output: [255]
|
|
||||||
seed: !!binary ""
|
|
||||||
- input: [4, 6, 2, 6, 1, 4, 6, 2, 1, 5]
|
|
||||||
output: [1, 6, 4, 1, 6, 6, 2, 2, 4, 5]
|
|
||||||
seed: !!binary ""
|
|
||||||
- input: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]
|
|
||||||
output: [4, 7, 10, 13, 3, 1, 2, 9, 12, 6, 11, 8, 5]
|
|
||||||
seed: !!binary ""
|
|
||||||
- input: [65, 6, 2, 6, 1, 4, 6, 2, 1, 5]
|
|
||||||
output: [6, 65, 2, 5, 4, 2, 6, 6, 1, 1]
|
|
||||||
seed: !!binary |
|
|
||||||
JlAYJ5H2j8g7PLiPHZI/rTS1uAvKiieOrifPN6Moso0=
|
|
||||||
```
|
|
||||||
|
|
@ -0,0 +1,168 @@
|
||||||
|
# General test format
|
||||||
|
|
||||||
|
This document defines the YAML format and structure used for ETH 2.0 testing.
|
||||||
|
|
||||||
|
## ToC
|
||||||
|
|
||||||
|
* [About](#about)
|
||||||
|
* [Glossary](#glossary)
|
||||||
|
* [Test format philosophy](#test-format-philosophy)
|
||||||
|
* [Test Suite](#yaml-suite)
|
||||||
|
* [Config](#config)
|
||||||
|
* [Fork-timeline](#fork-timeline)
|
||||||
|
* [Config sourcing](#config-sourcing)
|
||||||
|
* [Test structure](#test-structure)
|
||||||
|
|
||||||
|
## About
|
||||||
|
|
||||||
|
Ethereum 2.0 uses YAML as the format for all cross client tests. This document describes at a high level the general format to which all test files should conform.
|
||||||
|
|
||||||
|
The particular formats of specific types of tests (test suites) are defined in separate documents.
|
||||||
|
|
||||||
|
## Glossary
|
||||||
|
|
||||||
|
- `generator`: a program that outputs one or more `suite` files.
|
||||||
|
- `type`: the specialization of one single `generator`.
|
||||||
|
- `suite`: a YAML file with:
|
||||||
|
- a header: describes the `suite`, and defines what the `suite` is for
|
||||||
|
- a list of test cases
|
||||||
|
- `runner`: where a generator is a "producer", this is the 1-to-1 "consumer". A `runner` focuses on one `type`.
|
||||||
|
- `handler`: a `runner` may be too limited sometimes, you may have a `suite` with a specific focus that requires a different format.
|
||||||
|
To facilitate this, you specify a `handler`: the runner can deal with the format by using the specified handler.
|
||||||
|
Using a `handler` in a `runner` is optional.
|
||||||
|
- `case`: a test case, an entry in the `test_cases` list of a `suite`. A case can be anything in general,
|
||||||
|
but its format should be well-defined in the documentation corresponding to the `type` (and `handler`).\
|
||||||
|
A test has the same exact configuration and fork context as the other entries in the `case` list of its `suite`.
|
||||||
|
- `forks_timeline`: a fork timeline definition, a YAML file containing a key for each fork-name, and a slot number as value.
|
||||||
|
|
||||||
|
## Test format philosophy
|
||||||
|
|
||||||
|
### Config design
|
||||||
|
|
||||||
|
After long discussion, the following types of configured constants were identified:
|
||||||
|
- Never changing: genesis data
|
||||||
|
- Changing, but reliant on old value: e.g. a slot time may change, but if you want to do the conversion
|
||||||
|
`(genesis data, timestamp) -> slot number` you end up needing both constants.
|
||||||
|
- Changing, but kept around during fork transition: finalization may take a while,
|
||||||
|
e.g. an executable has to deal with new deposits and old deposits at the same time. Another example may be economic constants.
|
||||||
|
- Additional, back-wards compatible: new constants are introduced for later phases
|
||||||
|
- Changing: there is a very small chance some constant may really be *replaced*.
|
||||||
|
In this off-chance, it is likely better to include it as an additional variable,
|
||||||
|
and some clients may simply stop supporting the old one, if they do not want to sync from genesis.
|
||||||
|
|
||||||
|
Based on these types of changes, we model the config as a list of key value pairs,
|
||||||
|
that only grows with every fork (they may change in development versions of forks however, git manages this).
|
||||||
|
With this approach, configurations are backwards compatible (older clients ignore unknown variables), and easy to maintain.
|
||||||
|
|
||||||
|
### Fork config design
|
||||||
|
|
||||||
|
There are two types of fork-data:
|
||||||
|
1) timeline: when does a fork take place?
|
||||||
|
2) coverage: what forks are covered by a test?
|
||||||
|
|
||||||
|
The first is neat to have as a separate form: we prevent duplication, and can run with different presets
|
||||||
|
(e.g. fork timeline for a minimal local test, for a public testnet, or for main-net)
|
||||||
|
|
||||||
|
The second is still somewhat ambiguous: some tests may want cover multiple forks, and can do so in different ways:
|
||||||
|
- run one test, transitioning from one to the other
|
||||||
|
- run the same test for both
|
||||||
|
- run a test for every transition from one fork to the other
|
||||||
|
- more
|
||||||
|
|
||||||
|
There is a common factor here however: the options are exclusive, and give a clear idea on what test suites need to be ran to cover testing for a specific fork.
|
||||||
|
The way this list of forks is interpreted, is up to the test-runner:
|
||||||
|
State-transition test suites may want to just declare forks that are being covered in the test suite,
|
||||||
|
whereas shuffling test suites may want to declare a list of forks to test the shuffling algorithm for individually.
|
||||||
|
|
||||||
|
### Test completeness
|
||||||
|
|
||||||
|
We want tests to be independent from any sync-data. If one wants to run a test, the input data should be available from the YAML.
|
||||||
|
The aim is to provide clients with a well-defined scope of work to run a particular set of test-suites.
|
||||||
|
|
||||||
|
- Clients that are complete are expected to contribute to testing, seeking for better resources to get conformance with the spec, and other clients.
|
||||||
|
- Clients that are not complete in functionality can choose to ignore suites that use certain test-runners, or specific handlers of these test-runners.
|
||||||
|
- Clients that are on older versions can test there work based on older releases of the generated tests, and catch up with newer releases when possible.
|
||||||
|
|
||||||
|
## Test Suite
|
||||||
|
|
||||||
|
```
|
||||||
|
title: <required, string, short, one line> -- Display name for the test suite
|
||||||
|
summary: <required, string, average, 1-3 lines> -- Summarizes the test suite
|
||||||
|
forks_timeline: <required, string, reference to a fork definition file, without extension> -- Used to determine the forking timeline
|
||||||
|
forks: <required, list of strings> -- Runner decides what to do: run for each fork, or run for all at once, each fork transition, etc.
|
||||||
|
- ... <required, string, first the phase name, then the spec version>
|
||||||
|
config: <required, string, reference to a config file, without extension> -- Used to determine which set of constants to run (possibly compile time) with
|
||||||
|
runner: <required, string, no spaces, python-like naming format> *MUST be consistent with folder structure*
|
||||||
|
handler: <optional, string, no spaces, python-like naming format> *MUST be consistent with folder structure*
|
||||||
|
|
||||||
|
test_cases: <list, values being maps defining a test case each>
|
||||||
|
...
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
## Config
|
||||||
|
|
||||||
|
A configuration is a separate YAML file.
|
||||||
|
Separation of configuration and tests aims to:
|
||||||
|
- prevent duplication of a minimal set of tests
|
||||||
|
- make all tests easy to upgrade when a new config constant is introduced.
|
||||||
|
- clearly define which constants to use
|
||||||
|
- share-able between clients, for cross-client short or long lived test-nets
|
||||||
|
- minimize the amounts of different constants permutations to compile as a client. \**
|
||||||
|
|
||||||
|
\**: Some clients prefer compile-time constants and optimizations.
|
||||||
|
They should compile for each configuration once, and run the corresponding tests per build target.
|
||||||
|
|
||||||
|
|
||||||
|
## Fork-timeline
|
||||||
|
|
||||||
|
A fork timeline is (preferably) loaded in as a configuration object into a client, as opposed to the constants configuration:
|
||||||
|
- we do not allocate or optimize any code based on slot numbers
|
||||||
|
- when we transition from one fork to the other, it is preferred to stay online.
|
||||||
|
- we may decide on a slot number for a fork based on external events (e.g. Eth1 log event),
|
||||||
|
a client should be able to activate a fork dynamically.
|
||||||
|
|
||||||
|
Note that phases are considered to be "super forks",
|
||||||
|
i.e. they will just have a fork name, and be more heavy on changes.
|
||||||
|
|
||||||
|
## Config sourcing
|
||||||
|
|
||||||
|
The constants configurations are located in:
|
||||||
|
|
||||||
|
```
|
||||||
|
<specs repo root>/configs/contants/<config name>.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
And copied by CI for testing purposes to:
|
||||||
|
|
||||||
|
```
|
||||||
|
<tests repo root>/configs/contants/<config name>.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
The fork timelines are located in:
|
||||||
|
|
||||||
|
```
|
||||||
|
<specs repo root>/configs/fork_timelines/<timeline name>.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
And copied by CI for testing purposes to:
|
||||||
|
|
||||||
|
```
|
||||||
|
<tests repo root>/configs/fork_timelines/<timeline name>.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Test structure
|
||||||
|
|
||||||
|
To prevent parsing of hundreds of different YAML files to test a specific test type,
|
||||||
|
or even more specific, just a handler, tests should be structured in the following nested form:
|
||||||
|
|
||||||
|
```
|
||||||
|
. <--- root of eth2.0 tests repository
|
||||||
|
├── bls <--- collection of handler for a specific test-runner, example runner: "bls"
|
||||||
|
│ ├── signing <--- collection of test suites for a specific handler, example handler: "signing". If no handler, use a dummy folder "main"
|
||||||
|
│ │ ├── sign_msg.yml <--- an entry list of test suites
|
||||||
|
│ │ ... <--- more suite files (optional)
|
||||||
|
│ ... <--- more handlers
|
||||||
|
... <--- more test types
|
||||||
|
```
|
||||||
Loading…
Reference in New Issue