diff --git a/specs/test-format.md b/specs/test-format.md deleted file mode 100644 index d4256ef72..000000000 --- a/specs/test-format.md +++ /dev/null @@ -1,71 +0,0 @@ -# General test format [WIP] - -This document defines the general YAML format to which all tests should conform. Testing specifications in Eth2.0 are still a work in progress. _Expect breaking changes_ - -## ToC - -* [About](#about) -* [YAML Fields](#yaml-fields) -* [Example test suite](#example-test-suite) - -## About -Ethereum 2.0 uses YAML as the format for all cross client tests. This document describes at a high level the general format to which all test files should conform. - -The particular formats of specific types of tests (test suites) are defined in separate documents. - -## YAML fields -`title` _(required)_ - -`summary` _(optional)_ - -`test_suite` _(required)_ string defining the test suite to which the test cases conform - -`fork` _(required)_ production release versioning - -`version` _(required)_ version for particular test document - -`test_cases` _(required)_ list of test cases each of which is formatted to conform to the `test_case` standard defined by `test_suite`. All test cases have optional `name` and `description` string fields. - -## Example test suite -`shuffle` is a test suite that defines test cases for the `shuffle()` helper function defined in the `beacon-chain` spec. - -Test cases that conform to the `shuffle` test suite have the following fields: - -* `input` _(required)_ the list of items passed into `shuffle()` -* `output` _(required)_ the expected list returned by `shuffle()` -* `seed` _(required)_ the seed of entropy passed into `shuffle()` - -As for all test cases, `name` and `description` are optional string fields. - -The following is a sample YAML document for the `shuffle` test suite: - -```yaml -title: Shuffling Algorithm Tests -summary: Test vectors for shuffling a list based upon a seed using `shuffle` -test_suite: shuffle -fork: tchaikovsky -version: 1.0 - -test_cases: -- input: [] - output: [] - seed: !!binary "" -- name: boring_list - description: List with a single element, 0 - input: [0] - output: [0] - seed: !!binary "" -- input: [255] - output: [255] - seed: !!binary "" -- input: [4, 6, 2, 6, 1, 4, 6, 2, 1, 5] - output: [1, 6, 4, 1, 6, 6, 2, 2, 4, 5] - seed: !!binary "" -- input: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] - output: [4, 7, 10, 13, 3, 1, 2, 9, 12, 6, 11, 8, 5] - seed: !!binary "" -- input: [65, 6, 2, 6, 1, 4, 6, 2, 1, 5] - output: [6, 65, 2, 5, 4, 2, 6, 6, 1, 1] - seed: !!binary | - JlAYJ5H2j8g7PLiPHZI/rTS1uAvKiieOrifPN6Moso0= -``` diff --git a/specs/test_formats/README.md b/specs/test_formats/README.md new file mode 100644 index 000000000..fec018b4a --- /dev/null +++ b/specs/test_formats/README.md @@ -0,0 +1,168 @@ +# General test format + +This document defines the YAML format and structure used for ETH 2.0 testing. + +## ToC + +* [About](#about) +* [Glossary](#glossary) +* [Test format philosophy](#test-format-philosophy) +* [Test Suite](#yaml-suite) +* [Config](#config) +* [Fork-timeline](#fork-timeline) +* [Config sourcing](#config-sourcing) +* [Test structure](#test-structure) + +## About + +Ethereum 2.0 uses YAML as the format for all cross client tests. This document describes at a high level the general format to which all test files should conform. + +The particular formats of specific types of tests (test suites) are defined in separate documents. + +## Glossary + +- `generator`: a program that outputs one or more `suite` files. +- `type`: the specialization of one single `generator`. +- `suite`: a YAML file with: + - a header: describes the `suite`, and defines what the `suite` is for + - a list of test cases +- `runner`: where a generator is a "producer", this is the 1-to-1 "consumer". A `runner` focuses on one `type`. +- `handler`: a `runner` may be too limited sometimes, you may have a `suite` with a specific focus that requires a different format. + To facilitate this, you specify a `handler`: the runner can deal with the format by using the specified handler. + Using a `handler` in a `runner` is optional. +- `case`: a test case, an entry in the `test_cases` list of a `suite`. A case can be anything in general, + but its format should be well-defined in the documentation corresponding to the `type` (and `handler`).\ + A test has the same exact configuration and fork context as the other entries in the `case` list of its `suite`. +- `forks_timeline`: a fork timeline definition, a YAML file containing a key for each fork-name, and a slot number as value. + +## Test format philosophy + +### Config design + +After long discussion, the following types of configured constants were identified: +- Never changing: genesis data +- Changing, but reliant on old value: e.g. a slot time may change, but if you want to do the conversion + `(genesis data, timestamp) -> slot number` you end up needing both constants. +- Changing, but kept around during fork transition: finalization may take a while, + e.g. an executable has to deal with new deposits and old deposits at the same time. Another example may be economic constants. +- Additional, back-wards compatible: new constants are introduced for later phases +- Changing: there is a very small chance some constant may really be *replaced*. + In this off-chance, it is likely better to include it as an additional variable, + and some clients may simply stop supporting the old one, if they do not want to sync from genesis. + +Based on these types of changes, we model the config as a list of key value pairs, + that only grows with every fork (they may change in development versions of forks however, git manages this). +With this approach, configurations are backwards compatible (older clients ignore unknown variables), and easy to maintain. + +### Fork config design + +There are two types of fork-data: +1) timeline: when does a fork take place? +2) coverage: what forks are covered by a test? + +The first is neat to have as a separate form: we prevent duplication, and can run with different presets + (e.g. fork timeline for a minimal local test, for a public testnet, or for main-net) + +The second is still somewhat ambiguous: some tests may want cover multiple forks, and can do so in different ways: +- run one test, transitioning from one to the other +- run the same test for both +- run a test for every transition from one fork to the other +- more + +There is a common factor here however: the options are exclusive, and give a clear idea on what test suites need to be ran to cover testing for a specific fork. +The way this list of forks is interpreted, is up to the test-runner: +State-transition test suites may want to just declare forks that are being covered in the test suite, + whereas shuffling test suites may want to declare a list of forks to test the shuffling algorithm for individually. + +### Test completeness + +We want tests to be independent from any sync-data. If one wants to run a test, the input data should be available from the YAML. +The aim is to provide clients with a well-defined scope of work to run a particular set of test-suites. + +- Clients that are complete are expected to contribute to testing, seeking for better resources to get conformance with the spec, and other clients. +- Clients that are not complete in functionality can choose to ignore suites that use certain test-runners, or specific handlers of these test-runners. +- Clients that are on older versions can test there work based on older releases of the generated tests, and catch up with newer releases when possible. + +## Test Suite + +``` +title: -- Display name for the test suite +summary: -- Summarizes the test suite +forks_timeline: -- Used to determine the forking timeline +forks: -- Runner decides what to do: run for each fork, or run for all at once, each fork transition, etc. + - ... +config: -- Used to determine which set of constants to run (possibly compile time) with +runner: *MUST be consistent with folder structure* +handler: *MUST be consistent with folder structure* + +test_cases: + ... + +``` + +## Config + +A configuration is a separate YAML file. +Separation of configuration and tests aims to: +- prevent duplication of a minimal set of tests +- make all tests easy to upgrade when a new config constant is introduced. +- clearly define which constants to use +- share-able between clients, for cross-client short or long lived test-nets +- minimize the amounts of different constants permutations to compile as a client. \** + +\**: Some clients prefer compile-time constants and optimizations. +They should compile for each configuration once, and run the corresponding tests per build target. + + +## Fork-timeline + +A fork timeline is (preferably) loaded in as a configuration object into a client, as opposed to the constants configuration: + - we do not allocate or optimize any code based on slot numbers + - when we transition from one fork to the other, it is preferred to stay online. + - we may decide on a slot number for a fork based on external events (e.g. Eth1 log event), + a client should be able to activate a fork dynamically. + +Note that phases are considered to be "super forks", + i.e. they will just have a fork name, and be more heavy on changes. + +## Config sourcing + +The constants configurations are located in: + +``` +/configs/contants/.yaml +``` + +And copied by CI for testing purposes to: + +``` +/configs/contants/.yaml +``` + + +The fork timelines are located in: + +``` +/configs/fork_timelines/.yaml +``` + +And copied by CI for testing purposes to: + +``` +/configs/fork_timelines/.yaml +``` + +## Test structure + +To prevent parsing of hundreds of different YAML files to test a specific test type, + or even more specific, just a handler, tests should be structured in the following nested form: + +``` +. <--- root of eth2.0 tests repository +├── bls <--- collection of handler for a specific test-runner, example runner: "bls" +│   ├── signing <--- collection of test suites for a specific handler, example handler: "signing". If no handler, use a dummy folder "main" +│   │   ├── sign_msg.yml <--- an entry list of test suites +│   │   ... <--- more suite files (optional) +│   ... <--- more handlers +... <--- more test types +```