diff --git a/specs/test_formats/README.md b/specs/test_formats/README.md index 196315185..0b9d64fa5 100644 --- a/specs/test_formats/README.md +++ b/specs/test_formats/README.md @@ -49,22 +49,20 @@ Test formats: ## Glossary -- `generator`: a program that outputs one or more `suite` files. - - A generator should only output one `type` of test. - - A generator is free to output multiple `suite` files, optionally with different `handler`s. -- `type`: the specialization of one single `generator`. -- `suite`: a YAML file with: - - a header: describes the `suite`, and defines what the `suite` is for - - a list of test cases +- `generator`: a program that outputs one or more test-cases, each organized into a `config > runner > handler > suite` hierarchy. +- `config`: tests are grouped by configuration used for spec presets. In addition to the standard configurations, + `general` may be used as a catch-all for tests not restricted to one configuration. (E.g. BLS). +- `type`: the specialization of one single `generator`. E.g. epoch processing. - `runner`: where a generator is a *"producer"*, this is the *"consumer"*. - A `runner` focuses on *only one* `type`, and each type has *only one* `runner`. -- `handler`: a `runner` may be too limited sometimes, you may have a `suite` with a specific focus that requires a different format. +- `handler`: a `runner` may be too limited sometimes, you may have a set of tests with a specific focus that requires a different format. To facilitate this, you specify a `handler`: the runner can deal with the format by using the specified handler. - Using a `handler` in a `runner` is optional. -- `case`: a test case, an entry in the `test_cases` list of a `suite`. A case can be anything in general, - but its format should be well-defined in the documentation corresponding to the `type` (and `handler`).\ - A test has the same exact configuration and fork context as the other entries in the `case` list of its `suite`. - +- `suite`: a directory containing test cases that are coherent. Each `suite` under the same `handler` shares the same format. + This is an organizational/cosmetic hierarchy layer. +- `case`: a test case, a directory in a `suite`. A case can be anything in general, + but its format should be well-defined in the documentation corresponding to the `type` (and `handler`). +- `case part`: a test case consists of different files, possibly in different formats, to facilitate the specific test case format better. + Optionally, a `meta.yaml` is included to declare meta-data for the test, e.g. BLS requirements. ## Test format philosophy @@ -121,10 +119,12 @@ The well known bls/shuffling/ssz_static/operations/epoch_processing/etc. Handler ### `/` Specialization within category. All suites in here will have the same test case format. +Using a `handler` in a `runner` is optional. A `core` (or other generic) handler may be used if the `runner` does not have different formats. ### `/` -Suites are split up. Suite size does not change memory bounds, and makes lookups of particular tests fast to find and load. +Suites are split up. Suite size (i.e. the amount of tests) does not change the maximum memory requirement, as test cases can be loaded one by one. +This also makes filtered sets of tests fast and easy to load. ### `/` diff --git a/specs/test_formats/ssz_static/README.md b/specs/test_formats/ssz_static/README.md index 1df2cb5f6..1dfe0c23f 100644 --- a/specs/test_formats/ssz_static/README.md +++ b/specs/test_formats/ssz_static/README.md @@ -3,6 +3,6 @@ This set of test-suites provides static testing for SSZ: to instantiate just the known Eth 2.0 SSZ types from binary data. -This series of tests is based on the spec-maintained `minimal_ssz.py`, i.e. fully consistent with the SSZ spec. +This series of tests is based on the spec-maintained `eth2spec/utils/ssz/ssz_impl.py`, i.e. fully consistent with the SSZ spec. Test format documentation can be found here: [core test format](./core.md). diff --git a/specs/test_formats/ssz_static/core.md b/specs/test_formats/ssz_static/core.md index f24a225b0..1816e7d4d 100644 --- a/specs/test_formats/ssz_static/core.md +++ b/specs/test_formats/ssz_static/core.md @@ -4,28 +4,51 @@ The goal of this type is to provide clients with a solid reference for how the k Each object described in the Phase 0 spec is covered. This is important, as many of the clients aiming to serialize/deserialize objects directly into structs/classes do not support (or have alternatives for) generic SSZ encoding/decoding. + This test-format ensures these direct serializations are covered. +Note that this test suite does not cover the invalid-encoding case: + SSZ implementations should be hardened against invalid inputs with the other SSZ tests as guide, along with fuzzing. + ## Test case format +Each SSZ type is a `handler`, since the format is semantically different: the type of the data is different. + +One can iterate over the handlers, and select the type based on the handler name. +Suites are then the same format, but each specialized in one randomization mode. +Some randomization modes may only produce a single test case (e.g. the all-zeroes case). + +The output parts are: `meta.yaml`, `serialized.ssz`, `value.yaml` + +### `meta.yaml` + +For non-container SSZ type: + ```yaml -SomeObjectName: -- key, object name, formatted as in spec. E.g. "BeaconBlock". - value: dynamic -- the YAML-encoded value, of the type specified by type_name. - serialized: bytes -- string, SSZ-serialized data, hex encoded, with prefix 0x - root: bytes32 -- string, hash-tree-root of the value, hex encoded, with prefix 0x - signing_root: bytes32 -- string, signing-root of the value, hex encoded, with prefix 0x. Optional, present if type contains ``signature`` field +root: bytes32 -- string, hash-tree-root of the value, hex encoded, with prefix 0x +signing_root: bytes32 -- string, signing-root of the value, hex encoded, with prefix 0x. + Optional, present if type is a container and ends with a ``signature`` field. ``` +### `serialized.ssz` + +The raw encoded bytes. + +### `value.yaml` + +The same value as `serialized.ssz`, represented as YAML. + + ## Condition A test-runner can implement the following assertions: - Serialization: After parsing the `value`, SSZ-serialize it: the output should match `serialized` -- Hash-tree-root: After parsing the `value`, Hash-tree-root it: the output should match `root` - - Optionally also check signing-root, if present. +- Hash-tree-root: After parsing the `value` (or deserializing `serialized`), Hash-tree-root it: the output should match `root` + - Optionally also check `signing_root`, if present. - Deserialization: SSZ-deserialize the `serialized` value, and see if it matches the parsed `value` -## References +## References **`serialized`**—[SSZ serialization](../../simple-serialize.md#serialization) **`root`**—[hash_tree_root](../../simple-serialize.md#merkleization) function