From d64a4f248eb38f407fcfb2654a7adceefb9a6f4f Mon Sep 17 00:00:00 2001 From: protolambda Date: Mon, 15 Apr 2019 22:39:07 +1000 Subject: [PATCH] forks coverage description cleanup --- specs/test_formats/README.md | 24 +++++-------------- specs/test_formats/bls/aggregate_pubkeys.md | 4 ---- specs/test_formats/bls/aggregate_sigs.md | 4 ---- .../bls/msg_hash_g2_compressed.md | 4 ---- .../bls/msg_hash_g2_uncompressed.md | 4 ---- specs/test_formats/bls/priv_to_pub.md | 4 ---- specs/test_formats/bls/sign_msg.md | 4 ---- specs/test_formats/operations/deposits.md | 8 ------- specs/test_formats/shuffling/README.md | 4 ---- specs/test_formats/ssz/uint.md | 4 ---- 10 files changed, 6 insertions(+), 58 deletions(-) diff --git a/specs/test_formats/README.md b/specs/test_formats/README.md index 18c7329d9..6b9533056 100644 --- a/specs/test_formats/README.md +++ b/specs/test_formats/README.md @@ -75,22 +75,11 @@ There are two types of fork-data: The first is neat to have as a separate form: we prevent duplication, and can run with different presets (e.g. fork timeline for a minimal local test, for a public testnet, or for mainnet) -The second is still somewhat ambiguous: some tests may want cover multiple forks, and can do so in different ways: -- run one test, transitioning from one to the other -- run the same test for both -- run a test for every transition from one fork to the other -- more - -There is a common factor here however: the options are exclusive, and give a clear idea on what test suites need to be ran to cover testing for a specific fork. -The way this list of forks is interpreted, is up to the test-runner: -State-transition test suites may want to just declare forks that are being covered in the test suite, - whereas shuffling test suites may want to declare a list of forks to test the shuffling algorithm for individually. - -Test-formats specify the following `forks` interpretation rules: - -- `collective`: the test suite applies to all specified forks, and only needs to run once -- `individual`: the test suite should be ran against every fork -- more types may be specified with future test types. +The second does not affect the result of the tests, it just states what is covered by the tests, + so that the right suites can be executed to see coverage for a certain fork. +For some types of tests, it may be beneficial to ensure it runs exactly the same, with any given fork "active". +Test-formats can be explicit on the need to repeat a test with different forks being "active", + but generally tests run only once. ### Test completeness @@ -107,8 +96,7 @@ The aim is to provide clients with a well-defined scope of work to run a particu title: -- Display name for the test suite summary: -- Summarizes the test suite forks_timeline: -- Used to determine the forking timeline -forks: -- Runner decides what to do: run for each fork, or run for all at once, each fork transition, etc. - - ... +forks: -- Defines the coverage. Test-runner code may decide to re-run with the different forks "activated", when applicable. config: -- Used to determine which set of constants to run (possibly compile time) with runner: *MUST be consistent with folder structure* handler: *MUST be consistent with folder structure* diff --git a/specs/test_formats/bls/aggregate_pubkeys.md b/specs/test_formats/bls/aggregate_pubkeys.md index 9a6f1cc25..43c7d6c6d 100644 --- a/specs/test_formats/bls/aggregate_pubkeys.md +++ b/specs/test_formats/bls/aggregate_pubkeys.md @@ -15,7 +15,3 @@ output: BLS Pubkey -- expected output, single BLS pubkey ## Condition The `aggregate_pubkeys` handler should aggregate the keys in the `input`, and the result should match the expected `output`. - -## Forks - -Forks-interpretation: `collective` diff --git a/specs/test_formats/bls/aggregate_sigs.md b/specs/test_formats/bls/aggregate_sigs.md index 1588e26cb..6690c3344 100644 --- a/specs/test_formats/bls/aggregate_sigs.md +++ b/specs/test_formats/bls/aggregate_sigs.md @@ -15,7 +15,3 @@ output: BLS Signature -- expected output, single BLS signature ## Condition The `aggregate_sigs` handler should aggregate the signatures in the `input`, and the result should match the expected `output`. - -## Forks - -Forks-interpretation: `collective` diff --git a/specs/test_formats/bls/msg_hash_g2_compressed.md b/specs/test_formats/bls/msg_hash_g2_compressed.md index 51c64e28b..4e194e90b 100644 --- a/specs/test_formats/bls/msg_hash_g2_compressed.md +++ b/specs/test_formats/bls/msg_hash_g2_compressed.md @@ -17,7 +17,3 @@ All byte(s) fields are encoded as strings, hexadecimal encoding, prefixed with ` ## Condition The `msg_hash_g2_compressed` handler should hash the `message`, with the given `domain`, to G2 with compression, and the result should match the expected `output`. - -## Forks - -Forks-interpretation: `collective` diff --git a/specs/test_formats/bls/msg_hash_g2_uncompressed.md b/specs/test_formats/bls/msg_hash_g2_uncompressed.md index b7d2caa02..f42ea9998 100644 --- a/specs/test_formats/bls/msg_hash_g2_uncompressed.md +++ b/specs/test_formats/bls/msg_hash_g2_uncompressed.md @@ -17,7 +17,3 @@ All byte(s) fields are encoded as strings, hexadecimal encoding, prefixed with ` ## Condition The `msg_hash_g2_uncompressed` handler should hash the `message`, with the given `domain`, to G2, without compression, and the result should match the expected `output`. - -## Forks - -Forks-interpretation: `collective` diff --git a/specs/test_formats/bls/priv_to_pub.md b/specs/test_formats/bls/priv_to_pub.md index 9265b83ed..7af148d0f 100644 --- a/specs/test_formats/bls/priv_to_pub.md +++ b/specs/test_formats/bls/priv_to_pub.md @@ -15,7 +15,3 @@ All byte(s) fields are encoded as strings, hexadecimal encoding, prefixed with ` ## Condition The `priv_to_pub` handler should compute the public key for the given private key `input`, and the result should match the expected `output`. - -## Forks - -Forks-interpretation: `collective` diff --git a/specs/test_formats/bls/sign_msg.md b/specs/test_formats/bls/sign_msg.md index 3a6d63fa2..dd93174f2 100644 --- a/specs/test_formats/bls/sign_msg.md +++ b/specs/test_formats/bls/sign_msg.md @@ -18,7 +18,3 @@ All byte(s) fields are encoded as strings, hexadecimal encoding, prefixed with ` ## Condition The `sign_msg` handler should sign the given `message`, with `domain`, using the given `privkey`, and the result should match the expected `output`. - -## Forks - -Forks-interpretation: `collective` diff --git a/specs/test_formats/operations/deposits.md b/specs/test_formats/operations/deposits.md index 2bce84e30..b9dce318a 100644 --- a/specs/test_formats/operations/deposits.md +++ b/specs/test_formats/operations/deposits.md @@ -16,11 +16,3 @@ post: BeaconState -- state after applying the deposit. No value if deposit pr A `deposits` handler of the `operations` should process these cases, calling the implementation of the `process_deposit(state, deposit)` functionality described in the spec. The resulting state should match the expected `post` state, or no change if the `post` state is left blank. - -## Forks - -Forks-interpretation: `collective` - -Pre and post state contain slot numbers, and are time sensitive. -Additional tests will be added for future forks to cover fork-specific behavior based on input data - (including suites with deposits on fork transition blocks, covering multiple forks) diff --git a/specs/test_formats/shuffling/README.md b/specs/test_formats/shuffling/README.md index efc1b7b1a..514baf15a 100644 --- a/specs/test_formats/shuffling/README.md +++ b/specs/test_formats/shuffling/README.md @@ -30,7 +30,3 @@ Seed is the raw shuffling seed, passed to permute-index (or optimized shuffling The resulting list should match the expected output `shuffled` after shuffling the implied input, using the given `seed`. -## Forks - -Forks-interpretation: `collective` - diff --git a/specs/test_formats/ssz/uint.md b/specs/test_formats/ssz/uint.md index f71ddecb8..fd7cf3221 100644 --- a/specs/test_formats/ssz/uint.md +++ b/specs/test_formats/ssz/uint.md @@ -17,7 +17,3 @@ tags: List[string] -- description of test case, in the form of a list of labels Two-way testing can be implemented in the test-runner: - Encoding: After encoding the given input number `value`, the output should match `ssz` - Decoding: After decoding the given `ssz` bytes, it should match the input number `value` - -## Forks - -Forks-interpretation: `collective`