Merge pull request #1114 from ethereum/finish-state-tests-intro

Finish state tests intro

specs/test_formats/epoch_processing/README.md

@@ -0,0 +1,34 @@
+# Epoch processing tests
+
+The different epoch sub-transitions are tested individually with test handlers.
+The format is similar to block-processing state-transition tests.
+There is no "change" factor however, the transitions are pure functions with just the pre-state as input.
+Hence, the format is shared between each test-handler. (See test condition documentation on how to run the tests.)
+
+## Test case format
+
+```yaml
+description: string    -- description of test case, purely for debugging purposes
+bls_required: bool     -- optional, true if the test validity is strictly dependent on BLS being ON. False otherwise.
+bls_ignored: bool      -- optional, true if the test validity is strictly dependent on BLS being OFF. False otherwise.
+pre: BeaconState       -- state before running the sub-transition
+post: BeaconState      -- state after applying the epoch sub-transition.
+```
+
+Note: if both `bls_required` and `bls_ignored` are false (or simply not included),
+ then the test consumer can freely choose to run with BLS ON or OFF.
+One may choose for OFF for performance reasons during repeated testing. Otherwise it is recommended to run with BLS ON.
+
+## Condition
+
+A handler of the `epoch_processing` test-runner should process these cases, 
+ calling the corresponding processing implementation.
+
+Sub-transitions:
+
+| *`sub-transition-name`* | *`processing call`*               |
+|-------------------------|-----------------------------------|
+| `crosslinks`            | `process_crosslinks(state)`       |
+| `registry_updates`      | `process_registry_updates(state)` |
+
+The resulting state should match the expected `post` state.

specs/test_formats/operations/README.md

@@ -2,9 +2,39 @@
 
 The different kinds of operations ("transactions") are tested individually with test handlers.
 
-The tested operation kinds are:
-- [`deposits`](./deposits.md)
-- More tests are work-in-progress.
+## Test case format
 
+```yaml
+description: string                    -- description of test case, purely for debugging purposes
+bls_required: bool                     -- optional, true if the test validity is strictly dependent on BLS being ON. False otherwise.
+bls_ignored: bool                      -- optional, true if the test validity is strictly dependent on BLS being OFF. False otherwise.
+pre: BeaconState                       -- state before applying the operation
+<operation-name>: <operation-object>   -- the YAML encoded operation, e.g. a "ProposerSlashing", or "Deposit".
+post: BeaconState                      -- state after applying the operation. No value if operation processing is aborted.
+```
 
+Note: if both `bls_required` and `bls_ignored` are false (or simply not included),
+ then the test consumer can freely choose to run with BLS ON or OFF.
+One may choose for OFF for performance reasons during repeated testing. Otherwise it is recommended to run with BLS ON.
 
+## Condition
+
+A handler of the `operations` test-runner should process these cases, 
+ calling the corresponding processing implementation.
+
+Operations:
+
+| *`operation-name`*      | *`operation-object`* | *`input name`*       | *`processing call`*                                    |
+|-------------------------|----------------------|----------------------|--------------------------------------------------------|
+| `attestation`           | `Attestation`        | `attestation`        | `process_attestation(state, attestation)`                  |
+| `attester_slashing`     | `AttesterSlashing`   | `attester_slashing`  | `process_attester_slashing(state, attester_slashing)`            |
+| `block_header`          | `Block`              | `block`              | `process_block_header(state, block)`                   |
+| `deposit`               | `Deposit`            | `deposit`            | `process_deposit(state, deposit)`                      |
+| `proposer_slashing`     | `ProposerSlashing`   | `proposer_slashing`  | `process_proposer_slashing(state, proposer_slashing)`  |
+| `transfer`              | `Transfer`           | `transfer`           | `process_transfer(state, transfer)`                    |
+| `voluntary_exit`        | `VoluntaryExit`      | `voluntary_exit`     | `process_voluntary_exit(state, voluntary_exit)`        |
+
+Note that `block_header` is not strictly an operation (and is a full `Block`), but processed in the same manner, and hence included here. 
+
+The resulting state should match the expected `post` state, or if the `post` state is left blank,
+ the handler should reject the input operation as invalid.

specs/test_formats/operations/deposits.md

@@ -1,18 +0,0 @@
-# Test format: Deposit operations
-
-A deposit is a form of an operation (or "transaction"), modifying the state.
-
-## Test case format
-
-```yaml
-description: string  -- description of test case, purely for debugging purposes
-pre: BeaconState     -- state before applying the deposit
-deposit: Deposit     -- the deposit
-post: BeaconState    -- state after applying the deposit. No value if deposit processing is aborted.
-```
-
-## Condition
-
-A `deposits` handler of the `operations` should process these cases, 
- calling the implementation of the `process_deposit(state, deposit)` functionality described in the spec.
-The resulting state should match the expected `post` state, or if the `post` state is left blank, the handler should reject the inputs as invalid.

specs/test_formats/sanity/README.md

@@ -0,0 +1,7 @@
+# Sanity tests
+
+The aim of the sanity tests is to set a base-line on what really needs to pass, i.e. the essentials.
+
+There are two handlers, documented individually:
+- [`slots`](./slots.md): transitions of one or more slots (and epoch transitions within)
+- [`blocks`](./blocks.md): transitions triggered by one or more blocks

specs/test_formats/sanity/blocks.md

@@ -0,0 +1,19 @@
+# Sanity blocks testing
+
+Sanity tests to cover a series of one or more blocks being processed, aiming to cover common changes.
+
+## Test case format
+
+```yaml
+description: string    -- description of test case, purely for debugging purposes
+bls_required: bool     -- optional, true if the test validity is strictly dependent on BLS being ON. False otherwise.
+bls_ignored: bool      -- optional, true if the test validity is strictly dependent on BLS being OFF. False otherwise.
+pre: BeaconState       -- state before running through the transitions triggered by the blocks.
+blocks: [BeaconBlock]  -- blocks to process, in given order, following the main transition function (i.e. process slot and epoch transitions in between blocks as normal)
+post: BeaconState      -- state after applying all the transitions triggered by the blocks.
+```
+
+## Condition
+
+The resulting state should match the expected `post` state, or if the `post` state is left blank,
+ the handler should reject the series of blocks as invalid.

specs/test_formats/sanity/slots.md

@@ -0,0 +1,24 @@
+# Sanity slots testing
+
+Sanity tests to cover a series of one or more empty-slot transitions being processed, aiming to cover common changes.
+
+## Test case format
+
+```yaml
+description: string    -- description of test case, purely for debugging purposes
+bls_required: bool     -- optional, true if the test validity is strictly dependent on BLS being ON. False otherwise.
+bls_ignored: bool      -- optional, true if the test validity is strictly dependent on BLS being OFF. False otherwise.
+pre: BeaconState       -- state before running through the transitions.
+slots: N               -- amount of slots to process, N being a positive numer.
+post: BeaconState      -- state after applying all the transitions.
+```
+
+The transition with pure time, no blocks, is known as `state_transition_to(state, slot)` in the spec.
+This runs state-caching (pure slot transition) and epoch processing (every E slots).
+
+To process the data, call `state_transition_to(pre, pre.slot + N)`. And see if `pre` mutated into the equivalent of `post`.
+
+
+## Condition
+
+The resulting state should match the expected `post` state.

test_generators/epoch_processing/README.md

@@ -0,0 +1,11 @@
+# Epoch processing
+
+Epoch processing covers the sub-transitions during an epoch change.
+
+An epoch-processing test-runner can consume these sub-transition test-suites,
+ and handle different kinds of epoch sub-transitions by processing the cases using the specified test handler.
+
+Information on the format of the tests can be found in the [epoch-processing test formats documentation](../../specs/test_formats/epoch_processing/README.md).
+
+ 
+

test_generators/epoch_processing/main.py

@@ -0,0 +1,38 @@
+from typing import Callable, Iterable
+
+from eth2spec.phase0 import spec
+from eth2spec.test.epoch_processing import (
+    test_process_crosslinks,
+    test_process_registry_updates
+)
+from gen_base import gen_runner, gen_suite, gen_typing
+from gen_from_tests.gen import generate_from_tests
+from preset_loader import loader
+
+
+def create_suite(transition_name: str, config_name: str, get_cases: Callable[[], Iterable[gen_typing.TestCase]]) \
+        -> Callable[[str], gen_typing.TestSuiteOutput]:
+    def suite_definition(configs_path: str) -> gen_typing.TestSuiteOutput:
+        presets = loader.load_presets(configs_path, config_name)
+        spec.apply_constants_preset(presets)
+
+        return ("%s_%s" % (transition_name, config_name), transition_name, gen_suite.render_suite(
+            title="%s epoch processing" % transition_name,
+            summary="Test suite for %s type epoch processing" % transition_name,
+            forks_timeline="testing",
+            forks=["phase0"],
+            config=config_name,
+            runner="epoch_processing",
+            handler=transition_name,
+            test_cases=get_cases()))
+
+    return suite_definition
+
+
+if __name__ == "__main__":
+    gen_runner.run_generator("epoch_processing", [
+        create_suite('crosslinks', 'minimal', lambda: generate_from_tests(test_process_crosslinks)),
+        create_suite('crosslinks', 'mainnet', lambda: generate_from_tests(test_process_crosslinks)),
+        create_suite('registry_updates', 'minimal', lambda: generate_from_tests(test_process_registry_updates)),
+        create_suite('registry_updates', 'mainnet', lambda: generate_from_tests(test_process_registry_updates)),
+    ])

test_generators/epoch_processing/requirements.txt

@@ -0,0 +1,4 @@
+eth-utils==1.6.0
+../../test_libs/gen_helpers
+../../test_libs/config_helpers
+../../test_libs/pyspec

test_generators/operations/README.md

@@ -3,7 +3,6 @@
 Operations (or "transactions" in previous spec iterations),
  are atomic changes to the state, introduced by embedding in blocks.
 
-This generator provides a series of test suites, divided into handler, for each operation type.
 An operation test-runner can consume these operation test-suites,
  and handle different kinds of operations by processing the cases using the specified test handler.
 

test_generators/operations/main.py

@@ -1,3 +1,5 @@
+from typing import Callable, Iterable
+
 from eth2spec.test.block_processing import (
     test_process_attestation,
     test_process_attester_slashing,
@@ -8,9 +10,29 @@ from eth2spec.test.block_processing import (
     test_process_voluntary_exit
 )
 
-from gen_base import gen_runner
+from gen_base import gen_runner, gen_suite, gen_typing
+from gen_from_tests.gen import generate_from_tests
+from preset_loader import loader
+from eth2spec.phase0 import spec
+
+
+def create_suite(operation_name: str, config_name: str, get_cases: Callable[[], Iterable[gen_typing.TestCase]]) \
+        -> Callable[[str], gen_typing.TestSuiteOutput]:
+    def suite_definition(configs_path: str) -> gen_typing.TestSuiteOutput:
+        presets = loader.load_presets(configs_path, config_name)
+        spec.apply_constants_preset(presets)
+
+        return ("%s_%s" % (operation_name, config_name), operation_name, gen_suite.render_suite(
+            title="%s operation" % operation_name,
+            summary="Test suite for %s type operation processing" % operation_name,
+            forks_timeline="testing",
+            forks=["phase0"],
+            config=config_name,
+            runner="operations",
+            handler=operation_name,
+            test_cases=get_cases()))
+    return suite_definition
 
-from suite_creator import generate_from_tests, create_suite
 
 if __name__ == "__main__":
     gen_runner.run_generator("operations", [

test_generators/operations/suite_creator.py

@@ -1,39 +0,0 @@
-from typing import Callable, Iterable
-from inspect import getmembers, isfunction
-from gen_base import gen_suite, gen_typing
-from preset_loader import loader
-from eth2spec.phase0 import spec
-
-
-def generate_from_tests(pkg):
-    fn_names = [
-        name for (name, _) in getmembers(pkg, isfunction)
-        if name.startswith('test_')
-    ]
-    out = []
-    print("generating test vectors from tests package: %s" % pkg.__name__)
-    for name in fn_names:
-        tfn = getattr(pkg, name)
-        try:
-            out.append(tfn(generator_mode=True, bls_active=True))
-        except AssertionError:
-            print("ERROR: failed to generate vector from test: %s (pkg: %s)" % (name, pkg.__name__))
-    return out
-
-
-def create_suite(operation_name: str, config_name: str, get_cases: Callable[[], Iterable[gen_typing.TestCase]])\
-        -> Callable[[str], gen_typing.TestSuiteOutput]:
-    def suite_definition(configs_path: str) -> gen_typing.TestSuiteOutput:
-        presets = loader.load_presets(configs_path, config_name)
-        spec.apply_constants_preset(presets)
-
-        return ("%s_%s" % (operation_name, config_name), operation_name, gen_suite.render_suite(
-            title="%s operation" % operation_name,
-            summary="Test suite for %s type operation processing" % operation_name,
-            forks_timeline="testing",
-            forks=["phase0"],
-            config=config_name,
-            runner="operations",
-            handler=operation_name,
-            test_cases=get_cases()))
-    return suite_definition

test_generators/sanity/README.md

@@ -0,0 +1,8 @@
+# Sanity tests
+
+Sanity tests cover regular state-transitions in a common block-list format, to ensure the basics work.
+
+Information on the format of the tests can be found in the [sanity test formats documentation](../../specs/test_formats/sanity/README.md).
+
+ 
+

test_generators/sanity/main.py

@@ -0,0 +1,35 @@
+from typing import Callable, Iterable
+
+from eth2spec.test.sanity import test_blocks, test_slots
+
+from gen_base import gen_runner, gen_suite, gen_typing
+from gen_from_tests.gen import generate_from_tests
+from preset_loader import loader
+from eth2spec.phase0 import spec
+
+
+def create_suite(handler_name: str, config_name: str, get_cases: Callable[[], Iterable[gen_typing.TestCase]]) \
+        -> Callable[[str], gen_typing.TestSuiteOutput]:
+    def suite_definition(configs_path: str) -> gen_typing.TestSuiteOutput:
+        presets = loader.load_presets(configs_path, config_name)
+        spec.apply_constants_preset(presets)
+
+        return ("%sanity_s_%s" % (handler_name, config_name), handler_name, gen_suite.render_suite(
+            title="sanity testing",
+            summary="Sanity test suite, %s type, generated from pytests" % handler_name,
+            forks_timeline="testing",
+            forks=["phase0"],
+            config=config_name,
+            runner="sanity",
+            handler=handler_name,
+            test_cases=get_cases()))
+    return suite_definition
+
+
+if __name__ == "__main__":
+    gen_runner.run_generator("sanity", [
+        create_suite('blocks', 'minimal', lambda: generate_from_tests(test_blocks)),
+        create_suite('blocks', 'mainnet', lambda: generate_from_tests(test_blocks)),
+        create_suite('slots', 'minimal', lambda: generate_from_tests(test_slots)),
+        create_suite('slots', 'mainnet', lambda: generate_from_tests(test_slots)),
+    ])

test_generators/sanity/requirements.txt

@@ -0,0 +1,4 @@
+eth-utils==1.6.0
+../../test_libs/gen_helpers
+../../test_libs/config_helpers
+../../test_libs/pyspec

test_libs/gen_helpers/gen_from_tests/gen.py

@@ -0,0 +1,25 @@
+from inspect import getmembers, isfunction
+
+def generate_from_tests(src, bls_active=True):
+    """
+    Generate a list of test cases by running tests from the given src in generator-mode.
+    :param src: to retrieve tests from (discovered using inspect.getmembers)
+    :param bls_active: optional, to override BLS switch preference. Defaults to True.
+    :return: the list of test cases.
+    """
+    fn_names = [
+        name for (name, _) in getmembers(src, isfunction)
+        if name.startswith('test_')
+    ]
+    out = []
+    print("generating test vectors from tests source: %s" % src.__name__)
+    for name in fn_names:
+        tfn = getattr(src, name)
+        try:
+            test_case = tfn(generator_mode=True, bls_active=bls_active)
+            # If no test case data is returned, the test is ignored.
+            if test_case is not None:
+                out.append(test_case)
+        except AssertionError:
+            print("ERROR: failed to generate vector from test: %s (src: %s)" % (name, src.__name__))
+    return out

test_libs/gen_helpers/setup.py

@@ -2,7 +2,7 @@ from distutils.core import setup
 
 setup(
     name='gen_helpers',
-    packages=['gen_base'],
+    packages=['gen_base', 'gen_from_tests'],
     install_requires=[
         "ruamel.yaml==0.15.96",
         "eth-utils==1.6.0"

test_libs/pyspec/eth2spec/test/block_processing/test_process_attestation.py

@@ -75,7 +75,7 @@ def test_success_previous_epoch(state):
 @always_bls
 @spec_state_test
 def test_invalid_attestation_signature(state):
-    attestation = get_valid_attestation(state, signed=False)
+    attestation = get_valid_attestation(state)
     state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY
 
     yield from run_attestation_processing(state, attestation, False)
@@ -105,7 +105,7 @@ def test_old_source_epoch(state):
     state.finalized_epoch = 2
     state.previous_justified_epoch = 3
     state.current_justified_epoch = 4
-    attestation = get_valid_attestation(state, slot=(spec.SLOTS_PER_EPOCH * 3) + 1, signed=False)
+    attestation = get_valid_attestation(state, slot=(spec.SLOTS_PER_EPOCH * 3) + 1)
 
     # test logic sanity check: make sure the attestation is pointing to oldest known source epoch
     assert attestation.data.source_epoch == state.previous_justified_epoch
@@ -120,7 +120,7 @@ def test_old_source_epoch(state):
 
 @spec_state_test
 def test_wrong_shard(state):
-    attestation = get_valid_attestation(state, signed=False)
+    attestation = get_valid_attestation(state)
     state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY
 
     attestation.data.shard += 1
@@ -132,7 +132,7 @@ def test_wrong_shard(state):
 
 @spec_state_test
 def test_new_source_epoch(state):
-    attestation = get_valid_attestation(state, signed=False)
+    attestation = get_valid_attestation(state)
     state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY
 
     attestation.data.source_epoch += 1
@@ -144,7 +144,7 @@ def test_new_source_epoch(state):
 
 @spec_state_test
 def test_source_root_is_target_root(state):
-    attestation = get_valid_attestation(state, signed=False)
+    attestation = get_valid_attestation(state)
     state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY
 
     attestation.data.source_root = attestation.data.target_root
@@ -165,7 +165,7 @@ def test_invalid_current_source_root(state):
     state.current_justified_epoch = 4
     state.current_justified_root = b'\xff' * 32
 
-    attestation = get_valid_attestation(state, slot=(spec.SLOTS_PER_EPOCH * 3) + 1, signed=False)
+    attestation = get_valid_attestation(state, slot=(spec.SLOTS_PER_EPOCH * 3) + 1)
     state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY
 
     # Test logic sanity checks:
@@ -182,7 +182,7 @@ def test_invalid_current_source_root(state):
 
 @spec_state_test
 def test_bad_source_root(state):
-    attestation = get_valid_attestation(state, signed=False)
+    attestation = get_valid_attestation(state)
     state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY
 
     attestation.data.source_root = b'\x42' * 32
@@ -194,7 +194,7 @@ def test_bad_source_root(state):
 
 @spec_state_test
 def test_non_zero_crosslink_data_root(state):
-    attestation = get_valid_attestation(state, signed=False)
+    attestation = get_valid_attestation(state)
     state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY
 
     attestation.data.crosslink_data_root = b'\x42' * 32
@@ -221,7 +221,7 @@ def test_bad_previous_crosslink(state):
 
 @spec_state_test
 def test_inconsistent_bitfields(state):
-    attestation = get_valid_attestation(state, signed=False)
+    attestation = get_valid_attestation(state)
     state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY
 
     attestation.custody_bitfield = deepcopy(attestation.aggregation_bitfield) + b'\x00'
@@ -233,7 +233,7 @@ def test_inconsistent_bitfields(state):
 
 @spec_state_test
 def test_non_empty_custody_bitfield(state):
-    attestation = get_valid_attestation(state, signed=False)
+    attestation = get_valid_attestation(state)
     state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY
 
     attestation.custody_bitfield = deepcopy(attestation.aggregation_bitfield)
@@ -245,7 +245,7 @@ def test_non_empty_custody_bitfield(state):
 
 @spec_state_test
 def test_empty_aggregation_bitfield(state):
-    attestation = get_valid_attestation(state, signed=False)
+    attestation = get_valid_attestation(state)
     state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY
 
     attestation.aggregation_bitfield = b'\x00' * len(attestation.aggregation_bitfield)

test_libs/pyspec/eth2spec/test/block_processing/test_process_block_header.py

@@ -50,13 +50,13 @@ def test_success_block_header(state):
 @always_bls
 @spec_state_test
 def test_invalid_sig_block_header(state):
-    block = build_empty_block_for_next_slot(state, signed=False)
+    block = build_empty_block_for_next_slot(state)
     yield from run_block_header_processing(state, block, valid=False)
 
 
 @spec_state_test
 def test_invalid_slot_block_header(state):
-    block = build_empty_block_for_next_slot(state, signed=False)
+    block = build_empty_block_for_next_slot(state)
     block.slot = state.slot + 2  # invalid slot
     sign_block(state, block)
 
@@ -65,7 +65,7 @@ def test_invalid_slot_block_header(state):
 
 @spec_state_test
 def test_invalid_previous_block_root(state):
-    block = build_empty_block_for_next_slot(state, signed=False)
+    block = build_empty_block_for_next_slot(state)
     block.previous_block_root = b'\12' * 32  # invalid prev root
     sign_block(state, block)
 

test_libs/pyspec/eth2spec/test/block_processing/test_process_deposit.py

@@ -69,7 +69,7 @@ def test_invalid_sig_new_deposit(state):
     # fresh deposit = next validator index = validator appended to registry
     validator_index = len(state.validator_registry)
     amount = spec.MAX_EFFECTIVE_BALANCE
-    deposit = prepare_state_and_deposit(state, validator_index, amount, signed=False)
+    deposit = prepare_state_and_deposit(state, validator_index, amount)
     yield from run_deposit_processing(state, deposit, validator_index, valid=True, effective=False)
 
 
@@ -87,7 +87,7 @@ def test_success_top_up(state):
 def test_invalid_sig_top_up(state):
     validator_index = 0
     amount = spec.MAX_EFFECTIVE_BALANCE // 4
-    deposit = prepare_state_and_deposit(state, validator_index, amount, signed=False)
+    deposit = prepare_state_and_deposit(state, validator_index, amount)
 
     # invalid signatures, in top-ups, are allowed!
     yield from run_deposit_processing(state, deposit, validator_index, valid=True, effective=True)
@@ -97,7 +97,7 @@ def test_invalid_sig_top_up(state):
 def test_wrong_index(state):
     validator_index = len(state.validator_registry)
     amount = spec.MAX_EFFECTIVE_BALANCE
-    deposit = prepare_state_and_deposit(state, validator_index, amount, signed=False)
+    deposit = prepare_state_and_deposit(state, validator_index, amount)
 
     # mess up deposit_index
     deposit.index = state.deposit_index + 1
@@ -114,7 +114,7 @@ def test_wrong_index(state):
 def test_bad_merkle_proof(state):
     validator_index = len(state.validator_registry)
     amount = spec.MAX_EFFECTIVE_BALANCE
-    deposit = prepare_state_and_deposit(state, validator_index, amount, signed=False)
+    deposit = prepare_state_and_deposit(state, validator_index, amount)
 
     # mess up merkle branch
     deposit.proof[-1] = spec.ZERO_HASH

test_libs/pyspec/eth2spec/test/block_processing/test_process_transfer.py

@@ -86,7 +86,7 @@ def test_success_active_above_max_effective_fee(state):
 @always_bls
 @spec_state_test
 def test_invalid_signature(state):
-    transfer = get_valid_transfer(state, signed=False)
+    transfer = get_valid_transfer(state)
     # un-activate so validator can transfer
     state.validator_registry[transfer.sender].activation_eligibility_epoch = spec.FAR_FUTURE_EPOCH
     

test_libs/pyspec/eth2spec/test/block_processing/test_process_voluntary_exit.py

@@ -62,7 +62,7 @@ def test_invalid_signature(state):
     validator_index = get_active_validator_indices(state, current_epoch)[0]
     privkey = pubkey_to_privkey[state.validator_registry[validator_index].pubkey]
 
-    voluntary_exit = build_voluntary_exit(state, current_epoch, validator_index, privkey, signed=False)
+    voluntary_exit = build_voluntary_exit(state, current_epoch, validator_index, privkey)
 
     yield from run_voluntary_exit_processing(state, voluntary_exit, False)
 

test_libs/pyspec/eth2spec/test/epoch_processing/test_process_crosslinks.py

@@ -21,6 +21,7 @@ from eth2spec.test.helpers.attestations import (
     fill_aggregate_attestation,
     get_crosslink_committee,
     get_valid_attestation,
+    sign_attestation,
 )
 
 
@@ -33,7 +34,7 @@ def run_process_crosslinks(state, valid=True):
     """
     # transition state to slot before state transition
     slot = state.slot + (spec.SLOTS_PER_EPOCH - state.slot % spec.SLOTS_PER_EPOCH) - 1
-    block = build_empty_block_for_next_slot(state, signed=False)
+    block = build_empty_block_for_next_slot(state)
     block.slot = slot
     sign_block(state, block)
     state_transition(state, block)
@@ -103,19 +104,24 @@ def test_single_crosslink_update_from_previous_epoch(state):
 
 @spec_state_test
 def test_double_late_crosslink(state):
+    if spec.get_epoch_committee_count(state, spec.get_current_epoch(state)) < spec.SHARD_COUNT:
+        print("warning: ignoring test, test-assumptions are incompatible with configuration")
+        return
+
     next_epoch(state)
     state.slot += 4
 
     attestation_1 = get_valid_attestation(state, signed=True)
     fill_aggregate_attestation(state, attestation_1)
 
-    # add attestation_1 in the next epoch
+    # add attestation_1 to next epoch
     next_epoch(state)
     add_attestation_to_state(state, attestation_1, state.slot + 1)
 
     for slot in range(spec.SLOTS_PER_EPOCH):
-        attestation_2 = get_valid_attestation(state, signed=True)
+        attestation_2 = get_valid_attestation(state)
         if attestation_2.data.shard == attestation_1.data.shard:
+            sign_attestation(state, attestation_2)
             break
         next_slot(state)
     apply_empty_block(state)

test_libs/pyspec/eth2spec/test/epoch_processing/test_process_registry_updates.py

@@ -3,11 +3,41 @@ import eth2spec.phase0.spec as spec
 from eth2spec.phase0.spec import (
     get_current_epoch,
     is_active_validator,
+    process_registry_updates
 )
+from eth2spec.phase0.state_transition import state_transition
+from eth2spec.test.helpers.block import build_empty_block_for_next_slot, sign_block
 from eth2spec.test.helpers.state import next_epoch
 from eth2spec.test.context import spec_state_test
 
 
+def run_process_registry_updates(state, valid=True):
+    """
+    Run ``process_crosslinks``, yielding:
+      - pre-state ('pre')
+      - post-state ('post').
+    If ``valid == False``, run expecting ``AssertionError``
+    """
+    # transition state to slot before state transition
+    slot = state.slot + (spec.SLOTS_PER_EPOCH - state.slot % spec.SLOTS_PER_EPOCH) - 1
+    block = build_empty_block_for_next_slot(state)
+    block.slot = slot
+    sign_block(state, block)
+    state_transition(state, block)
+
+    # cache state before epoch transition
+    spec.cache_state(state)
+
+    # process components of epoch transition before registry update
+    spec.process_justification_and_finalization(state)
+    spec.process_crosslinks(state)
+    spec.process_rewards_and_penalties(state)
+
+    yield 'pre', state
+    process_registry_updates(state)
+    yield 'post', state
+
+
 @spec_state_test
 def test_activation(state):
     index = 0
@@ -19,17 +49,10 @@ def test_activation(state):
     state.validator_registry[index].effective_balance = spec.MAX_EFFECTIVE_BALANCE
     assert not is_active_validator(state.validator_registry[index], get_current_epoch(state))
 
-    yield 'pre', state
-
-    blocks = []
     for _ in range(spec.ACTIVATION_EXIT_DELAY + 1):
-        block = next_epoch(state)
-        blocks.append(block)
+        next_epoch(state)
 
-    # provide extra type hinting here, since it is wrapped in a list.
-    yield 'blocks', blocks, [spec.BeaconBlock]
-
-    yield 'post', state
+    yield from run_process_registry_updates(state)
 
     assert state.validator_registry[index].activation_eligibility_epoch != spec.FAR_FUTURE_EPOCH
     assert state.validator_registry[index].activation_epoch != spec.FAR_FUTURE_EPOCH
@@ -48,17 +71,10 @@ def test_ejection(state):
     # Mock an ejection
     state.validator_registry[index].effective_balance = spec.EJECTION_BALANCE
 
-    yield 'pre', state
-
-    blocks = []
     for _ in range(spec.ACTIVATION_EXIT_DELAY + 1):
-        block = next_epoch(state)
-        blocks.append(block)
+        next_epoch(state)
 
-    # provide extra type hinting here, since it is wrapped in a list.
-    yield 'blocks', blocks, [spec.BeaconBlock]
-
-    yield 'post', state
+    yield from run_process_registry_updates(state)
 
     assert state.validator_registry[index].exit_epoch != spec.FAR_FUTURE_EPOCH
     assert not is_active_validator(

test_libs/pyspec/eth2spec/test/helpers/attestations.py

@@ -138,7 +138,7 @@ def fill_aggregate_attestation(state, attestation):
 
 
 def add_attestation_to_state(state, attestation, slot):
-    block = build_empty_block_for_next_slot(state, signed=False)
+    block = build_empty_block_for_next_slot(state)
     block.slot = slot
     block.body.attestations.append(attestation)
     state_transition_to(state, block.slot)

test_libs/pyspec/eth2spec/test/sanity/test_blocks.py

@@ -13,42 +13,20 @@ from eth2spec.phase0.spec import (
     get_block_root_at_slot,
     get_current_epoch,
     get_domain,
-    advance_slot,
-    cache_state,
 )
 from eth2spec.phase0.state_transition import (
     state_transition,
 )
-from .helpers.state import (
-    get_balance,
-    get_state_root
-)
-from .helpers.transfers import get_valid_transfer
-from .helpers.block import build_empty_block_for_next_slot, sign_block
-from .helpers.keys import (
-    privkeys,
-    pubkeys,
-)
-from .helpers.attester_slashings import get_valid_attester_slashing
-from .helpers.proposer_slashings import get_valid_proposer_slashing
-from .helpers.attestations import get_valid_attestation
-from .helpers.deposits import prepare_state_and_deposit
+from eth2spec.test.helpers.state import get_balance
+from eth2spec.test.helpers.transfers import get_valid_transfer
+from eth2spec.test.helpers.block import build_empty_block_for_next_slot, sign_block
+from eth2spec.test.helpers.keys import privkeys, pubkeys
+from eth2spec.test.helpers.attester_slashings import get_valid_attester_slashing
+from eth2spec.test.helpers.proposer_slashings import get_valid_proposer_slashing
+from eth2spec.test.helpers.attestations import get_valid_attestation
+from eth2spec.test.helpers.deposits import prepare_state_and_deposit
 
-from .context import spec_state_test, never_bls
-
-
-@spec_state_test
-def test_slot_transition(state):
-    pre_slot = state.slot
-    pre_root = state.hash_tree_root()
-    yield 'pre', state
-
-    cache_state(state)
-    advance_slot(state)
-    yield 'post', state
-
-    assert state.slot == pre_slot + 1
-    assert get_state_root(state, pre_slot) == pre_root
+from eth2spec.test.context import spec_state_test, never_bls
 
 
 @never_bls
@@ -75,7 +53,7 @@ def test_skipped_slots(state):
     pre_slot = state.slot
     yield 'pre', state
 
-    block = build_empty_block_for_next_slot(state, signed=False)
+    block = build_empty_block_for_next_slot(state)
     block.slot += 3
     sign_block(state, block)
     yield 'blocks', [block], [spec.BeaconBlock]
@@ -93,7 +71,7 @@ def test_empty_epoch_transition(state):
     pre_slot = state.slot
     yield 'pre', state
 
-    block = build_empty_block_for_next_slot(state, signed=False)
+    block = build_empty_block_for_next_slot(state)
     block.slot += spec.SLOTS_PER_EPOCH
     sign_block(state, block)
     yield 'blocks', [block], [spec.BeaconBlock]
@@ -112,7 +90,7 @@ def test_empty_epoch_transition(state):
 #     pre_state = deepcopy(state)
 #     yield 'pre', state
 #
-#     block = build_empty_block_for_next_slot(state, signed=False)
+#     block = build_empty_block_for_next_slot(state)
 #     block.slot += spec.SLOTS_PER_EPOCH * 5
 #     sign_block(state, block, proposer_index=0)
 #     yield 'blocks', [block], [spec.BeaconBlock]
@@ -140,7 +118,7 @@ def test_proposer_slashing(state):
     #
     # Add to state via block transition
     #
-    block = build_empty_block_for_next_slot(state, signed=False)
+    block = build_empty_block_for_next_slot(state)
     block.body.proposer_slashings.append(proposer_slashing)
     sign_block(state, block)
     yield 'blocks', [block], [spec.BeaconBlock]
@@ -173,7 +151,7 @@ def test_attester_slashing(state):
     #
     # Add to state via block transition
     #
-    block = build_empty_block_for_next_slot(state, signed=False)
+    block = build_empty_block_for_next_slot(state)
     block.body.attester_slashings.append(attester_slashing)
     sign_block(state, block)
     yield 'blocks', [block], [spec.BeaconBlock]
@@ -209,7 +187,7 @@ def test_deposit_in_block(state):
 
     yield 'pre', state
 
-    block = build_empty_block_for_next_slot(state, signed=False)
+    block = build_empty_block_for_next_slot(state)
     block.body.deposits.append(deposit)
     sign_block(state, block)
 
@@ -236,7 +214,7 @@ def test_deposit_top_up(state):
 
     yield 'pre', state
 
-    block = build_empty_block_for_next_slot(state, signed=False)
+    block = build_empty_block_for_next_slot(state)
     block.body.deposits.append(deposit)
     sign_block(state, block)
 
@@ -260,7 +238,7 @@ def test_attestation(state):
 
     # Add to state via block transition
     pre_current_attestations_len = len(state.current_epoch_attestations)
-    attestation_block = build_empty_block_for_next_slot(state, signed=False)
+    attestation_block = build_empty_block_for_next_slot(state)
     attestation_block.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY
     attestation_block.body.attestations.append(attestation)
     sign_block(state, attestation_block)
@@ -271,7 +249,7 @@ def test_attestation(state):
     # Epoch transition should move to previous_epoch_attestations
     pre_current_attestations_root = spec.hash_tree_root(state.current_epoch_attestations)
 
-    epoch_block = build_empty_block_for_next_slot(state, signed=False)
+    epoch_block = build_empty_block_for_next_slot(state)
     epoch_block.slot += spec.SLOTS_PER_EPOCH
     sign_block(state, epoch_block)
     state_transition(state, epoch_block)
@@ -309,7 +287,7 @@ def test_voluntary_exit(state):
     )
 
     # Add to state via block transition
-    initiate_exit_block = build_empty_block_for_next_slot(state, signed=False)
+    initiate_exit_block = build_empty_block_for_next_slot(state)
     initiate_exit_block.body.voluntary_exits.append(voluntary_exit)
     sign_block(state, initiate_exit_block)
     state_transition(state, initiate_exit_block)
@@ -317,7 +295,7 @@ def test_voluntary_exit(state):
     assert state.validator_registry[validator_index].exit_epoch < spec.FAR_FUTURE_EPOCH
 
     # Process within epoch transition
-    exit_block = build_empty_block_for_next_slot(state, signed=False)
+    exit_block = build_empty_block_for_next_slot(state)
     exit_block.slot += spec.SLOTS_PER_EPOCH
     sign_block(state, exit_block)
     state_transition(state, exit_block)
@@ -346,7 +324,7 @@ def test_transfer(state):
     yield 'pre', state
 
     # Add to state via block transition
-    block = build_empty_block_for_next_slot(state, signed=False)
+    block = build_empty_block_for_next_slot(state)
     block.body.transfers.append(transfer)
     sign_block(state, block)
 
@@ -374,7 +352,7 @@ def test_balance_driven_status_transitions(state):
     yield 'pre', state
 
     # trigger epoch transition
-    block = build_empty_block_for_next_slot(state, signed=False)
+    block = build_empty_block_for_next_slot(state)
     block.slot += spec.SLOTS_PER_EPOCH
     sign_block(state, block)
     state_transition(state, block)
@@ -412,13 +390,13 @@ def test_historical_batch(state):
 #
 #     blocks = []
 #     for _ in range(spec.SLOTS_PER_ETH1_VOTING_PERIOD - 1):
-#         block = build_empty_block_for_next_slot(state, signed=False)
+#         block = build_empty_block_for_next_slot(state)
 #         state_transition(state, block)
 #         expected_votes += 1
 #         assert len(state.eth1_data_votes) == expected_votes
 #         blocks.append(block)
 #
-#     block = build_empty_block_for_next_slot(state, signed=False)
+#     block = build_empty_block_for_next_slot(state)
 #     blocks.append(block)
 #
 #     state_transition(state, block)

test_libs/pyspec/eth2spec/test/sanity/test_slots.py

@@ -0,0 +1,58 @@
+import eth2spec.phase0.spec as spec
+
+from eth2spec.phase0.state_transition import state_transition_to
+from eth2spec.test.helpers.state import get_state_root
+from eth2spec.test.context import spec_state_test
+
+
+@spec_state_test
+def test_slots_1(state):
+    pre_slot = state.slot
+    pre_root = state.hash_tree_root()
+    yield 'pre', state
+
+    slots = 1
+    yield 'slots', slots
+    state_transition_to(state, state.slot + slots)
+
+    yield 'post', state
+    assert state.slot == pre_slot + 1
+    assert get_state_root(state, pre_slot) == pre_root
+
+
+@spec_state_test
+def test_slots_2(state):
+    yield 'pre', state
+    slots = 2
+    yield 'slots', slots
+    state_transition_to(state, state.slot + slots)
+    yield 'post', state
+
+
+@spec_state_test
+def test_empty_epoch(state):
+    yield 'pre', state
+    slots = spec.SLOTS_PER_EPOCH
+    yield 'slots', slots
+    state_transition_to(state, state.slot + slots)
+    yield 'post', state
+
+
+@spec_state_test
+def test_double_empty_epoch(state):
+    yield 'pre', state
+    slots = spec.SLOTS_PER_EPOCH * 2
+    yield 'slots', slots
+    state_transition_to(state, state.slot + slots)
+    yield 'post', state
+
+
+@spec_state_test
+def test_over_epoch_boundary(state):
+    state_transition_to(state, state.slot + (spec.SLOTS_PER_EPOCH // 2))
+    yield 'pre', state
+    slots = spec.SLOTS_PER_EPOCH
+    yield 'slots', slots
+    state_transition_to(state, state.slot + slots)
+    yield 'post', state
+

test_libs/pyspec/eth2spec/test/utils.py

@@ -19,8 +19,10 @@ def spectest(description: str = None):
                 else:
                     # description can be explicit
                     out['description'] = description
+                has_contents = False
                 # put all generated data into a dict.
                 for data in fn(*args, **kw):
+                    has_contents = True
                     # If there is a type argument, encode it as that type.
                     if len(data) == 3:
                         (key, value, typ) = data
@@ -32,11 +34,15 @@ def spectest(description: str = None):
                             out[key] = encode(value, value.__class__)
                         else:
                             out[key] = value
-                return out
+                if has_contents:
+                    return out
+                else:
+                    return None
             else:
                 # just complete the function, ignore all yielded data, we are not using it
                 for _ in fn(*args, **kw):
                     continue
+                return None
         return entry
     return runner
 
@@ -54,7 +60,7 @@ def with_tags(tags: Dict[str, Any]):
             fn_out = fn(*args, **kw)
             # do not add tags if the function is not returning a dict at all (i.e. not in generator mode)
             if fn_out is None:
-                return fn_out
+                return None
             return {**tags, **fn_out}
         return entry
     return runner