Merge pull request #1238 from ethereum/update-generators

Update test vector generators

specs/test_formats/bls/msg_hash_g2_compressed.md

@@ -7,7 +7,7 @@ A BLS compressed-hash to G2.
 ```yaml
 input: 
   message: bytes32,
-  domain: bytes -- any number
+  domain: uint64 -- the BLS domain
 output: List[bytes48] -- length of two
 ```
 

specs/test_formats/bls/msg_hash_g2_uncompressed.md

@@ -6,8 +6,8 @@ A BLS uncompressed-hash to G2.
 
 ```yaml
 input: 
-  message: bytes32,
+  message: bytes32
-  domain: bytes -- any number
+  domain: uint64 -- the BLS domain
 output: List[List[bytes48]] -- 3 lists, each a length of two
 ```
 

specs/test_formats/bls/sign_msg.md

@@ -8,7 +8,7 @@ Message signing with BLS should produce a signature.
 input:
   privkey: bytes32 -- the private key used for signing
   message: bytes32 -- input message to sign (a hash)
-  domain: bytes    -- BLS domain
+  domain: uint64 -- the BLS domain
 output: bytes96    -- expected signature
 ```
 

specs/test_formats/epoch_processing/README.md

@@ -17,13 +17,17 @@ post: BeaconState      -- state after applying the epoch sub-transition.
 ## Condition
 
 A handler of the `epoch_processing` test-runner should process these cases, 
- calling the corresponding processing implementation.
+ calling the corresponding processing implementation (same name, prefixed with `process_`).
+This excludes the other parts of the epoch-transition.
+The provided pre-state is already transitioned to just before the specific sub-transition of focus of the handler.
 
 Sub-transitions:
 
-| *`sub-transition-name`* | *`processing call`*               |
+- `justification_and_finalization`
-|-------------------------|-----------------------------------|
+- `crosslinks`
-| `crosslinks`            | `process_crosslinks(state)`       |
+- *`rewards_and_penalties` - planned testing extension*
-| `registry_updates`      | `process_registry_updates(state)` |
+- `registry_updates`
+- `slashings`
+- `final_updates`
 
 The resulting state should match the expected `post` state.

specs/test_formats/genesis/README.md

@@ -0,0 +1,8 @@
+# Genesis tests
+
+The aim of the genesis tests is to provide a baseline to test genesis-state initialization and test
+ if the proposed genesis-validity conditions are working.
+
+There are two handlers, documented individually:
+- [`validity`](./validity.md): Tests if a genesis state is valid, i.e. if it counts as trigger to launch.
+- [`initialization`](./initialization.md): Tests the initialization of a genesis state based on Eth1 data.

specs/test_formats/genesis/initialization.md

@@ -0,0 +1,22 @@
+# Genesis creation testing
+
+Tests the initialization of a genesis state based on Eth1 data.
+
+## Test case format
+
+```yaml
+description: string            -- description of test case, purely for debugging purposes
+bls_setting: int               -- see general test-format spec.
+eth1_block_hash: Bytes32       -- the root of the Eth-1 block, hex encoded, with prefix 0x
+eth1_timestamp: int            -- the timestamp of the block, in seconds.
+deposits: [Deposit]            -- list of deposits to build the genesis state with
+state: BeaconState             -- the expected genesis state.
+```
+
+To process this test, build a genesis state with the provided `eth1_block_hash`, `eth1_timestamp` and `deposits`:
+`initialize_beacon_state_from_eth1(eth1_block_hash, eth1_timestamp, deposits)`,
+ as described in the Beacon Chain specification.
+
+## Condition
+
+The resulting state should match the expected `state`.

specs/test_formats/genesis/validity.md

@@ -0,0 +1,19 @@
+# Genesis validity testing
+
+Tests if a genesis state is valid, i.e. if it counts as trigger to launch.
+
+## Test case format
+
+```yaml
+description: string    -- description of test case, purely for debugging purposes
+bls_setting: int       -- see general test-format spec.
+genesis: BeaconState   -- state to validate.
+is_valid: bool         -- true if the genesis state is deemed valid as to launch with, false otherwise.
+```
+
+To process the data, call `is_valid_genesis_state(genesis)`.
+
+
+## Condition
+
+The result of calling `is_valid_genesis_state(genesis)` should match the expected `is_valid` boolean.

specs/test_formats/operations/README.md

@@ -16,13 +16,14 @@ post: BeaconState                      -- state after applying the operation. No
 
 A handler of the `operations` test-runner should process these cases,
  calling the corresponding processing implementation.
+This excludes the other parts of the block-transition.
 
 Operations:
 
 | *`operation-name`*      | *`operation-object`* | *`input name`*       | *`processing call`*                                    |
 |-------------------------|----------------------|----------------------|--------------------------------------------------------|
-| `attestation`           | `Attestation`        | `attestation`        | `process_attestation(state, attestation)`                  |
+| `attestation`           | `Attestation`        | `attestation`        | `process_attestation(state, attestation)`              |
-| `attester_slashing`     | `AttesterSlashing`   | `attester_slashing`  | `process_attester_slashing(state, attester_slashing)`            |
+| `attester_slashing`     | `AttesterSlashing`   | `attester_slashing`  | `process_attester_slashing(state, attester_slashing)`  |
 | `block_header`          | `Block`              | `block`              | `process_block_header(state, block)`                   |
 | `deposit`               | `Deposit`            | `deposit`            | `process_deposit(state, deposit)`                      |
 | `proposer_slashing`     | `ProposerSlashing`   | `proposer_slashing`  | `process_proposer_slashing(state, proposer_slashing)`  |

specs/test_formats/sanity/slots.md

@@ -12,11 +12,10 @@ slots: N               -- amount of slots to process, N being a positive number.
 post: BeaconState      -- state after applying all the transitions.
 ```
 
-The transition with pure time, no blocks, is known as `state_transition_to(state, slot)` in the spec.
+The transition with pure time, no blocks, is known as `process_slots(state, slot)` in the spec.
 This runs state-caching (pure slot transition) and epoch processing (every E slots).
 
-To process the data, call `state_transition_to(pre, pre.slot + N)`. And see if `pre` mutated into the equivalent of `post`.
+To process the data, call `process_slots(pre, pre.slot + N)`.
-
 
 ## Condition
 

test_generators/bls/main.py

@@ -27,9 +27,6 @@ def hex_to_int(x: str) -> int:
     return int(x, 16)
 
 
-# Note: even though a domain is only an uint64,
-# To avoid issues with YAML parsers that are limited to 53-bit (JS language limit)
-# It is serialized as an hex string as well.
 DOMAINS = [
     0,
     1,
@@ -92,7 +89,7 @@ def case01_message_hash_G2_uncompressed():
             yield {
                 'input': {
                     'message': '0x' + msg.hex(),
-                    'domain': int_to_hex(domain)
+                    'domain': domain
                 },
                 'output': hash_message(msg, domain)
             }
@@ -104,7 +101,7 @@ def case02_message_hash_G2_compressed():
             yield {
                 'input': {
                     'message': '0x' + msg.hex(),
-                    'domain': int_to_hex(domain)
+                    'domain': domain
                 },
                 'output': hash_message_compressed(msg, domain)
             }
@@ -129,7 +126,7 @@ def case04_sign_messages():
                     'input': {
                         'privkey': int_to_hex(privkey),
                         'message': '0x' + message.hex(),
-                        'domain': int_to_hex(domain)
+                        'domain': domain
                     },
                     'output': '0x' + sig.hex()
                 }

test_generators/epoch_processing/main.py

@@ -4,7 +4,10 @@ from eth2spec.phase0 import spec as spec_phase0
 from eth2spec.phase1 import spec as spec_phase1
 from eth2spec.test.phase_0.epoch_processing import (
     test_process_crosslinks,
-    test_process_registry_updates
+    test_process_final_updates,
+    test_process_justification_and_finalization,
+    test_process_registry_updates,
+    test_process_slashings
 )
 from gen_base import gen_runner, gen_suite, gen_typing
 from gen_from_tests.gen import generate_from_tests
@@ -35,8 +38,16 @@ if __name__ == "__main__":
     gen_runner.run_generator("epoch_processing", [
         create_suite('crosslinks', 'minimal', lambda: generate_from_tests(test_process_crosslinks, 'phase0')),
         create_suite('crosslinks', 'mainnet', lambda: generate_from_tests(test_process_crosslinks, 'phase0')),
+        create_suite('final_updates', 'minimal', lambda: generate_from_tests(test_process_final_updates, 'phase0')),
+        create_suite('final_updates', 'mainnet', lambda: generate_from_tests(test_process_final_updates, 'phase0')),
+        create_suite('justification_and_finalization', 'minimal',
+                     lambda: generate_from_tests(test_process_justification_and_finalization, 'phase0')),
+        create_suite('justification_and_finalization', 'mainnet',
+                     lambda: generate_from_tests(test_process_justification_and_finalization, 'phase0')),
         create_suite('registry_updates', 'minimal',
                      lambda: generate_from_tests(test_process_registry_updates, 'phase0')),
         create_suite('registry_updates', 'mainnet',
                      lambda: generate_from_tests(test_process_registry_updates, 'phase0')),
+        create_suite('slashings', 'minimal', lambda: generate_from_tests(test_process_slashings, 'phase0')),
+        create_suite('slashings', 'mainnet', lambda: generate_from_tests(test_process_slashings, 'phase0')),
     ])

test_generators/genesis/README.md

@@ -0,0 +1,8 @@
+# Genesis test generator
+
+Genesis tests cover the initialization and validity-based launch trigger for the Beacon Chain genesis state.
+
+Information on the format of the tests can be found in the [genesis test formats documentation](../../specs/test_formats/genesis/README.md).
+
+ 
+

test_generators/genesis/main.py

@@ -0,0 +1,33 @@
+from typing import Callable, Iterable
+
+from eth2spec.test.genesis import test_initialization, test_validity
+
+from gen_base import gen_runner, gen_suite, gen_typing
+from gen_from_tests.gen import generate_from_tests
+from preset_loader import loader
+from eth2spec.phase0 import spec as spec
+
+
+def create_suite(handler_name: str, config_name: str, get_cases: Callable[[], Iterable[gen_typing.TestCase]]) \
+        -> Callable[[str], gen_typing.TestSuiteOutput]:
+    def suite_definition(configs_path: str) -> gen_typing.TestSuiteOutput:
+        presets = loader.load_presets(configs_path, config_name)
+        spec.apply_constants_preset(presets)
+
+        return ("genesis_%s_%s" % (handler_name, config_name), handler_name, gen_suite.render_suite(
+            title="genesis testing",
+            summary="Genesis test suite, %s type, generated from pytests" % handler_name,
+            forks_timeline="testing",
+            forks=["phase0"],
+            config=config_name,
+            runner="genesis",
+            handler=handler_name,
+            test_cases=get_cases()))
+    return suite_definition
+
+
+if __name__ == "__main__":
+    gen_runner.run_generator("genesis", [
+        create_suite('initialization', 'minimal', lambda: generate_from_tests(test_initialization, 'phase0')),
+        create_suite('validity', 'minimal', lambda: generate_from_tests(test_validity, 'phase0')),
+    ])

test_generators/genesis/requirements.txt

@@ -0,0 +1,4 @@
+eth-utils==1.6.0
+../../test_libs/gen_helpers
+../../test_libs/config_helpers
+../../test_libs/pyspec

test_generators/operations/main.py

@@ -49,7 +49,9 @@ if __name__ == "__main__":
         create_suite('proposer_slashing', 'minimal', lambda: generate_from_tests(test_process_proposer_slashing, 'phase0')),
         create_suite('proposer_slashing', 'mainnet', lambda: generate_from_tests(test_process_proposer_slashing, 'phase0')),
         create_suite('transfer',          'minimal', lambda: generate_from_tests(test_process_transfer, 'phase0')),
-        create_suite('transfer',          'mainnet', lambda: generate_from_tests(test_process_transfer, 'phase0')),
+        # Disabled, due to the high amount of different transfer tests, this produces a shocking size of tests.
+        # Unnecessarily, as transfer are disabled currently, so not a priority.
+        # create_suite('transfer',          'mainnet', lambda: generate_from_tests(test_process_transfer, 'phase0')),
         create_suite('voluntary_exit',    'minimal', lambda: generate_from_tests(test_process_voluntary_exit, 'phase0')),
         create_suite('voluntary_exit',    'mainnet', lambda: generate_from_tests(test_process_voluntary_exit, 'phase0')),
     ])

test_generators/shuffling/main.py

@@ -10,7 +10,7 @@ from preset_loader import loader
 def shuffling_case(seed, count):
     yield 'seed', '0x' + seed.hex()
     yield 'count', count
-    yield 'shuffled', [spec.shuffle_index(i, count, seed) for i in range(count)]
+    yield 'shuffled', [int(spec.shuffled_index(i, count, seed)) for i in range(count)]
 
 
 @to_tuple

test_libs/pyspec/eth2spec/debug/encode.py

@@ -1,10 +1,10 @@
 from eth2spec.utils.ssz.ssz_impl import hash_tree_root
 from eth2spec.utils.ssz.ssz_typing import (
-    SSZValue, uint, Container, boolean
+    uint, Container, boolean
 )
 
 
-def encode(value: SSZValue, include_hash_tree_roots=False):
+def encode(value, include_hash_tree_roots=False):
     if isinstance(value, uint):
         # Larger uints are boxed and the class declares their byte length
         if value.type().byte_len > 8:

test_libs/pyspec/eth2spec/test/genesis/test_validity.py

@@ -16,13 +16,13 @@ def create_valid_beacon_state(spec):
 def run_is_valid_genesis_state(spec, state, valid=True):
     """
     Run ``is_valid_genesis_state``, yielding:
-      - state ('state')
+      - genesis ('state')
       - is_valid ('is_valid')
-    If ``valid == False``, run expecting ``AssertionError``
     """
-    yield state
+    yield 'genesis', state
     is_valid = spec.is_valid_genesis_state(state)
     yield 'is_valid', is_valid
+    assert is_valid == valid
 
 
 @with_phases(['phase0'])
@@ -39,7 +39,7 @@ def test_is_valid_genesis_state_false_invalid_timestamp(spec):
     state = create_valid_beacon_state(spec)
     state.genesis_time = spec.MIN_GENESIS_TIME - 1
 
-    yield from run_is_valid_genesis_state(spec, state, valid=True)
+    yield from run_is_valid_genesis_state(spec, state, valid=False)
 
 
 @with_phases(['phase0'])
@@ -51,13 +51,14 @@ def test_is_valid_genesis_state_true_more_balance(spec):
     yield from run_is_valid_genesis_state(spec, state, valid=True)
 
 
-@with_phases(['phase0'])
+# TODO: not part of the genesis function yet. Erroneously merged.
-@spectest_with_bls_switch
+# @with_phases(['phase0'])
-def test_is_valid_genesis_state_false_not_enough_balance(spec):
+# @spectest_with_bls_switch
-    state = create_valid_beacon_state(spec)
+# def test_is_valid_genesis_state_false_not_enough_balance(spec):
-    state.validators[0].effective_balance = spec.MAX_EFFECTIVE_BALANCE - 1
+#     state = create_valid_beacon_state(spec)
-
+#     state.validators[0].effective_balance = spec.MAX_EFFECTIVE_BALANCE - 1
-    yield from run_is_valid_genesis_state(spec, state, valid=False)
+#
+#     yield from run_is_valid_genesis_state(spec, state, valid=False)
 
 
 @with_phases(['phase0'])

test_libs/pyspec/eth2spec/test/phase_0/block_processing/test_process_attestation.py

@@ -1,6 +1,7 @@
 from eth2spec.test.context import spec_state_test, expect_assertion_error, always_bls, with_all_phases, with_phases
 from eth2spec.test.helpers.attestations import (
     get_valid_attestation,
+    sign_aggregate_attestation,
     sign_attestation,
 )
 from eth2spec.test.helpers.state import (
@@ -59,6 +60,7 @@ def test_success(spec, state):
 @spec_state_test
 def test_success_previous_epoch(spec, state):
     attestation = get_valid_attestation(spec, state, signed=True)
+    state.slot = spec.SLOTS_PER_EPOCH - 1
     next_epoch(spec, state)
     apply_empty_block(spec, state)
 
@@ -68,6 +70,9 @@ def test_success_previous_epoch(spec, state):
 @with_all_phases
 @spec_state_test
 def test_success_since_max_epochs_per_crosslink(spec, state):
+    # Do not run mainnet (64 epochs), that would mean the equivalent of ~7 hours chain simulation.
+    if spec.MAX_EPOCHS_PER_CROSSLINK > 4:
+        return
     for _ in range(spec.MAX_EPOCHS_PER_CROSSLINK + 2):
         next_epoch(spec, state)
     apply_empty_block(spec, state)
@@ -87,6 +92,9 @@ def test_success_since_max_epochs_per_crosslink(spec, state):
 @with_all_phases
 @spec_state_test
 def test_wrong_end_epoch_with_max_epochs_per_crosslink(spec, state):
+    # Do not run mainnet (64 epochs), that would mean the equivalent of ~7 hours chain simulation.
+    if spec.MAX_EPOCHS_PER_CROSSLINK > 4:
+        return
     for _ in range(spec.MAX_EPOCHS_PER_CROSSLINK + 2):
         next_epoch(spec, state)
     apply_empty_block(spec, state)
@@ -130,8 +138,9 @@ def test_before_inclusion_delay(spec, state):
 @spec_state_test
 def test_after_epoch_slots(spec, state):
     attestation = get_valid_attestation(spec, state, signed=True)
+    state.slot = spec.SLOTS_PER_EPOCH - 1
     # increment past latest inclusion slot
-    spec.process_slots(state, state.slot + spec.SLOTS_PER_EPOCH + 1)
+    spec.process_slots(state, state.slot + 2)
     apply_empty_block(spec, state)
 
     yield from run_attestation_processing(spec, state, attestation, False)
@@ -203,10 +212,17 @@ def test_future_target_epoch(spec, state):
 
     attestation = get_valid_attestation(spec, state)
 
-    state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY
+    participants = spec.get_attesting_indices(
-
+        state,
+        attestation.data,
+        attestation.aggregation_bits
+    )
     attestation.data.target.epoch = spec.get_current_epoch(state) + 1  # target epoch will be too new to handle
-    sign_attestation(spec, state, attestation)
+
+    # manually add signature for correct participants
+    attestation.signature = sign_aggregate_attestation(spec, state, attestation.data, participants)
+
+    state.slot += spec.MIN_ATTESTATION_INCLUSION_DELAY
 
     yield from run_attestation_processing(spec, state, attestation, False)
 
@@ -290,15 +306,17 @@ def test_non_zero_crosslink_data_root(spec, state):
 @with_all_phases
 @spec_state_test
 def test_bad_parent_crosslink(spec, state):
+    state.slot = spec.SLOTS_PER_EPOCH - 1
     next_epoch(spec, state)
     apply_empty_block(spec, state)
 
-    attestation = get_valid_attestation(spec, state, signed=True)
+    attestation = get_valid_attestation(spec, state, signed=False)
     for _ in range(spec.MIN_ATTESTATION_INCLUSION_DELAY):
         next_slot(spec, state)
     apply_empty_block(spec, state)
 
     attestation.data.crosslink.parent_root = b'\x27' * 32
+    sign_attestation(spec, state, attestation)
 
     yield from run_attestation_processing(spec, state, attestation, False)
 
@@ -306,15 +324,17 @@ def test_bad_parent_crosslink(spec, state):
 @with_all_phases
 @spec_state_test
 def test_bad_crosslink_start_epoch(spec, state):
+    state.slot = spec.SLOTS_PER_EPOCH - 1
     next_epoch(spec, state)
     apply_empty_block(spec, state)
 
-    attestation = get_valid_attestation(spec, state, signed=True)
+    attestation = get_valid_attestation(spec, state, signed=False)
     for _ in range(spec.MIN_ATTESTATION_INCLUSION_DELAY):
         next_slot(spec, state)
     apply_empty_block(spec, state)
 
     attestation.data.crosslink.start_epoch += 1
+    sign_attestation(spec, state, attestation)
 
     yield from run_attestation_processing(spec, state, attestation, False)
 
@@ -322,15 +342,17 @@ def test_bad_crosslink_start_epoch(spec, state):
 @with_all_phases
 @spec_state_test
 def test_bad_crosslink_end_epoch(spec, state):
+    state.slot = spec.SLOTS_PER_EPOCH - 1
     next_epoch(spec, state)
     apply_empty_block(spec, state)
 
-    attestation = get_valid_attestation(spec, state, signed=True)
+    attestation = get_valid_attestation(spec, state, signed=False)
     for _ in range(spec.MIN_ATTESTATION_INCLUSION_DELAY):
         next_slot(spec, state)
     apply_empty_block(spec, state)
 
     attestation.data.crosslink.end_epoch += 1
+    sign_attestation(spec, state, attestation)
 
     yield from run_attestation_processing(spec, state, attestation, False)
 

test_libs/pyspec/eth2spec/test/utils.py

@@ -1,6 +1,6 @@
 from typing import Dict, Any, Callable, Iterable
 from eth2spec.debug.encode import encode
-from eth2spec.utils.ssz.ssz_typing import Container
+from eth2spec.utils.ssz.ssz_typing import SSZValue
 
 
 def spectest(description: str = None):
@@ -29,10 +29,12 @@ def spectest(description: str = None):
                         (key, value, typ) = data
                         out[key] = encode(value, typ)
                     else:
-                        # Otherwise, try to infer the type, but keep it as-is if it's not a SSZ container.
+                        # Otherwise, try to infer the type, but keep it as-is if it's not a SSZ type or bytes.
                         (key, value) = data
-                        if isinstance(value, Container):
+                        if isinstance(value, (SSZValue, bytes)):
-                            out[key] = encode(value, value.__class__)
+                            out[key] = encode(value)
+                        elif isinstance(value, list) and all([isinstance(el, (SSZValue, bytes)) for el in value]):
+                            out[key] = [encode(el) for el in value]
                         else:
                             # not a ssz value.
                             # It could be vector or bytes still, but it is a rare case,