diff --git a/test_generators/README.md b/test_generators/README.md new file mode 100644 index 000000000..2d6160c03 --- /dev/null +++ b/test_generators/README.md @@ -0,0 +1,151 @@ +# Eth2.0 Test Generators + +This directory of contains all the generators for YAML tests, consumed by Eth 2.0 client implementations. + +Any issues with the generators and/or generated tests should be filed + in the repository that hosts the generator outputs, here: [ethereum/eth2.0-tests](https://github.com/ethereum/eth2.0-tests/). + +Whenever a release is made, the new tests are automatically built and +[eth2TestGenBot](https://github.com/eth2TestGenBot) commits the changes to the test repository. + +## How to run generators + +pre-requisites: +- Python 3 installed +- PIP 3 +- GNU make + +### Cleaning + +This removes the existing virtual environments (`/test_generators/.venvs/`), and generated tests (`/yaml_tests/`). + +```bash +make clean +``` + +### Running all test generators + +This runs all the generators. + +```bash +make all +``` + +### Running a single generator + +The make file auto-detects generators in the `test_generators/` directory, + and provides a tests-gen target for each generator, see example. + +```bash +make ./tests/shuffling/ +``` + +## Developing a generator + +Simply open up the generator (not all at once) of choice in your favorite IDE/editor, and run: + +```bash +# Create a virtual environment (any venv/.venv/.venvs is git-ignored) +python3 -m venv .venv +# Activate the venv, this is where dependencies are installed for the generator +. .venv/bin/activate +``` + +Now that you have a virtual environment, write your generator. +It's recommended to extend the base-generator. + +Create a `requirements.txt` in the root of your generator directory: +``` +eth-utils==1.4.1 +../test_libs/gen_helpers +``` + +Install all the necessary requirements (re-run when you add more): +```bash +pip3 install -r requirements.txt --user +``` + +And write your initial test generator, extending the base generator: + +Write a `main.py` file, here's an example: + +```python +from gen_base import gen_runner, gen_suite, gen_typing + +from eth_utils import ( + to_dict, to_tuple +) + + +@to_dict +def bar_test_case(v: int): + yield "bar_v", v + yield "bar_v_plus_1", v + 1 + yield "bar_list", list(range(v)) + + +@to_tuple +def generate_bar_test_cases(): + for i in range(10): + yield bar_test_case(i) + + +def bar_test_suite() -> gen_typing.TestSuite: + return gen_suite.render_suite( + title="bar_minimal", + summary="Minimal example suite, testing bar.", + fork="v0.5.1", + config="minimal", + test_cases=generate_bar_test_cases()) + + +if __name__ == "__main__": + gen_runner.run_generator("foo", [bar_test_suite]) + +``` + +Recommendations: +- you can have more than just 1 generator, e.g. ` gen_runner.run_generator("foo", [bar_test_suite, abc_test_suite, example_test_suite])` +- you can concatenate lists of test cases, if you don't want to split it up in suites. +- you can split your suite generators into different python files/packages, good for code organization. +- use config "minimal" for performance. But also implement a suite with the default config where necessary +- the test-generator accepts `--output` and `--force` (overwrite output) + +## How to add a new test generator + +In order to add a new test generator that builds `New Tests`: + +1. Create a new directory `new_tests`, within the `test_generators` directory. + Note that `new_tests` is also the name of the directory in which the tests will appear in the tests repository later. +2. Your generator is assumed to have a `requirements.txt` file, + with any dependencies it may need. Leave it empty if your generator has none. +3. Your generator is assumed to have a `main.py` file in its root. + By adding the base generator to your requirements, you can make a generator really easily. See docs below. +4. Your generator is called with `-o some/file/path/for_testing/can/be_anything`. + The base generator helps you handle this; you only have to define suite headers, + and a list of tests for each suite you generate. +5. Finally, add any linting or testing commands to the + [circleci config file](https://github.com/ethereum/eth2.0-test-generators/blob/master/.circleci/config.yml) + if desired to increase code quality. + +Note: you do not have to change the makefile. +However, if necessary (e.g. not using python, or mixing in other languages), submit an issue, and it can be a special case. +Do note that generators should be easy to maintain, lean, and based on the spec. + +All of this should be done in a pull request to the master branch. + +To deploy new tests to the testing repository: + +1. Create a release tag with a new version number on Github. +2. Increment either the: + - major version, to indicate a change in the general testing format + - minor version, if a new test generator has been added + - path version, in other cases. + +## How to remove a test generator + +If a test generator is not needed anymore, undo the steps described above and make a new release: + +1. remove the generator folder +2. remove the generated tests in the `eth2.0-tests` repository by opening a PR there. +3. make a new release diff --git a/test_generators/bls/README.md b/test_generators/bls/README.md new file mode 100644 index 000000000..9ce1b2f6c --- /dev/null +++ b/test_generators/bls/README.md @@ -0,0 +1,20 @@ +# BLS Test Generator + +Explanation of BLS12-381 type hierarchy +The base unit is bytes48 of which only 381 bits are used + +- FQ: uint381 modulo field modulus +- FQ2: (FQ, FQ) +- G2: (FQ2, FQ2, FQ2) + +## Resources + +- [Eth2.0 spec](https://github.com/ethereum/eth2.0-specs/blob/master/specs/bls_signature.md) +- [Finite Field Arithmetic](http://www.springeronline.com/sgw/cda/pageitems/document/cda_downloaddocument/0,11996,0-0-45-110359-0,00.pdf) +- Chapter 2 of [Elliptic Curve Cryptography](http://cacr.uwaterloo.ca/ecc/). Darrel Hankerson, Alfred Menezes, and Scott Vanstone +- [Zcash BLS parameters](https://github.com/zkcrypto/pairing/tree/master/src/bls12_381) +- [Trinity implementation](https://github.com/ethereum/trinity/blob/master/eth2/_utils/bls.py) + +## Comments + +Compared to Zcash, Ethereum specs always requires the compressed form (c_flag / most significant bit always set). \ No newline at end of file diff --git a/test_generators/bls/main.py b/test_generators/bls/main.py new file mode 100644 index 000000000..3c2ab454b --- /dev/null +++ b/test_generators/bls/main.py @@ -0,0 +1,192 @@ +""" +BLS test vectors generator +Usage: + "python tgen_bls path/to/output.yml" +""" + +# Standard library +import sys +from typing import Tuple + +# Third-party +import yaml + +# Ethereum +from eth_utils import int_to_big_endian, big_endian_to_int + +# Local imports +from py_ecc import bls + + +def int_to_hex(n: int) -> str: + return '0x' + int_to_big_endian(n).hex() + + +def hex_to_int(x: str) -> int: + return int(x, 16) + + +# Note: even though a domain is only an uint64, +# To avoid issues with YAML parsers that are limited to 53-bit (JS language limit) +# It is serialized as an hex string as well. +DOMAINS = [ + 0, + 1, + 1234, + 2**32-1, + 2**64-1 +] + +MESSAGES = [ + b'\x00' * 32, + b'\x56' * 32, + b'\xab' * 32, +] + +PRIVKEYS = [ + # Curve order is 256 so private keys are 32 bytes at most. + # Also not all integers is a valid private key, so using pre-generated keys + hex_to_int('0x00000000000000000000000000000000263dbd792f5b1be47ed85f8938c0f29586af0d3ac7b977f21c278fe1462040e3'), + hex_to_int('0x0000000000000000000000000000000047b8192d77bf871b62e87859d653922725724a5c031afeabc60bcef5ff665138'), + hex_to_int('0x00000000000000000000000000000000328388aff0d4a5b7dc9205abd374e7e98f3cd9f3418edb4eafda5fb16473d216'), +] + + +def hash_message(msg: bytes, + domain: int) ->Tuple[Tuple[str, str], Tuple[str, str], Tuple[str, str]]: + """ + Hash message + Input: + - Message as bytes + - domain as uint64 + Output: + - Message hash as a G2 point + """ + return [ + [ + int_to_hex(fq2.coeffs[0]), + int_to_hex(fq2.coeffs[1]), + ] + for fq2 in bls.utils.hash_to_G2(msg, domain) + ] + + +def hash_message_compressed(msg: bytes, domain: int) -> Tuple[str, str]: + """ + Hash message + Input: + - Message as bytes + - domain as uint64 + Output: + - Message hash as a compressed G2 point + """ + z1, z2 = bls.utils.compress_G2(bls.utils.hash_to_G2(msg, domain)) + return [int_to_hex(z1), int_to_hex(z2)] + + +if __name__ == '__main__': + + # Order not preserved - https://github.com/yaml/pyyaml/issues/110 + metadata = { + 'title': 'BLS signature and aggregation tests', + 'summary': 'Test vectors for BLS signature', + 'test_suite': 'bls', + 'fork': 'phase0-0.5.0', + } + + case01_message_hash_G2_uncompressed = [] + for msg in MESSAGES: + for domain in DOMAINS: + case01_message_hash_G2_uncompressed.append({ + 'input': {'message': '0x' + msg.hex(), 'domain': int_to_hex(domain)}, + 'output': hash_message(msg, domain) + }) + + case02_message_hash_G2_compressed = [] + for msg in MESSAGES: + for domain in DOMAINS: + case02_message_hash_G2_compressed.append({ + 'input': {'message': '0x' + msg.hex(), 'domain': int_to_hex(domain)}, + 'output': hash_message_compressed(msg, domain) + }) + + case03_private_to_public_key = [] + #  Used in later cases + pubkeys = [bls.privtopub(privkey) for privkey in PRIVKEYS] + #  Used in public key aggregation + pubkeys_serial = ['0x' + pubkey.hex() for pubkey in pubkeys] + case03_private_to_public_key = [ + { + 'input': int_to_hex(privkey), + 'output': pubkey_serial, + } + for privkey, pubkey_serial in zip(PRIVKEYS, pubkeys_serial) + ] + + case04_sign_messages = [] + sigs = [] # used in verify + for privkey in PRIVKEYS: + for message in MESSAGES: + for domain in DOMAINS: + sig = bls.sign(message, privkey, domain) + case04_sign_messages.append({ + 'input': { + 'privkey': int_to_hex(privkey), + 'message': '0x' + message.hex(), + 'domain': int_to_hex(domain) + }, + 'output': '0x' + sig.hex() + }) + sigs.append(sig) + + # TODO: case05_verify_messages: Verify messages signed in case04 + # It takes too long, empty for now + + case06_aggregate_sigs = [] + for domain in DOMAINS: + for message in MESSAGES: + sigs = [] + for privkey in PRIVKEYS: + sig = bls.sign(message, privkey, domain) + sigs.append(sig) + case06_aggregate_sigs.append({ + 'input': ['0x' + sig.hex() for sig in sigs], + 'output': '0x' + bls.aggregate_signatures(sigs).hex(), + }) + + case07_aggregate_pubkeys = [ + { + 'input': pubkeys_serial, + 'output': '0x' + bls.aggregate_pubkeys(pubkeys).hex(), + } + ] + + # TODO + # Aggregate verify + + # TODO + # Proof-of-possession + + with open(sys.argv[2] + "test_bls.yml", 'w') as outfile: + # Dump at top level + yaml.dump(metadata, outfile, default_flow_style=False) + # default_flow_style will unravel "ValidatorRecord" and "committee" line, + # exploding file size + yaml.dump( + {'case01_message_hash_G2_uncompressed': case01_message_hash_G2_uncompressed}, + outfile, + ) + yaml.dump( + {'case02_message_hash_G2_compressed': case02_message_hash_G2_compressed}, + outfile, + ) + yaml.dump( + {'case03_private_to_public_key': case03_private_to_public_key}, + outfile, + ) + yaml.dump({'case04_sign_messages': case04_sign_messages}, outfile) + + # Too time consuming to generate + # yaml.dump({'case05_verify_messages': case05_verify_messages}, outfile) + yaml.dump({'case06_aggregate_sigs': case06_aggregate_sigs}, outfile) + yaml.dump({'case07_aggregate_pubkeys': case07_aggregate_pubkeys}, outfile) diff --git a/test_generators/bls/requirements.txt b/test_generators/bls/requirements.txt new file mode 100644 index 000000000..3989a3a0f --- /dev/null +++ b/test_generators/bls/requirements.txt @@ -0,0 +1,2 @@ +py-ecc==1.6.0 +PyYAML==4.2b1 diff --git a/test_generators/shuffling/README.md b/test_generators/shuffling/README.md new file mode 100644 index 000000000..047a1b872 --- /dev/null +++ b/test_generators/shuffling/README.md @@ -0,0 +1,16 @@ +# Shuffling Test Generator + +``` +2018 Status Research & Development GmbH +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). + +This work uses public domain work under CC0 from the Ethereum Foundation +https://github.com/ethereum/eth2.0-specs +``` + + +This file implements a test vectors generator for the shuffling algorithm described in the Ethereum +[specs](https://github.com/ethereum/eth2.0-specs/blob/2983e68f0305551083fac7fcf9330c1fc9da3411/specs/core/0_beacon-chain.md#get_new_shuffling) + +Utilizes 'swap or not' shuffling found in [An Enciphering Scheme Based on a Card Shuffle](https://link.springer.com/content/pdf/10.1007%2F978-3-642-32009-5_1.pdf). +See the `Generalized domain` algorithm on page 3. diff --git a/test_generators/shuffling/constants.py b/test_generators/shuffling/constants.py new file mode 100644 index 000000000..92862f898 --- /dev/null +++ b/test_generators/shuffling/constants.py @@ -0,0 +1,6 @@ +SLOTS_PER_EPOCH = 2**6 # 64 slots, 6.4 minutes +FAR_FUTURE_EPOCH = 2**64 - 1 # uint64 max +SHARD_COUNT = 2**10 # 1024 +TARGET_COMMITTEE_SIZE = 2**7 # 128 validators +ACTIVATION_EXIT_DELAY = 2**2 # 4 epochs +SHUFFLE_ROUND_COUNT = 90 diff --git a/test_generators/shuffling/core_helpers.py b/test_generators/shuffling/core_helpers.py new file mode 100644 index 000000000..c424b771e --- /dev/null +++ b/test_generators/shuffling/core_helpers.py @@ -0,0 +1,95 @@ +from typing import Any, List, NewType + +from constants import SLOTS_PER_EPOCH, SHARD_COUNT, TARGET_COMMITTEE_SIZE, SHUFFLE_ROUND_COUNT +from utils import hash +from yaml_objects import Validator + +Epoch = NewType("Epoch", int) +ValidatorIndex = NewType("ValidatorIndex", int) +Bytes32 = NewType("Bytes32", bytes) + + +def int_to_bytes1(x): + return x.to_bytes(1, 'little') + + +def int_to_bytes4(x): + return x.to_bytes(4, 'little') + + +def bytes_to_int(data: bytes) -> int: + return int.from_bytes(data, 'little') + + +def is_active_validator(validator: Validator, epoch: Epoch) -> bool: + """ + Check if ``validator`` is active. + """ + return validator.activation_epoch <= epoch < validator.exit_epoch + + +def get_active_validator_indices(validators: List[Validator], epoch: Epoch) -> List[ValidatorIndex]: + """ + Get indices of active validators from ``validators``. + """ + return [i for i, v in enumerate(validators) if is_active_validator(v, epoch)] + + +def split(values: List[Any], split_count: int) -> List[List[Any]]: + """ + Splits ``values`` into ``split_count`` pieces. + """ + list_length = len(values) + return [ + values[(list_length * i // split_count): (list_length * (i + 1) // split_count)] + for i in range(split_count) + ] + + +def get_epoch_committee_count(active_validator_count: int) -> int: + """ + Return the number of committees in one epoch. + """ + return max( + 1, + min( + SHARD_COUNT // SLOTS_PER_EPOCH, + active_validator_count // SLOTS_PER_EPOCH // TARGET_COMMITTEE_SIZE, + ) + ) * SLOTS_PER_EPOCH + + +def get_permuted_index(index: int, list_size: int, seed: Bytes32) -> int: + """ + Return `p(index)` in a pseudorandom permutation `p` of `0...list_size-1` with ``seed`` as entropy. + + Utilizes 'swap or not' shuffling found in + https://link.springer.com/content/pdf/10.1007%2F978-3-642-32009-5_1.pdf + See the 'generalized domain' algorithm on page 3. + """ + for round in range(SHUFFLE_ROUND_COUNT): + pivot = bytes_to_int(hash(seed + int_to_bytes1(round))[0:8]) % list_size + flip = (pivot - index) % list_size + position = max(index, flip) + source = hash(seed + int_to_bytes1(round) + int_to_bytes4(position // 256)) + byte = source[(position % 256) // 8] + bit = (byte >> (position % 8)) % 2 + index = flip if bit else index + + return index + + +def get_shuffling(seed: Bytes32, + validators: List[Validator], + epoch: Epoch) -> List[List[ValidatorIndex]]: + """ + Shuffle active validators and split into crosslink committees. + Return a list of committees (each a list of validator indices). + """ + # Shuffle active validator indices + active_validator_indices = get_active_validator_indices(validators, epoch) + length = len(active_validator_indices) + shuffled_indices = [active_validator_indices[get_permuted_index(i, length, seed)] for i in range(length)] + + # Split the shuffled active validator indices + return split(shuffled_indices, get_epoch_committee_count(length)) diff --git a/test_generators/shuffling/main.py b/test_generators/shuffling/main.py new file mode 100644 index 000000000..03352944a --- /dev/null +++ b/test_generators/shuffling/main.py @@ -0,0 +1,160 @@ +import random +import sys +import os + +import yaml + +from constants import ACTIVATION_EXIT_DELAY, FAR_FUTURE_EPOCH +from core_helpers import get_shuffling +from yaml_objects import Validator + + +def noop(self, *args, **kw): + # Prevent !!str or !!binary tags + pass + + +yaml.emitter.Emitter.process_tag = noop + + +EPOCH = 1000 # The epoch, also a mean for the normal distribution + +# Standard deviation, around 8% validators will activate or exit within +# ENTRY_EXIT_DELAY inclusive from EPOCH thus creating an edge case for validator +# shuffling +RAND_EPOCH_STD = 35 +MAX_EXIT_EPOCH = 5000 # Maximum exit_epoch for easier reading + + +def active_exited_validators_generator(): + """ + Random cases with variety of validator's activity status + """ + # Order not preserved - https://github.com/yaml/pyyaml/issues/110 + metadata = { + 'title': 'Shuffling Algorithm Tests 1', + 'summary': 'Test vectors for validator shuffling with different validator\'s activity status.' + ' Note: only relevant validator fields are defined.', + 'test_suite': 'shuffle', + 'fork': 'phase0-0.5.0', + } + + # Config + random.seed(int("0xEF00BEAC", 16)) + num_cases = 10 + + test_cases = [] + + for case in range(num_cases): + seedhash = bytes(random.randint(0, 255) for byte in range(32)) + idx_max = random.randint(128, 512) + + validators = [] + for idx in range(idx_max): + v = Validator(original_index=idx) + # 4/5 of all validators are active + if random.random() < 0.8: + # Choose a normally distributed epoch number + rand_epoch = round(random.gauss(EPOCH, RAND_EPOCH_STD)) + + # for 1/2 of *active* validators rand_epoch is the activation epoch + if random.random() < 0.5: + v.activation_epoch = rand_epoch + + # 1/4 of active validators will exit in forseeable future + if random.random() < 0.5: + v.exit_epoch = random.randint( + rand_epoch + ACTIVATION_EXIT_DELAY + 1, MAX_EXIT_EPOCH) + # 1/4 of active validators in theory remain in the set indefinitely + else: + v.exit_epoch = FAR_FUTURE_EPOCH + # for the other active 1/2 rand_epoch is the exit epoch + else: + v.activation_epoch = random.randint( + 0, rand_epoch - ACTIVATION_EXIT_DELAY) + v.exit_epoch = rand_epoch + + # The remaining 1/5 of all validators is not activated + else: + v.activation_epoch = FAR_FUTURE_EPOCH + v.exit_epoch = FAR_FUTURE_EPOCH + + validators.append(v) + + input_ = { + 'validators': validators, + 'epoch': EPOCH + } + output = get_shuffling( + seedhash, validators, input_['epoch']) + + test_cases.append({ + 'seed': '0x' + seedhash.hex(), 'input': input_, 'output': output + }) + + return { + 'metadata': metadata, + 'filename': 'test_vector_shuffling.yml', + 'test_cases': test_cases + } + + +def validators_set_size_variety_generator(): + """ + Different validator set size cases, inspired by removed manual `permutated_index` tests + https://github.com/ethereum/eth2.0-test-generators/tree/bcd9ab2933d9f696901d1dfda0828061e9d3093f/permutated_index + """ + # Order not preserved - https://github.com/yaml/pyyaml/issues/110 + metadata = { + 'title': 'Shuffling Algorithm Tests 2', + 'summary': 'Test vectors for validator shuffling with different validator\'s set size.' + ' Note: only relevant validator fields are defined.', + 'test_suite': 'shuffle', + 'fork': 'tchaikovsky', + 'version': 1.0 + } + + # Config + random.seed(int("0xEF00BEAC", 16)) + + test_cases = [] + + seedhash = bytes(random.randint(0, 255) for byte in range(32)) + idx_max = 4096 + set_sizes = [1, 2, 3, 1024, idx_max] + + for size in set_sizes: + validators = [] + for idx in range(size): + v = Validator(original_index=idx) + v.activation_epoch = EPOCH + v.exit_epoch = FAR_FUTURE_EPOCH + validators.append(v) + input_ = { + 'validators': validators, + 'epoch': EPOCH + } + output = get_shuffling( + seedhash, validators, input_['epoch']) + + test_cases.append({ + 'seed': '0x' + seedhash.hex(), 'input': input_, 'output': output + }) + + return { + 'metadata': metadata, + 'filename': 'shuffling_set_size.yml', + 'test_cases': test_cases + } + + +if __name__ == '__main__': + output_dir = sys.argv[2] + for generator in [active_exited_validators_generator, validators_set_size_variety_generator]: + result = generator() + filename = os.path.join(output_dir, result['filename']) + with open(filename, 'w') as outfile: + # Dump at top level + yaml.dump(result['metadata'], outfile, default_flow_style=False) + # default_flow_style will unravel "ValidatorRecord" and "committee" line, exploding file size + yaml.dump({'test_cases': result['test_cases']}, outfile) diff --git a/test_generators/shuffling/requirements.txt b/test_generators/shuffling/requirements.txt new file mode 100644 index 000000000..dde2fb67d --- /dev/null +++ b/test_generators/shuffling/requirements.txt @@ -0,0 +1,4 @@ +eth-hash[pycryptodome]==0.2.0 +eth-typing==2.0.0 +eth-utils==1.4.1 +PyYAML==4.2b1 diff --git a/test_generators/shuffling/utils.py b/test_generators/shuffling/utils.py new file mode 100644 index 000000000..bcd2c6a3c --- /dev/null +++ b/test_generators/shuffling/utils.py @@ -0,0 +1,6 @@ +from eth_typing import Hash32 +from eth_utils import keccak + + +def hash(x: bytes) -> Hash32: + return keccak(x) diff --git a/test_generators/shuffling/yaml_objects.py b/test_generators/shuffling/yaml_objects.py new file mode 100644 index 000000000..18e45220e --- /dev/null +++ b/test_generators/shuffling/yaml_objects.py @@ -0,0 +1,25 @@ +from typing import Any + +import yaml + + +class Validator(yaml.YAMLObject): + """ + A validator stub containing only the fields relevant for get_shuffling() + """ + fields = { + 'activation_epoch': 'uint64', + 'exit_epoch': 'uint64', + # Extra index field to ease testing/debugging + 'original_index': 'uint64', + } + + def __init__(self, **kwargs): + for k in self.fields.keys(): + setattr(self, k, kwargs.get(k)) + + def __setattr__(self, name: str, value: Any) -> None: + super().__setattr__(name, value) + + def __getattribute__(self, name: str) -> Any: + return super().__getattribute__(name) diff --git a/test_generators/ssz/__init__.py b/test_generators/ssz/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/test_generators/ssz/main.py b/test_generators/ssz/main.py new file mode 100644 index 000000000..d19ec12b4 --- /dev/null +++ b/test_generators/ssz/main.py @@ -0,0 +1,84 @@ +import argparse +import pathlib +import sys + +from ruamel.yaml import ( + YAML, +) + +from uint_test_generators import ( + generate_uint_bounds_test, + generate_uint_random_test, + generate_uint_wrong_length_test, +) + +test_generators = [ + generate_uint_random_test, + generate_uint_wrong_length_test, + generate_uint_bounds_test, +] + + +def make_filename_for_test(test): + title = test["title"] + filename = title.lower().replace(" ", "_") + ".yaml" + return pathlib.Path(filename) + + +def validate_output_dir(path_str): + path = pathlib.Path(path_str) + + if not path.exists(): + raise argparse.ArgumentTypeError("Output directory must exist") + + if not path.is_dir(): + raise argparse.ArgumentTypeError("Output path must lead to a directory") + + return path + + +parser = argparse.ArgumentParser( + prog="gen-ssz-tests", + description="Generate YAML test files for SSZ and tree hashing", +) +parser.add_argument( + "-o", + "--output-dir", + dest="output_dir", + required=True, + type=validate_output_dir, + help="directory into which the generated YAML files will be dumped" +) +parser.add_argument( + "-f", + "--force", + action="store_true", + default=False, + help="if set overwrite test files if they exist", +) + + +if __name__ == "__main__": + args = parser.parse_args() + output_dir = args.output_dir + if not args.force: + file_mode = "x" + else: + file_mode = "w" + + yaml = YAML(pure=True) + + print(f"generating {len(test_generators)} test files...") + for test_generator in test_generators: + test = test_generator() + + filename = make_filename_for_test(test) + path = output_dir / filename + + try: + with path.open(file_mode) as f: + yaml.dump(test, f) + except IOError as e: + sys.exit(f'Error when dumping test "{test["title"]}" ({e})') + + print("done.") diff --git a/test_generators/ssz/renderers.py b/test_generators/ssz/renderers.py new file mode 100644 index 000000000..e551ab14c --- /dev/null +++ b/test_generators/ssz/renderers.py @@ -0,0 +1,102 @@ +from collections.abc import ( + Mapping, + Sequence, +) + +from eth_utils import ( + encode_hex, + to_dict, +) + +from ssz.sedes import ( + BaseSedes, + Boolean, + Bytes, + BytesN, + Container, + List, + UInt, +) + + +def render_value(value): + if isinstance(value, bool): + return value + elif isinstance(value, int): + return str(value) + elif isinstance(value, bytes): + return encode_hex(value) + elif isinstance(value, Sequence): + return tuple(render_value(element) for element in value) + elif isinstance(value, Mapping): + return render_dict_value(value) + else: + raise ValueError(f"Cannot render value {value}") + + +@to_dict +def render_dict_value(value): + for key, value in value.items(): + yield key, render_value(value) + + +def render_type_definition(sedes): + if isinstance(sedes, Boolean): + return "bool" + + elif isinstance(sedes, UInt): + return f"uint{sedes.length * 8}" + + elif isinstance(sedes, BytesN): + return f"bytes{sedes.length}" + + elif isinstance(sedes, Bytes): + return f"bytes" + + elif isinstance(sedes, List): + return [render_type_definition(sedes.element_sedes)] + + elif isinstance(sedes, Container): + return { + field_name: render_type_definition(field_sedes) + for field_name, field_sedes in sedes.fields + } + + elif isinstance(sedes, BaseSedes): + raise Exception("Unreachable: All sedes types have been checked") + + else: + raise TypeError("Expected BaseSedes") + + +@to_dict +def render_test_case(*, sedes, valid, value=None, serial=None, description=None, tags=None): + value_and_serial_given = value is not None and serial is not None + if valid: + if not value_and_serial_given: + raise ValueError("For valid test cases, both value and ssz must be present") + else: + if value_and_serial_given: + raise ValueError("For invalid test cases, either value or ssz must not be present") + + if tags is None: + tags = [] + + yield "type", render_type_definition(sedes) + yield "valid", valid + if value is not None: + yield "value", render_value(value) + if serial is not None: + yield "ssz", encode_hex(serial) + if description is not None: + yield description + yield "tags", tags + + +@to_dict +def render_test(*, title, summary, fork, test_cases): + yield "title", title, + if summary is not None: + yield "summary", summary + yield "fork", fork + yield "test_cases", test_cases diff --git a/test_generators/ssz/requirements.txt b/test_generators/ssz/requirements.txt new file mode 100644 index 000000000..88193a01d --- /dev/null +++ b/test_generators/ssz/requirements.txt @@ -0,0 +1,2 @@ +ruamel.yaml==0.15.87 +ssz==0.1.0a2 diff --git a/test_generators/ssz/uint_test_generators.py b/test_generators/ssz/uint_test_generators.py new file mode 100644 index 000000000..c8c841fe7 --- /dev/null +++ b/test_generators/ssz/uint_test_generators.py @@ -0,0 +1,132 @@ +import random + +from eth_utils import ( + to_tuple, +) + +import ssz +from ssz.sedes import ( + UInt, +) +from renderers import ( + render_test, + render_test_case, +) + +random.seed(0) + + +BIT_SIZES = [i for i in range(8, 512 + 1, 8)] +RANDOM_TEST_CASES_PER_BIT_SIZE = 10 +RANDOM_TEST_CASES_PER_LENGTH = 3 + + +def get_random_bytes(length): + return bytes(random.randint(0, 255) for _ in range(length)) + + +def generate_uint_bounds_test(): + test_cases = generate_uint_bounds_test_cases() + generate_uint_out_of_bounds_test_cases() + + return render_test( + title="UInt Bounds", + summary="Integers right at or beyond the bounds of the allowed value range", + fork="phase0-0.2.0", + test_cases=test_cases, + ) + + +def generate_uint_random_test(): + test_cases = generate_random_uint_test_cases() + + return render_test( + title="UInt Random", + summary="Random integers chosen uniformly over the allowed value range", + fork="phase0-0.2.0", + test_cases=test_cases, + ) + + +def generate_uint_wrong_length_test(): + test_cases = generate_uint_wrong_length_test_cases() + + return render_test( + title="UInt Wrong Length", + summary="Serialized integers that are too short or too long", + fork="phase0-0.2.0", + test_cases=test_cases, + ) + + +@to_tuple +def generate_random_uint_test_cases(): + for bit_size in BIT_SIZES: + sedes = UInt(bit_size) + + for _ in range(RANDOM_TEST_CASES_PER_BIT_SIZE): + value = random.randrange(0, 2 ** bit_size) + serial = ssz.encode(value, sedes) + # note that we need to create the tags in each loop cycle, otherwise ruamel will use + # YAML references which makes the resulting file harder to read + tags = tuple(["atomic", "uint", "random"]) + yield render_test_case( + sedes=sedes, + valid=True, + value=value, + serial=serial, + tags=tags, + ) + + +@to_tuple +def generate_uint_wrong_length_test_cases(): + for bit_size in BIT_SIZES: + sedes = UInt(bit_size) + lengths = sorted({ + 0, + sedes.length // 2, + sedes.length - 1, + sedes.length + 1, + sedes.length * 2, + }) + for length in lengths: + for _ in range(RANDOM_TEST_CASES_PER_LENGTH): + tags = tuple(["atomic", "uint", "wrong_length"]) + yield render_test_case( + sedes=sedes, + valid=False, + serial=get_random_bytes(length), + tags=tags, + ) + + +@to_tuple +def generate_uint_bounds_test_cases(): + common_tags = ("atomic", "uint") + for bit_size in BIT_SIZES: + sedes = UInt(bit_size) + + for value, tag in ((0, "uint_lower_bound"), (2 ** bit_size - 1, "uint_upper_bound")): + serial = ssz.encode(value, sedes) + yield render_test_case( + sedes=sedes, + valid=True, + value=value, + serial=serial, + tags=common_tags + (tag,), + ) + + +@to_tuple +def generate_uint_out_of_bounds_test_cases(): + common_tags = ("atomic", "uint") + for bit_size in BIT_SIZES: + sedes = UInt(bit_size) + + for value, tag in ((-1, "uint_underflow"), (2 ** bit_size, "uint_overflow")): + yield render_test_case( + sedes=sedes, + valid=False, + value=value, + tags=common_tags + (tag,), + )