Merge pull request #71 from CarlBeek/carl_interop

KeyGen test vectors for mock interop start

interop/mocked_start/README.md

@@ -0,0 +1,79 @@
+# Mocked start of Eth2 for interoperability testing
+
+This document represents a set of standards to aid in the start of short-lived testnets _without_ an eth1 network as a source of initial deposits.
+
+A network start consists of the following:
+
+1. A `BeaconState`
+2. Network configuration
+3. Shared validator pubkey/privkeys
+4. Validator distribution across nodes
+
+## Beacon state
+
+### Quick-start genesis
+
+Quick-start is a simple method to create and run a common genesis `BeaconState` with from two parameters -- `genesis_time` and `validator_count`. These parameters can be specified either in a YAML file or as command-line params. This method is appealing in many testing scenarios because it is both simple and succinct. The main drawback of this method is that all validators are initialized with `MAX_EFFECTIVE_BALANCE` to start.
+
+#### Generate deposits
+
+A list of `validator_count` `deposits` is derived using the first `validator_count` pubkey/privkey pairs from a shared pubkey/privkey rainbow table of valid pubkey/privkeys generated in the [method below](#pubkeyprivkey-generation). `withdrawal_credentials` for each are set to `BLS_WITHDRAWAL_PREFIX + hash(deposit.data.pubkey)[1:]`. `amount` for each is set to `MAX_EFFECTIVE_BALANCE`. 
+
+#### Create genesis state
+
+Clients must create the genesis state by calling `initialize_beacon_state_from_eth1` with coordinated junk values for `eth1_block_hash = b'\x42'*32` and `eth1_timestamp = 2**40`, and with the list of `deposits` created in the [prior section](generate-deposits). The returned `BeaconState` must then be modified with `state.genesis_time = genesis_time`.
+
+Clients must _not_ run `is_valid_genesis_state` as this state is already considered valid. Specifically, we do not check nor care about `MIN_GENESIS_TIME` in these coordinated starts.
+
+### Start chain from specified state
+
+In all interop testing scenarios, we must start from a specified `BeaconState`. This state might be a genesis state or some arbitrary state at any point in a chain's history. Specifically, when debugging testnets or testing very specific scenarios, a testnet might be started from a specified non-genesis state.
+
+As with the state transition conformance tests, the `BeaconState` can be specified as either `.yaml` or `.ssz` format.
+
+In most clients, it makes sense to just pipe the output of [quick-start genesis](quick-start-genesis) into this generic chain start method.
+
+
+## Network configuration
+
+A shared testnet must agree upon a common configuration of constants. The specs repo currently contains two [configuration presets](https://github.com/ethereum/eth2.0-specs/tree/master/configs) -- [`mainnet`](https://github.com/ethereum/eth2.0-specs/blob/master/configs/mainnet.yaml) and [`minimal`](https://github.com/ethereum/eth2.0-specs/blob/master/configs/minimal.yaml). `minimal` will serve as a primary configuration for most interop tests.
+
+If there are components of this configuration that do not serve a specific need, we will add more configurations accordingly.
+
+
+## Pubkey/privkey generation
+
+For interop testing, we use a common set of public/private keypairs to populate validator records and new deposits. We use the following method to generate the shared keypairs.
+
+There is a compute/storage tradeoff to be made here between calculating the required validators and reading them from a YAML file ([./keygen_10000_validators.yaml](./keygen_10000_validators.yaml)) and it is left up to implementors to choose which they prefer.
+
+The following script is used to generated pubkey/privkeys for the first `N` validators. The `i`-th deposit/validator index uses the `validator_index_to_pubkey[i]` pubkey and associated privkey.
+
+```python
+CURVE_ORDER = 52435875175126190479447740508185965837690552500527637822603658699938581184513
+validator_index_to_pubkey = {}
+pubkey_to_privkey = {}
+privkey_to_pubkey = {}
+for index in range(N):
+    privkey = int.from_bytes(
+        sha256(int_to_bytes(n=index, length=32)),
+        byteorder='little'
+    ) % CURVE_ORDER
+    pubkey = bls.privtopubkey(privkey)
+    pubkey_to_privkey[pubkey] = privkey
+    privkey_to_pubkey[privkey] = pubkey
+    validator_index_to_pubkey[index] = pubkey
+```
+
+### Test vectors
+
+[./keygen_test_vector.yaml](./keygen_test_vector.yaml) is a YAML file containing a list of the first 10 validators and their key pairs. The list index corresponds to the validator number. For the generation of more or fewer indices, see the script used to generate it at [./keygen.py](./keygen.py)
+
+
+## Distribution of validators across nodes
+
+Nodes should support running some subset of validators within a testnet configuration. For ease of distribution, validators should normally be split across nodes in contiguous ranges (eg validators 0-9 on node `A`, validators 10-15 on node `B`). 
+
+To generally support this functionality, a node should accept a tuple of `validator_start_index` and `num_validators`. For example `(validator_start_index=5, num_validators=3` would provision a node to control validator indices `(5, 6, 7)`.
+
+Nodes can also support specifying the entire list of indices, rather than the succint range form. This will allow for more novel distributions of validator indices at the cost of verbosity in specifying.

interop/mocked_start/keygen.py

@@ -0,0 +1,29 @@
+from py_ecc.bls.api import privtopub
+from hashlib import sha256 as _sha256
+from typing import List, Dict
+from yaml import dump, Dumper
+
+CURVE_ORDER = 52435875175126190479447740508185965837690552500527637822603658699938581184513
+
+
+def sha256(x):
+    return _sha256(x).digest()
+
+
+def generate_validator_keypairs(N: int) -> List[Dict]:
+    keypairs = []
+    for index in range(N):
+        privkey = int.from_bytes(
+            sha256(index.to_bytes(length=32, byteorder='little')),
+            byteorder='little'
+        ) % CURVE_ORDER
+        pubkey = privtopub(privkey)
+        keypairs.append({'privkey': privkey, 'pubkey': pubkey})
+    return keypairs
+
+
+if __name__ == '__main__':
+    keypairs = generate_validator_keypairs(10)
+    keypairs_yaml = dump(keypairs, Dumper=Dumper)
+    with open('keygen_test_vector.yaml', 'w') as f:
+        f.write(keypairs_yaml)

interop/mocked_start/keygen_test_vector.yaml

@@ -0,0 +1,30 @@
+- privkey: 16808672146709759238327133555736750089977066230599028589193936481731504400486
+  pubkey: !!binary |
+    qZp27XeW974i1bfoXe63xWd+iOUR4LM3YY+MTrYTSbS/LRU/ZJ97UzWf6LlKOORM
+- privkey: 37006103240406073079686739739280712467525465637222501547219594975923976982528
+  pubkey: !!binary |
+    uJvrxpl2lyajGMjplxvTFxKXxhrqSmV4p6T5S1R9y6W6wWqJEItrah/jaV0ah0oL
+- privkey: 22330876536127119444572216874798222843352868708084730796787004036811744442455
+  pubkey: !!binary |
+    o6MrD4tN24PxoKhT2B3XJd/ld9T0w9uOzlLOKwJuyoSBXBp+jpKk3j11VzO/fkqb
+- privkey: 17048462031355941381150076874414096388968985457797372268770826099852902060945
+  pubkey: !!binary |
+    iMFB33fNnY16cadcgmxBqcnwPG7hsYDz54UvaigAmd7TUbWNZuZTr45CgWpNj1Mu
+- privkey: 28647806952216650698330424381872693846361470773871570637461872359310549743691
+  pubkey: !!binary |
+    gSg7eiDhykYOvZu9dwBdVXNwyrsfmkT1MMTExmIw9nX434tMKBiFGqfXeoDKWkpe
+- privkey: 2416304019107052589452838695606585506736351107897780798170812672519914514344
+  pubkey: !!binary |
+    qwvdoPhfhC9DG+rM8SUL8f17pRtBAP1kNktkAf2oW7AGmz5xW1iBloTn/AsQpyo0
+- privkey: 7300215445567548136411883691093515822872548648751398235557229381530420545683
+  pubkey: !!binary |
+    mXfxyLcxqNVVgUa/uGyuomQ088WHi1ib8oCkLJFZ5wDp3w5AhilsILAR0ueMJ9Nz
+- privkey: 26495790445032093722332687600112008700915252495659977774957922313678954054133
+  pubkey: !!binary |
+    qNTHwneVpyWWExfvWVOnAy7W2Dc524sOinI1PRuLRDlCf376LInKoDzJ8o+Muris
+- privkey: 2908643403277969554503670470854573663206729491025062456164283925661321952518
+  pubkey: !!binary |
+    ptMQ27+rmiJFD1mZP4ekzl22Ij87Xx8w0sTscYki1ADgs8d0HejlmWD3JBGg7hCn
+- privkey: 19554639423851580804889717218680781396599791537051606512605582393920758869044
+  pubkey: !!binary |
+    mJNBPAAoOj+e2f2YRd2hzqOCKNIlZ/lUHczDV+VKLWpuIEEDySVky8BfSQWsfEk6