Merge branch 'dev' into proto-merge-test-gen

specs/core/0_beacon-chain.md

@@ -67,8 +67,8 @@
         - [`get_permuted_index`](#get_permuted_index)
         - [`get_split_offset`](#get_split_offset)
         - [`get_epoch_committee_count`](#get_epoch_committee_count)
+        - [`get_shard_delta`](#get_shard_delta)
         - [`compute_committee`](#compute_committee)
-        - [`get_current_epoch_committee_count`](#get_current_epoch_committee_count)
         - [`get_crosslink_committees_at_slot`](#get_crosslink_committees_at_slot)
         - [`get_block_root`](#get_block_root)
         - [`get_state_root`](#get_state_root)
@@ -96,7 +96,6 @@
         - [`bls_verify`](#bls_verify)
         - [`bls_verify_multiple`](#bls_verify_multiple)
         - [`bls_aggregate_pubkeys`](#bls_aggregate_pubkeys)
-        - [`process_deposit`](#process_deposit)
         - [Routines for updating validator status](#routines-for-updating-validator-status)
             - [`activate_validator`](#activate_validator)
             - [`initiate_validator_exit`](#initiate_validator_exit)
@@ -140,10 +139,6 @@
                 - [Voluntary exits](#voluntary-exits)
                 - [Transfers](#transfers)
             - [State root verification](#state-root-verification)
-- [References](#references)
-    - [Normative](#normative)
-    - [Informative](#informative)
-- [Copyright](#copyright)
 
 <!-- /TOC -->
 
@@ -172,7 +167,7 @@ Code snippets appearing in `this style` are to be interpreted as Python code.
 * **Crosslink** - a set of signatures from a committee attesting to a block in a shard chain that can be included into the beacon chain. Crosslinks are the main means by which the beacon chain "learns about" the updated state of shard chains.
 * **Slot** - a period during which one proposer has the ability to create a beacon chain block and some attesters have the ability to make attestations.
 * **Epoch** - an aligned span of slots during which all [validators](#dfn-validator) get exactly one chance to make an attestation.
-* **Finalized**, **justified** - see Casper FFG finalization [[casper-ffg]](#ref-casper-ffg).
+* **Finalized**, **justified** - see the [Casper FFG paper](https://arxiv.org/abs/1710.09437).
 * **Withdrawal period** - the number of slots between a [validator](#dfn-validator) exit and the [validator](#dfn-validator) balance being withdrawable.
 * **Genesis time** - the Unix time of the genesis beacon chain block at slot 0.
 
@@ -843,19 +838,27 @@ def get_split_offset(list_size: int, chunks: int, index: int) -> int:
 ### `get_epoch_committee_count`
 
 ```python
-def get_epoch_committee_count(active_validator_count: int) -> int:
+def get_epoch_committee_count(state: BeaconState, epoch: Epoch) -> int:
     """
     Return the number of committees in one epoch.
     """
+    active_validators = get_active_validator_indices(state.validator_registry, epoch)
     return max(
         1,
         min(
             SHARD_COUNT // SLOTS_PER_EPOCH,
-            active_validator_count // SLOTS_PER_EPOCH // TARGET_COMMITTEE_SIZE,
+            len(active_validators) // SLOTS_PER_EPOCH // TARGET_COMMITTEE_SIZE,
         )
     ) * SLOTS_PER_EPOCH
 ```
 
+### `get_shard_delta`
+
+```python
+def get_shard_delta(state: BeaconState, epoch: Epoch) -> int:
+    return min(get_epoch_committee_count(state, epoch), SHARD_COUNT - SHARD_COUNT // SLOTS_PER_EPOCH)
+```
+
 ### `compute_committee`
 
 ```python
@@ -877,20 +880,6 @@ def compute_committee(validator_indices: List[ValidatorIndex],
 
 **Note**: this definition and the next few definitions are highly inefficient as algorithms, as they re-calculate many sub-expressions. Production implementations are expected to appropriately use caching/memoization to avoid redoing work.
 
-### `get_current_epoch_committee_count`
-
-```python
-def get_current_epoch_committee_count(state: BeaconState) -> int:
-    """
-    Return the number of committees in the current epoch of the given ``state``.
-    """
-    current_active_validators = get_active_validator_indices(
-        state.validator_registry,
-        get_current_epoch(state),
-    )
-    return get_epoch_committee_count(len(current_active_validators))
-```
-
 ### `get_crosslink_committees_at_slot`
 
 ```python
@@ -909,16 +898,17 @@ def get_crosslink_committees_at_slot(state: BeaconState,
         state.validator_registry,
         epoch,
     )
-    committees_per_epoch = get_epoch_committee_count(len(indices))
 
     if epoch == current_epoch:
         start_shard = state.latest_start_shard
     elif epoch == previous_epoch:
-        start_shard = (state.latest_start_shard - committees_per_epoch) % SHARD_COUNT
+        previous_shard_delta = get_shard_delta(state, previous_epoch)
+        start_shard = (state.latest_start_shard - previous_shard_delta) % SHARD_COUNT
     elif epoch == next_epoch:
-        current_epoch_committees = get_current_epoch_committee_count(state)
+        current_shard_delta = get_shard_delta(state, current_epoch)
-        start_shard = (state.latest_start_shard + current_epoch_committees) % SHARD_COUNT
+        start_shard = (state.latest_start_shard + current_shard_delta) % SHARD_COUNT
 
+    committees_per_epoch = get_epoch_committee_count(state, epoch)
     committees_per_slot = committees_per_epoch // SLOTS_PER_EPOCH
     offset = slot % SLOTS_PER_EPOCH
     slot_start_shard = (start_shard + committees_per_slot * offset) % SHARD_COUNT
@@ -1044,7 +1034,7 @@ def verify_merkle_branch(leaf: Bytes32, proof: List[Bytes32], depth: int, index:
 
 ```python
 def get_crosslink_committee_for_attestation(state: BeaconState,
-                                        attestation_data: AttestationData) -> List[ValidatorIndex]:
+                                            attestation_data: AttestationData) -> List[ValidatorIndex]:
     """
     Return the crosslink committee corresponding to ``attestation_data``.
     """                                        
@@ -1199,7 +1189,7 @@ def verify_indexed_attestation(state: BeaconState, indexed_attestation: IndexedA
     if len(custody_bit_1_indices) > 0:  # [TO BE REMOVED IN PHASE 1]
         return False
 
-    total_attesting_indices = len(custody_bit_0_indices + custody_bit_1_indices)
+    total_attesting_indices = len(custody_bit_0_indices) + len(custody_bit_1_indices)
     if not (1 <= total_attesting_indices <= MAX_ATTESTATION_PARTICIPANTS):
         return False
 
@@ -1290,75 +1280,6 @@ def get_delayed_activation_exit_epoch(epoch: Epoch) -> Epoch:
 
 `bls_aggregate_pubkeys` is a function for aggregating multiple BLS public keys into a single aggregate key, defined in the [BLS Signature spec](https://github.com/ethereum/eth2.0-specs/blob/master/specs/bls_signature.md#bls_aggregate_pubkeys).
 
-### `process_deposit`
-
-Used to add a [validator](#dfn-validator) or top up an existing [validator](#dfn-validator)'s balance by some `deposit` amount:
-
-```python
-def process_deposit(state: BeaconState, deposit: Deposit) -> None:
-    """
-    Process a deposit from Ethereum 1.0.
-    Note that this function mutates ``state``.
-    """
-    # Deposits must be processed in order
-    assert deposit.index == state.deposit_index
-
-    # Verify the Merkle branch
-    merkle_branch_is_valid = verify_merkle_branch(
-        leaf=hash(serialize(deposit.data)),  # 48 + 32 + 8 + 96 = 184 bytes serialization
-        proof=deposit.proof,
-        depth=DEPOSIT_CONTRACT_TREE_DEPTH,
-        index=deposit.index,
-        root=state.latest_eth1_data.deposit_root,
-    )
-    assert merkle_branch_is_valid
-
-    # Increment the next deposit index we are expecting. Note that this
-    # needs to be done here because while the deposit contract will never
-    # create an invalid Merkle branch, it may admit an invalid deposit
-    # object, and we need to be able to skip over it
-    state.deposit_index += 1
-
-    validator_pubkeys = [v.pubkey for v in state.validator_registry]
-    pubkey = deposit.data.pubkey
-    amount = deposit.data.amount
-
-    if pubkey not in validator_pubkeys:
-        # Verify the proof of possession
-        proof_is_valid = bls_verify(
-            pubkey=pubkey,
-            message_hash=signing_root(deposit.data),
-            signature=deposit.data.proof_of_possession,
-            domain=get_domain(
-                state.fork,
-                get_current_epoch(state),
-                DOMAIN_DEPOSIT,
-            )
-        )
-        if not proof_is_valid:
-            return
-
-        # Add new validator
-        validator = Validator(
-            pubkey=pubkey,
-            withdrawal_credentials=deposit.data.withdrawal_credentials,
-            activation_epoch=FAR_FUTURE_EPOCH,
-            exit_epoch=FAR_FUTURE_EPOCH,
-            withdrawable_epoch=FAR_FUTURE_EPOCH,
-            initiated_exit=False,
-            slashed=False,
-            high_balance=0
-        )
-
-        # Note: In phase 2 registry indices that have been withdrawn for a long time will be recycled.
-        state.validator_registry.append(validator)
-        state.balances.append(0)
-        set_balance(state, len(state.validator_registry) - 1, amount)
-    else:
-        # Increase balance by deposit amount
-        index = validator_pubkeys.index(pubkey)
-        increase_balance(state, index, amount)
-```
 
 ### Routines for updating validator status
 
@@ -1817,8 +1738,8 @@ def update_justification_and_finalization(state: BeaconState) -> None:
     new_justified_epoch = state.current_justified_epoch
     new_finalized_epoch = state.finalized_epoch
 
-    # Rotate the justification bitfield up one epoch to make room for the current epoch
+    # Rotate the justification bitfield up one epoch to make room for the current epoch (and limit to 64 bits)
-    state.justification_bitfield <<= 1
+    state.justification_bitfield = (state.justification_bitfield << 1) % 2**64
     # If the previous epoch gets justified, fill the second last bit
     previous_boundary_attesting_balance = get_attesting_balance(state, get_previous_epoch_boundary_attestations(state))
     if previous_boundary_attesting_balance * 3 >= get_previous_total_balance(state) * 2:
@@ -2088,7 +2009,7 @@ def update_registry(state: BeaconState) -> None:
         update_validator_registry(state)
     state.latest_start_shard = (
         state.latest_start_shard +
-        get_current_epoch_committee_count(state)
+        get_shard_delta(state, get_current_epoch(state))
     ) % SHARD_COUNT
 ```
 
@@ -2368,7 +2289,76 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None:
 
 Verify that `len(block.body.deposits) == min(MAX_DEPOSITS, latest_eth1_data.deposit_count - state.deposit_index)`.
 
-For each `deposit` in `block.body.deposits`, run `process_deposit(state, deposit)`.
+For each `deposit` in `block.body.deposits`, run the following function:
+
+```python
+def process_deposit(state: BeaconState, deposit: Deposit) -> None:
+    """
+    Process a deposit from Ethereum 1.0.
+    Used to add a validator or top up an existing validator's
+    balance by some ``deposit`` amount.
+
+    Note that this function mutates ``state``.
+    """
+    # Deposits must be processed in order
+    assert deposit.index == state.deposit_index
+
+    # Verify the Merkle branch
+    merkle_branch_is_valid = verify_merkle_branch(
+        leaf=hash(serialize(deposit.data)),  # 48 + 32 + 8 + 96 = 184 bytes serialization
+        proof=deposit.proof,
+        depth=DEPOSIT_CONTRACT_TREE_DEPTH,
+        index=deposit.index,
+        root=state.latest_eth1_data.deposit_root,
+    )
+    assert merkle_branch_is_valid
+
+    # Increment the next deposit index we are expecting. Note that this
+    # needs to be done here because while the deposit contract will never
+    # create an invalid Merkle branch, it may admit an invalid deposit
+    # object, and we need to be able to skip over it
+    state.deposit_index += 1
+
+    validator_pubkeys = [v.pubkey for v in state.validator_registry]
+    pubkey = deposit.data.pubkey
+    amount = deposit.data.amount
+
+    if pubkey not in validator_pubkeys:
+        # Verify the proof of possession
+        proof_is_valid = bls_verify(
+            pubkey=pubkey,
+            message_hash=signing_root(deposit.data),
+            signature=deposit.data.proof_of_possession,
+            domain=get_domain(
+                state.fork,
+                get_current_epoch(state),
+                DOMAIN_DEPOSIT,
+            )
+        )
+        if not proof_is_valid:
+            return
+
+        # Add new validator
+        validator = Validator(
+            pubkey=pubkey,
+            withdrawal_credentials=deposit.data.withdrawal_credentials,
+            activation_epoch=FAR_FUTURE_EPOCH,
+            exit_epoch=FAR_FUTURE_EPOCH,
+            withdrawable_epoch=FAR_FUTURE_EPOCH,
+            initiated_exit=False,
+            slashed=False,
+            high_balance=0
+        )
+
+        # Note: In phase 2 registry indices that have been withdrawn for a long time will be recycled.
+        state.validator_registry.append(validator)
+        state.balances.append(0)
+        set_balance(state, len(state.validator_registry) - 1, amount)
+    else:
+        # Increase balance by deposit amount
+        index = validator_pubkeys.index(pubkey)
+        increase_balance(state, index, amount)
+```
 
 ##### Voluntary exits
 
@@ -2459,17 +2449,3 @@ Verify the block's `state_root` by running the following function:
 def verify_block_state_root(state: BeaconState, block: BeaconBlock) -> None:
     assert block.state_root == hash_tree_root(state)
 ```
-
-# References
-
-This section is divided into Normative and Informative references.  Normative references are those that must be read in order to implement this specification, while Informative references are merely helpful information.  An example of the former might be the details of a required consensus algorithm, and an example of the latter might be a pointer to research that demonstrates why a particular consensus algorithm might be better suited for inclusion in the standard than another.
-
-## Normative
-
-## Informative
-<a id="ref-casper-ffg"></a> _**casper-ffg**_ </br> &nbsp; _Casper the Friendly Finality Gadget_. V. Buterin and V. Griffith. URL: https://arxiv.org/abs/1710.09437
-
-<a id="ref-python-poc"></a> _**python-poc**_ </br> &nbsp; _Python proof-of-concept implementation_. Ethereum Foundation. URL: https://github.com/ethereum/beacon_chain
-
-# Copyright
-Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).

specs/core/1_shard-data-chains.md

@@ -20,6 +20,7 @@
         - [`ShardAttestation`](#shardattestation)
     - [Helper functions](#helper-functions)
         - [`get_period_committee`](#get_period_committee)
+        - [`get_switchover_epoch`](#get_switchover_epoch)
         - [`get_persistent_committee`](#get_persistent_committee)
         - [`get_shard_proposer_index`](#get_shard_proposer_index)
         - [`get_shard_header`](#get_shard_header)
@@ -137,6 +138,14 @@ def get_period_committee(state: BeaconState,
     )
 ```
 
+### `get_switchover_epoch`
+
+```python
+def get_switchover_epoch(state: BeaconState, epoch: Epoch, index: ValidatorIndex):
+    earlier_start_epoch = epoch - (epoch % PERSISTENT_COMMITTEE_PERIOD) - PERSISTENT_COMMITTEE_PERIOD * 2
+    return bytes_to_int(hash(generate_seed(state, earlier_start_epoch) + bytes3(index))[0:8]) % PERSISTENT_COMMITTEE_PERIOD
+```
+
 ### `get_persistent_committee`
 
 ```python
@@ -146,6 +155,7 @@ def get_persistent_committee(state: BeaconState,
     """
     Return the persistent committee for the given ``shard`` at the given ``slot``.
     """
+    epoch = slot_to_epoch(slot)
     earlier_start_epoch = epoch - (epoch % PERSISTENT_COMMITTEE_PERIOD) - PERSISTENT_COMMITTEE_PERIOD * 2
     later_start_epoch = epoch - (epoch % PERSISTENT_COMMITTEE_PERIOD) - PERSISTENT_COMMITTEE_PERIOD
 
@@ -160,14 +170,11 @@ def get_persistent_committee(state: BeaconState,
     earlier_committee = get_period_committee(state, shard, earlier_start_epoch, index, committee_count)
     later_committee = get_period_committee(state, shard, later_start_epoch, index, committee_count)
 
-    def get_switchover_epoch(index):
-        return bytes_to_int(hash(earlier_seed + bytes3(index))[0:8]) % PERSISTENT_COMMITTEE_PERIOD
-
     # Take not-yet-cycled-out validators from earlier committee and already-cycled-in validators from
     # later committee; return a sorted list of the union of the two, deduplicated
     return sorted(list(set(
-        [i for i in earlier_committee if epoch % PERSISTENT_COMMITTEE_PERIOD < get_switchover_epoch(i)] +
+        [i for i in earlier_committee if epoch % PERSISTENT_COMMITTEE_PERIOD < get_switchover_epoch(state, epoch, i)] +
-        [i for i in later_committee if epoch % PERSISTENT_COMMITTEE_PERIOD >= get_switchover_epoch(i)]
+        [i for i in later_committee if epoch % PERSISTENT_COMMITTEE_PERIOD >= get_switchover_epoch(state, epoch, i)]
     )))
 ```
 

specs/simple-serialize.md

@@ -115,10 +115,10 @@ Let `value` be a self-signed container object. The convention is that the signat
 | Language | Project | Maintainer | Implementation |
 |-|-|-|-|
 | Python | Ethereum 2.0 | Ethereum Foundation | [https://github.com/ethereum/py-ssz](https://github.com/ethereum/py-ssz) |
-| Rust | Lighthouse | Sigma Prime | [https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz](https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz) |
+| Rust | Lighthouse | Sigma Prime | [https://github.com/sigp/lighthouse/tree/master/eth2/utils/ssz](https://github.com/sigp/lighthouse/tree/master/eth2/utils/ssz) |
 | Nim | Nimbus | Status | [https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) |
 | Rust | Shasper | ParityTech | [https://github.com/paritytech/shasper/tree/master/util/ssz](https://github.com/paritytech/shasper/tree/master/util/ssz) |
-| Javascript | Lodestart | Chain Safe Systems | [https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js](https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js) |
+| TypeScript | Lodestar | ChainSafe Systems | [https://github.com/ChainSafe/ssz-js](https://github.com/ChainSafe/ssz-js) |
 | Java | Cava | ConsenSys | [https://www.github.com/ConsenSys/cava/tree/master/ssz](https://www.github.com/ConsenSys/cava/tree/master/ssz) |
 | Go | Prysm | Prysmatic Labs | [https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz](https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz) |
 | Swift | Yeeth | Dean Eigenmann | [https://github.com/yeeth/SimpleSerialize.swift](https://github.com/yeeth/SimpleSerialize.swift) |

specs/test_formats/README.md

@@ -7,7 +7,7 @@ This document defines the YAML format and structure used for ETH 2.0 testing.
 * [About](#about)
 * [Glossary](#glossary)
 * [Test format philosophy](#test-format-philosophy)
-* [Test Suite](#yaml-suite)
+* [Test Suite](#test-suite)
 * [Config](#config)
 * [Fork-timeline](#fork-timeline)
 * [Config sourcing](#config-sourcing)
@@ -28,7 +28,7 @@ The particular formats of specific types of tests (test suites) are defined in s
 - `suite`: a YAML file with:
   - a header: describes the `suite`, and defines what the `suite` is for
   - a list of test cases
-- `runner`: where a generator is a *"producer"*, this is the *"consumer"**.
+- `runner`: where a generator is a *"producer"*, this is the *"consumer"*.
   - A `runner` focuses on *only one* `type`, and each type has *only one* `runner`.
 - `handler`: a `runner` may be too limited sometimes, you may have a `suite` with a specific focus that requires a different format.
   To facilitate this, you specify a `handler`: the runner can deal with the format by using the specified handler.