Persistent committee size per slot reduced to max 128 (#734)

* Persistent committee size per slot target 128 max 256

Cuts down the cost of verifying a shard chain and aggregating signatures for a shard chain, and also makes the shard chain signatures more usable by light clients for verification as they would only need to keep track of a max 256-sized committee.

specs/core/1_shard-data-chains.md

@@ -17,41 +17,51 @@ At the current stage, Phase 1, while fundamentally feature-complete, is still su
             - [Time parameters](#time-parameters)
             - [Max operations per block](#max-operations-per-block)
             - [Signature domains](#signature-domains)
-    - [Shard chains and crosslink data](#shard-chains-and-crosslink-data)
+- [Shard chains and crosslink data](#shard-chains-and-crosslink-data)
-      - [Helper functions](#helper-functions)
+    - [Helper functions](#helper-functions)
             - [`get_split_offset`](#get_split_offset)
             - [`get_shuffled_committee`](#get_shuffled_committee)
             - [`get_persistent_committee`](#get_persistent_committee)
             - [`get_shard_proposer_index`](#get_shard_proposer_index)
-      - [Data Structures](#data-structures)
+    - [Data Structures](#data-structures)
         - [Shard chain blocks](#shard-chain-blocks)
-      - [Shard block processing](#shard-block-processing)
+    - [Shard block processing](#shard-block-processing)
         - [Verifying shard block data](#verifying-shard-block-data)
         - [Verifying a crosslink](#verifying-a-crosslink)
         - [Shard block fork choice rule](#shard-block-fork-choice-rule)
-    - [Updates to the beacon chain](#updates-to-the-beacon-chain)
+- [Updates to the beacon chain](#updates-to-the-beacon-chain)
-      - [Data structures](#data-structures)
+    - [Data structures](#data-structures)
         - [`Validator`](#validator)
         - [`BeaconBlockBody`](#beaconblockbody)
+        - [`BeaconState`](#beaconstate)
         - [`BranchChallenge`](#branchchallenge)
         - [`BranchResponse`](#branchresponse)
         - [`BranchChallengeRecord`](#branchchallengerecord)
+        - [`InteractiveCustodyChallengeRecord`](#interactivecustodychallengerecord)
+        - [`InteractiveCustodyChallengeInitiation`](#interactivecustodychallengeinitiation)
+        - [`InteractiveCustodyChallengeResponse`](#interactivecustodychallengeresponse)
+        - [`InteractiveCustodyChallengeContinuation`](#interactivecustodychallengecontinuation)
         - [`SubkeyReveal`](#subkeyreveal)
     - [Helpers](#helpers)
-        - [`get_attestation_data_merkle_depth`](#get_attestation_data_merkle_depth)
+        - [`get_branch_challenge_record_by_id`](#get_branch_challenge_record_by_id)
+        - [`get_custody_challenge_record_by_id`](#get_custody_challenge_record_by_id)
+        - [`get_attestation_merkle_depth`](#get_attestation_merkle_depth)
         - [`epoch_to_custody_period`](#epoch_to_custody_period)
         - [`slot_to_custody_period`](#slot_to_custody_period)
         - [`get_current_custody_period`](#get_current_custody_period)
         - [`verify_custody_subkey_reveal`](#verify_custody_subkey_reveal)
-        - [`prepare_validator_for_withdrawal`](#prepare_validator_for_withdrawal)
+        - [`verify_signed_challenge_message`](#verify_signed_challenge_message)
         - [`penalize_validator`](#penalize_validator)
-      - [Per-slot processing](#per-slot-processing)
+    - [Per-slot processing](#per-slot-processing)
         - [Operations](#operations)
             - [Branch challenges](#branch-challenges)
             - [Branch responses](#branch-responses)
             - [Subkey reveals](#subkey-reveals)
-      - [Per-epoch processing](#per-epoch-processing)
+            - [Interactive custody challenge initiations](#interactive-custody-challenge-initiations)
-      - [One-time phase 1 initiation transition](#one-time-phase-1-initiation-transition)
+            - [Interactive custody challenge responses](#interactive-custody-challenge-responses)
+            - [Interactive custody challenge continuations](#interactive-custody-challenge-continuations)
+    - [Per-epoch processing](#per-epoch-processing)
+    - [One-time phase 1 initiation transition](#one-time-phase-1-initiation-transition)
 
 <!-- /TOC -->
 
@@ -128,16 +138,27 @@ def get_split_offset(list_size: int, chunks: int, index: int) -> int:
 ```python
 def get_shuffled_committee(state: BeaconState,
                            shard: Shard,
-                           committee_start_epoch: Epoch) -> List[ValidatorIndex]:
+                           committee_start_epoch: Epoch,
+                           index: int,
+                           committee_count: int) -> List[ValidatorIndex]:
     """
     Return shuffled committee.
     """
-    validator_indices = get_active_validator_indices(state.validators, committee_start_epoch)
+    active_validator_indices = get_active_validator_indices(state.validator_registry, committee_start_epoch)
+    length = len(active_validator_indices)
     seed = generate_seed(state, committee_start_epoch)
-    start_offset = get_split_offset(len(validator_indices), SHARD_COUNT, shard)
+    start_offset = get_split_offset(
-    end_offset = get_split_offset(len(validator_indices), SHARD_COUNT, shard + 1)
+        length,
+        SHARD_COUNT * committee_count,
+        shard * committee_count + index,
+    )
+    end_offset = get_split_offset(
+        length,
+        SHARD_COUNT * committee_count,
+        shard * committee_count + index + 1,
+    )
     return [
-        validator_indices[get_permuted_index(i, len(validator_indices), seed)]
+        active_validator_indices[get_permuted_index(i, length, seed)]
         for i in range(start_offset, end_offset)
     ]
 ```
@@ -147,15 +168,24 @@ def get_shuffled_committee(state: BeaconState,
 ```python
 def get_persistent_committee(state: BeaconState,
                              shard: Shard,
-                             epoch: Epoch) -> List[ValidatorIndex]:
+                             slot: Slot) -> List[ValidatorIndex]:
     """
-    Return the persistent committee for the given ``shard`` at the given ``epoch``.
+    Return the persistent committee for the given ``shard`` at the given ``slot``.
     """
-    earlier_committee_start_epoch = epoch - (epoch % PERSISTENT_COMMITTEE_PERIOD) - PERSISTENT_COMMITTEE_PERIOD * 2
+        
-    earlier_committee = get_shuffled_committee(state, shard, earlier_committee_start_epoch)
+    earlier_start_epoch = epoch - (epoch % PERSISTENT_COMMITTEE_PERIOD) - PERSISTENT_COMMITTEE_PERIOD * 2
+    later_start_epoch = epoch - (epoch % PERSISTENT_COMMITTEE_PERIOD) - PERSISTENT_COMMITTEE_PERIOD
 
-    later_committee_start_epoch = epoch - (epoch % PERSISTENT_COMMITTEE_PERIOD) - PERSISTENT_COMMITTEE_PERIOD
+    committee_count = max(
-    later_committee = get_shuffled_committee(state, shard, later_committee_start_epoch)
+        len(get_active_validator_indices(state.validator_registry, earlier_start_epoch)) //
+        (SHARD_COUNT * TARGET_COMMITTEE_SIZE),
+        len(get_active_validator_indices(state.validator_registry, later_start_epoch)) //
+        (SHARD_COUNT * TARGET_COMMITTEE_SIZE),
+    ) + 1
+    
+    index = slot % committee_count
+    earlier_committee = get_shuffled_committee(state, shard, earlier_start_epoch, index, committee_count)
+    later_committee = get_shuffled_committee(state, shard, later_start_epoch, index, committee_count)
 
     def get_switchover_epoch(index):
         return (
@@ -170,6 +200,7 @@ def get_persistent_committee(state: BeaconState,
         [i for i in later_committee if epoch % PERSISTENT_COMMITTEE_PERIOD >= get_switchover_epoch(i)]
     )))
 ```
+
 #### `get_shard_proposer_index`
 
 ```python
@@ -181,14 +212,14 @@ def get_shard_proposer_index(state: BeaconState,
         int_to_bytes8(shard) +
         int_to_bytes8(slot)
     )
-    persistent_committee = get_persistent_committee(state, shard, slot_to_epoch(slot))
+    persistent_committee = get_persistent_committee(state, shard, slot)
     # Default proposer
     index = bytes_to_int(seed[0:8]) % len(persistent_committee)
     # If default proposer exits, try the other proposers in order; if all are exited
     # return None (ie. no block can be proposed)
     validators_to_try = persistent_committee[index:] + persistent_committee[:index]
     for index in validators_to_try:
-        if is_active_validator(state.validators[index], get_current_epoch(state)):
+        if is_active_validator(state.validator_registry[index], get_current_epoch(state)):
             return index
     return None
 ```
@@ -233,14 +264,14 @@ To validate a block header on shard `shard_block.shard_id`, compute as follows:
 * Verify that `shard_block.beacon_chain_ref` is the hash of a block in the (canonical) beacon chain with slot less than or equal to `slot`.
 * Verify that `shard_block.beacon_chain_ref` is equal to or a descendant of the `shard_block.beacon_chain_ref` specified in the `ShardBlock` pointed to by `shard_block.parent_root`.
 * Let `state` be the state of the beacon chain block referred to by `shard_block.beacon_chain_ref`.
-* Let `persistent_committee = get_persistent_committee(state, shard_block.shard_id, slot_to_epoch(shard_block.slot))`.
+* Let `persistent_committee = get_persistent_committee(state, shard_block.shard_id, shard_block.slot)`.
 * Assert `verify_bitfield(shard_block.participation_bitfield, len(persistent_committee))`
-* For every `i in range(len(persistent_committee))` where `is_active_validator(state.validators[persistent_committee[i]], get_current_epoch(state))` returns `False`, verify that `get_bitfield_bit(shard_block.participation_bitfield, i) == 0`
+* For every `i in range(len(persistent_committee))` where `is_active_validator(state.validator_registry[persistent_committee[i]], get_current_epoch(state))` returns `False`, verify that `get_bitfield_bit(shard_block.participation_bitfield, i) == 0`
 * Let `proposer_index = get_shard_proposer_index(state, shard_block.shard_id, shard_block.slot)`.
 * Verify that `proposer_index` is not `None`.
 * Let `msg` be the `shard_block` but with `shard_block.signature` set to `[0, 0]`.
 * Verify that `bls_verify(pubkey=validators[proposer_index].pubkey, message_hash=hash(msg), signature=shard_block.signature, domain=get_domain(state, slot_to_epoch(shard_block.slot), DOMAIN_SHARD_PROPOSER))` passes.
-* Let `group_public_key = bls_aggregate_pubkeys([state.validators[index].pubkey for i, index in enumerate(persistent_committee) if get_bitfield_bit(shard_block.participation_bitfield, i) is True])`.
+* Let `group_public_key = bls_aggregate_pubkeys([state.validator_registry[index].pubkey for i, index in enumerate(persistent_committee) if get_bitfield_bit(shard_block.participation_bitfield, i) is True])`.
 * Verify that `bls_verify(pubkey=group_public_key, message_hash=shard_block.parent_root, sig=shard_block.aggregate_signature, domain=get_domain(state, slot_to_epoch(shard_block.slot), DOMAIN_SHARD_ATTESTER))` passes.
 
 ### Verifying shard block data