Merge pull request #488 from ethereum/JustinDrake-patch-1

Reduce and revamp custody placeholder logic

specs/core/0_beacon-chain.md

@@ -26,7 +26,7 @@
                 - [`ProposerSlashing`](#proposerslashing)
             - [Casper slashings](#casper-slashings)
                 - [`CasperSlashing`](#casperslashing)
-                - [`SlashableVoteData`](#slashablevotedata)
+                - [`SlashableVote`](#slashablevote)
             - [Attestations](#attestations)
                 - [`Attestation`](#attestation)
                 - [`AttestationData`](#attestationdata)
@@ -85,7 +85,9 @@
             - [`get_effective_balance`](#get_effective_balance)
             - [`get_fork_version`](#get_fork_version)
             - [`get_domain`](#get_domain)
-            - [`verify_slashable_vote_data`](#verify_slashable_vote_data)
+            - [`get_bitfield_bit`](#get_bitfield_bit)
+            - [`verify_bitfield`](#verify_bitfield)
+            - [`verify_slashable_vote`](#verify_slashable_vote)
             - [`is_double_vote`](#is_double_vote)
             - [`is_surround_vote`](#is_surround_vote)
             - [`integer_squareroot`](#integer_squareroot)
@@ -110,7 +112,6 @@
             - [Attestations](#attestations-1)
             - [Deposits](#deposits-1)
             - [Exits](#exits-1)
-            - [Custody](#custody)
     - [Per-epoch processing](#per-epoch-processing)
         - [Helpers](#helpers)
         - [Eth1 data](#eth1-data-1)
@@ -171,7 +172,7 @@ Code snippets appearing in `this style` are to be interpreted as Python code. Be
 | `EJECTION_BALANCE` | `2**4 * 1e9` (= 16,000,000,000) | Gwei |
 | `MAX_BALANCE_CHURN_QUOTIENT` | `2**5` (= 32) | - |
 | `BEACON_CHAIN_SHARD_NUMBER` | `2**64 - 1` | - |
-| `MAX_CASPER_VOTES` | `2**10` (= 1,024) | votes |
+| `MAX_INDICES_PER_SLASHABLE_VOTE` | `2**12` (= 4,096) | votes |
 | `MAX_WITHDRAWALS_PER_EPOCH` | `2**2` (= 4) | withdrawals |
 
 * For the safety of crosslinks `TARGET_COMMITTEE_SIZE` exceeds [the recommended minimum committee size of 111](https://vitalik.ca/files/Ithaca201807_Sharding.pdf); with sufficient active validators (at least `EPOCH_LENGTH * TARGET_COMMITTEE_SIZE`), the shuffling algorithm ensures committee sizes at least `TARGET_COMMITTEE_SIZE`. (Unbiasable randomness with a Verifiable Delay Function (VDF) will improve committee robustness and lower the safe minimum committee size.)
@@ -245,7 +246,7 @@ Code snippets appearing in `this style` are to be interpreted as Python code. Be
 | Name | Value |
 | - | - |
 | `MAX_PROPOSER_SLASHINGS` | `2**4` (= 16) |
-| `MAX_CASPER_SLASHINGS` | `2**4` (= 16) |
+| `MAX_CASPER_SLASHINGS` | `2**0` (= 1) |
 | `MAX_ATTESTATIONS` | `2**7` (= 128) |
 | `MAX_DEPOSITS` | `2**4` (= 16) |
 | `MAX_EXITS` | `2**4` (= 16) |
@@ -273,7 +274,7 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git
 ```python
 {
     # Proposer index
-    'proposer_index': 'uint24',
+    'proposer_index': 'uint64',
     # First proposal data
     'proposal_data_1': ProposalSignedData,
     # First proposal signature
@@ -292,20 +293,20 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git
 ```python
 {
     # First batch of votes
-    'slashable_vote_data_1': SlashableVoteData,
+    'slashable_vote_1': SlashableVote,
     # Second batch of votes
-    'slashable_vote_data_2': SlashableVoteData,
+    'slashable_vote_2': SlashableVote,
 }
 ```
 
-##### `SlashableVoteData`
+##### `SlashableVote`
 
 ```python
 {
-    # Validator indices with custody bit equal to 0
+    # Validator indices
-    'custody_bit_0_indices': ['uint24'],
+    'validator_indices': '[uint64]',
-    # Validator indices with custody bit equal to 1
+    # Custody bitfield
-    'custody_bit_1_indices': ['uint24'],
+    'custody_bitfield': 'bytes',
     # Attestation data
     'data': AttestationData,
     # Aggregate signature
@@ -358,9 +359,9 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git
 ```python
 {
     # Attestation data
-    data: AttestationData,
+    'data': AttestationData,
     # Custody bit
-    custody_bit: bool,
+    'custody_bit': 'bool',
 }
 ```
 
@@ -414,7 +415,7 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git
     # Minimum epoch for processing exit
     'epoch': 'uint64',
     # Index of the exiting validator
-    'validator_index': 'uint24',
+    'validator_index': 'uint64',
     # Validator signature
     'signature': 'bytes96',
 }
@@ -446,16 +447,11 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git
     'proposer_slashings': [ProposerSlashing],
     'casper_slashings': [CasperSlashing],
     'attestations': [Attestation],
-    'custody_reseeds': [CustodyReseed],
-    'custody_challenges': [CustodyChallenge],
-    'custody_responses': [CustodyResponse],
     'deposits': [Deposit],
     'exits': [Exit],
 }
 ```
 
-`CustodyReseed`, `CustodyChallenge`, and `CustodyResponse` will be defined in phase 1; for now, put dummy classes as these lists will remain empty throughout phase 0.
-
 #### `ProposalSignedData`
 
 ```python
@@ -488,7 +484,6 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git
 
     # Randomness and committees
     'latest_randao_mixes': ['bytes32'],
-    'latest_vdf_outputs': ['bytes32'],
     'previous_epoch_start_shard': 'uint64',
     'current_epoch_start_shard': 'uint64',
     'previous_calculation_epoch': 'uint64',
@@ -496,9 +491,6 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git
     'previous_epoch_seed': 'bytes32',
     'current_epoch_seed': 'bytes32',
 
-    # Custody challenges
-    'custody_challenges': [CustodyChallenge],
-
     # Finality
     'previous_justified_epoch': 'uint64',
     'justified_epoch': 'uint64',
@@ -539,10 +531,6 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git
     'exit_count': 'uint64',
     # Status flags
     'status_flags': 'uint64',
-    # Slot of latest custody reseed
-    'latest_custody_reseed_slot': 'uint64',
-    # Slot of second-latest custody reseed
-    'penultimate_custody_reseed_slot': 'uint64',
 }
 ```
 
@@ -611,16 +599,16 @@ The following data structures are defined as [SimpleSerialize (SSZ)](https://git
 
 We define the following Python custom types for type hinting and readability:
 
-| Name | Type | Description |
+| Name | SSZ equivalent | Description |
 | - | - | - |
-| `SlotNumber` | unsigned 64-bit integer | the number of a slot |
+| `SlotNumber` | `uint64` | a slot number |
-| `EpochNumber` | unsigned 64-bit integer | the number of an epoch |
+| `EpochNumber` | `uint64` | an epoch number |
-| `ShardNumber` | unsigned 64-bit integer | the number of a shard |
+| `ShardNumber` | `uint64` | a shard number |
-| `ValidatorIndex` | unsigned 24-bit integer | the index number of a validator in the registry |
+| `ValidatorIndex` | `uint64` | an index in the validator registry |
-| `Gwei` | unsigned 64-bit integer | an amount in Gwei |
+| `Gwei` | `uint64` | an amount in Gwei |
-| `Bytes32` | 32-byte data | binary data with 32-byte length |
+| `Bytes32` | `bytes32` | 32 bytes of binary data |
-| `BLSPubkey` | 48-byte data | a public key in BLS signature scheme |
+| `BLSPubkey` | `bytes48` | a BLS public key |
-| `BLSSignature` | 96-byte data | a signature in BLS signature scheme |
+| `BLSSignature` | `bytes96` | a BLS signature |
 
 ## Ethereum 1.0 deposit contract
 
@@ -1101,23 +1089,23 @@ def merkle_root(values: List[Bytes32]) -> Bytes32:
 ```python
 def get_attestation_participants(state: BeaconState,
                                  attestation_data: AttestationData,
-                                 aggregation_bitfield: bytes) -> List[ValidatorIndex]:
+                                 bitfield: bytes) -> List[ValidatorIndex]:
     """
-    Returns the participant indices at for the ``attestation_data`` and ``aggregation_bitfield``.
+    Returns the participant indices at for the ``attestation_data`` and ``bitfield``.
     """
-
     # Find the committee in the list with the desired shard
     crosslink_committees = get_crosslink_committees_at_slot(state, attestation_data.slot)
 
     assert attestation_data.shard in [shard for _, shard in crosslink_committees]
     crosslink_committee = [committee for committee, shard in crosslink_committees if shard == attestation_data.shard][0]
-    assert len(aggregation_bitfield) == (len(crosslink_committee) + 7) // 8
+
+    assert verify_bitfield(bitfield, len(crosslink_committee))
 
     # Find the participating attesters in the committee
     participants = []
     for i, validator_index in enumerate(crosslink_committee):
-        aggregation_bit = (aggregation_bitfield[i // 8] >> (7 - (i % 8))) % 2
+        aggregation_bit = get_bitfield_bit(bitfield, i)
-        if aggregation_bit == 1:
+        if aggregation_bit == 0b1:
             participants.append(validator_index)
     return participants
 ```
@@ -1159,23 +1147,74 @@ def get_domain(fork: Fork,
     ) * 2**32 + domain_type
 ```
 
-#### `verify_slashable_vote_data`
+#### `get_bitfield_bit`
 
 ```python
-def verify_slashable_vote_data(state: BeaconState, vote_data: SlashableVoteData) -> bool:
+def get_bitfield_bit(bitfield: bytes, i: int) -> int:
-    if len(vote_data.custody_bit_0_indices) + len(vote_data.custody_bit_1_indices) > MAX_CASPER_VOTES:
+    """
+    Extract the bit in ``bitfield`` at position ``i``.
+    """
+    return (bitfield[i // 8] >> (7 - (i % 8))) % 2
+```
+
+#### `verify_bitfield`
+
+```python
+def verify_bitfield(bitfield: bytes, committee_size: int) -> bool:
+    """
+    Verify ``bitfield`` against the ``committee_size``.
+    """
+    if len(bitfield) != (committee_size + 7) // 8:
         return False
 
-    return bls_verify_multiple(
+    for i in range(committee_size + 1, committee_size - committee_size % 8 + 8):
+        if get_bitfield_bit(bitfield, i) == 0b1:
+            return False
+
+    return True
+```
+
+#### `verify_slashable_vote`
+
+```python
+def verify_slashable_vote(state: BeaconState, slashable_vote: SlashableVote) -> bool:
+    """
+    Verify validity of ``slashable_vote`` fields.
+    """
+    if slashable_vote.custody_bitfield != b'\x00' * len(slashable_vote.custody_bitfield):  # [TO BE REMOVED IN PHASE 1]
+        return False
+
+    if len(slashable_vote.validator_indices) == 0:
+        return False
+
+    for i in range(len(slashable_vote.validator_indices) - 1):
+        if slashable_vote.validator_indices[i] >= slashable_vote.validator_indices[i + 1]:
+            return False
+
+    if not verify_bitfield(slashable_vote.custody_bitfield, len(slashable_vote.validator_indices)):
+        return False
+
+    if len(slashable_vote.validator_indices) > MAX_INDICES_PER_SLASHABLE_VOTE:
+        return False
+
+    custody_bit_0_indices = []
+    custody_bit_1_indices = []
+    for i, validator_index in enumerate(slashable_vote.validator_indices):
+        if get_bitfield_bit(slashable_vote.custody_bitfield, i) == 0b0:
+            custody_bit_0_indices.append(validator_index)
+        else:
+            custody_bit_1_indices.append(validator_index)
+
+    return bls_verify(
         pubkeys=[
-            bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in vote_data.custody_bit_0_indices]),
+            bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in custody_bit_0_indices]),
-            bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in vote_data.custody_bit_1_indices]),
+            bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in custody_bit_1_indices]),
         ],
         messages=[
-            hash_tree_root(AttestationDataAndCustodyBit(vote_data.data, False)),
+            hash_tree_root(AttestationDataAndCustodyBit(attestation_data=slashable_vote.data, custody_bit=0b0)),
-            hash_tree_root(AttestationDataAndCustodyBit(vote_data.data, True)),
+            hash_tree_root(AttestationDataAndCustodyBit(attestation_data=slashable_vote.data, custody_bit=0b1)),
         ],
-        signature=vote_data.aggregate_signature,
+        signature=slashable_vote.aggregate_signature,
         domain=get_domain(
             state.fork,
             slot_to_epoch(vote_data.data.slot),
@@ -1280,9 +1319,6 @@ A valid block with slot `GENESIS_SLOT` (a "genesis block") has the following val
         proposer_slashings=[],
         casper_slashings=[],
         attestations=[],
-        custody_reseeds=[],
-        custody_challenges=[],
-        custody_responses=[],
         deposits=[],
         exits=[],
     ),
@@ -1313,7 +1349,6 @@ def get_initial_beacon_state(initial_validator_deposits: List[Deposit],
 
         # Randomness and committees
         latest_randao_mixes=[ZERO_HASH for _ in range(LATEST_RANDAO_MIXES_LENGTH)],
-        latest_vdf_outputs=[ZERO_HASH for _ in range(LATEST_RANDAO_MIXES_LENGTH // EPOCH_LENGTH)],
         previous_epoch_start_shard=GENESIS_START_SHARD,
         current_epoch_start_shard=GENESIS_START_SHARD,
         previous_calculation_epoch=GENESIS_EPOCH,
@@ -1321,9 +1356,6 @@ def get_initial_beacon_state(initial_validator_deposits: List[Deposit],
         previous_epoch_seed=ZERO_HASH,
         current_epoch_seed=ZERO_HASH,
 
-        # Custody challenges
-        custody_challenges=[],
-
         # Finality
         previous_justified_epoch=GENESIS_EPOCH,
         justified_epoch=GENESIS_EPOCH,
@@ -1424,8 +1456,6 @@ def process_deposit(state: BeaconState,
             penalized_epoch=FAR_FUTURE_EPOCH,
             exit_count=0,
             status_flags=0,
-            latest_custody_reseed_slot=GENESIS_SLOT,
-            penultimate_custody_reseed_slot=GENESIS_SLOT,
         )
 
         # Note: In phase 2 registry indices that have been withdrawn for a long time will be recycled.
@@ -1551,15 +1581,14 @@ Verify that `len(block.body.casper_slashings) <= MAX_CASPER_SLASHINGS`.
 
 For each `casper_slashing` in `block.body.casper_slashings`:
 
-* Let `slashable_vote_data_1 = casper_slashing.slashable_vote_data_1`.
+* Let `slashable_vote_1 = casper_slashing.slashable_vote_1`.
-* Let `slashable_vote_data_2 = casper_slashing.slashable_vote_data_2`.
+* Let `slashable_vote_2 = casper_slashing.slashable_vote_2`.
-* Let `indices(slashable_vote_data) = slashable_vote_data.custody_bit_0_indices + slashable_vote_data.custody_bit_1_indices`.
+* Let `intersection = [x for x in slashable_vote_1.validator_indices if x in slashable_vote_2.validator_indices]`.
-* Let `intersection = [x for x in indices(slashable_vote_data_1) if x in indices(slashable_vote_data_2)]`.
 * Verify that `len(intersection) >= 1`.
-* Verify that `slashable_vote_data_1.data != slashable_vote_data_2.data`.
+* Verify that `slashable_vote_1.data != slashable_vote_2.data`.
-* Verify that `is_double_vote(slashable_vote_data_1.data, slashable_vote_data_2.data)` or `is_surround_vote(slashable_vote_data_1.data, slashable_vote_data_2.data)`.
+* Verify that `is_double_vote(slashable_vote_1.data, slashable_vote_2.data)` or `is_surround_vote(slashable_vote_1.data, slashable_vote_2.data)`.
-* Verify that `verify_slashable_vote_data(state, slashable_vote_data_1)`.
+* Verify that `verify_slashable_vote(state, slashable_vote_1)`.
-* Verify that `verify_slashable_vote_data(state, slashable_vote_data_2)`.
+* Verify that `verify_slashable_vote(state, slashable_vote_2)`.
 * For each [validator](#dfn-validator) index `i` in `intersection` run `penalize_validator(state, i)` if `state.validator_registry[i].penalized_epoch > get_current_epoch(state)`.
 
 #### Attestations
@@ -1573,10 +1602,33 @@ For each `attestation` in `block.body.attestations`:
 * Verify that `attestation.data.justified_epoch` is equal to `state.justified_epoch if attestation.data.slot >= get_epoch_start_slot(get_current_epoch(state)) else state.previous_justified_epoch`.
 * Verify that `attestation.data.justified_block_root` is equal to `get_block_root(state, get_epoch_start_slot(attestation.data.justified_epoch))`.
 * Verify that either `attestation.data.latest_crosslink_root` or `attestation.data.shard_block_root` equals `state.latest_crosslinks[shard].shard_block_root`.
-* `aggregate_signature` verification:
+* Verify bitfields and aggregate signature:
-    * Let `participants = get_attestation_participants(state, attestation.data, attestation.aggregation_bitfield)`.
+
-    * Let `group_public_key = bls_aggregate_pubkeys([state.validator_registry[v].pubkey for v in participants])`.
+```python
-    * Verify that `bls_verify(pubkey=group_public_key, message=hash_tree_root(AttestationDataAndCustodyBit(attestation.data, False)), signature=attestation.aggregate_signature, domain=get_domain(state.fork, slot_to_epoch(attestation.data.slot), DOMAIN_ATTESTATION))`.
+    assert attestation.custody_bitfield == b'\x00' * len(attestation.custody_bitfield)  # [TO BE REMOVED IN PHASE 1]
+
+    for i in range(len(crosslink_committee)):
+        if get_bitfield_bit(attestation.aggregation_bitfield) == 0b0:
+            assert get_bitfield_bit(attestation.custody_bitfield) == 0b0
+
+    participants = get_attestation_participants(state, attestation.data, attestation.aggregation_bitfield)
+    custody_bit_1_participants = get_attestation_participants(state, attestation.data, attestation.custody_bitfield)
+    custody_bit_0_participants = [i in participants for i not in custody_bit_1_participants]
+
+    assert bls_verify_multiple(
+        pubkeys=[
+            bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in custody_bit_0_participants]),
+            bls_aggregate_pubkeys([state.validator_registry[i].pubkey for i in custody_bit_1_participants]),
+        ],
+        messages=[
+            hash_tree_root(AttestationDataAndCustodyBit(data=attestation.data, custody_bit=0b0)),
+            hash_tree_root(AttestationDataAndCustodyBit(data=attestation.data, custody_bit=0b1)),
+        ],
+        signature=attestation.aggregate_signature,
+        domain=get_domain(state.fork, slot_to_epoch(attestation.data.slot), DOMAIN_ATTESTATION),
+    ) 
+```
+
 * [TO BE REMOVED IN PHASE 1] Verify that `attestation.data.shard_block_root == ZERO_HASH`.
 * Append `PendingAttestation(data=attestation.data, aggregation_bitfield=attestation.aggregation_bitfield, custody_bitfield=attestation.custody_bitfield, slot_included=state.slot)` to `state.latest_attestations`.
 
@@ -1628,10 +1680,6 @@ For each `exit` in `block.body.exits`:
 * Verify that `bls_verify(pubkey=validator.pubkey, message=exit_message, signature=exit.signature, domain=get_domain(state.fork, exit.epoch, DOMAIN_EXIT))`.
 * Run `initiate_validator_exit(state, exit.validator_index)`.
 
-#### Custody
-
-[TO BE REMOVED IN PHASE 1] Verify that `len(block.body.custody_reseeds) == len(block.body.custody_challenges) == len(block.body.custody_responses) == 0`.
-
 ## Per-epoch processing
 
 The steps below happen when `(state.slot + 1) % EPOCH_LENGTH == 0`.