Merge pull request #1440 from ethereum/naive-aggregation

Naive aggregation strategy

scripts/build_spec.py

@@ -28,6 +28,7 @@ from eth2spec.utils.ssz.ssz_typing import (
     Bytes1, Bytes4, Bytes8, Bytes32, Bytes48, Bytes96, Bitlist, Bitvector,
 )
 from eth2spec.utils.bls import (
+    bls_aggregate_signatures,
     bls_aggregate_pubkeys,
     bls_verify,
     bls_verify_multiple,

specs/networking/p2p-interface.md

@@ -115,9 +115,10 @@ This section outlines constants that are used in this spec.
 |---|---|---|
 | `GOSSIP_MAX_SIZE` | `2**20` (= 1048576, 1 MiB) | The maximum allowed size of uncompressed gossip messages. |
 | `MAX_CHUNK_SIZE` | `2**20` (1048576, 1 MiB) | The maximum allowed size of uncompressed req/resp chunked responses. |
-| `SHARD_SUBNET_COUNT` | `TODO` | The number of shard subnets used in the gossipsub protocol. |
+| `ATTESTATION_SUBNET_COUNT` | `64` | The number of attestation subnets used in the gossipsub protocol. |
 | `TTFB_TIMEOUT` | `5s` | The maximum time to wait for first byte of request response (time-to-first-byte). |
 | `RESP_TIMEOUT` | `10s` | The maximum time for complete response transfer. |
+| `ATTESTATION_PROPAGATION_SLOT_RANGE` | `32` | The maximum number of slots during which an attestation can be propagated. |
 
 ## The gossip domain: gossipsub
 
@@ -140,54 +141,72 @@ The following gossipsub [parameters](https://github.com/libp2p/specs/tree/master
 - `gossip_history` (number of heartbeat intervals to retain message IDs): 5
 - `heartbeat_interval` (frequency of heartbeat, seconds): 1
 
-### Topics
+### Topics and messages
 
-Topics are plain UTF-8 strings and are encoded on the wire as determined by protobuf (gossipsub messages are enveloped in protobuf messages).
+Topics are plain UTF-8 strings and are encoded on the wire as determined by protobuf (gossipsub messages are enveloped in protobuf messages). Topic strings have form: `/eth2/TopicName/TopicEncoding`. This defines both the type of data being sent on the topic and how the data field of the message is encoded.
 
-Topic strings have form: `/eth2/TopicName/TopicEncoding`. This defines both the type of data being sent on the topic and how the data field of the message is encoded. (Further details can be found in [Messages](#Messages)).
+Each gossipsub [message](https://github.com/libp2p/go-libp2p-pubsub/blob/master/pb/rpc.proto#L17-L24) has a maximum size of `GOSSIP_MAX_SIZE`. Clients MUST reject (fail validation) messages that are over this size limit. Likewise, clients MUST NOT emit or propagate messages larger than this limit.
 
-There are two main topics used to propagate attestations and beacon blocks to all nodes on the network. Their `TopicName`s are:
+The payload is carried in the `data` field of a gossipsub message, and varies depending on the topic:
+
+| Topic                                  | Message Type      |
+|----------------------------------------|-------------------|
+| beacon_block                           | BeaconBlock       |
+| beacon_aggregate_and_proof             | AggregateAndProof |
+| beacon_attestation\*                   | Attestation       |
+| committee_index{subnet_id}\_beacon_attestation | Attestation       |
+| voluntary_exit                         | VoluntaryExit     |
+| proposer_slashing                      | ProposerSlashing  |
+| attester_slashing                      | AttesterSlashing  |
+
+Clients MUST reject (fail validation) messages containing an incorrect type, or invalid payload.
+
+When processing incoming gossip, clients MAY descore or disconnect peers who fail to observe these constraints.
+
+\* The `beacon_attestation` topic is only for interop and will be removed prior to mainnet.
+
+#### Global topics
+
+There are two primary global topics used to propagate beacon blocks and aggregate attestations to all nodes on the network. Their `TopicName`s are:
 
 - `beacon_block` - This topic is used solely for propagating new beacon blocks to all nodes on the networks. Blocks are sent in their entirety. Clients MUST validate the block proposer signature before forwarding it across the network.
-- `beacon_attestation` - This topic is used to propagate aggregated attestations (in their entirety) to subscribing nodes (typically block proposers) to be included in future blocks. Clients MUST validate that the block being voted for passes validation before forwarding the attestation on the network (TODO: [additional validations](https://github.com/ethereum/eth2.0-specs/issues/1332)).
+- `beacon_aggregate_and_proof` - This topic is used to propagate aggregated attestations (as `AggregateAndProof`s) to subscribing nodes (typically validators) to be included in future blocks. The following validations MUST pass before forwarding the `aggregate_and_proof` on the network.
+    - The aggregate attestation defined by `hash_tree_root(aggregate_and_proof.aggregate)` has _not_ already been seen (via aggregate gossip, within a block, or through the creation of an equivalent aggregate locally).
+    - The block being voted for (`aggregate_and_proof.aggregate.data.beacon_block_root`) passes validation.
+    - `aggregate_and_proof.aggregate.data.slot` is within the last `ATTESTATION_PROPAGATION_SLOT_RANGE` slots (`aggregate_and_proof.aggregate.data.slot + ATTESTATION_PROPAGATION_SLOT_RANGE >= current_slot >= aggregate_and_proof.aggregate.data.slot`).
+    - The validator index is within the aggregate's committee -- i.e. `aggregate_and_proof.index in get_attesting_indices(state, aggregate_and_proof.aggregate.data, aggregate_and_proof.aggregate.aggregation_bits)`.
+    - `aggregate_and_proof.selection_proof` selects the validator as an aggregator for the slot -- i.e. `is_aggregator(state, aggregate_and_proof.aggregate.data.index, aggregate_and_proof.selection_proof)` returns `True`.
+    - The `aggregate_and_proof.selection_proof` is a valid signature of the `aggregate_and_proof.aggregate.data.slot` by the validator with index `aggregate_and_proof.index`.
+    - The signature of `aggregate_and_proof.aggregate` is valid.
 
-Additional topics are used to propagate lower frequency validator messages. Their `TopicName`s are:
+Additional global topics are used to propagate lower frequency validator messages. Their `TopicName`s are:
 
 - `voluntary_exit` - This topic is used solely for propagating voluntary validator exits to proposers on the network. Voluntary exits are sent in their entirety. Clients who receive a voluntary exit on this topic MUST validate the conditions within `process_voluntary_exit` before forwarding it across the network.
 - `proposer_slashing` - This topic is used solely for propagating proposer slashings to proposers on the network. Proposer slashings are sent in their entirety. Clients who receive a proposer slashing on this topic MUST validate the conditions within `process_proposer_slashing` before forwarding it across the network.
 - `attester_slashing` - This topic is used solely for propagating attester slashings to proposers on the network. Attester slashings are sent in their entirety. Clients who receive an attester slashing on this topic MUST validate the conditions within `process_attester_slashing` before forwarding it across the network.
 
+#### Attestation subnets
+
+Attestation subnets are used to propagate unaggregated attestations to subsections of the network. Their `TopicName`s are:
+
+- `committee_index{subnet_id}_beacon_attestation` - These topics are used to propagate unaggregated attestations to the subnet `subnet_id` (typically beacon and persistent committees) to be aggregated before being gossiped to `beacon_aggregate_and_proof`. The following validations MUST pass before forwarding the `attestation` on the subnet.
+    - The attestation's committee index (`attestation.data.index`) is for the correct subnet.
+    - The attestation is unaggregated -- that is, it has exactly one participating validator (`len([bit for bit in attestation.aggregation_bits if bit == 0b1]) == 1`).
+    - The block being voted for (`attestation.data.beacon_block_root`) passes validation.
+    - `attestation.data.slot` is within the last `ATTESTATION_PROPAGATION_SLOT_RANGE` slots (`attestation.data.slot + ATTESTATION_PROPAGATION_SLOT_RANGE >= current_slot >= attestation.data.slot`).
+    - The signature of `attestation` is valid.
+
 #### Interop
 
-Unaggregated and aggregated attestations from all shards are sent to the `beacon_attestation` topic. Clients are not required to publish aggregate attestations but must be able to process them. All validating clients SHOULD try to perform local attestation aggregation to prepare for block proposing.
+Unaggregated and aggregated attestations from all shards are sent as `Attestation`s to the `beacon_attestation` topic. Clients are not required to publish aggregate attestations but must be able to process them. All validating clients SHOULD try to perform local attestation aggregation to prepare for block proposing.
 
 #### Mainnet
 
-Shards are grouped into their own subnets (defined by a shard topic). The number of shard subnets is defined via `SHARD_SUBNET_COUNT` and the shard `shard_number % SHARD_SUBNET_COUNT` is assigned to the topic: `shard{shard_number % SHARD_SUBNET_COUNT}_beacon_attestation`. Unaggregated attestations are sent to the subnet topic. Aggregated attestations are sent to the `beacon_attestation` topic.
+Attestation broadcasting is grouped into subnets defined by a topic. The number of subnets is defined via `ATTESTATION_SUBNET_COUNT`. For the `committee_index{subnet_id}_beacon_attestation` topics, `subnet_id` is set to `index % ATTESTATION_SUBNET_COUNT`, where `index` is the `CommitteeIndex` of the given committee.
 
-TODO: [aggregation strategy](https://github.com/ethereum/eth2.0-specs/issues/1331)
+Unaggregated attestations are sent to the subnet topic, `committee_index{attestation.data.index % ATTESTATION_SUBNET_COUNT}_beacon_attestation` as `Attestation`s.
 
-### Messages
+Aggregated attestations are sent to the `beacon_aggregate_and_proof` topic as `AggregateAndProof`s.
-
-Each gossipsub [message](https://github.com/libp2p/go-libp2p-pubsub/blob/master/pb/rpc.proto#L17-L24) has a maximum size of `GOSSIP_MAX_SIZE`.
-
-Clients MUST reject (fail validation) messages that are over this size limit. Likewise, clients MUST NOT emit or propagate messages larger than this limit.
-
-The payload is carried in the `data` field of a gossipsub message, and varies depending on the topic:
-
-
-| Topic                        | Message Type      |
-|------------------------------|-------------------|
-| beacon_block                 | BeaconBlock       |
-| beacon_attestation           | Attestation       |
-| shard{N}\_beacon_attestation | Attestation       |
-| voluntary_exit               | VoluntaryExit     |
-| proposer_slashing            | ProposerSlashing  |
-| attester_slashing            | AttesterSlashing  |
-
-Clients MUST reject (fail validation) messages containing an incorrect type, or invalid payload.
-
-When processing incoming gossip, clients MAY descore or disconnect peers who fail to observe these constraints.
 
 ### Encodings
 
@@ -199,7 +218,7 @@ Topics are post-fixed with an encoding. Encodings define how the payload of a go
 
 #### Mainnet
 
-- `ssz_snappy` - All objects are SSZ-encoded and then compressed with [Snappy](https://github.com/google/snappy). Example: The beacon attestation topic string is `/eth2/beacon_attestation/ssz_snappy`, and the data field of a gossipsub message is an `Attestation` that has been SSZ-encoded and then compressed with Snappy.
+- `ssz_snappy` - All objects are SSZ-encoded and then compressed with [Snappy](https://github.com/google/snappy). Example: The beacon aggregate attestation topic string is `/eth2/beacon_aggregate_and_proof/ssz_snappy`, and the data field of a gossipsub message is an `AggregateAndProof` that has been SSZ-encoded and then compressed with Snappy.
 
 Implementations MUST use a single encoding. Changing an encoding will require coordination between participating implementations.
 
@@ -654,9 +673,21 @@ No security or privacy guarantees are lost as a result of choosing plaintext top
 
 Furthermore, the Eth2 topic names are shorter than their digest equivalents (assuming SHA-256 hash), so hashing topics would bloat messages unnecessarily.
 
-### Why are there `SHARD_SUBNET_COUNT` subnets, and why is this not defined?
+### Why are there `ATTESTATION_SUBNET_COUNT` attestation subnets?
 
-Depending on the number of validators, it may be more efficient to group shard subnets and might provide better stability for the gossipsub channel. The exact grouping will be dependent on more involved network tests. This constant allows for more flexibility in setting up the network topology for attestation aggregation (as aggregation should happen on each subnet).
+Depending on the number of validators, it may be more efficient to group shard subnets and might provide better stability for the gossipsub channel. The exact grouping will be dependent on more involved network tests. This constant allows for more flexibility in setting up the network topology for attestation aggregation (as aggregation should happen on each subnet). The value is currently set to to be equal `MAX_COMMITTEES_PER_SLOT` until network tests indicate otherwise.
+
+### Why are attestations limited to be broadcast on gossip channels within `SLOTS_PER_EPOCH` slots?
+
+Attestations can only be included on chain within an epoch's worth of slots so this is the natural cutoff. There is no utility to the chain to broadcast attestations older than one epoch, and because validators have a chance to make a new attestation each epoch, there is minimal utility to the fork choice to relay old attestations as a new latest message can soon be created by each validator.
+
+In addition to this, relaying attestations requires validating the attestation in the context of the `state` during which it was created. Thus, validating arbitrarily old attestations would put additional requirements on which states need to be readily available to the node. This would result in a higher resource burden and could serve as a DoS vector.
+
+### Why are aggregate attestations broadcast to the global topic as `AggregateAndProof`s rather than just as `Attestation`s?
+
+The dominant strategy for an individual validator is to always broadcast an aggregate containing their own attestation to the global channel to ensure that proposers see their attestation for inclusion. Using a private selection criteria and providing this proof of selection alongside the gossiped aggregate ensures that this dominant strategy will not flood the global channel.
+
+Also, an attacker can create any number of honest-looking aggregates and broadcast them to the global pubsub channel. Thus without some sort of proof of selection as an aggregator, the global channel can trivially be spammed.
 
 ### Why are we sending entire objects in the pubsub and not just hashes?
 

specs/validator/0_beacon-chain-validator.md

@@ -37,8 +37,9 @@
                 - [Attestations](#attestations)
                 - [Deposits](#deposits)
                 - [Voluntary exits](#voluntary-exits)
-        - [Attestations](#attestations-1)
+        - [Attesting](#attesting)
             - [Attestation data](#attestation-data)
+                - [General](#general)
                 - [LMD GHOST vote](#lmd-ghost-vote)
                 - [FFG vote](#ffg-vote)
             - [Construct attestation](#construct-attestation)
@@ -46,6 +47,16 @@
                 - [Aggregation bits](#aggregation-bits)
                 - [Custody bits](#custody-bits)
                 - [Aggregate signature](#aggregate-signature)
+            - [Broadcast attestation](#broadcast-attestation)
+        - [Attestation aggregation](#attestation-aggregation)
+            - [Aggregation selection](#aggregation-selection)
+            - [Construct aggregate](#construct-aggregate)
+                - [Data](#data-1)
+                - [Aggregation bits](#aggregation-bits-1)
+                - [Custody bits](#custody-bits-1)
+                - [Aggregate signature](#aggregate-signature-1)
+            - [Broadcast aggregate](#broadcast-aggregate)
+                - [`AggregateAndProof`](#aggregateandproof)
     - [How to avoid slashing](#how-to-avoid-slashing)
         - [Proposer slashing](#proposer-slashing)
         - [Attester slashing](#attester-slashing)
@@ -69,6 +80,7 @@ All terminology, constants, functions, and protocol mechanics defined in the [Ph
 | Name | Value | Unit | Duration |
 | - | - | :-: | :-: |
 | `ETH1_FOLLOW_DISTANCE` | `2**10` (= 1,024) | blocks | ~4 hours |
+| `TARGET_AGGREGATORS_PER_COMMITTEE` | `2**4` (= 16) | validators | |
 
 ## Becoming a validator
 
@@ -273,11 +285,11 @@ The `proof` for each deposit must be constructed against the deposit root contai
 
 Up to `MAX_VOLUNTARY_EXITS`, [`VoluntaryExit`](../core/0_beacon-chain.md#voluntaryexit) objects can be included in the `block`. The exits must satisfy the verification conditions found in [exits processing](../core/0_beacon-chain.md#voluntary-exits).
 
-### Attestations
+### Attesting
 
 A validator is expected to create, sign, and broadcast an attestation during each epoch. The `committee`, assigned `index`, and assigned `slot` for which the validator performs this role during an epoch are defined by `get_committee_assignment(state, epoch, validator_index)`.
 
-A validator should create and broadcast the attestation halfway through the `slot` during which the validator is assigned―that is, `SECONDS_PER_SLOT * 0.5` seconds after the start of `slot`.
+A validator should create and broadcast the `attestation` to the associated attestation subnet one-third of the way through the `slot` during which the validator is assigned―that is, `SECONDS_PER_SLOT / 3` seconds after the start of `slot`.
 
 #### Attestation data
 
@@ -315,7 +327,7 @@ Set `attestation.data = attestation_data` where `attestation_data` is the `Attes
 
 ##### Aggregation bits
 
-- Let `attestation.aggregation_bits` be a `Bitlist[MAX_VALIDATORS_PER_COMMITTEE]` where the bits at the index in the aggregated validator's `committee` is set to `0b1`.
+- Let `attestation.aggregation_bits` be a `Bitlist[MAX_VALIDATORS_PER_COMMITTEE]` of length `len(committee)`, where the bit of the index of the validator in the `committee` is set to `0b1`.
 
 *Note*: Calling `get_attesting_indices(state, attestation.data, attestation.aggregation_bits)` should return a list of length equal to 1, containing `validator_index`.
 
@@ -340,6 +352,81 @@ def get_signed_attestation_data(state: BeaconState, attestation: IndexedAttestat
     return bls_sign(privkey, hash_tree_root(attestation_data_and_custody_bit), domain)
 ```
 
+#### Broadcast attestation
+
+Finally, the validator broadcasts `attestation` to the associated attestation subnet -- the `index{attestation.data.index % ATTESTATION_SUBNET_COUNT}_beacon_attestation` pubsub topic.
+
+### Attestation aggregation
+
+Some validators are selected to locally aggregate attestations with a similar `attestation_data` to their constructed `attestation` for the assigned `slot`.
+
+#### Aggregation selection
+
+A validator is selected to aggregate based upon the return value of `is_aggregator()`.
+
+```python
+def slot_signature(state: BeaconState, slot: Slot, privkey: int) -> BLSSignature:
+    domain = get_domain(state, DOMAIN_BEACON_ATTESTER, compute_epoch_at_slot(slot))
+    return bls_sign(privkey, hash_tree_root(slot), domain)
+```
+
+```python
+def is_aggregator(state: BeaconState, slot: Slot, index: CommitteeIndex, slot_signature: BLSSignature) -> bool:
+    committee = get_beacon_committee(state, slot, index)
+    modulo = max(1, len(committee) // TARGET_AGGREGATORS_PER_COMMITTEE)
+    return bytes_to_int(hash(slot_signature)[0:8]) % modulo == 0
+```
+
+#### Construct aggregate
+
+If the validator is selected to aggregate (`is_aggregator()`), they construct an aggregate attestation via the following.
+
+Collect `attestations` seen via gossip during the `slot` that have an equivalent `attestation_data` to that constructed by the validator, and create an `aggregate_attestation: Attestation` with the following fields.
+
+##### Data
+
+Set `aggregate_attestation.data = attestation_data` where `attestation_data` is the `AttestationData` object that is the same for each individual attestation being aggregated.
+
+##### Aggregation bits
+
+Let `aggregate_attestation.aggregation_bits` be a `Bitlist[MAX_VALIDATORS_PER_COMMITTEE]` of length `len(committee)`, where each bit set from each individual attestation is set to `0b1`.
+
+##### Custody bits
+
+- Let `aggregate_attestation.custody_bits` be a `Bitlist[MAX_VALIDATORS_PER_COMMITTEE]` filled with zeros of length `len(committee)`.
+
+*Note*: This is a stub for Phase 0.
+
+##### Aggregate signature
+
+Set `aggregate_attestation.signature = aggregate_signature` where `aggregate_signature` is obtained from:
+
+```python
+def get_aggregate_signature(attestations: Sequence[Attestation]) -> BLSSignature:
+    signatures = [attestation.signature for attestation in attestations]
+    return bls_aggregate_signatures(signatures)
+```
+
+#### Broadcast aggregate
+
+If the validator is selected to aggregate (`is_aggregator`), then they broadcast their best aggregate to the global aggregate channel (`beacon_aggregate_and_proof`) two-thirds of the way through the `slot`-that is, `SECONDS_PER_SLOT * 2 / 3` seconds after the start of `slot`.
+
+Aggregate attestations are broadcast as `AggregateAndProof` objects to prove to the gossip channel that the validator has been selected as an aggregator.
+
+##### `AggregateAndProof`
+
+```python
+class AggregateAndProof(Container):
+    index: ValidatorIndex
+    selection_proof: BLSSignature
+    aggregate: Attestation
+```
+
+Where
+* `index` is the validator's `validator_index`.
+* `selection_proof` is the signature of the slot (`slot_signature()`).
+* `aggregate` is the `aggregate_attestation` constructed in the previous section.
+
 ## How to avoid slashing
 
 "Slashing" is the burning of some amount of validator funds and immediate ejection from the active validator set. In Phase 0, there are two ways in which funds can be slashed: [proposer slashing](#proposer-slashing) and [attester slashing](#attester-slashing). Although being slashed has serious repercussions, it is simple enough to avoid being slashed all together by remaining _consistent_ with respect to the messages a validator has previously signed.