Merge pull request #2302 from ethereum/shardblobs

Shard blob, headers and slashings

configs/mainnet/sharding.yaml

@@ -18,6 +18,8 @@ MAX_SHARDS: 1024
 INITIAL_ACTIVE_SHARDS: 64
 # 2**3 (= 8)
 GASPRICE_ADJUSTMENT_COEFFICIENT: 8
+# 2**4 (= 16)
+MAX_SHARD_PROPOSER_SLASHINGS: 16
 
 # Shard block configs
 # ---------------------------------------------------------------
@@ -41,5 +43,5 @@ SHARD_COMMITTEE_PERIOD: 256
 
 # Signature domains
 # ---------------------------------------------------------------
-DOMAIN_SHARD_PROPOSAL: 0x80000000
+DOMAIN_SHARD_PROPOSER: 0x80000000
 DOMAIN_SHARD_COMMITTEE: 0x81000000

configs/minimal/sharding.yaml

@@ -18,6 +18,8 @@ MAX_SHARDS: 8
 INITIAL_ACTIVE_SHARDS: 2
 # 2**3 (= 8)
 GASPRICE_ADJUSTMENT_COEFFICIENT: 8
+# [customized] reduced for testing
+MAX_SHARD_PROPOSER_SLASHINGS: 4
 
 # Shard block configs
 # ---------------------------------------------------------------
@@ -41,6 +43,5 @@ SHARD_COMMITTEE_PERIOD: 256
 
 # Signature domains
 # ---------------------------------------------------------------
-DOMAIN_SHARD_PROPOSAL: 0x80000000
+DOMAIN_SHARD_PROPOSER: 0x80000000
 DOMAIN_SHARD_COMMITTEE: 0x81000000
-DOMAIN_LIGHT_CLIENT: 0x82000000

specs/sharding/beacon-chain.md

@@ -24,9 +24,13 @@
   - [`BeaconState`](#beaconstate)
 - [New containers](#new-containers)
   - [`DataCommitment`](#datacommitment)
-  - [`ShardHeader`](#shardheader)
+  - [`ShardBlobBodySummary`](#shardblobbodysummary)
-  - [`SignedShardHeader`](#signedshardheader)
+  - [`ShardBlobHeader`](#shardblobheader)
+  - [`SignedShardBlobHeader`](#signedshardblobheader)
   - [`PendingShardHeader`](#pendingshardheader)
+  - [`ShardBlobReference`](#shardblobreference)
+  - [`SignedShardBlobReference`](#signedshardblobreference)
+  - [`ShardProposerSlashing`](#shardproposerslashing)
 - [Helper functions](#helper-functions)
   - [Misc](#misc-1)
     - [`next_power_of_two`](#next_power_of_two)
@@ -48,6 +52,7 @@
     - [Updated `process_attestation`](#updated-process_attestation)
     - [`update_pending_votes`](#update_pending_votes)
     - [`process_shard_header`](#process_shard_header)
+      - [Shard Proposer slashings](#shard-proposer-slashings)
   - [Epoch transition](#epoch-transition)
     - [Pending headers](#pending-headers)
     - [Shard epoch increment](#shard-epoch-increment)
@@ -94,6 +99,7 @@ The following values are (non-configurable) constants used throughout the specif
 | `INITIAL_ACTIVE_SHARDS` | `uint64(2**6)` (= 64) | Initial shard count |
 | `GASPRICE_ADJUSTMENT_COEFFICIENT` | `uint64(2**3)` (= 8) | Gasprice may decrease/increase by at most exp(1 / this value) *per epoch* |
 | `MAX_SHARD_HEADERS_PER_SHARD` | `4` | |
+| `MAX_SHARD_PROPOSER_SLASHINGS` | `2**4` (= 16) | Maximum amount of shard proposer slashing operations per block |
 
 ### Shard block configs
 
@@ -127,7 +133,7 @@ The following values are (non-configurable) constants used throughout the specif
 
 | Name | Value |
 | - | - |
-| `DOMAIN_SHARD_HEADER` | `DomainType('0x80000000')` |
+| `DOMAIN_SHARD_PROPOSER` | `DomainType('0x80000000')` |
 | `DOMAIN_SHARD_COMMITTEE` | `DomainType('0x81000000')` |
 
 ## Updated containers
@@ -153,7 +159,8 @@ class AttestationData(Container):
 
 ```python
 class BeaconBlockBody(merge.BeaconBlockBody):  # [extends The Merge block body]
-    shard_headers: List[SignedShardHeader, MAX_SHARDS * MAX_SHARD_HEADERS_PER_SHARD]
+    shard_proposer_slashings: List[ShardProposerSlashing, MAX_SHARD_PROPOSER_SLASHINGS]
+    shard_headers: List[SignedShardBlobHeader, MAX_SHARDS * MAX_SHARD_HEADERS_PER_SHARD]
 ```
 
 ### `BeaconState`
@@ -186,26 +193,37 @@ class DataCommitment(Container):
     length: uint64
 ```
 
-### `ShardHeader`
+### `ShardBlobBodySummary`
 
 ```python
-class ShardHeader(Container):
+class ShardBlobBodySummary(Container):
-    # Slot and shard that this header is intended for
-    slot: Slot
-    shard: Shard
     # The actual data commitment
     commitment: DataCommitment
     # Proof that the degree < commitment.length
     degree_proof: BLSCommitment
+    # Hash-tree-root as summary of the data field
+    data_root: Root
+    # Latest block root of the Beacon Chain, before shard_blob.slot
+    beacon_block_root: Root
 ```
 
-TODO: add shard-proposer-index to shard headers, similar to optimization done with beacon-blocks.
+### `ShardBlobHeader`
-
-### `SignedShardHeader`
 
 ```python
-class SignedShardHeader(Container):
+class ShardBlobHeader(Container):
-    message: ShardHeader
+    # Slot and shard that this header is intended for
+    slot: Slot
+    shard: Shard
+    body_summary: ShardBlobBodySummary
+    # Proposer of the shard-blob
+    proposer_index: ValidatorIndex
+```
+
+### `SignedShardBlobHeader`
+
+```python
+class SignedShardBlobHeader(Container):
+    message: ShardBlobHeader
     signature: BLSSignature
 ```
 
@@ -226,6 +244,35 @@ class PendingShardHeader(Container):
     confirmed: boolean
 ```
 
+### `ShardBlobReference`
+
+```python
+class ShardBlobReference(Container):
+    # Slot and shard that this reference is intended for
+    slot: Slot
+    shard: Shard
+    # Hash-tree-root of commitment data
+    body_root: Root
+    # Proposer of the shard-blob
+    proposer_index: ValidatorIndex
+```
+
+### `SignedShardBlobReference`
+
+```python
+class SignedShardBlobReference(Container):
+    message: ShardBlobReference
+    signature: BLSSignature
+```
+
+### `ShardProposerSlashing`
+
+```python
+class ShardProposerSlashing(Container):
+    signed_reference_1: SignedShardBlobReference
+    signed_reference_2: SignedShardBlobReference
+```
+
 ## Helper functions
 
 ### Misc
@@ -435,6 +482,8 @@ def process_operations(state: BeaconState, body: BeaconBlockBody) -> None:
 
     for_ops(body.proposer_slashings, process_proposer_slashing)
     for_ops(body.attester_slashings, process_attester_slashing)
+    # New shard proposer slashing processing
+    for_ops(body.shard_proposer_slashings, process_shard_proposer_slashing)
     # Limit is dynamic based on active shard count
     assert len(body.shard_headers) <= MAX_SHARD_HEADERS_PER_SHARD * get_active_shard_count(state, get_current_epoch(state))
     for_ops(body.shard_headers, process_shard_header)
@@ -499,30 +548,40 @@ def update_pending_votes(state: BeaconState, attestation: Attestation) -> None:
 
 ```python
 def process_shard_header(state: BeaconState,
-                         signed_header: SignedShardHeader) -> None:
+                         signed_header: SignedShardBlobHeader) -> None:
     header = signed_header.message
-    header_root = hash_tree_root(header)
+    # Verify the header is not 0, and not from the future.
-    assert compute_epoch_at_slot(header.slot) in [get_previous_epoch(state), get_current_epoch(state)]
+    assert Slot(0) < header.slot <= state.slot
-
+    header_epoch = compute_epoch_at_slot(header.slot)
+    # Verify that the header is within the processing time window
+    assert header_epoch in [get_previous_epoch(state), get_current_epoch(state)]
+    # Verify that the shard is active
+    assert header.shard < get_active_shard_count(state, header_epoch)
+    # Verify that the block root matches,
+    # to ensure the header will only be included in this specific Beacon Chain sub-tree.
+    assert header.beacon_block_root == get_block_root_at_slot(state, header.slot - 1)
+    # Verify proposer
+    assert header.proposer_index == get_shard_proposer_index(state, header.slot, header.shard)
     # Verify signature
-    signer_index = get_shard_proposer_index(state, header.slot, header.shard)
     signing_root = compute_signing_root(header, get_domain(state, DOMAIN_SHARD_HEADER))
-    assert bls.Verify(state.validators[signer_index].pubkey, signing_root, signed_header.signature)
+    assert bls.Verify(state.validators[header.proposer_index].pubkey, signing_root, signed_header.signature)
 
     # Verify the length by verifying the degree.
-    if header.commitment.length == 0:
+    body_summary = header.body_summary
-        assert header.degree_proof == G1_SETUP[0]
+    if body_summary.commitment.length == 0:
+        assert body_summary.degree_proof == G1_SETUP[0]
     assert (
-        bls.Pairing(header.degree_proof, G2_SETUP[0])
+        bls.Pairing(body_summary.degree_proof, G2_SETUP[0])
-        == bls.Pairing(header.commitment.point, G2_SETUP[-header.commitment.length])
+        == bls.Pairing(body_summary.commitment.point, G2_SETUP[-body_summary.commitment.length])
     )
 
     # Get the correct pending header list
-    if compute_epoch_at_slot(header.slot) == get_current_epoch(state):
+    if header_epoch == get_current_epoch(state):
         pending_headers = state.current_epoch_pending_shard_headers
     else:
         pending_headers = state.previous_epoch_pending_shard_headers
 
+    header_root = hash_tree_root(header)
     # Check that this header is not yet in the pending list
     assert header_root not in [pending_header.root for pending_header in pending_headers]
 
@@ -532,7 +591,7 @@ def process_shard_header(state: BeaconState,
     pending_headers.append(PendingShardHeader(
         slot=header.slot,
         shard=header.shard,
-        commitment=header.commitment,
+        commitment=body_summary.commitment,
         root=header_root,
         votes=Bitlist[MAX_VALIDATORS_PER_COMMITTEE]([0] * committee_length),
         confirmed=False,
@@ -544,6 +603,33 @@ the length proof is the commitment to the polynomial `B(X) * X**(MAX_DEGREE + 1
 where `MAX_DEGREE` is the maximum power of `s` available in the setup, which is `MAX_DEGREE = len(G2_SETUP) - 1`.
 The goal is to ensure that a proof can only be constructed if `deg(B) < l` (there are not hidden higher-order terms in the polynomial, which would thwart reconstruction).
 
+##### Shard Proposer slashings
+
+```python
+def process_shard_proposer_slashing(state: BeaconState, proposer_slashing: ShardProposerSlashing) -> None:
+    reference_1 = proposer_slashing.signed_reference_1.message
+    reference_2 = proposer_slashing.signed_reference_2.message
+
+    # Verify header slots match
+    assert reference_1.slot == reference_2.slot
+    # Verify header shards match
+    assert reference_1.shard == reference_2.shard
+    # Verify header proposer indices match
+    assert reference_1.proposer_index == reference_2.proposer_index
+    # Verify the headers are different (i.e. different body)
+    assert reference_1 != reference_2
+    # Verify the proposer is slashable
+    proposer = state.validators[reference_1.proposer_index]
+    assert is_slashable_validator(proposer, get_current_epoch(state))
+    # Verify signatures
+    for signed_header in (proposer_slashing.signed_reference_1, proposer_slashing.signed_reference_2):
+        domain = get_domain(state, DOMAIN_SHARD_PROPOSER, compute_epoch_at_slot(signed_header.message.slot))
+        signing_root = compute_signing_root(signed_header.message, domain)
+        assert bls.Verify(proposer.pubkey, signing_root, signed_header.signature)
+
+    slash_validator(state, reference_1.proposer_index)
+```
+
 ### Epoch transition
 
 This epoch transition overrides the Merge epoch transition:

specs/sharding/p2p-interface.md

@@ -10,12 +10,14 @@
 
 - [Introduction](#introduction)
 - [New containers](#new-containers)
+  - [ShardBlobBody](#shardblobbody)
   - [ShardBlob](#shardblob)
   - [SignedShardBlob](#signedshardblob)
 - [Gossip domain](#gossip-domain)
   - [Topics and messages](#topics-and-messages)
     - [Shard blobs: `shard_blob_{shard}`](#shard-blobs-shard_blob_shard)
     - [Shard header: `shard_header`](#shard-header-shard_header)
+    - [Shard proposer slashing: `shard_proposer_slashing`](#shard-proposer-slashing-shard_proposer_slashing)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
 <!-- /TOC -->
@@ -29,30 +31,41 @@ The adjustments and additions for Shards are outlined in this document.
 
 ## New containers
 
-### ShardBlob
+### ShardBlobBody
 
-Network-only.
+```python
+class ShardBlobBody(Container):
+    # The actual data commitment
+    commitment: DataCommitment
+    # Proof that the degree < commitment.length
+    degree_proof: BLSCommitment
+    # The actual data. Should match the commitment and degree proof.
+    data: List[BLSPoint, POINTS_PER_SAMPLE * MAX_SAMPLES_PER_BLOCK]
+    # Latest block root of the Beacon Chain, before shard_blob.slot
+    beacon_block_root: Root
+```
+
+The user MUST always verify the commitments in the `body` are valid for the `data` in the `body`.
+
+### ShardBlob
 
 ```python
 class ShardBlob(Container):
     # Slot and shard that this blob is intended for
     slot: Slot
     shard: Shard
-    # The actual data. Represented in header as data commitment and degree proof
+    body: ShardBlobBody
-    data: List[BLSPoint, POINTS_PER_SAMPLE * MAX_SAMPLES_PER_BLOCK]
+    # Proposer of the shard-blob
+    proposer_index: ValidatorIndex
 ```
 
-Note that the hash-tree-root of the `ShardBlob` does not match the `ShardHeader`,
+This is the expanded form of the `ShardBlobHeader` type.
-since the blob deals with full data, whereas the header includes the KZG commitment and degree proof instead.
 
 ### SignedShardBlob
 
-Network-only.
-
 ```python
 class SignedShardBlob(Container):
     message: ShardBlob
-    # The signature, the message is the commitment on the blob
     signature: BLSSignature
 ```
 
@@ -66,6 +79,7 @@ Following the same scheme as the [Phase0 gossip topics](../phase0/p2p-interface.
 |----------------------------------|---------------------------|
 | `shard_blob_{shard}`             | `SignedShardBlob`         |
 | `shard_header`                   | `SignedShardHeader`       |
+| `shard_proposer_slashing`        | `ShardProposerSlashing`   |
 
 The [DAS network specification](./das-p2p.md) defines additional topics.
 
@@ -73,22 +87,49 @@ The [DAS network specification](./das-p2p.md) defines additional topics.
 
 Shard block data, in the form of a `SignedShardBlob` is published to the `shard_blob_{shard}` subnets.
 
-The following validations MUST pass before forwarding the `signed_blob` (with inner `blob`) on the horizontal subnet or creating samples for it.
+The following validations MUST pass before forwarding the `signed_blob` (with inner `message` as `blob`) on the horizontal subnet or creating samples for it.
 - _[REJECT]_ `blob.shard` MUST match the topic `{shard}` parameter. (And thus within valid shard index range)
 - _[IGNORE]_ The `blob` is not from a future slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
   i.e. validate that `blob.slot <= current_slot`
   (a client MAY queue future blobs for processing at the appropriate slot).
-- _[IGNORE]_ The blob is the first blob with valid signature received for the proposer for the `(slot, shard)` combination.
+- _[IGNORE]_ The `blob` is new enough to be still be processed --
+  i.e. validate that `compute_epoch_at_slot(blob.slot) >= get_previous_epoch(state)`
+- _[IGNORE]_ The blob is the first blob with valid signature received for the `(blob.proposer_index, blob.slot, blob.shard)` combination.
 - _[REJECT]_ As already limited by the SSZ list-limit, it is important the blob is well-formatted and not too large.
-- _[REJECT]_ The `blob.data` MUST NOT contain any point `p >= MODULUS`. Although it is a `uint256`, not the full 256 bit range is valid.
+- _[REJECT]_ The `blob.body.data` MUST NOT contain any point `p >= MODULUS`. Although it is a `uint256`, not the full 256 bit range is valid.
-- _[REJECT]_ The proposer signature, `signed_blob.signature`, is valid with respect to the `proposer_index` pubkey, signed over the SSZ output of `commit_to_data(blob.data)`.
+- _[REJECT]_ The proposer signature, `signed_blob.signature`, is valid with respect to the `proposer_index` pubkey.
-- _[REJECT]_ The blob is proposed by the expected `proposer_index` for the blob's slot.
+- _[REJECT]_ The blob is proposed by the expected `proposer_index` for the blob's slot
+  in the context of the current shuffling (defined by `blob.body.beacon_block_root`/`slot`).
+  If the `proposer_index` cannot immediately be verified against the expected shuffling,
+  the block MAY be queued for later processing while proposers for the blob's branch are calculated --
+  in such a case _do not_ `REJECT`, instead `IGNORE` this message.
 
-TODO: make double blob proposals slashable?
 
 #### Shard header: `shard_header`
 
-Shard header data, in the form of a `SignedShardHeader` is published to the global `shard_header` subnet.
+Shard header data, in the form of a `SignedShardBlobHeader` is published to the global `shard_header` subnet.
 
-TODO: validation conditions.
+The following validations MUST pass before forwarding the `signed_shard_header` (with inner `message` as `header`) on the network.
+- _[IGNORE]_ The `header` is not from a future slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
+  i.e. validate that `header.slot <= current_slot`
+  (a client MAY queue future headers for processing at the appropriate slot).
+- _[IGNORE]_ The `header` is new enough to be still be processed --
+  i.e. validate that `compute_epoch_at_slot(header.slot) >= get_previous_epoch(state)`
+- _[IGNORE]_ The header is the first header with valid signature received for the `(header.proposer_index, header.slot, header.shard)` combination.
+- _[REJECT]_ The proposer signature, `signed_shard_header.signature`, is valid with respect to the `proposer_index` pubkey.
+- _[REJECT]_ The header is proposed by the expected `proposer_index` for the block's slot
+  in the context of the current shuffling (defined by `header.body_summary.beacon_block_root`/`slot`).
+  If the `proposer_index` cannot immediately be verified against the expected shuffling,
+  the block MAY be queued for later processing while proposers for the block's branch are calculated --
+  in such a case _do not_ `REJECT`, instead `IGNORE` this message.
 
+
+#### Shard proposer slashing: `shard_proposer_slashing`
+
+Shard proposer slashings, in the form of `ShardProposerSlashing`, are published to the global `shard_proposer_slashing` topic.
+
+The following validations MUST pass before forwarding the `shard_proposer_slashing` on to the network.
+- _[IGNORE]_ The shard proposer slashing is the first valid shard proposer slashing received
+  for the proposer with index `proposer_slashing.signed_header_1.message.proposer_index`.
+  The `slot` and `shard` are ignored, there are no per-shard slashings.
+- _[REJECT]_ All of the conditions within `process_shard_proposer_slashing` pass validation.