414 lines
15 KiB
Markdown
414 lines
15 KiB
Markdown
# Sharding -- The Beacon Chain
|
|
|
|
**Notice**: This document is a work-in-progress for researchers and implementers.
|
|
|
|
## Table of contents
|
|
|
|
<!-- TOC -->
|
|
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
|
|
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
|
|
|
|
- [Introduction](#introduction)
|
|
- [Glossary](#glossary)
|
|
- [Constants](#constants)
|
|
- [Misc](#misc)
|
|
- [Domain types](#domain-types)
|
|
- [Preset](#preset)
|
|
- [Misc](#misc-1)
|
|
- [Time parameters](#time-parameters)
|
|
- [Shard blob samples](#shard-blob-samples)
|
|
- [Configuration](#configuration)
|
|
- [Time parameters](#time-parameters-1)
|
|
- [Containers](#containers)
|
|
- [New Containers](#new-containers)
|
|
- [`BuilderBlockBid`](#builderblockbid)
|
|
- [`BuilderBlockBidWithRecipientAddress`](#builderblockbidwithrecipientaddress)
|
|
- [`ShardedCommitmentsContainer`](#shardedcommitmentscontainer)
|
|
- [`ShardSample`](#shardsample)
|
|
- [Extended Containers](#extended-containers)
|
|
- [`BeaconState`](#beaconstate)
|
|
- [`BuilderBlockData`](#builderblockdata)
|
|
- [`BeaconBlockBody`](#beaconblockbody)
|
|
- [Helper functions](#helper-functions)
|
|
- [Block processing](#block-processing)
|
|
- [`is_builder_block_slot`](#is_builder_block_slot)
|
|
- [Beacon state accessors](#beacon-state-accessors)
|
|
- [`get_active_shard_count`](#get_active_shard_count)
|
|
- [Beacon chain state transition function](#beacon-chain-state-transition-function)
|
|
- [Block processing](#block-processing-1)
|
|
- [`process_block`](#process_block)
|
|
- [Block header](#block-header)
|
|
- [Builder Block Bid](#builder-block-bid)
|
|
- [Sharded data](#sharded-data)
|
|
- [Execution payload](#execution-payload)
|
|
|
|
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
|
|
<!-- /TOC -->
|
|
|
|
|
|
## Introduction
|
|
|
|
This document describes the extensions made to the Phase 0 design of The Beacon Chain to support data sharding,
|
|
based on the ideas [here](https://notes.ethereum.org/@dankrad/new_sharding) and more broadly [here](https://arxiv.org/abs/1809.09044),
|
|
using KZG10 commitments to commit to data to remove any need for fraud proofs (and hence, safety-critical synchrony assumptions) in the design.
|
|
|
|
### Glossary
|
|
|
|
- **Data**: A list of KZG points, to translate a byte string into
|
|
- **Blob**: Data with commitments and meta-data, like a flattened bundle of L2 transactions.
|
|
|
|
|
|
## Constants
|
|
|
|
The following values are (non-configurable) constants used throughout the specification.
|
|
|
|
### Misc
|
|
|
|
| Name | Value | Notes |
|
|
| - | - | - |
|
|
| `FIELD_ELEMENTS_PER_SAMPLE` | `uint64(2**4)` (= 16) | 31 * 16 = 496 bytes |
|
|
|
|
### Domain types
|
|
|
|
| Name | Value |
|
|
| - | - |
|
|
| `DOMAIN_SHARD_SAMPLE` | `DomainType('0x10000000')` |
|
|
|
|
## Preset
|
|
|
|
### Misc
|
|
|
|
| Name | Value | Notes |
|
|
| - | - | - |
|
|
| `MAX_SHARDS` | `uint64(2**12)` (= 4,096) | Theoretical max shard count (used to determine data structure sizes) |
|
|
| `ACTIVE_SHARDS` | `uint64(2**8)` (= 256) | Initial shard count |
|
|
| `MAX_PROPOSER_BLOCKS_BETWEEN_BUILDER_BLOCKS` | `uint64(2**4)` (= 16) | TODO: Need to define what happens if there were more blocks without builder blocks |
|
|
|
|
### Time parameters
|
|
|
|
With the introduction of builder blocks the number of slots per epoch is doubled (it counts beacon blocks and builder blocks).
|
|
|
|
| Name | Value | Unit | Duration |
|
|
| - | - | :-: | :-: |
|
|
| `SLOTS_PER_EPOCH` | `uint64(2**6)` (= 64) | slots | 8:32 minutes |
|
|
|
|
### Shard blob samples
|
|
|
|
| Name | Value | Notes |
|
|
| - | - | - |
|
|
| `SAMPLES_PER_BLOB` | `uint64(2**9)` (= 512) | 248 * 512 = 126,976 bytes |
|
|
|
|
## Configuration
|
|
|
|
Note: Some preset variables may become run-time configurable for testnets, but default to a preset while the spec is unstable.
|
|
E.g. `ACTIVE_SHARDS` and `SAMPLES_PER_BLOB`.
|
|
|
|
### Time parameters
|
|
|
|
| Name | Value | Unit | Duration |
|
|
| - | - | :-: | :-: |
|
|
| `SECONDS_PER_SLOT` | `uint64(8)` | seconds | 8 seconds |
|
|
|
|
## Containers
|
|
|
|
### New Containers
|
|
|
|
#### `BuilderBlockBid`
|
|
|
|
```python
|
|
class BuilderBlockBid(Container):
|
|
slot: Slot
|
|
parent_block_root: Root
|
|
|
|
execution_payload_root: Root
|
|
|
|
sharded_data_commitment_root: Root # Root of the sharded data (only data, not beacon/builder block commitments)
|
|
|
|
sharded_data_commitment_count: uint64 # Count of sharded data commitments
|
|
|
|
bid: Gwei # Block builder bid paid to proposer
|
|
|
|
validator_index: ValidatorIndex # Validator index for this bid
|
|
|
|
# Block builders use an Eth1 address -- need signature as
|
|
# block bid and data gas base fees will be charged to this address
|
|
signature_y_parity: bool
|
|
signature_r: uint256
|
|
signature_s: uint256
|
|
```
|
|
|
|
#### `BuilderBlockBidWithRecipientAddress`
|
|
|
|
```python
|
|
class BuilderBlockBidWithRecipientAddress(Container):
|
|
builder_block_bid: Union[None, BuilderBlockBid]
|
|
recipient_address: ExecutionAddress # Address to receive the block builder bid
|
|
```
|
|
|
|
#### `ShardedCommitmentsContainer`
|
|
|
|
```python
|
|
class ShardedCommitmentsContainer(Container):
|
|
sharded_commitments: List[KZGCommitment, 2 * MAX_SHARDS]
|
|
|
|
# Aggregate degree proof for all sharded_commitments
|
|
degree_proof: KZGCommitment
|
|
|
|
# The sizes of the blocks encoded in the commitments (last builder and all beacon blocks since)
|
|
included_block_sizes: List[uint64, MAX_PROPOSER_BLOCKS_BETWEEN_BUILDER_BLOCKS + 1]
|
|
|
|
# Number of commitments that are for sharded data (no blocks)
|
|
included_sharded_data_commitments: uint64
|
|
|
|
# Random evaluation of beacon blocks + execution payload (this helps with quick verification)
|
|
block_verification_kzg_proof: KZGCommitment
|
|
```
|
|
|
|
#### `ShardSample`
|
|
|
|
```python
|
|
class ShardSample(Container):
|
|
slot: Slot
|
|
row: uint64
|
|
column: uint64
|
|
data: Vector[BLSFieldElement, FIELD_ELEMENTS_PER_SAMPLE]
|
|
proof: KZGCommitment
|
|
builder: ValidatorIndex
|
|
signature: BLSSignature
|
|
```
|
|
|
|
### Extended Containers
|
|
|
|
#### `BeaconState`
|
|
|
|
```python
|
|
class BeaconState(bellatrix.BeaconState):
|
|
blocks_since_builder_block: List[BeaconBlock, MAX_PROPOSER_BLOCKS_BETWEEN_BUILDER_BLOCKS]
|
|
```
|
|
|
|
#### `BuilderBlockData`
|
|
|
|
```python
|
|
class BuilderBlockData(Container):
|
|
execution_payload: ExecutionPayload
|
|
sharded_commitments_container: ShardedCommitmentsContainer
|
|
```
|
|
|
|
#### `BeaconBlockBody`
|
|
|
|
```python
|
|
class BeaconBlockBody(altair.BeaconBlockBody):
|
|
payload_data: Union[BuilderBlockBid, BuilderBlockData]
|
|
```
|
|
|
|
## Helper functions
|
|
|
|
### Block processing
|
|
|
|
#### `is_builder_block_slot`
|
|
|
|
```python
|
|
def is_builder_block_slot(slot: Slot) -> bool:
|
|
return slot % 2 == 1
|
|
```
|
|
|
|
### Beacon state accessors
|
|
|
|
#### `get_active_shard_count`
|
|
|
|
```python
|
|
def get_active_shard_count(state: BeaconState, epoch: Epoch) -> uint64:
|
|
"""
|
|
Return the number of active shards.
|
|
Note that this puts an upper bound on the number of committees per slot.
|
|
"""
|
|
return ACTIVE_SHARDS
|
|
```
|
|
|
|
## Beacon chain state transition function
|
|
|
|
### Block processing
|
|
|
|
#### `process_block`
|
|
|
|
```python
|
|
def process_block(state: BeaconState, block: BeaconBlock) -> None:
|
|
process_block_header(state, block)
|
|
verify_builder_block_bid(state, block)
|
|
process_sharded_data(state, block)
|
|
process_execution_payload(state, block, EXECUTION_ENGINE)
|
|
|
|
if not is_builder_block_slot(block.slot):
|
|
process_randao(state, block.body)
|
|
|
|
process_eth1_data(state, block.body)
|
|
process_operations(state, block.body)
|
|
process_sync_aggregate(state, block.body.sync_aggregate)
|
|
|
|
if is_builder_block_slot(block.slot):
|
|
state.blocks_since_builder_block = []
|
|
state.blocks_since_builder_block.append(block)
|
|
```
|
|
|
|
#### Block header
|
|
|
|
```python
|
|
def process_block_header(state: BeaconState, block: BeaconBlock) -> None:
|
|
# Verify that the slots match
|
|
assert block.slot == state.slot
|
|
# Verify that the block is newer than latest block header
|
|
assert block.slot > state.latest_block_header.slot
|
|
# Verify that proposer index is the correct index
|
|
if not is_builder_block_slot(block.slot):
|
|
assert block.proposer_index == get_beacon_proposer_index(state)
|
|
# Verify that the parent matches
|
|
assert block.parent_root == hash_tree_root(state.latest_block_header)
|
|
# Cache current block as the new latest block
|
|
state.latest_block_header = BeaconBlockHeader(
|
|
slot=block.slot,
|
|
proposer_index=block.proposer_index,
|
|
parent_root=block.parent_root,
|
|
state_root=Bytes32(), # Overwritten in the next process_slot call
|
|
body_root=hash_tree_root(block.body),
|
|
)
|
|
|
|
# Verify proposer is not slashed
|
|
proposer = state.validators[block.proposer_index]
|
|
assert not proposer.slashed
|
|
```
|
|
|
|
#### Builder Block Bid
|
|
|
|
```python
|
|
def verify_builder_block_bid(state: BeaconState, block: BeaconBlock) -> None:
|
|
if is_builder_block_slot(block.slot):
|
|
# Get last builder block bid
|
|
assert state.blocks_since_builder_block[-1].body.payload_data.selector == 0
|
|
builder_block_bid = state.blocks_since_builder_block[-1].body.payload_data.value.builder_block_bid
|
|
assert builder_block_bid.slot + 1 == block.slot
|
|
|
|
assert block.body.payload_data.selector == 1 # Verify that builder block does not contain bid
|
|
|
|
builder_block_data = block.body.payload_data.value
|
|
|
|
assert builder_block_bid.execution_payload_root == hash_tree_root(builder_block_data.execution_payload)
|
|
|
|
assert builder_block_bid.sharded_data_commitment_count == builder_block_data.included_sharded_data_commitments
|
|
|
|
assert builder_block_bid.sharded_data_commitment_root == hash_tree_root(builder_block_data.sharded_commitments[-builder_block_bid.included_sharded_data_commitments:])
|
|
|
|
assert builder_block_bid.validator_index == block.proposer_index
|
|
|
|
else:
|
|
assert block.body.payload_data.selector == 0
|
|
|
|
builder_block_bid = block.body.payload_data.value.builder_block_bid
|
|
assert builder_block_bid.slot == block.slot
|
|
assert builder_block_bid.parent_block_root == block.parent_root
|
|
# We do not check that the builder address exists or has sufficient balance here.
|
|
# If it does not have sufficient balance, the block proposer loses out, so it is their
|
|
# responsibility to check.
|
|
|
|
# Check that the builder is a slashable validator. We can probably reduce this requirement and only
|
|
# ensure that they have 1 ETH in their account as a DOS protection.
|
|
builder = state.validators[builder_block_bid.validator_index]
|
|
assert is_slashable_validator(builder, get_current_epoch(state))
|
|
```
|
|
|
|
#### Sharded data
|
|
|
|
```python
|
|
def process_sharded_data(state: BeaconState, block: BeaconBlock) -> None:
|
|
if is_builder_block_slot(block.slot):
|
|
assert block.body.payload_data.selector == 1
|
|
sharded_commitments_container = block.body.payload_data.value.sharded_commitments_container
|
|
|
|
# Verify not too many commitments
|
|
assert len(sharded_commitments_container.sharded_commitments) // 2 <= get_active_shard_count(state, get_current_epoch(state))
|
|
|
|
# Verify the degree proof
|
|
r = hash_to_bls_field(sharded_commitments_container.sharded_commitments, 0)
|
|
r_powers = compute_powers(r, len(sharded_commitments_container.sharded_commitments))
|
|
combined_commitment = elliptic_curve_lincomb(sharded_commitments_container.sharded_commitments, r_powers)
|
|
|
|
payload_field_elements_per_blob = SAMPLES_PER_BLOB * FIELD_ELEMENTS_PER_SAMPLE // 2
|
|
|
|
verify_degree_proof(combined_commitment, payload_field_elements_per_blob, sharded_commitments_container.degree_proof)
|
|
|
|
# Verify that the 2*N commitments lie on a degree < N polynomial
|
|
low_degree_check(sharded_commitments_container.sharded_commitments)
|
|
|
|
# Verify that blocks since the last builder block have been included
|
|
blocks_chunked = [bytes_to_field_elements(ssz_serialize(block)) for block in state.blocks_since_builder_block]
|
|
block_vectors = []
|
|
|
|
for block_chunked in blocks_chunked:
|
|
for i in range(0, len(block_chunked), payload_field_elements_per_blob):
|
|
block_vectors.append(block_chunked[i:i + payload_field_elements_per_blob])
|
|
|
|
number_of_blobs = len(block_vectors)
|
|
r = hash_to_bls_field(sharded_commitments_container.sharded_commitments[:number_of_blobs], 0)
|
|
x = hash_to_bls_field(sharded_commitments_container.sharded_commitments[:number_of_blobs], 1)
|
|
|
|
r_powers = compute_powers(r, number_of_blobs)
|
|
combined_vector = vector_lincomb(block_vectors, r_powers)
|
|
combined_commitment = elliptic_curve_lincomb(sharded_commitments_container.sharded_commitments[:number_of_blobs], r_powers)
|
|
y = evaluate_polynomial_in_evaluation_form(combined_vector, x)
|
|
|
|
verify_kzg_proof(combined_commitment, x, y, sharded_commitments_container.block_verification_kzg_proof)
|
|
|
|
# Verify that number of sharded data commitments is correctly indicated
|
|
assert 2 * (number_of_blobs + included_sharded_data_commitments) == len(sharded_commitments_container.sharded_commitments)
|
|
```
|
|
|
|
#### Execution payload
|
|
|
|
```python
|
|
def process_execution_payload(state: BeaconState, block: BeaconBlock, execution_engine: ExecutionEngine) -> None:
|
|
if is_builder_block_slot(block.slot):
|
|
assert block.body.payload_data.selector == 1
|
|
payload = block.body.payload_data.value.execution_payload
|
|
|
|
# Verify consistency of the parent hash with respect to the previous execution payload header
|
|
assert payload.parent_hash == state.latest_execution_payload_header.block_hash
|
|
# Verify random
|
|
assert payload.random == get_randao_mix(state, get_current_epoch(state))
|
|
# Verify timestamp
|
|
assert payload.timestamp == compute_timestamp_at_slot(state, state.slot)
|
|
|
|
# Get sharded data commitments
|
|
sharded_commitments_container = block.body.sharded_commitments_container
|
|
sharded_data_commitments = sharded_commitments_container.sharded_commitments[-sharded_commitments_container.included_sharded_data_commitments:]
|
|
|
|
# Get all unprocessed builder block bids
|
|
unprocessed_builder_block_bid_with_recipient_addresses = []
|
|
for block in state.blocks_since_builder_block[1:]:
|
|
unprocessed_builder_block_bid_with_recipient_addresses.append(block.body.builder_block_bid_with_recipient_address.value)
|
|
|
|
# Verify the execution payload is valid
|
|
# The execution engine gets two extra payloads: One for the sharded data commitments (these are needed to verify type 3 transactions)
|
|
# and one for all so far unprocessed builder block bids:
|
|
# * The execution engine needs to transfer the balance from the bidder to the proposer.
|
|
# * The execution engine needs to deduct data gas fees from the bidder balances
|
|
assert execution_engine.execute_payload(payload,
|
|
sharded_data_commitments,
|
|
unprocessed_builder_block_bid_with_recipient_addresses)
|
|
|
|
# Cache execution payload header
|
|
state.latest_execution_payload_header = ExecutionPayloadHeader(
|
|
parent_hash=payload.parent_hash,
|
|
fee_recipient=payload.fee_recipient,
|
|
state_root=payload.state_root,
|
|
receipt_root=payload.receipt_root,
|
|
logs_bloom=payload.logs_bloom,
|
|
random=payload.random,
|
|
block_number=payload.block_number,
|
|
gas_limit=payload.gas_limit,
|
|
gas_used=payload.gas_used,
|
|
timestamp=payload.timestamp,
|
|
extra_data=payload.extra_data,
|
|
base_fee_per_gas=payload.base_fee_per_gas,
|
|
block_hash=payload.block_hash,
|
|
transactions_root=hash_tree_root(payload.transactions),
|
|
)
|
|
``` |