24 KiB

Raw Blame History

Ethereum 2.0 Phase 1 -- The Beacon Chain with Shards

Notice: This document is a work-in-progress for researchers and implementers.

Introduction
Custom types
Configuration
Updated containers
New containers
Helper functions

Introduction

This document describes the extensions made to the Phase 0 design of The Beacon Chain to support data sharding, based on the ideas here and more broadly here, using KZG10 commitments to commit to data to remove any need for fraud proofs (and hence, safety-critical synchrony assumptions) in the design.

Custom types

We define the following Python custom types for type hinting and readability:

Name	SSZ equivalent	Description
`Shard`	`uint64`	A shard number
`BLSCommitment`	`bytes48`	A G1 curve point

Configuration

Misc

Name	Value	Notes
`MAX_SHARDS`	`uint64(2**10)` (= 1024)	Theoretical max shard count (used to determine data structure sizes)
`INITIAL_ACTIVE_SHARDS`	`uint64(2**6)` (= 64)	Initial shard count
`GASPRICE_ADJUSTMENT_COEFFICIENT`	`uint64(2**3)` (= 8)	Gasprice may decrease/increase by at most exp(1 / this value) per epoch
`MAX_SHARD_HEADERS_PER_SHARD`	`4`
`MAX_SHARD_HEADERS`	`MAX_SHARDS * MAX_SHARD_HEADERS_PER_SHARD`
`PRIMITIVE_ROOT_OF_UNITY`	`5`	Primitive root of unity of the BLS12_381 (inner) modulus
`DATA_AVAILABILITY_INVERSE_CODING_RATE`	`2**1` (=2)	Factor by which samples are extended for data availability encoding

Shard block configs

Name	Value	Notes
`POINTS_PER_SAMPLE`	`uint64(2**3)` (= 8)	31 * 8 = 248 bytes
`MAX_SAMPLES_PER_BLOCK`	`uint64(2**11)` (= 2,048)	248 * 2,048 = 507,904 bytes
`TARGET_SAMPLES_PER_BLOCK`	`uint64(2**10)` (= 1,024)	248 * 1,024 = 253,952 bytes

Precomputed size verification points

Name	Value
`G2_SETUP`	Type `List[G2]`. The G2-side trusted setup `[G, Gs, Gs**2....]`; note that the first point is the generator.
`MODULUS`	`0x73eda753299d7d483339d80809a1d80553bda402fffe5bfeffffffff00000001` (curve order of BLS12_381)
`ROOT_OF_UNITY`	`pow(PRIMITIVE_ROOT_OF_UNITY, (MODULUS - 1) // (MAX_SAMPLES_PER_BLOCK * POINTS_PER_SAMPLE, MODULUS)`

Gwei values

Name	Value	Unit	Description
`MAX_GASPRICE`	`Gwei(2**33)` (= 8,589,934,592)	Gwei	Max gasprice charged for a TARGET-sized shard block
`MIN_GASPRICE`	`Gwei(2**3)` (= 8)	Gwei	Min gasprice charged for a TARGET-sized shard block

Time parameters

Name	Value	Unit	Duration
`SHARD_COMMITTEE_PERIOD`	`Epoch(2**8)` (= 256)	epochs	~27 hours

Domain types

Name	Value
`DOMAIN_SHARD_HEADER`	`DomainType('0x80000000')`

Updated containers

The following containers have updated definitions in Phase 1.

`AttestationData`

class AttestationData(Container):
    slot: Slot
    index: CommitteeIndex
    # LMD GHOST vote
    beacon_block_root: Root
    # FFG vote
    source: Checkpoint
    target: Checkpoint
    # Shard header root
    shard_header_root: Root

`BeaconBlock`

class BeaconBlock(phase0.BeaconBlock):
    shard_headers: List[Signed[ShardHeader], MAX_SHARD_HEADERS]

`BeaconState`

class BeaconState(phase0.BeaconState):
    # Updated fields
    previous_epoch_attestations: List[PendingAttestation, MAX_ATTESTATIONS * SLOTS_PER_EPOCH]
    current_epoch_attestations: List[PendingAttestation, MAX_ATTESTATIONS * SLOTS_PER_EPOCH]
    # New fields
    current_epoch_pending_shard_headers: List[PendingShardHeader, MAX_SHARD_HEADERS * SLOTS_PER_EPOCH]
    previous_epoch_pending_shard_headers: List[PendingShardHeader, MAX_SHARD_HEADERS * SLOTS_PER_EPOCH]
    grandparent_epoch_confirmed_commitments: Vector[Vector[DataCommitment, SLOTS_PER_EPOCH], MAX_SHARDS]
    shard_gasprice: uint64
    current_epoch_start_shard: Shard

New containers

The following containers are new in Phase 1.

`DataCommitment`

class DataCommitment(Container):
    # KZG10 commitment to the data
    point: BLSCommitment
    # Length of the data in samples
    length: uint64

`ShardHeader`

class ShardHeader(Container):
    # Slot and shard that this header is intended for
    slot: Slot
    shard: Shard
    # The actual data commitment
    commitment: DataCommitment
    # Proof that the degree < commitment.length
    degree_proof: BLSCommitment

`PendingShardHeader`

class PendingShardHeader(Container):
    # Slot and shard that this header is intended for
    slot: Slot
    shard: Shard
    # KZG10 commitment to the data
    commitment: BLSCommitment
    # hash_tree_root of the ShardHeader (stored so that attestations
    # can be checked against it)
    root: Root
    # Length of the data in samples
    length: uint64
    # Who voted for the header
    votes: Bitlist[MAX_VALIDATORS_PER_COMMITTEE]
    # Has this header been confirmed?
    confirmed: bool

Helper functions

Misc

`next_power_of_two`

def next_power_of_two(x):
    return 2 ** ((x - 1).bit_length())

`reverse_bit_order`

def reverse_bit_order(n, order):
    """
    Reverse the bit order of an integer n
    """
    assert is_power_of_two(order)
    return int(('{:0' + str(order.bit_length() - 1) + 'b}').format(n)[::-1], 2)

`compute_previous_slot`

def compute_previous_slot(slot: Slot) -> Slot:
    if slot > 0:
        return Slot(slot - 1)
    else:
        return Slot(0)

`compute_shard_from_committee_index`

def compute_shard_from_committee_index(state: BeaconState, index: CommitteeIndex, slot: Slot) -> Shard:
    active_shards = get_active_shard_count(state)
    return Shard((index + get_start_shard(state, slot)) % active_shards)

`compute_updated_gasprice`

def compute_updated_gasprice(prev_gasprice: Gwei, shard_block_length: uint64, adjustment_quotient: uint64) -> Gwei:
    if shard_block_length > TARGET_SAMPLES_PER_BLOCK:
        delta = max(1, prev_gasprice * (shard_block_length - TARGET_SAMPLES_PER_BLOCK)
                       // TARGET_SAMPLES_PER_BLOCK // adjustment_quotient)
        return min(prev_gasprice + delta, MAX_GASPRICE)
    else:
        delta = max(1, prev_gasprice * (TARGET_SAMPLES_PER_BLOCK - shard_block_length)
                       // TARGET_SAMPLES_PER_BLOCK // adjustment_quotient)
        return max(prev_gasprice, MIN_GASPRICE + delta) - delta

`compute_committee_source_epoch`

def compute_committee_source_epoch(epoch: Epoch, period: uint64) -> Epoch:
    """
    Return the source epoch for computing the committee.
    """
    source_epoch = Epoch(epoch - epoch % period)
    if source_epoch >= period:
        source_epoch -= period  # `period` epochs lookahead
    return source_epoch

Beacon state accessors

Updated `get_committee_count_per_slot`

def get_committee_count_per_slot(state: BeaconState, epoch: Epoch) -> uint64:
    """
    Return the number of committees in each slot for the given ``epoch``.
    """
    return max(uint64(1), min(
        get_active_shard_count(state, epoch),
        uint64(len(get_active_validator_indices(state, epoch))) // SLOTS_PER_EPOCH // TARGET_COMMITTEE_SIZE,
    ))

`get_active_shard_count`

def get_active_shard_count(state: BeaconState, epoch: Epoch) -> uint64:
    """
    Return the number of active shards.
    Note that this puts an upper bound on the number of committees per slot.
    """
    return INITIAL_ACTIVE_SHARDS

`get_shard_committee`

def get_shard_committee(beacon_state: BeaconState, epoch: Epoch, shard: Shard) -> Sequence[ValidatorIndex]:
    """
    Return the shard committee of the given ``epoch`` of the given ``shard``.
    """
    source_epoch = compute_committee_source_epoch(epoch, SHARD_COMMITTEE_PERIOD)
    active_validator_indices = get_active_validator_indices(beacon_state, source_epoch)
    seed = get_seed(beacon_state, source_epoch, DOMAIN_SHARD_COMMITTEE)
    return compute_committee(
        indices=active_validator_indices,
        seed=seed,
        index=shard,
        count=get_active_shard_count(beacon_state, epoch),
    )

`compute_proposer_index`

Updated version to get a proposer index that will only allow proposers with a certain minimum balance, ensuring that the balance is always sufficient to cover gas costs.

def compute_proposer_index(state: BeaconState, indices: Sequence[ValidatorIndex], seed: Bytes32, min_effective_balance: GWei = GWei(0)) -> ValidatorIndex:
    """
    Return from ``indices`` a random index sampled by effective balance.
    """
    assert len(indices) > 0
    MAX_RANDOM_BYTE = 2**8 - 1
    i = uint64(0)
    total = uint64(len(indices))
    while True:
        candidate_index = indices[compute_shuffled_index(i % total, total, seed)]
        random_byte = hash(seed + uint_to_bytes(uint64(i // 32)))[i % 32]
        effective_balance = state.validators[candidate_index].effective_balance
        if effective_balance * MAX_RANDOM_BYTE >= MAX_EFFECTIVE_BALANCE * random_byte and effective_balance > min_effective_balance:
            return candidate_index
        i += 1

`get_shard_proposer_index`

def get_shard_proposer_index(beacon_state: BeaconState, slot: Slot, shard: Shard) -> ValidatorIndex:
    """
    Return the proposer's index of shard block at ``slot``.
    """
    epoch = compute_epoch_at_slot(slot)
    committee = get_shard_committee(beacon_state, epoch, shard)
    seed = hash(get_seed(state, epoch, DOMAIN_BEACON_PROPOSER) + uint_to_bytes(state.slot))
    EFFECTIVE_BALANCE_MAX_DOWNWARD_DEVIATION = EFFECTIVE_BALANCE_INCREMENT - EFFECTIVE_BALANCE_INCREMENT * HYSTERESIS_DOWNWARD_MULTIPLIER // HYSTERESIS_QUOTIENT

    return compute_proposer_index(state, committee, seed,
        state.shard_gasprice * MAX_SAMPLES_PER_BLOCK // TARGET_SAMPLES_PER_BLOCK + EFFECTIVE_BALANCE_MAX_DOWNWARD_DEVIATION)

`get_start_shard`

def get_start_shard(state: BeaconState, slot: Slot) -> Shard:
    """
    Return the start shard at ``slot``.
    """
    current_epoch_start_slot = compute_start_slot_at_epoch(get_current_epoch(state))
    shard = state.current_epoch_start_shard
    if slot > current_epoch_start_slot:
        # Current epoch or the next epoch lookahead
        for _slot in range(current_epoch_start_slot, slot):
            committee_count = get_committee_count_per_slot(state, compute_epoch_at_slot(Slot(_slot)))
            active_shard_count = get_active_shard_count(state, compute_epoch_at_slot(Slot(_slot)))
            shard = (shard + committee_count) % active_shard_count
        return Shard(shard)
    elif slot < current_epoch_start_slot:
        # Previous epoch
        for _slot in list(range(slot, current_epoch_start_slot))[::-1]:
            committee_count = get_committee_count_per_slot(state, compute_epoch_at_slot(Slot(_slot)))
            active_shard_count = get_active_shard_count(state, compute_epoch_at_slot(Slot(_slot)))
            # Ensure positive
            shard = (shard + active_shard_count - committee_count) % active_shard_count
    return Shard(shard)

Block processing

def process_block(state: BeaconState, block: BeaconBlock) -> None:
    process_block_header(state, block)
    process_randao(state, block.body)
    process_eth1_data(state, block.body)
    process_light_client_aggregate(state, block.body)
    process_operations(state, block.body)

Operations

def process_operations(state: BeaconState, body: BeaconBlockBody) -> None:
    # Verify that outstanding deposits are processed up to the maximum number of deposits
    assert len(body.deposits) == min(MAX_DEPOSITS, state.eth1_data.deposit_count - state.eth1_deposit_index)

    def for_ops(operations: Sequence[Any], fn: Callable[[BeaconState, Any], None]) -> None:
        for operation in operations:
            fn(state, operation)

    for_ops(body.proposer_slashings, process_proposer_slashing)
    for_ops(body.attester_slashings, process_attester_slashing)
    # New attestation processing
    for_ops(body.attestations, process_attestation)
    for_ops(body.deposits, process_deposit)
    for_ops(body.voluntary_exits, process_voluntary_exit)
    # Limit is dynamic based on active shard count
    assert len(body.shard_headers) <= MAX_SHARD_HEADERS_PER_SHARD * get_active_shard_count(state, get_current_epoch(state))
    for_ops(body.shard_headers, process_shard_header)

    # See custody game spec.
    process_custody_game_operations(state, body)

    process_shard_transitions(state, body.shard_transitions, body.attestations)

    # TODO process_operations(body.shard_receipt_proofs, process_shard_receipt_proofs)

New Attestation processing

Updated `process_attestation`

def process_attestation(state: BeaconState, attestation: Attestation) -> None:
    phase0.process_attestation(state, attestation)
    update_pending_votes(
        state=state,
        attestation: Attestation,
        root=,
        aggregation_bits=attestation.aggregation_bits
    )

`update_pending_votes`

def update_pending_votes(state: BeaconState,
                         attestation: Attestation) -> None:
    if compute_epoch_at_slot(slot) == get_current_epoch(state):
        pending_headers = state.current_epoch_pending_shard_headers
    else:
        pending_headers = state.previous_epoch_pending_shard_headers
    # Create or update the PendingShardHeader object
    pending_header = None
    for header in pending_headers:
        if header.root == attestation.data.shard_header_root:
            pending_header = header
    assert pending_header is not None
    assert pending_header.slot == attestation.data.slot + 1
    assert pending_header.shard == compute_shard_from_committee_index(
        state,
        attestation.data.index,
        attestation.data.slot
    )
    pending_header.votes = bitwise_or(
        pending_header.votes,
        attestation.aggregation_bits
    )

    # Check if the PendingShardHeader is eligible for expedited confirmation
    # Requirement 1: nothing else confirmed
    all_candidates = [
        c for c in pending_headers if
        (c.slot, c.shard) == (pending_header.slot, pending_header.shard)
    ]
    if True not in [c.confirmed for c in all_candidates]:
        # Requirement 2: >= 2/3 of balance attesting
        participants = get_attesting_indices(state, attestationg.data, pending_commitment.votes)
        participants_balance = get_total_balance(state, participants)
        full_committee = get_beacon_committee(state, attestationg.data.slot, attestationg.data.shard)
        full_committee_balance = get_total_balance(state, full_committee)
        if participants_balance * 3 > full_committee_balance * 2:
            pending_header.confirmed = True

`process_shard_header`

def process_shard_header(state: BeaconState,
                         signed_header: Signed[ShardDataHeader]) -> None:
    header = signed_header.message
    header_root = hash_tree_root(header)
    # Verify signature
    signer_index = get_shard_proposer_index(state, header.slot, header.shard)
    assert bls.Verify(
        state.validators[signer_index].pubkey,
        compute_signing_root(header, get_domain(state, DOMAIN_SHARD_HEADER)),
        signed_header.signature
    )
    # Verify length of the header, and simultaneously verify degree.
    assert (
        bls.Pairing(header.degree_proof, G2_SETUP[0]) ==
        bls.Pairing(header.commitment.point, G2_SETUP[-header.commitment.length]))
    )
    # Get the correct pending header list
    if compute_epoch_at_slot(header.slot) == get_current_epoch(state):
        pending_headers = state.current_epoch_pending_shard_headers
    else:
        pending_headers = state.previous_epoch_pending_shard_headers
        
    # Check that this header is not yet in the pending list
    for pending_header in pending_headers:
        assert header_root != pending_header.root
    # Include it in the pending list
    committee_length = len(get_beacon_committee(state, header.slot, header.shard))
    pending_headers.append(PendingShardHeader(
        slot=header.slot,
        shard=header.shard,
        commitment=header.commitment,
        root=header_root,
        votes=Bitlist[MAX_VALIDATORS_PER_COMMITTEE]([0] * committee_length),
        confirmed=False
    ))

The degree proof works as follows. For a block B with length l (so l values in [0...l - 1], seen as a polynomial B(X) which takes these values), the length proof is the commitment to the polynomial B(X) * X**(MAX_DEGREE + 1 - l), where MAX_DEGREE the the maximum power of s available in the setup, which is MAX_DEGREE = len(G2_SETUP) - 1. The goal is to ensure that a proof can only be constructed if deg(B) < l (there are not hidden higher-order terms in the polynomial, which would thwart reconstruction).

Shard transition processing

Epoch transition

This epoch transition overrides the phase0 epoch transition:

def process_epoch(state: BeaconState) -> None:
    process_justification_and_finalization(state)
    process_rewards_and_penalties(state)
    process_registry_updates(state)

    # Proof of custody
    process_reveal_deadlines(state)
    process_challenge_deadlines(state)

    process_slashings(state)

    # Sharding
    process_pending_headers(state)
    charge_confirmed_header_fees(state)
    reset_pending_headers(state)

    # Final updates
    # Phase 0
    process_eth1_data_reset(state)
    process_effective_balance_updates(state)
    process_slashings_reset(state)
    process_randao_mixes_reset(state)
    process_historical_roots_update(state)
    process_participation_record_updates(state)
    # Proof of custody
    process_custody_final_updates(state)
    # Update current_epoch_start_shard
    state.current_epoch_start_shard = get_start_shard(state, Slot(state.slot + 1))

Pending headers


def process_pending_headers(state: BeaconState):
    for slot in range(SLOTS_PER_EPOCH):
        for shard in range(SHARD_COUNT):
            # Pending headers for this (slot, shard) combo
            candidates = [
                c for c in state.previous_epoch_pending_shard_headers if
                (c.slot, c.shard) == (slot, shard)
            ]
            if True not in [c.confirmed for c in candidates]:
                # The entire committee (and its balance)
                full_committee = get_beacon_committee(state, slot, shard)
                full_committee_balance = get_total_balance(state, full_committee)
                # The set of voters who voted for each header
                # (and their total balances)
                voting_sets = [
                    [v for i, v in enumerate(full_committee) if c.votes[i]]
                    for c in candidates
                ]
                voting_balances = [
                    get_total_balance(state, voters)
                    for voters in voting_sets
                ]
                # Get the index with the most total balance voting for them.
                # NOTE: if two choices get exactly the same voting balance,
                # the candidate earlier in the list wins
                if max(voting_balances) > 0:
                    winning_index = voting_balances.index(max(voting_balances))
                else:
                    # If no votes, zero wins
                    winning_index = [c.root for c in candidates].index(Root())
                candidates[winning_index].confirmed = True
    for slot in range(SLOTS_PER_EPOCH):
        for shard in range(SHARD_COUNT):
            state.grandparent_epoch_confirmed_commitments[shard][slot] = DataCommitment()
    for c in state.previous_epoch_pending_shard_headers:
        if c.confirmed:
            state.grandparent_epoch_confirmed_commitments[c.shard][c.slot % SLOTS_PER_EPOCH] = c.commitment

def charge_confirmed_header_fees(state: BeaconState) -> None:
    new_gasprice = state.shard_gasprice
    adjustment_quotient = get_active_shard_count(state) * SLOTS_PER_EPOCH * GASPRICE_ADJUSTMENT_COEFFICIENT
    for slot in range(SLOTS_PER_EPOCH):
        for shard in range(SHARD_COUNT):
            confirmed_candidates = [
                c for c in state.previous_epoch_pending_shard_headers if
                (c.slot, c.shard, c.confirmed) == (slot, shard, True)
            ]
            if confirmed_candidates:
                candidate = confirmed_candidates[0]
                # Charge EIP 1559 fee
                proposer = get_shard_proposer(state, slot, shard)
                fee = (
                    (state.shard_gasprice * candidates[i].commitment.length) //
                    TARGET_SAMPLES_PER_BLOCK
                )
                decrease_balance(state, proposer, fee)
                new_gasprice = compute_updated_gasprice(
                    new_gasprice,
                    candidates[i].commitment.length,
                    adjustment_quotient
                )
    state.shard_gasprice = new_gasprice

def reset_pending_headers(state: BeaconState):
    state.previous_epoch_pending_shard_headers = state.current_epoch_pending_shard_headers
    shards = [
        compute_shard_from_committee_index(state, index, slot)
        for i in range()
        state,
        attestation.data.index,
        attestation.data.slot
    )
    state.current_epoch_pending_shard_headers = []
    # Add dummy "empty" PendingAttestations
    # (default to vote for if no shard header available)
    for slot in range(SLOTS_IN_EPOCH):
        for index in range(get_committee_count_per_slot(get_current_epoch(state))):
            shard = compute_shard_from_committee_index(state, index, slot)
            committee_length = len(get_beacon_committee(
                state,
                header.slot,
                header.shard
            ))
            state.current_epoch_pending_shard_headers.append(PendingShardHeader(
                slot=slot,
                shard=shard,
                commitment=DataCommitment(),
                root=Root(),
                votes=Bitlist[MAX_VALIDATORS_PER_COMMITTEE]([0] * committee_length),
                confirmed=False
            ))

Custody game updates

process_reveal_deadlines, process_challenge_deadlines and process_custody_final_updates are defined in the Custody Game spec.

24 KiB Raw Blame History

Ethereum 2.0 Phase 1 -- The Beacon Chain with Shards

Table of contents

Introduction

Custom types

Configuration

Misc

Shard block configs

Precomputed size verification points

Gwei values

Time parameters

Domain types

Updated containers

AttestationData

BeaconBlock

BeaconState

New containers

DataCommitment

ShardHeader

PendingShardHeader

Helper functions

Misc

next_power_of_two

reverse_bit_order

compute_previous_slot

compute_shard_from_committee_index

compute_updated_gasprice

compute_committee_source_epoch

Beacon state accessors

Updated get_committee_count_per_slot

get_active_shard_count

get_shard_committee

compute_proposer_index

get_shard_proposer_index

get_start_shard

Block processing

Operations

New Attestation processing

Updated process_attestation

update_pending_votes

process_shard_header

Shard transition processing

Epoch transition

Pending headers

Custody game updates

24 KiB

Raw Blame History

`AttestationData`

`BeaconBlock`

`BeaconState`

`DataCommitment`

`ShardHeader`

`PendingShardHeader`

`next_power_of_two`

`reverse_bit_order`

`compute_previous_slot`

`compute_shard_from_committee_index`

`compute_updated_gasprice`

`compute_committee_source_epoch`

Updated `get_committee_count_per_slot`

`get_active_shard_count`

`get_shard_committee`

`compute_proposer_index`

`get_shard_proposer_index`

`get_start_shard`

Updated `process_attestation`

`update_pending_votes`

`process_shard_header`