eth2.0-specs/specs/deneb/validator.md

195 lines
7.1 KiB
Markdown
Raw Normal View History

2023-02-07 23:22:28 +00:00
# Deneb -- Honest Validator
2022-03-10 05:31:11 +00:00
**Notice**: This document is a work-in-progress for researchers and implementers.
## Table of contents
<!-- TOC -->
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
- [Introduction](#introduction)
- [Prerequisites](#prerequisites)
- [Helpers](#helpers)
- [`BlobsBundle`](#blobsbundle)
- [Modified `GetPayloadResponse`](#modified-getpayloadresponse)
- [Protocol](#protocol)
- [`ExecutionEngine`](#executionengine)
- [Modified `get_payload`](#modified-get_payload)
2022-03-10 05:31:11 +00:00
- [Beacon chain responsibilities](#beacon-chain-responsibilities)
2022-10-22 15:13:14 +00:00
- [Block and sidecar proposal](#block-and-sidecar-proposal)
2022-03-10 05:31:11 +00:00
- [Constructing the `BeaconBlockBody`](#constructing-the-beaconblockbody)
2023-06-12 18:29:07 +00:00
- [ExecutionPayload](#executionpayload)
- [Blob KZG commitments](#blob-kzg-commitments)
2023-10-27 09:10:29 +00:00
- [Constructing the `BlobSidecar`s](#constructing-the-blobsidecars)
2023-10-27 13:16:10 +00:00
- [Sidecar](#sidecar)
2022-03-10 05:31:11 +00:00
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
<!-- /TOC -->
## Introduction
2023-02-07 23:22:28 +00:00
This document represents the changes to be made in the code of an "honest validator" to implement Deneb.
2022-03-10 05:31:11 +00:00
## Prerequisites
This document is an extension of the [Capella -- Honest Validator](../capella/validator.md) guide.
2022-03-10 05:31:11 +00:00
All behaviors and definitions defined in this document, and documents it extends, carry over unless explicitly noted or overridden.
2023-02-07 23:22:28 +00:00
All terminology, constants, functions, and protocol mechanics defined in the updated [Beacon Chain doc of Deneb](./beacon-chain.md) are requisite for this document and used throughout.
2022-03-10 05:31:11 +00:00
Please see related Beacon Chain doc before continuing and use them as a reference throughout.
## Helpers
### `BlobsBundle`
2023-06-08 07:05:46 +00:00
*[New in Deneb:EIP4844]*
```python
@dataclass
class BlobsBundle(object):
commitments: Sequence[KZGCommitment]
proofs: Sequence[KZGProof]
blobs: Sequence[Blob]
```
### Modified `GetPayloadResponse`
```python
@dataclass
class GetPayloadResponse(object):
execution_payload: ExecutionPayload
block_value: uint256
2023-06-08 07:05:46 +00:00
blobs_bundle: BlobsBundle # [New in Deneb:EIP4844]
```
wip wip Add `TARGET_NUMBER_OF_PEERS` Add networking spec draft fix simplification Rename `DoYouHave` to `GetCustodyStatus` Add DataLineSidecar design Apply suggestions from code review Co-authored-by: dankrad <mail@dankradfeist.de> Co-authored-by: danny <dannyjryan@gmail.com> Revamp after reviews and discussion Remove `CustodyStatus` minor fix Change`DataColumn` to `List[DataCell, MAX_BLOBS_PER_BLOCK]` Move folder Replace `DataColumnByRootAndIndex` with `DataColumnSidecarByRoot` message. Add extended data description Remove `DataRow` Apply suggestions from @jacobkaufmann code review Co-authored-by: Jacob Kaufmann <jacobkaufmann18@gmail.com> Represent matrix in `BLSFieldElement` form Add `assert time >= store.time` to `on_tick` Revert the spec. Only handle it in tests Remove extra tick cleanup leftover Add randomized block cases Specify RPC byRoot blocks-sidecars elegibility fix typo Update specs/phase0/p2p-interface.md Co-authored-by: Mikhail Kalinin <noblesse.knight@gmail.com> Update specs/deneb/p2p-interface.md Co-authored-by: Mikhail Kalinin <noblesse.knight@gmail.com> add failed on_block condition rephrase Update specs/phase0/p2p-interface.md Co-authored-by: Mikhail Kalinin <noblesse.knight@gmail.com> apply suggestion Update specs/deneb/p2p-interface.md Co-authored-by: danny <dannyjryan@gmail.com> Update specs/deneb/p2p-interface.md Co-authored-by: danny <dannyjryan@gmail.com> remove the last consider from on_block to state_transition simplify and add a new rule Update specs/phase0/p2p-interface.md Co-authored-by: Mikhail Kalinin <noblesse.knight@gmail.com> Update specs/deneb/p2p-interface.md Co-authored-by: Mikhail Kalinin <noblesse.knight@gmail.com> Update specs/deneb/p2p-interface.md Co-authored-by: danny <dannyjryan@gmail.com> remove gossip failure rules Apply suggestions from code review bump version to v1.4.0-beta.5 Move `blob_sidecar_{subnet_id}` to `Blob subnets` section Misc minor fix Add linter support Add column subnet validation. Split `verify_column_sidecar` into two functions Fix `get_data_column_sidecars` by using `compute_samples_and_proofs` Apply suggestions from code review Co-authored-by: danny <dannyjryan@gmail.com> Do not assign row custody Apply suggestions from code review Co-authored-by: danny <dannyjryan@gmail.com> Revamp reconstruction section Use depth as the primary preset for inclusion proof. Fix `get_data_column_sidecars` and add tests for merkle proof Change `SAMPLES_PER_SLOT` to 8 and add tests (requirement TBD) Apply PR feedback from @ppopth and @jtraglia Fix `get_data_column_sidecars` Co-authored-by: Pop Chunhapanya <haxx.pop@gmail.com> Apply suggestions from code review Co-authored-by: Pop Chunhapanya <haxx.pop@gmail.com> Apply suggestions from code review Co-authored-by: fradamt <104826920+fradamt@users.noreply.github.com> Co-authored-by: Jacob Kaufmann <jacobkaufmann18@gmail.com> Fix `get_data_column_sidecars` and `get_custody_lines` Apply suggestions from code review Co-authored-by: Jacob Kaufmann <jacobkaufmann18@gmail.com> Enhance tests fix typo Co-authored-by: fradamt <104826920+fradamt@users.noreply.github.com> Remove `epoch` from `get_custody_lines` fix fix
2023-11-14 09:04:42 +00:00
```python
def compute_signed_block_header(signed_block: SignedBeaconBlock) -> SignedBeaconBlockHeader:
block = signed_block.message
block_header = BeaconBlockHeader(
slot=block.slot,
proposer_index=block.proposer_index,
parent_root=block.parent_root,
state_root=block.state_root,
body_root=hash_tree_root(block.body),
)
return SignedBeaconBlockHeader(message=block_header, signature=signed_block.signature)
```
## Protocol
### `ExecutionEngine`
#### Modified `get_payload`
Given the `payload_id`, `get_payload` returns the most recent version of the execution payload that
has been built since the corresponding call to `notify_forkchoice_updated` method.
2022-11-11 05:21:56 +00:00
```python
def get_payload(self: ExecutionEngine, payload_id: PayloadId) -> GetPayloadResponse:
"""
Return ExecutionPayload, uint256, BlobsBundle objects.
"""
2022-11-28 10:51:38 +00:00
# pylint: disable=unused-argument
2022-11-11 05:21:56 +00:00
...
```
2022-03-10 05:31:11 +00:00
## Beacon chain responsibilities
All validator responsibilities remain unchanged other than those noted below.
### Block and sidecar proposal
2022-03-10 05:31:11 +00:00
#### Constructing the `BeaconBlockBody`
2023-06-12 18:29:07 +00:00
##### ExecutionPayload
`prepare_execution_payload` is updated from the Capella specs to provide the parent beacon block root.
*Note*: In this section, `state` is the state of the slot for the block proposal _without_ the block yet applied.
That is, `state` is the `previous_state` processed through any empty slots up to the assigned slot using `process_slots(previous_state, slot)`.
*Note*: The only change made to `prepare_execution_payload` is to add the parent beacon block root as an additional
parameter to the `PayloadAttributes`.
```python
def prepare_execution_payload(state: BeaconState,
safe_block_hash: Hash32,
finalized_block_hash: Hash32,
suggested_fee_recipient: ExecutionAddress,
execution_engine: ExecutionEngine) -> Optional[PayloadId]:
# Verify consistency of the parent hash with respect to the previous execution payload header
parent_hash = state.latest_execution_payload_header.block_hash
# Set the forkchoice head and initiate the payload build process
payload_attributes = PayloadAttributes(
timestamp=compute_timestamp_at_slot(state, state.slot),
prev_randao=get_randao_mix(state, get_current_epoch(state)),
suggested_fee_recipient=suggested_fee_recipient,
withdrawals=get_expected_withdrawals(state),
parent_beacon_block_root=hash_tree_root(state.latest_block_header), # [New in Deneb:EIP4788]
)
return execution_engine.notify_forkchoice_updated(
head_block_hash=parent_hash,
safe_block_hash=safe_block_hash,
finalized_block_hash=finalized_block_hash,
payload_attributes=payload_attributes,
)
```
##### Blob KZG commitments
2022-03-10 05:31:11 +00:00
2023-06-08 07:05:46 +00:00
*[New in Deneb:EIP4844]*
2023-06-07 09:45:39 +00:00
1. The execution payload is obtained from the execution engine as defined above using `payload_id`. The response also includes a `blobs_bundle` entry containing the corresponding `blobs`, `commitments`, and `proofs`.
2. Set `block.body.blob_kzg_commitments = commitments`.
2022-03-10 05:31:11 +00:00
2023-10-27 09:10:29 +00:00
#### Constructing the `BlobSidecar`s
2023-06-08 07:05:46 +00:00
*[New in Deneb:EIP4844]*
2023-10-27 09:10:29 +00:00
To construct a `BlobSidecar`, a `blob_sidecar` is defined with the necessary context for block and sidecar proposal.
2022-03-10 05:31:11 +00:00
##### Sidecar
2022-03-10 05:31:11 +00:00
2023-10-27 09:10:29 +00:00
Blobs associated with a block are packaged into sidecar objects for distribution to the associated sidecar topic, the `blob_sidecar_{subnet_id}` pubsub topic.
Free the blobs This PR reintroduces and further decouples blocks and blobs in EIP-4844, so as to improve network and processing performance. Block and blob processing, for the purpose of gossip validation, are independent: they can both be propagated and gossip-validated in parallel - the decoupled design allows 4 important optimizations (or, if you are so inclined, removes 4 unnecessary pessimizations): * Blocks and blobs travel on independent meshes allowing for better parallelization and utilization of high-bandwidth peers * Re-broadcasting after validation can start earlier allowing more efficient use of upload bandwidth - blocks for example can be rebroadcast to peers while blobs are still being downloaded * bandwidth-reduction techniques such as per-peer deduplication are more efficient because of the smaller message size * gossip verification happens independently for blocks and blobs, allowing better sharing / use of CPU and I/O resources in clients With growing block sizes and additional blob data to stream, the network streaming time becomes a dominant factor in propagation times - on a 100mbit line, streaming 1mb to 8 peers takes ~1s - this process is repeated for each hop in both incoming and outgoing directions. This design in particular sends each blob on a separate subnet, thus maximising the potential for parallelisation and providing a natural path for growing the number of blobs per block should the network be judged to be able to handle it. Changes compared to the current design include: * `BlobsSidecar` is split into individual `BlobSidecar` containers - each container is signed individually by the proposer * the signature is used during gossip validation but later dropped. * KZG commitment verification is moved out of the gossip pipeline and instead done before fork choice addition, when both block and sidecars have arrived * clients may verify individual blob commitments earlier * more generally and similar to block verification, gossip propagation is performed solely based on trivial consistency checks and proposer signature verification * by-root blob requests are done per-blob, so as to retain the ability to fill in blobs one-by-one assuming clients generally receive blobs from gossip * by-range blob requests are done per-block, so as to simplify historical sync * range and root requests are limited to `128` entries for both blocks and blobs - practically, the current higher limit of `1024` for blocks does not get used and keeping the limits consistent simplifies implementation - with the merge, block sizes have grown significantly and clients generally fetch smaller chunks.
2023-02-07 09:55:51 +00:00
Each `sidecar` is obtained from:
2022-03-10 05:31:11 +00:00
```python
2023-10-27 09:10:29 +00:00
def get_blob_sidecars(signed_block: SignedBeaconBlock,
2023-03-12 23:05:01 +00:00
blobs: Sequence[Blob],
blob_kzg_proofs: Sequence[KZGProof]) -> Sequence[BlobSidecar]:
2023-10-27 14:07:50 +00:00
block = signed_block.message
wip wip Add `TARGET_NUMBER_OF_PEERS` Add networking spec draft fix simplification Rename `DoYouHave` to `GetCustodyStatus` Add DataLineSidecar design Apply suggestions from code review Co-authored-by: dankrad <mail@dankradfeist.de> Co-authored-by: danny <dannyjryan@gmail.com> Revamp after reviews and discussion Remove `CustodyStatus` minor fix Change`DataColumn` to `List[DataCell, MAX_BLOBS_PER_BLOCK]` Move folder Replace `DataColumnByRootAndIndex` with `DataColumnSidecarByRoot` message. Add extended data description Remove `DataRow` Apply suggestions from @jacobkaufmann code review Co-authored-by: Jacob Kaufmann <jacobkaufmann18@gmail.com> Represent matrix in `BLSFieldElement` form Add `assert time >= store.time` to `on_tick` Revert the spec. Only handle it in tests Remove extra tick cleanup leftover Add randomized block cases Specify RPC byRoot blocks-sidecars elegibility fix typo Update specs/phase0/p2p-interface.md Co-authored-by: Mikhail Kalinin <noblesse.knight@gmail.com> Update specs/deneb/p2p-interface.md Co-authored-by: Mikhail Kalinin <noblesse.knight@gmail.com> add failed on_block condition rephrase Update specs/phase0/p2p-interface.md Co-authored-by: Mikhail Kalinin <noblesse.knight@gmail.com> apply suggestion Update specs/deneb/p2p-interface.md Co-authored-by: danny <dannyjryan@gmail.com> Update specs/deneb/p2p-interface.md Co-authored-by: danny <dannyjryan@gmail.com> remove the last consider from on_block to state_transition simplify and add a new rule Update specs/phase0/p2p-interface.md Co-authored-by: Mikhail Kalinin <noblesse.knight@gmail.com> Update specs/deneb/p2p-interface.md Co-authored-by: Mikhail Kalinin <noblesse.knight@gmail.com> Update specs/deneb/p2p-interface.md Co-authored-by: danny <dannyjryan@gmail.com> remove gossip failure rules Apply suggestions from code review bump version to v1.4.0-beta.5 Move `blob_sidecar_{subnet_id}` to `Blob subnets` section Misc minor fix Add linter support Add column subnet validation. Split `verify_column_sidecar` into two functions Fix `get_data_column_sidecars` by using `compute_samples_and_proofs` Apply suggestions from code review Co-authored-by: danny <dannyjryan@gmail.com> Do not assign row custody Apply suggestions from code review Co-authored-by: danny <dannyjryan@gmail.com> Revamp reconstruction section Use depth as the primary preset for inclusion proof. Fix `get_data_column_sidecars` and add tests for merkle proof Change `SAMPLES_PER_SLOT` to 8 and add tests (requirement TBD) Apply PR feedback from @ppopth and @jtraglia Fix `get_data_column_sidecars` Co-authored-by: Pop Chunhapanya <haxx.pop@gmail.com> Apply suggestions from code review Co-authored-by: Pop Chunhapanya <haxx.pop@gmail.com> Apply suggestions from code review Co-authored-by: fradamt <104826920+fradamt@users.noreply.github.com> Co-authored-by: Jacob Kaufmann <jacobkaufmann18@gmail.com> Fix `get_data_column_sidecars` and `get_custody_lines` Apply suggestions from code review Co-authored-by: Jacob Kaufmann <jacobkaufmann18@gmail.com> Enhance tests fix typo Co-authored-by: fradamt <104826920+fradamt@users.noreply.github.com> Remove `epoch` from `get_custody_lines` fix fix
2023-11-14 09:04:42 +00:00
signed_block_header = compute_signed_block_header(signed_block)
return [
BlobSidecar(
index=index,
blob=blob,
2023-10-27 14:07:50 +00:00
kzg_commitment=block.body.blob_kzg_commitments[index],
2023-03-12 23:05:01 +00:00
kzg_proof=blob_kzg_proofs[index],
signed_block_header=signed_block_header,
kzg_commitment_inclusion_proof=compute_merkle_proof(
2023-10-27 14:07:50 +00:00
block.body,
get_generalized_index(BeaconBlockBody, 'blob_kzg_commitments', index),
2023-10-27 09:10:29 +00:00
),
)
for index, blob in enumerate(blobs)
]
2022-03-10 05:31:11 +00:00
```
The `subnet_id` for the `blob_sidecar` is calculated with:
- Let `blob_index = blob_sidecar.index`.
- Let `subnet_id = compute_subnet_for_blob_sidecar(blob_index)`.
```python
2023-05-09 14:03:59 +00:00
def compute_subnet_for_blob_sidecar(blob_index: BlobIndex) -> SubnetID:
return SubnetID(blob_index % BLOB_SIDECAR_SUBNET_COUNT)
```
After publishing the peers on the network may request the sidecar through sync-requests, or a local user may be interested.
The validator MUST hold on to sidecars for `MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS` epochs and serve when capable,
2022-03-10 05:31:11 +00:00
to ensure the data-availability of these blobs throughout the network.
After `MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS` nodes MAY prune the sidecars and/or stop serving them.