2023-02-07 23:22:28 +00:00
# Deneb -- Honest Validator
2022-03-10 05:31:11 +00:00
**Notice**: This document is a work-in-progress for researchers and implementers.
## Table of contents
<!-- TOC -->
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE - RUN doctoc TO UPDATE -->
- [Introduction ](#introduction )
- [Prerequisites ](#prerequisites )
- [Helpers ](#helpers )
2022-11-11 05:21:56 +00:00
- [`get_blobs_and_kzg_commitments` ](#get_blobs_and_kzg_commitments )
2022-03-10 05:31:11 +00:00
- [Beacon chain responsibilities ](#beacon-chain-responsibilities )
2022-10-22 15:13:14 +00:00
- [Block and sidecar proposal ](#block-and-sidecar-proposal )
2022-03-10 05:31:11 +00:00
- [Constructing the `BeaconBlockBody` ](#constructing-the-beaconblockbody )
2022-07-13 10:13:30 +00:00
- [Blob KZG commitments ](#blob-kzg-commitments )
2023-02-10 09:40:43 +00:00
- [Constructing the `SignedBlobSidecar`s ](#constructing-the-signedblobsidecars )
2022-10-22 15:13:14 +00:00
- [Sidecar ](#sidecar )
2022-03-10 05:31:11 +00:00
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
<!-- /TOC -->
## Introduction
2023-02-07 23:22:28 +00:00
This document represents the changes to be made in the code of an "honest validator" to implement Deneb.
2022-03-10 05:31:11 +00:00
## Prerequisites
2022-10-20 14:30:49 +00:00
This document is an extension of the [Capella -- Honest Validator ](../capella/validator.md ) guide.
2022-03-10 05:31:11 +00:00
All behaviors and definitions defined in this document, and documents it extends, carry over unless explicitly noted or overridden.
2023-02-07 23:22:28 +00:00
All terminology, constants, functions, and protocol mechanics defined in the updated [Beacon Chain doc of Deneb ](./beacon-chain.md ) are requisite for this document and used throughout.
2022-03-10 05:31:11 +00:00
Please see related Beacon Chain doc before continuing and use them as a reference throughout.
## Helpers
2022-11-11 05:21:56 +00:00
### `get_blobs_and_kzg_commitments`
2022-10-20 14:30:49 +00:00
2022-11-11 05:21:56 +00:00
The interface to retrieve blobs and corresponding kzg commitments.
2022-10-20 14:30:49 +00:00
2022-11-11 05:21:56 +00:00
Note: This API is *unstable* . `get_blobs_and_kzg_commitments` and `get_payload` may be unified.
Implementers may also retrieve blobs individually per transaction.
2022-10-20 14:30:49 +00:00
2022-11-11 05:21:56 +00:00
```python
2023-03-14 22:22:23 +00:00
def get_blobs_and_kzg_commitments(
payload_id: PayloadId
) -> Tuple[Sequence[Blob], Sequence[KZGCommitment], Sequence[KZGProof]]:
2022-11-28 10:51:38 +00:00
# pylint: disable=unused-argument
2022-11-11 05:21:56 +00:00
...
```
2022-10-20 14:30:49 +00:00
2022-03-10 05:31:11 +00:00
## Beacon chain responsibilities
All validator responsibilities remain unchanged other than those noted below.
2022-10-22 00:33:55 +00:00
### Block and sidecar proposal
2022-03-10 05:31:11 +00:00
#### Constructing the `BeaconBlockBody`
2022-07-13 10:12:31 +00:00
##### Blob KZG commitments
2022-03-10 05:31:11 +00:00
2022-10-20 14:30:49 +00:00
1. After retrieving the execution payload from the execution engine as specified in Capella,
2022-07-15 15:29:22 +00:00
use the `payload_id` to retrieve `blobs` and `blob_kzg_commitments` via `get_blobs_and_kzg_commitments(payload_id)` .
2022-07-15 15:54:53 +00:00
2. Validate `blobs` and `blob_kzg_commitments` :
2022-03-10 05:31:11 +00:00
2022-07-15 15:29:22 +00:00
```python
2022-07-15 15:54:53 +00:00
def validate_blobs_and_kzg_commitments(execution_payload: ExecutionPayload,
2022-10-01 14:25:47 +00:00
blobs: Sequence[Blob],
2023-03-11 23:02:35 +00:00
blob_kzg_commitments: Sequence[KZGCommitment],
blob_kzg_proofs: Sequence[KZGProof]) -> None:
2022-07-15 15:29:22 +00:00
# Optionally sanity-check that the KZG commitments match the versioned hashes in the transactions
assert verify_kzg_commitments_against_transactions(execution_payload.transactions, blob_kzg_commitments)
# Optionally sanity-check that the KZG commitments match the blobs (as produced by the execution engine)
2023-03-11 23:02:35 +00:00
assert len(blob_kzg_commitments) == len(blobs) == len(blob_kzg_proofs)
assert verify_blob_kzg_proof_batch(blobs, blob_kzg_commitments, blob_kzg_proofs)
2022-07-13 10:12:31 +00:00
```
2022-03-10 05:31:11 +00:00
2022-07-15 15:29:22 +00:00
3. If valid, set `block.body.blob_kzg_commitments = blob_kzg_commitments` .
2022-03-10 05:31:11 +00:00
2023-02-10 09:40:43 +00:00
#### Constructing the `SignedBlobSidecar`s
Free the blobs
This PR reintroduces and further decouples blocks and blobs in EIP-4844,
so as to improve network and processing performance.
Block and blob processing, for the purpose of gossip validation, are
independent: they can both be propagated and gossip-validated
in parallel - the decoupled design allows 4 important optimizations
(or, if you are so inclined, removes 4 unnecessary pessimizations):
* Blocks and blobs travel on independent meshes allowing for better
parallelization and utilization of high-bandwidth peers
* Re-broadcasting after validation can start earlier allowing more
efficient use of upload bandwidth - blocks for example can be
rebroadcast to peers while blobs are still being downloaded
* bandwidth-reduction techniques such as per-peer deduplication are more
efficient because of the smaller message size
* gossip verification happens independently for blocks and blobs,
allowing better sharing / use of CPU and I/O resources in clients
With growing block sizes and additional blob data to stream, the network
streaming time becomes a dominant factor in propagation times - on a
100mbit line, streaming 1mb to 8 peers takes ~1s - this process is
repeated for each hop in both incoming and outgoing directions.
This design in particular sends each blob on a separate subnet, thus
maximising the potential for parallelisation and providing a natural
path for growing the number of blobs per block should the network be
judged to be able to handle it.
Changes compared to the current design include:
* `BlobsSidecar` is split into individual `BlobSidecar` containers -
each container is signed individually by the proposer
* the signature is used during gossip validation but later dropped.
* KZG commitment verification is moved out of the gossip pipeline and
instead done before fork choice addition, when both block and sidecars
have arrived
* clients may verify individual blob commitments earlier
* more generally and similar to block verification, gossip propagation
is performed solely based on trivial consistency checks and proposer
signature verification
* by-root blob requests are done per-blob, so as to retain the ability
to fill in blobs one-by-one assuming clients generally receive blobs
from gossip
* by-range blob requests are done per-block, so as to simplify
historical sync
* range and root requests are limited to `128` entries for both blocks
and blobs - practically, the current higher limit of `1024` for blocks
does not get used and keeping the limits consistent simplifies
implementation - with the merge, block sizes have grown significantly
and clients generally fetch smaller chunks.
2023-02-07 09:55:51 +00:00
To construct a `SignedBlobSidecar` , a `signed_blob_sidecar` is defined with the necessary context for block and sidecar proposal.
2022-03-10 05:31:11 +00:00
2022-10-22 00:33:55 +00:00
##### Sidecar
2022-03-10 05:31:11 +00:00
2023-02-10 09:40:43 +00:00
Blobs associated with a block are packaged into sidecar objects for distribution to the network.
Free the blobs
This PR reintroduces and further decouples blocks and blobs in EIP-4844,
so as to improve network and processing performance.
Block and blob processing, for the purpose of gossip validation, are
independent: they can both be propagated and gossip-validated
in parallel - the decoupled design allows 4 important optimizations
(or, if you are so inclined, removes 4 unnecessary pessimizations):
* Blocks and blobs travel on independent meshes allowing for better
parallelization and utilization of high-bandwidth peers
* Re-broadcasting after validation can start earlier allowing more
efficient use of upload bandwidth - blocks for example can be
rebroadcast to peers while blobs are still being downloaded
* bandwidth-reduction techniques such as per-peer deduplication are more
efficient because of the smaller message size
* gossip verification happens independently for blocks and blobs,
allowing better sharing / use of CPU and I/O resources in clients
With growing block sizes and additional blob data to stream, the network
streaming time becomes a dominant factor in propagation times - on a
100mbit line, streaming 1mb to 8 peers takes ~1s - this process is
repeated for each hop in both incoming and outgoing directions.
This design in particular sends each blob on a separate subnet, thus
maximising the potential for parallelisation and providing a natural
path for growing the number of blobs per block should the network be
judged to be able to handle it.
Changes compared to the current design include:
* `BlobsSidecar` is split into individual `BlobSidecar` containers -
each container is signed individually by the proposer
* the signature is used during gossip validation but later dropped.
* KZG commitment verification is moved out of the gossip pipeline and
instead done before fork choice addition, when both block and sidecars
have arrived
* clients may verify individual blob commitments earlier
* more generally and similar to block verification, gossip propagation
is performed solely based on trivial consistency checks and proposer
signature verification
* by-root blob requests are done per-blob, so as to retain the ability
to fill in blobs one-by-one assuming clients generally receive blobs
from gossip
* by-range blob requests are done per-block, so as to simplify
historical sync
* range and root requests are limited to `128` entries for both blocks
and blobs - practically, the current higher limit of `1024` for blocks
does not get used and keeping the limits consistent simplifies
implementation - with the merge, block sizes have grown significantly
and clients generally fetch smaller chunks.
2023-02-07 09:55:51 +00:00
Each `sidecar` is obtained from:
2022-03-10 05:31:11 +00:00
```python
2023-03-12 23:05:01 +00:00
def get_blob_sidecars(block: BeaconBlock,
blobs: Sequence[Blob],
blob_kzg_proofs: Sequence[KZGProof]) -> Sequence[BlobSidecar]:
2023-02-10 09:40:43 +00:00
return [
2023-02-10 10:16:51 +00:00
BlobSidecar(
2023-02-10 09:40:43 +00:00
block_root=hash_tree_root(block),
index=index,
slot=block.slot,
block_parent_root=block.parent_root,
blob=blob,
2023-02-14 12:32:18 +00:00
kzg_commitment=block.body.blob_kzg_commitments[index],
2023-03-12 23:05:01 +00:00
kzg_proof=blob_kzg_proofs[index],
2023-02-10 09:40:43 +00:00
)
for index, blob in enumerate(blobs)
]
2023-02-10 10:16:51 +00:00
2022-03-10 05:31:11 +00:00
```
2023-02-16 07:18:52 +00:00
Then for each sidecar, `signed_sidecar = SignedBlobSidecar(message=sidecar, signature=signature)` is constructed and published to the `blob_sidecar_{index}` topics according to its index.
2023-02-10 10:16:51 +00:00
`signature` is obtained from:
```python
def get_blob_sidecar_signature(state: BeaconState,
sidecar: BlobSidecar,
privkey: int) -> BLSSignature:
2023-02-14 12:39:59 +00:00
domain = get_domain(state, DOMAIN_BLOB_SIDECAR, compute_epoch_at_slot(sidecar.slot))
2023-02-10 10:16:51 +00:00
signing_root = compute_signing_root(sidecar, domain)
return bls.Sign(privkey, signing_root)
```
2022-03-10 05:31:11 +00:00
2022-10-22 00:33:55 +00:00
After publishing the peers on the network may request the sidecar through sync-requests, or a local user may be interested.
2023-02-10 10:16:51 +00:00
2023-03-01 17:10:58 +00:00
The validator MUST hold on to sidecars for `MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS` epochs and serve when capable,
2022-03-10 05:31:11 +00:00
to ensure the data-availability of these blobs throughout the network.
2023-03-01 17:10:58 +00:00
After `MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS` nodes MAY prune the sidecars and/or stop serving them.