eth2.0-specs/specs/deneb/p2p-interface.md

387 lines
17 KiB
Markdown
Raw Normal View History

2023-02-07 23:22:28 +00:00
# Deneb -- Networking
2022-03-10 05:31:11 +00:00
2023-02-07 23:22:28 +00:00
This document contains the consensus-layer networking specification for Deneb.
2022-03-10 05:31:11 +00:00
The specification of these changes continues in the same format as the network specifications of previous upgrades, and assumes them as pre-requisite.
## Table of contents
<!-- TOC -->
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
2023-04-19 11:10:46 +00:00
- [Modifications in Deneb](#modifications-in-deneb)
- [Configuration](#configuration)
- [Containers](#containers)
- [`BlobSidecar`](#blobsidecar)
- [`SignedBlobSidecar`](#signedblobsidecar)
- [`BlobIdentifier`](#blobidentifier)
- [Helpers](#helpers)
- [`verify_blob_sidecar_signature`](#verify_blob_sidecar_signature)
- [The gossip domain: gossipsub](#the-gossip-domain-gossipsub)
- [Topics and messages](#topics-and-messages)
- [Global topics](#global-topics)
- [`beacon_block`](#beacon_block)
- [`blob_sidecar_{subnet_id}`](#blob_sidecar_subnet_id)
2023-06-14 16:51:10 +00:00
- [`beacon_aggregate_and_proof`](#beacon_aggregate_and_proof)
- [Attestation subnets](#attestation-subnets)
2023-06-20 21:50:20 +00:00
- [`beacon_attestation_{subnet_id}`](#beacon_attestation_subnet_id)
2023-04-19 11:10:46 +00:00
- [Transitioning the gossip](#transitioning-the-gossip)
- [The Req/Resp domain](#the-reqresp-domain)
- [Messages](#messages)
- [BeaconBlocksByRange v2](#beaconblocksbyrange-v2)
- [BeaconBlocksByRoot v2](#beaconblocksbyroot-v2)
- [BlobSidecarsByRoot v1](#blobsidecarsbyroot-v1)
- [BlobSidecarsByRange v1](#blobsidecarsbyrange-v1)
2022-03-10 05:31:11 +00:00
- [Design decision rationale](#design-decision-rationale)
- [Why are blobs relayed as a sidecar, separate from beacon blocks?](#why-are-blobs-relayed-as-a-sidecar-separate-from-beacon-blocks)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
<!-- /TOC -->
2023-04-19 11:10:46 +00:00
## Modifications in Deneb
### Configuration
2022-03-10 05:31:11 +00:00
*[New in Deneb:EIP4844]*
| Name | Value | Description |
|------------------------------------------|-----------------------------------|---------------------------------------------------------------------|
| `MAX_REQUEST_BLOCKS_DENEB` | `2**7` (= 128) | Maximum number of blocks in a single request |
| `MAX_REQUEST_BLOB_SIDECARS` | `MAX_REQUEST_BLOCKS_DENEB * MAX_BLOBS_PER_BLOCK` | Maximum number of blob sidecars in a single request |
| `MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS` | `2**12` (= 4096 epochs, ~18 days) | The minimum epoch range over which a node must serve blob sidecars |
| `BLOB_SIDECAR_SUBNET_COUNT` | `6` | The number of blob sidecar subnets used in the gossipsub protocol. |
2022-03-10 05:31:11 +00:00
2023-04-19 11:10:46 +00:00
### Containers
2022-03-10 05:31:11 +00:00
2023-04-19 11:10:46 +00:00
#### `BlobSidecar`
2023-06-08 07:05:46 +00:00
*[New in Deneb:EIP4844]*
2023-06-07 09:45:39 +00:00
```python
Free the blobs This PR reintroduces and further decouples blocks and blobs in EIP-4844, so as to improve network and processing performance. Block and blob processing, for the purpose of gossip validation, are independent: they can both be propagated and gossip-validated in parallel - the decoupled design allows 4 important optimizations (or, if you are so inclined, removes 4 unnecessary pessimizations): * Blocks and blobs travel on independent meshes allowing for better parallelization and utilization of high-bandwidth peers * Re-broadcasting after validation can start earlier allowing more efficient use of upload bandwidth - blocks for example can be rebroadcast to peers while blobs are still being downloaded * bandwidth-reduction techniques such as per-peer deduplication are more efficient because of the smaller message size * gossip verification happens independently for blocks and blobs, allowing better sharing / use of CPU and I/O resources in clients With growing block sizes and additional blob data to stream, the network streaming time becomes a dominant factor in propagation times - on a 100mbit line, streaming 1mb to 8 peers takes ~1s - this process is repeated for each hop in both incoming and outgoing directions. This design in particular sends each blob on a separate subnet, thus maximising the potential for parallelisation and providing a natural path for growing the number of blobs per block should the network be judged to be able to handle it. Changes compared to the current design include: * `BlobsSidecar` is split into individual `BlobSidecar` containers - each container is signed individually by the proposer * the signature is used during gossip validation but later dropped. * KZG commitment verification is moved out of the gossip pipeline and instead done before fork choice addition, when both block and sidecars have arrived * clients may verify individual blob commitments earlier * more generally and similar to block verification, gossip propagation is performed solely based on trivial consistency checks and proposer signature verification * by-root blob requests are done per-blob, so as to retain the ability to fill in blobs one-by-one assuming clients generally receive blobs from gossip * by-range blob requests are done per-block, so as to simplify historical sync * range and root requests are limited to `128` entries for both blocks and blobs - practically, the current higher limit of `1024` for blocks does not get used and keeping the limits consistent simplifies implementation - with the merge, block sizes have grown significantly and clients generally fetch smaller chunks.
2023-02-07 09:55:51 +00:00
class BlobSidecar(Container):
block_root: Root
index: BlobIndex # Index of blob in block
Free the blobs This PR reintroduces and further decouples blocks and blobs in EIP-4844, so as to improve network and processing performance. Block and blob processing, for the purpose of gossip validation, are independent: they can both be propagated and gossip-validated in parallel - the decoupled design allows 4 important optimizations (or, if you are so inclined, removes 4 unnecessary pessimizations): * Blocks and blobs travel on independent meshes allowing for better parallelization and utilization of high-bandwidth peers * Re-broadcasting after validation can start earlier allowing more efficient use of upload bandwidth - blocks for example can be rebroadcast to peers while blobs are still being downloaded * bandwidth-reduction techniques such as per-peer deduplication are more efficient because of the smaller message size * gossip verification happens independently for blocks and blobs, allowing better sharing / use of CPU and I/O resources in clients With growing block sizes and additional blob data to stream, the network streaming time becomes a dominant factor in propagation times - on a 100mbit line, streaming 1mb to 8 peers takes ~1s - this process is repeated for each hop in both incoming and outgoing directions. This design in particular sends each blob on a separate subnet, thus maximising the potential for parallelisation and providing a natural path for growing the number of blobs per block should the network be judged to be able to handle it. Changes compared to the current design include: * `BlobsSidecar` is split into individual `BlobSidecar` containers - each container is signed individually by the proposer * the signature is used during gossip validation but later dropped. * KZG commitment verification is moved out of the gossip pipeline and instead done before fork choice addition, when both block and sidecars have arrived * clients may verify individual blob commitments earlier * more generally and similar to block verification, gossip propagation is performed solely based on trivial consistency checks and proposer signature verification * by-root blob requests are done per-blob, so as to retain the ability to fill in blobs one-by-one assuming clients generally receive blobs from gossip * by-range blob requests are done per-block, so as to simplify historical sync * range and root requests are limited to `128` entries for both blocks and blobs - practically, the current higher limit of `1024` for blocks does not get used and keeping the limits consistent simplifies implementation - with the merge, block sizes have grown significantly and clients generally fetch smaller chunks.
2023-02-07 09:55:51 +00:00
slot: Slot
block_parent_root: Root # Proposer shuffling determinant
Free the blobs This PR reintroduces and further decouples blocks and blobs in EIP-4844, so as to improve network and processing performance. Block and blob processing, for the purpose of gossip validation, are independent: they can both be propagated and gossip-validated in parallel - the decoupled design allows 4 important optimizations (or, if you are so inclined, removes 4 unnecessary pessimizations): * Blocks and blobs travel on independent meshes allowing for better parallelization and utilization of high-bandwidth peers * Re-broadcasting after validation can start earlier allowing more efficient use of upload bandwidth - blocks for example can be rebroadcast to peers while blobs are still being downloaded * bandwidth-reduction techniques such as per-peer deduplication are more efficient because of the smaller message size * gossip verification happens independently for blocks and blobs, allowing better sharing / use of CPU and I/O resources in clients With growing block sizes and additional blob data to stream, the network streaming time becomes a dominant factor in propagation times - on a 100mbit line, streaming 1mb to 8 peers takes ~1s - this process is repeated for each hop in both incoming and outgoing directions. This design in particular sends each blob on a separate subnet, thus maximising the potential for parallelisation and providing a natural path for growing the number of blobs per block should the network be judged to be able to handle it. Changes compared to the current design include: * `BlobsSidecar` is split into individual `BlobSidecar` containers - each container is signed individually by the proposer * the signature is used during gossip validation but later dropped. * KZG commitment verification is moved out of the gossip pipeline and instead done before fork choice addition, when both block and sidecars have arrived * clients may verify individual blob commitments earlier * more generally and similar to block verification, gossip propagation is performed solely based on trivial consistency checks and proposer signature verification * by-root blob requests are done per-blob, so as to retain the ability to fill in blobs one-by-one assuming clients generally receive blobs from gossip * by-range blob requests are done per-block, so as to simplify historical sync * range and root requests are limited to `128` entries for both blocks and blobs - practically, the current higher limit of `1024` for blocks does not get used and keeping the limits consistent simplifies implementation - with the merge, block sizes have grown significantly and clients generally fetch smaller chunks.
2023-02-07 09:55:51 +00:00
proposer_index: ValidatorIndex
blob: Blob
kzg_commitment: KZGCommitment
kzg_proof: KZGProof # Allows for quick verification of kzg_commitment
Free the blobs This PR reintroduces and further decouples blocks and blobs in EIP-4844, so as to improve network and processing performance. Block and blob processing, for the purpose of gossip validation, are independent: they can both be propagated and gossip-validated in parallel - the decoupled design allows 4 important optimizations (or, if you are so inclined, removes 4 unnecessary pessimizations): * Blocks and blobs travel on independent meshes allowing for better parallelization and utilization of high-bandwidth peers * Re-broadcasting after validation can start earlier allowing more efficient use of upload bandwidth - blocks for example can be rebroadcast to peers while blobs are still being downloaded * bandwidth-reduction techniques such as per-peer deduplication are more efficient because of the smaller message size * gossip verification happens independently for blocks and blobs, allowing better sharing / use of CPU and I/O resources in clients With growing block sizes and additional blob data to stream, the network streaming time becomes a dominant factor in propagation times - on a 100mbit line, streaming 1mb to 8 peers takes ~1s - this process is repeated for each hop in both incoming and outgoing directions. This design in particular sends each blob on a separate subnet, thus maximising the potential for parallelisation and providing a natural path for growing the number of blobs per block should the network be judged to be able to handle it. Changes compared to the current design include: * `BlobsSidecar` is split into individual `BlobSidecar` containers - each container is signed individually by the proposer * the signature is used during gossip validation but later dropped. * KZG commitment verification is moved out of the gossip pipeline and instead done before fork choice addition, when both block and sidecars have arrived * clients may verify individual blob commitments earlier * more generally and similar to block verification, gossip propagation is performed solely based on trivial consistency checks and proposer signature verification * by-root blob requests are done per-blob, so as to retain the ability to fill in blobs one-by-one assuming clients generally receive blobs from gossip * by-range blob requests are done per-block, so as to simplify historical sync * range and root requests are limited to `128` entries for both blocks and blobs - practically, the current higher limit of `1024` for blocks does not get used and keeping the limits consistent simplifies implementation - with the merge, block sizes have grown significantly and clients generally fetch smaller chunks.
2023-02-07 09:55:51 +00:00
```
2023-04-19 11:10:46 +00:00
#### `SignedBlobSidecar`
Free the blobs This PR reintroduces and further decouples blocks and blobs in EIP-4844, so as to improve network and processing performance. Block and blob processing, for the purpose of gossip validation, are independent: they can both be propagated and gossip-validated in parallel - the decoupled design allows 4 important optimizations (or, if you are so inclined, removes 4 unnecessary pessimizations): * Blocks and blobs travel on independent meshes allowing for better parallelization and utilization of high-bandwidth peers * Re-broadcasting after validation can start earlier allowing more efficient use of upload bandwidth - blocks for example can be rebroadcast to peers while blobs are still being downloaded * bandwidth-reduction techniques such as per-peer deduplication are more efficient because of the smaller message size * gossip verification happens independently for blocks and blobs, allowing better sharing / use of CPU and I/O resources in clients With growing block sizes and additional blob data to stream, the network streaming time becomes a dominant factor in propagation times - on a 100mbit line, streaming 1mb to 8 peers takes ~1s - this process is repeated for each hop in both incoming and outgoing directions. This design in particular sends each blob on a separate subnet, thus maximising the potential for parallelisation and providing a natural path for growing the number of blobs per block should the network be judged to be able to handle it. Changes compared to the current design include: * `BlobsSidecar` is split into individual `BlobSidecar` containers - each container is signed individually by the proposer * the signature is used during gossip validation but later dropped. * KZG commitment verification is moved out of the gossip pipeline and instead done before fork choice addition, when both block and sidecars have arrived * clients may verify individual blob commitments earlier * more generally and similar to block verification, gossip propagation is performed solely based on trivial consistency checks and proposer signature verification * by-root blob requests are done per-blob, so as to retain the ability to fill in blobs one-by-one assuming clients generally receive blobs from gossip * by-range blob requests are done per-block, so as to simplify historical sync * range and root requests are limited to `128` entries for both blocks and blobs - practically, the current higher limit of `1024` for blocks does not get used and keeping the limits consistent simplifies implementation - with the merge, block sizes have grown significantly and clients generally fetch smaller chunks.
2023-02-07 09:55:51 +00:00
2023-06-08 07:05:46 +00:00
*[New in Deneb:EIP4844]*
2023-06-07 09:45:39 +00:00
Free the blobs This PR reintroduces and further decouples blocks and blobs in EIP-4844, so as to improve network and processing performance. Block and blob processing, for the purpose of gossip validation, are independent: they can both be propagated and gossip-validated in parallel - the decoupled design allows 4 important optimizations (or, if you are so inclined, removes 4 unnecessary pessimizations): * Blocks and blobs travel on independent meshes allowing for better parallelization and utilization of high-bandwidth peers * Re-broadcasting after validation can start earlier allowing more efficient use of upload bandwidth - blocks for example can be rebroadcast to peers while blobs are still being downloaded * bandwidth-reduction techniques such as per-peer deduplication are more efficient because of the smaller message size * gossip verification happens independently for blocks and blobs, allowing better sharing / use of CPU and I/O resources in clients With growing block sizes and additional blob data to stream, the network streaming time becomes a dominant factor in propagation times - on a 100mbit line, streaming 1mb to 8 peers takes ~1s - this process is repeated for each hop in both incoming and outgoing directions. This design in particular sends each blob on a separate subnet, thus maximising the potential for parallelisation and providing a natural path for growing the number of blobs per block should the network be judged to be able to handle it. Changes compared to the current design include: * `BlobsSidecar` is split into individual `BlobSidecar` containers - each container is signed individually by the proposer * the signature is used during gossip validation but later dropped. * KZG commitment verification is moved out of the gossip pipeline and instead done before fork choice addition, when both block and sidecars have arrived * clients may verify individual blob commitments earlier * more generally and similar to block verification, gossip propagation is performed solely based on trivial consistency checks and proposer signature verification * by-root blob requests are done per-blob, so as to retain the ability to fill in blobs one-by-one assuming clients generally receive blobs from gossip * by-range blob requests are done per-block, so as to simplify historical sync * range and root requests are limited to `128` entries for both blocks and blobs - practically, the current higher limit of `1024` for blocks does not get used and keeping the limits consistent simplifies implementation - with the merge, block sizes have grown significantly and clients generally fetch smaller chunks.
2023-02-07 09:55:51 +00:00
```python
class SignedBlobSidecar(Container):
message: BlobSidecar
signature: BLSSignature
```
2023-04-19 11:10:46 +00:00
#### `BlobIdentifier`
2023-06-08 07:05:46 +00:00
*[New in Deneb:EIP4844]*
2023-06-07 09:45:39 +00:00
```python
class BlobIdentifier(Container):
block_root: Root
index: BlobIndex
```
2023-04-19 11:10:46 +00:00
#### Helpers
2023-03-12 23:05:01 +00:00
2023-04-19 11:10:46 +00:00
##### `verify_blob_sidecar_signature`
2023-03-12 23:05:01 +00:00
```python
def verify_blob_sidecar_signature(state: BeaconState, signed_blob_sidecar: SignedBlobSidecar) -> bool:
proposer = state.validators[signed_blob_sidecar.message.proposer_index]
signing_root = compute_signing_root(signed_blob_sidecar.message, get_domain(state, DOMAIN_BLOB_SIDECAR))
return bls.Verify(proposer.pubkey, signing_root, signed_blob_sidecar.signature)
```
2023-04-19 11:10:46 +00:00
### The gossip domain: gossipsub
2022-03-10 05:31:11 +00:00
2023-02-07 23:22:28 +00:00
Some gossip meshes are upgraded in the fork of Deneb to support upgraded types.
2022-03-10 05:31:11 +00:00
2023-04-19 11:10:46 +00:00
#### Topics and messages
2022-03-10 05:31:11 +00:00
Topics follow the same specification as in prior upgrades.
2023-06-14 16:51:10 +00:00
The `beacon_block` topic is modified to also support Deneb blocks and new topics are added per table below.
The `voluntary_exit` topic is implicitly modified due to the lock-in use of `CAPELLA_FORK_VERSION` for this message signature validation for EIP-7044.
The `beacon_aggregate_and_proof` and `beacon_attestation_{subnet_id}` topics are modified to support the gossip of attestations created in epoch `N` to be gossiped through the entire range of slots in epoch `N+1` rather than only through one epoch of slots for EIP-7045.
2022-03-10 05:31:11 +00:00
The specification around the creation, validation, and dissemination of messages has not changed from the Capella document unless explicitly noted here.
2022-03-10 05:31:11 +00:00
The derivation of the `message-id` remains stable.
The new topics along with the type of the `data` field of a gossipsub message are given in this table:
| Name | Message Type |
| - | - |
2023-06-07 09:45:39 +00:00
| `blob_sidecar_{subnet_id}` | `SignedBlobSidecar` [New in Deneb:EIP4844] |
2022-03-10 05:31:11 +00:00
2023-04-19 11:10:46 +00:00
##### Global topics
2022-03-10 05:31:11 +00:00
Deneb introduces new global topics for blob sidecars.
2022-03-10 05:31:11 +00:00
2023-04-19 11:10:46 +00:00
###### `beacon_block`
2023-06-14 16:51:10 +00:00
The *type* of the payload of this topic changes to the (modified) `SignedBeaconBlock` found in Deneb.
*[Modified in Deneb:EIP4844]*
New validation:
- _[REJECT]_ The length of KZG commitments is less than or equal to the limitation defined in Consensus Layer --
i.e. validate that `len(body.signed_beacon_block.message.blob_kzg_commitments) <= MAX_BLOBS_PER_BLOCK`
###### `blob_sidecar_{subnet_id}`
2022-11-15 17:29:03 +00:00
2023-06-08 07:05:46 +00:00
*[New in Deneb:EIP4844]*
2023-06-07 09:45:39 +00:00
2023-05-20 14:05:22 +00:00
This topic is used to propagate signed blob sidecars, where each blob index maps to some `subnet_id`.
2022-03-10 05:31:11 +00:00
2023-03-31 14:06:17 +00:00
The following validations MUST pass before forwarding the `signed_blob_sidecar` on the network, assuming the alias `sidecar = signed_blob_sidecar.message`:
2022-03-10 05:31:11 +00:00
- _[REJECT]_ The sidecar is for the correct subnet -- i.e. `compute_subnet_for_blob_sidecar(sidecar.index) == subnet_id`.
- _[REJECT]_ The sidecar's index is consistent with `MAX_BLOBS_PER_BLOCK` -- i.e. `sidecar.index < MAX_BLOBS_PER_BLOCK`.
- _[IGNORE]_ The sidecar is not from a future slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- i.e. validate that `sidecar.slot <= current_slot` (a client MAY queue future sidecars for processing at the appropriate slot).
- _[IGNORE]_ The sidecar is from a slot greater than the latest finalized slot -- i.e. validate that `sidecar.slot > compute_start_slot_at_epoch(state.finalized_checkpoint.epoch)`
2023-02-22 15:51:56 +00:00
- _[IGNORE]_ The sidecar's block's parent (defined by `sidecar.block_parent_root`) has been seen (via both gossip and non-gossip sources) (a client MAY queue sidecars for processing once the parent block is retrieved).
- _[REJECT]_ The sidecar's block's parent (defined by `sidecar.block_parent_root`) passes validation.
- _[REJECT]_ The sidecar is from a higher slot than the sidecar's block's parent (defined by `sidecar.block_parent_root`).
2023-03-31 14:06:17 +00:00
- _[REJECT]_ The proposer signature, `signed_blob_sidecar.signature`, is valid as verified by `verify_blob_sidecar_signature`.
- _[IGNORE]_ The sidecar is the only sidecar with valid signature received for the tuple `(sidecar.block_root, sidecar.index)`.
- _[REJECT]_ The sidecar is proposed by the expected `proposer_index` for the block's slot in the context of the current shuffling (defined by `block_parent_root`/`slot`).
If the `proposer_index` cannot immediately be verified against the expected shuffling, the sidecar MAY be queued for later processing while proposers for the block's branch are calculated -- in such a case _do not_ `REJECT`, instead `IGNORE` this message.
2022-03-10 05:31:11 +00:00
2023-06-14 16:51:10 +00:00
###### `beacon_aggregate_and_proof`
*[Modified in Deneb:EIP7045]*
The following validation is removed:
* _[IGNORE]_ `aggregate.data.slot` is within the last `ATTESTATION_PROPAGATION_SLOT_RANGE` slots (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
i.e. `aggregate.data.slot + ATTESTATION_PROPAGATION_SLOT_RANGE >= current_slot >= aggregate.data.slot`
(a client MAY queue future aggregates for processing at the appropriate slot).
The following validations are added in its place:
* _[IGNORE]_ `aggregate.data.slot` is equal to or earlier than the `current_slot` (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
i.e. `aggregate.data.slot <= current_slot`
(a client MAY queue future aggregates for processing at the appropriate slot).
* _[IGNORE]_ the epoch of `aggregate.data.slot` is either the current or previous epoch
(with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
i.e. `compute_epoch_at_slot(aggregate.data.slot) in (get_previous_epoch(state), get_current_epoch(state))`
##### Attestation subnets
2023-06-20 21:50:20 +00:00
###### `beacon_attestation_{subnet_id}`
2023-06-14 16:51:10 +00:00
*[Modified in Deneb:EIP7045]*
The following validation is removed:
* _[IGNORE]_ `attestation.data.slot` is within the last `ATTESTATION_PROPAGATION_SLOT_RANGE` slots (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
i.e. `attestation.data.slot + ATTESTATION_PROPAGATION_SLOT_RANGE >= current_slot >= attestation.data.slot`
(a client MAY queue future attestations for processing at the appropriate slot).
The following validations are added in its place:
* _[IGNORE]_ `attestation.data.slot` is equal to or earlier than the `current_slot` (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
i.e. `attestation.data.slot <= current_slot`
(a client MAY queue future attestation for processing at the appropriate slot).
* _[IGNORE]_ the epoch of `attestation.data.slot` is either the current or previous epoch
(with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
i.e. `compute_epoch_at_slot(attestation.data.slot) in (get_previous_epoch(state), get_current_epoch(state))`
2022-03-10 05:31:11 +00:00
2023-04-19 11:10:46 +00:00
#### Transitioning the gossip
2022-03-10 05:31:11 +00:00
See gossip transition details found in the [Altair document](../altair/p2p-interface.md#transitioning-the-gossip) for
details on how to handle transitioning gossip topics for this upgrade.
2022-03-10 05:31:11 +00:00
2023-04-19 11:10:46 +00:00
### The Req/Resp domain
2022-03-10 05:31:11 +00:00
2023-04-19 11:10:46 +00:00
#### Messages
2022-03-10 05:31:11 +00:00
2023-04-19 11:10:46 +00:00
##### BeaconBlocksByRange v2
2022-03-10 05:31:11 +00:00
**Protocol ID:** `/eth2/beacon_chain/req/beacon_blocks_by_range/2/`
2023-02-07 23:22:28 +00:00
The Deneb fork-digest is introduced to the `context` enum to specify Deneb beacon block type.
2022-03-10 05:31:11 +00:00
Per `context = compute_fork_digest(fork_version, genesis_validators_root)`:
[0]: # (eth2spec: skip)
| `fork_version` | Chunk SSZ type |
|--------------------------|-------------------------------|
| `GENESIS_FORK_VERSION` | `phase0.SignedBeaconBlock` |
| `ALTAIR_FORK_VERSION` | `altair.SignedBeaconBlock` |
| `BELLATRIX_FORK_VERSION` | `bellatrix.SignedBeaconBlock` |
2022-11-08 14:06:38 +00:00
| `CAPELLA_FORK_VERSION` | `capella.SignedBeaconBlock` |
| `DENEB_FORK_VERSION` | `deneb.SignedBeaconBlock` |
2022-03-10 05:31:11 +00:00
No more than `MAX_REQUEST_BLOCKS_DENEB` may be requested at a time.
Free the blobs This PR reintroduces and further decouples blocks and blobs in EIP-4844, so as to improve network and processing performance. Block and blob processing, for the purpose of gossip validation, are independent: they can both be propagated and gossip-validated in parallel - the decoupled design allows 4 important optimizations (or, if you are so inclined, removes 4 unnecessary pessimizations): * Blocks and blobs travel on independent meshes allowing for better parallelization and utilization of high-bandwidth peers * Re-broadcasting after validation can start earlier allowing more efficient use of upload bandwidth - blocks for example can be rebroadcast to peers while blobs are still being downloaded * bandwidth-reduction techniques such as per-peer deduplication are more efficient because of the smaller message size * gossip verification happens independently for blocks and blobs, allowing better sharing / use of CPU and I/O resources in clients With growing block sizes and additional blob data to stream, the network streaming time becomes a dominant factor in propagation times - on a 100mbit line, streaming 1mb to 8 peers takes ~1s - this process is repeated for each hop in both incoming and outgoing directions. This design in particular sends each blob on a separate subnet, thus maximising the potential for parallelisation and providing a natural path for growing the number of blobs per block should the network be judged to be able to handle it. Changes compared to the current design include: * `BlobsSidecar` is split into individual `BlobSidecar` containers - each container is signed individually by the proposer * the signature is used during gossip validation but later dropped. * KZG commitment verification is moved out of the gossip pipeline and instead done before fork choice addition, when both block and sidecars have arrived * clients may verify individual blob commitments earlier * more generally and similar to block verification, gossip propagation is performed solely based on trivial consistency checks and proposer signature verification * by-root blob requests are done per-blob, so as to retain the ability to fill in blobs one-by-one assuming clients generally receive blobs from gossip * by-range blob requests are done per-block, so as to simplify historical sync * range and root requests are limited to `128` entries for both blocks and blobs - practically, the current higher limit of `1024` for blocks does not get used and keeping the limits consistent simplifies implementation - with the merge, block sizes have grown significantly and clients generally fetch smaller chunks.
2023-02-07 09:55:51 +00:00
2023-04-19 11:10:46 +00:00
##### BeaconBlocksByRoot v2
2022-03-10 05:31:11 +00:00
**Protocol ID:** `/eth2/beacon_chain/req/beacon_blocks_by_root/2/`
Per `context = compute_fork_digest(fork_version, genesis_validators_root)`:
[1]: # (eth2spec: skip)
2022-12-13 16:07:37 +00:00
| `fork_version` | Chunk SSZ type |
|--------------------------|-------------------------------|
| `GENESIS_FORK_VERSION` | `phase0.SignedBeaconBlock` |
| `ALTAIR_FORK_VERSION` | `altair.SignedBeaconBlock` |
2022-03-10 05:31:11 +00:00
| `BELLATRIX_FORK_VERSION` | `bellatrix.SignedBeaconBlock` |
2022-11-08 14:06:38 +00:00
| `CAPELLA_FORK_VERSION` | `capella.SignedBeaconBlock` |
| `DENEB_FORK_VERSION` | `deneb.SignedBeaconBlock` |
2022-03-10 05:31:11 +00:00
No more than `MAX_REQUEST_BLOCKS_DENEB` may be requested at a time.
2022-03-10 05:31:11 +00:00
2023-04-19 11:10:46 +00:00
##### BlobSidecarsByRoot v1
Free the blobs This PR reintroduces and further decouples blocks and blobs in EIP-4844, so as to improve network and processing performance. Block and blob processing, for the purpose of gossip validation, are independent: they can both be propagated and gossip-validated in parallel - the decoupled design allows 4 important optimizations (or, if you are so inclined, removes 4 unnecessary pessimizations): * Blocks and blobs travel on independent meshes allowing for better parallelization and utilization of high-bandwidth peers * Re-broadcasting after validation can start earlier allowing more efficient use of upload bandwidth - blocks for example can be rebroadcast to peers while blobs are still being downloaded * bandwidth-reduction techniques such as per-peer deduplication are more efficient because of the smaller message size * gossip verification happens independently for blocks and blobs, allowing better sharing / use of CPU and I/O resources in clients With growing block sizes and additional blob data to stream, the network streaming time becomes a dominant factor in propagation times - on a 100mbit line, streaming 1mb to 8 peers takes ~1s - this process is repeated for each hop in both incoming and outgoing directions. This design in particular sends each blob on a separate subnet, thus maximising the potential for parallelisation and providing a natural path for growing the number of blobs per block should the network be judged to be able to handle it. Changes compared to the current design include: * `BlobsSidecar` is split into individual `BlobSidecar` containers - each container is signed individually by the proposer * the signature is used during gossip validation but later dropped. * KZG commitment verification is moved out of the gossip pipeline and instead done before fork choice addition, when both block and sidecars have arrived * clients may verify individual blob commitments earlier * more generally and similar to block verification, gossip propagation is performed solely based on trivial consistency checks and proposer signature verification * by-root blob requests are done per-blob, so as to retain the ability to fill in blobs one-by-one assuming clients generally receive blobs from gossip * by-range blob requests are done per-block, so as to simplify historical sync * range and root requests are limited to `128` entries for both blocks and blobs - practically, the current higher limit of `1024` for blocks does not get used and keeping the limits consistent simplifies implementation - with the merge, block sizes have grown significantly and clients generally fetch smaller chunks.
2023-02-07 09:55:51 +00:00
**Protocol ID:** `/eth2/beacon_chain/req/blob_sidecars_by_root/1/`
2023-06-08 07:05:46 +00:00
*[New in Deneb:EIP4844]*
The `<context-bytes>` field is calculated as `context = compute_fork_digest(fork_version, genesis_validators_root)`:
[1]: # (eth2spec: skip)
| `fork_version` | Chunk SSZ type |
|--------------------------|-------------------------------|
| `DENEB_FORK_VERSION` | `deneb.BlobSidecar` |
Request Content:
```
(
List[BlobIdentifier, MAX_REQUEST_BLOB_SIDECARS]
)
```
Response Content:
```
(
List[BlobSidecar, MAX_REQUEST_BLOB_SIDECARS]
)
```
Free the blobs This PR reintroduces and further decouples blocks and blobs in EIP-4844, so as to improve network and processing performance. Block and blob processing, for the purpose of gossip validation, are independent: they can both be propagated and gossip-validated in parallel - the decoupled design allows 4 important optimizations (or, if you are so inclined, removes 4 unnecessary pessimizations): * Blocks and blobs travel on independent meshes allowing for better parallelization and utilization of high-bandwidth peers * Re-broadcasting after validation can start earlier allowing more efficient use of upload bandwidth - blocks for example can be rebroadcast to peers while blobs are still being downloaded * bandwidth-reduction techniques such as per-peer deduplication are more efficient because of the smaller message size * gossip verification happens independently for blocks and blobs, allowing better sharing / use of CPU and I/O resources in clients With growing block sizes and additional blob data to stream, the network streaming time becomes a dominant factor in propagation times - on a 100mbit line, streaming 1mb to 8 peers takes ~1s - this process is repeated for each hop in both incoming and outgoing directions. This design in particular sends each blob on a separate subnet, thus maximising the potential for parallelisation and providing a natural path for growing the number of blobs per block should the network be judged to be able to handle it. Changes compared to the current design include: * `BlobsSidecar` is split into individual `BlobSidecar` containers - each container is signed individually by the proposer * the signature is used during gossip validation but later dropped. * KZG commitment verification is moved out of the gossip pipeline and instead done before fork choice addition, when both block and sidecars have arrived * clients may verify individual blob commitments earlier * more generally and similar to block verification, gossip propagation is performed solely based on trivial consistency checks and proposer signature verification * by-root blob requests are done per-blob, so as to retain the ability to fill in blobs one-by-one assuming clients generally receive blobs from gossip * by-range blob requests are done per-block, so as to simplify historical sync * range and root requests are limited to `128` entries for both blocks and blobs - practically, the current higher limit of `1024` for blocks does not get used and keeping the limits consistent simplifies implementation - with the merge, block sizes have grown significantly and clients generally fetch smaller chunks.
2023-02-07 09:55:51 +00:00
Requests sidecars by block root and index.
The response is a list of `BlobSidecar` whose length is less than or equal to the number of requests.
It may be less in the case that the responding peer is missing blocks or sidecars.
Free the blobs This PR reintroduces and further decouples blocks and blobs in EIP-4844, so as to improve network and processing performance. Block and blob processing, for the purpose of gossip validation, are independent: they can both be propagated and gossip-validated in parallel - the decoupled design allows 4 important optimizations (or, if you are so inclined, removes 4 unnecessary pessimizations): * Blocks and blobs travel on independent meshes allowing for better parallelization and utilization of high-bandwidth peers * Re-broadcasting after validation can start earlier allowing more efficient use of upload bandwidth - blocks for example can be rebroadcast to peers while blobs are still being downloaded * bandwidth-reduction techniques such as per-peer deduplication are more efficient because of the smaller message size * gossip verification happens independently for blocks and blobs, allowing better sharing / use of CPU and I/O resources in clients With growing block sizes and additional blob data to stream, the network streaming time becomes a dominant factor in propagation times - on a 100mbit line, streaming 1mb to 8 peers takes ~1s - this process is repeated for each hop in both incoming and outgoing directions. This design in particular sends each blob on a separate subnet, thus maximising the potential for parallelisation and providing a natural path for growing the number of blobs per block should the network be judged to be able to handle it. Changes compared to the current design include: * `BlobsSidecar` is split into individual `BlobSidecar` containers - each container is signed individually by the proposer * the signature is used during gossip validation but later dropped. * KZG commitment verification is moved out of the gossip pipeline and instead done before fork choice addition, when both block and sidecars have arrived * clients may verify individual blob commitments earlier * more generally and similar to block verification, gossip propagation is performed solely based on trivial consistency checks and proposer signature verification * by-root blob requests are done per-blob, so as to retain the ability to fill in blobs one-by-one assuming clients generally receive blobs from gossip * by-range blob requests are done per-block, so as to simplify historical sync * range and root requests are limited to `128` entries for both blocks and blobs - practically, the current higher limit of `1024` for blocks does not get used and keeping the limits consistent simplifies implementation - with the merge, block sizes have grown significantly and clients generally fetch smaller chunks.
2023-02-07 09:55:51 +00:00
The response is unsigned, i.e. `BlobSidecar`, as the signature of the beacon block proposer
may not be available beyond the initial distribution via gossip.
No more than `MAX_REQUEST_BLOB_SIDECARS` may be requested at a time.
`BlobSidecarsByRoot` is primarily used to recover recent blobs (e.g. when receiving a block with a transaction whose corresponding blob is missing).
The response MUST consist of zero or more `response_chunk`.
Free the blobs This PR reintroduces and further decouples blocks and blobs in EIP-4844, so as to improve network and processing performance. Block and blob processing, for the purpose of gossip validation, are independent: they can both be propagated and gossip-validated in parallel - the decoupled design allows 4 important optimizations (or, if you are so inclined, removes 4 unnecessary pessimizations): * Blocks and blobs travel on independent meshes allowing for better parallelization and utilization of high-bandwidth peers * Re-broadcasting after validation can start earlier allowing more efficient use of upload bandwidth - blocks for example can be rebroadcast to peers while blobs are still being downloaded * bandwidth-reduction techniques such as per-peer deduplication are more efficient because of the smaller message size * gossip verification happens independently for blocks and blobs, allowing better sharing / use of CPU and I/O resources in clients With growing block sizes and additional blob data to stream, the network streaming time becomes a dominant factor in propagation times - on a 100mbit line, streaming 1mb to 8 peers takes ~1s - this process is repeated for each hop in both incoming and outgoing directions. This design in particular sends each blob on a separate subnet, thus maximising the potential for parallelisation and providing a natural path for growing the number of blobs per block should the network be judged to be able to handle it. Changes compared to the current design include: * `BlobsSidecar` is split into individual `BlobSidecar` containers - each container is signed individually by the proposer * the signature is used during gossip validation but later dropped. * KZG commitment verification is moved out of the gossip pipeline and instead done before fork choice addition, when both block and sidecars have arrived * clients may verify individual blob commitments earlier * more generally and similar to block verification, gossip propagation is performed solely based on trivial consistency checks and proposer signature verification * by-root blob requests are done per-blob, so as to retain the ability to fill in blobs one-by-one assuming clients generally receive blobs from gossip * by-range blob requests are done per-block, so as to simplify historical sync * range and root requests are limited to `128` entries for both blocks and blobs - practically, the current higher limit of `1024` for blocks does not get used and keeping the limits consistent simplifies implementation - with the merge, block sizes have grown significantly and clients generally fetch smaller chunks.
2023-02-07 09:55:51 +00:00
Each _successful_ `response_chunk` MUST contain a single `BlobSidecar` payload.
2023-05-17 18:08:33 +00:00
Clients MUST support requesting sidecars since `minimum_request_epoch`, where `minimum_request_epoch = max(finalized_epoch, current_epoch - MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS, DENEB_FORK_EPOCH)`. If any root in the request content references a block earlier than `minimum_request_epoch`, peers MAY respond with error code `3: ResourceUnavailable` or not include the blob sidecar in the response.
Free the blobs This PR reintroduces and further decouples blocks and blobs in EIP-4844, so as to improve network and processing performance. Block and blob processing, for the purpose of gossip validation, are independent: they can both be propagated and gossip-validated in parallel - the decoupled design allows 4 important optimizations (or, if you are so inclined, removes 4 unnecessary pessimizations): * Blocks and blobs travel on independent meshes allowing for better parallelization and utilization of high-bandwidth peers * Re-broadcasting after validation can start earlier allowing more efficient use of upload bandwidth - blocks for example can be rebroadcast to peers while blobs are still being downloaded * bandwidth-reduction techniques such as per-peer deduplication are more efficient because of the smaller message size * gossip verification happens independently for blocks and blobs, allowing better sharing / use of CPU and I/O resources in clients With growing block sizes and additional blob data to stream, the network streaming time becomes a dominant factor in propagation times - on a 100mbit line, streaming 1mb to 8 peers takes ~1s - this process is repeated for each hop in both incoming and outgoing directions. This design in particular sends each blob on a separate subnet, thus maximising the potential for parallelisation and providing a natural path for growing the number of blobs per block should the network be judged to be able to handle it. Changes compared to the current design include: * `BlobsSidecar` is split into individual `BlobSidecar` containers - each container is signed individually by the proposer * the signature is used during gossip validation but later dropped. * KZG commitment verification is moved out of the gossip pipeline and instead done before fork choice addition, when both block and sidecars have arrived * clients may verify individual blob commitments earlier * more generally and similar to block verification, gossip propagation is performed solely based on trivial consistency checks and proposer signature verification * by-root blob requests are done per-blob, so as to retain the ability to fill in blobs one-by-one assuming clients generally receive blobs from gossip * by-range blob requests are done per-block, so as to simplify historical sync * range and root requests are limited to `128` entries for both blocks and blobs - practically, the current higher limit of `1024` for blocks does not get used and keeping the limits consistent simplifies implementation - with the merge, block sizes have grown significantly and clients generally fetch smaller chunks.
2023-02-07 09:55:51 +00:00
Clients MUST respond with at least one sidecar, if they have it.
Clients MAY limit the number of blocks and sidecars in the response.
2022-03-10 05:31:11 +00:00
2023-04-19 11:10:46 +00:00
##### BlobSidecarsByRange v1
2022-03-10 05:31:11 +00:00
Free the blobs This PR reintroduces and further decouples blocks and blobs in EIP-4844, so as to improve network and processing performance. Block and blob processing, for the purpose of gossip validation, are independent: they can both be propagated and gossip-validated in parallel - the decoupled design allows 4 important optimizations (or, if you are so inclined, removes 4 unnecessary pessimizations): * Blocks and blobs travel on independent meshes allowing for better parallelization and utilization of high-bandwidth peers * Re-broadcasting after validation can start earlier allowing more efficient use of upload bandwidth - blocks for example can be rebroadcast to peers while blobs are still being downloaded * bandwidth-reduction techniques such as per-peer deduplication are more efficient because of the smaller message size * gossip verification happens independently for blocks and blobs, allowing better sharing / use of CPU and I/O resources in clients With growing block sizes and additional blob data to stream, the network streaming time becomes a dominant factor in propagation times - on a 100mbit line, streaming 1mb to 8 peers takes ~1s - this process is repeated for each hop in both incoming and outgoing directions. This design in particular sends each blob on a separate subnet, thus maximising the potential for parallelisation and providing a natural path for growing the number of blobs per block should the network be judged to be able to handle it. Changes compared to the current design include: * `BlobsSidecar` is split into individual `BlobSidecar` containers - each container is signed individually by the proposer * the signature is used during gossip validation but later dropped. * KZG commitment verification is moved out of the gossip pipeline and instead done before fork choice addition, when both block and sidecars have arrived * clients may verify individual blob commitments earlier * more generally and similar to block verification, gossip propagation is performed solely based on trivial consistency checks and proposer signature verification * by-root blob requests are done per-blob, so as to retain the ability to fill in blobs one-by-one assuming clients generally receive blobs from gossip * by-range blob requests are done per-block, so as to simplify historical sync * range and root requests are limited to `128` entries for both blocks and blobs - practically, the current higher limit of `1024` for blocks does not get used and keeping the limits consistent simplifies implementation - with the merge, block sizes have grown significantly and clients generally fetch smaller chunks.
2023-02-07 09:55:51 +00:00
**Protocol ID:** `/eth2/beacon_chain/req/blob_sidecars_by_range/1/`
2022-03-10 05:31:11 +00:00
2023-06-08 07:05:46 +00:00
*[New in Deneb:EIP4844]*
2022-03-10 05:31:11 +00:00
The `<context-bytes>` field is calculated as `context = compute_fork_digest(fork_version, genesis_validators_root)`:
[1]: # (eth2spec: skip)
| `fork_version` | Chunk SSZ type |
|--------------------------|-------------------------------|
| `DENEB_FORK_VERSION` | `deneb.BlobSidecar` |
2022-03-10 05:31:11 +00:00
Request Content:
```
(
start_slot: Slot
count: uint64
)
```
Response Content:
```
(
List[BlobSidecar, MAX_REQUEST_BLOB_SIDECARS]
2022-03-10 05:31:11 +00:00
)
```
Requests blob sidecars in the slot range `[start_slot, start_slot + count)`, leading up to the current head block as selected by fork choice.
2022-03-10 05:31:11 +00:00
The response is unsigned, i.e. `BlobSidecarsByRange`, as the signature of the beacon block proposer may not be available beyond the initial distribution via gossip.
2022-03-10 05:31:11 +00:00
Before consuming the next response chunk, the response reader SHOULD verify the blob sidecar is well-formatted and correct w.r.t. the expected KZG commitments through `verify_blob_kzg_proof_batch`.
2022-03-10 05:31:11 +00:00
2023-02-17 18:59:56 +00:00
`BlobSidecarsByRange` is primarily used to sync blobs that may have been missed on gossip and to sync within the `MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS` window.
2022-03-10 05:31:11 +00:00
The request MUST be encoded as an SSZ-container.
The response MUST consist of zero or more `response_chunk`.
Each _successful_ `response_chunk` MUST contain a single `BlobSidecar` payload.
2022-03-10 05:31:11 +00:00
Let `blob_serve_range` be `[max(current_epoch - MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS, DENEB_FORK_EPOCH), current_epoch]`.
Clients MUST keep a record of signed blob sidecars seen on the epoch range `blob_serve_range`
2022-03-10 05:31:11 +00:00
where `current_epoch` is defined by the current wall-clock time,
2023-01-19 20:26:21 +00:00
and clients MUST support serving requests of blobs on this range.
2022-03-10 05:31:11 +00:00
Peers that are unable to reply to blob sidecar requests within the
range `blob_serve_range` SHOULD respond with error code `3: ResourceUnavailable`.
2022-03-10 05:31:11 +00:00
Such peers that are unable to successfully reply to this range of requests MAY get descored
or disconnected at any time.
*Note*: The above requirement implies that nodes that start from a recent weak subjectivity checkpoint
MUST backfill the local blobs database to at least the range `blob_serve_range`
2023-02-17 18:59:56 +00:00
to be fully compliant with `BlobSidecarsByRange` requests.
2022-03-10 05:31:11 +00:00
*Note*: Although clients that bootstrap from a weak subjectivity checkpoint can begin
participating in the networking immediately, other peers MAY
disconnect and/or temporarily ban such an un-synced or semi-synced client.
Clients MUST respond with at least the blob sidecars of the first blob-carrying block that exists in the range, if they have it, and no more than `MAX_REQUEST_BLOB_SIDECARS` sidecars.
2022-03-10 05:31:11 +00:00
Clients MUST include all blob sidecars of each block from which they include blob sidecars.
2022-03-10 05:31:11 +00:00
The following blob sidecars, where they exist, MUST be sent in consecutive `(slot, index)` order.
2022-03-10 05:31:11 +00:00
Slots that do not contain known blobs MUST be skipped, mimicking the behaviour
of the `BlocksByRange` request. Only response chunks with known blobs should
therefore be sent.
2023-01-10 18:28:19 +00:00
Clients MAY limit the number of blob sidecars in the response.
2022-03-10 05:31:11 +00:00
The response MUST contain no more than `count * MAX_BLOBS_PER_BLOCK` blob sidecars.
2022-03-10 05:31:11 +00:00
Clients MUST respond with blob sidecars from their view of the current fork choice
-- that is, blob sidecars as included by blocks from the single chain defined by the current head.
2022-03-10 05:31:11 +00:00
Of note, blocks from slots before the finalization MUST lead to the finalized block reported in the `Status` handshake.
Clients MUST respond with blob sidecars that are consistent from a single chain within the context of the request.
2022-03-10 05:31:11 +00:00
After the initial blob sidecar, clients MAY stop in the process of responding if their fork choice changes the view of the chain in the context of the request.
2022-03-10 05:31:11 +00:00
## Design decision rationale
2022-03-10 05:31:11 +00:00
### Why are blobs relayed as a sidecar, separate from beacon blocks?
2022-03-10 05:31:11 +00:00
This "sidecar" design provides forward compatibility for further data increases by black-boxing `is_data_available()`:
with full sharding `is_data_available()` can be replaced by data-availability-sampling (DAS)
thus avoiding all blobs being downloaded by all beacon nodes on the network.
2023-02-16 08:20:40 +00:00
Such sharding design may introduce an updated `BlobSidecar` to identify the shard,
2022-03-10 05:31:11 +00:00
but does not affect the `BeaconBlock` structure.