simplify blob verification, range request

* validate blobs using raw types
* remove `BlobSidecars` and send flattened list of `BlobSidecar`
instances instead

specs/deneb/fork-choice.md

@@ -27,37 +27,36 @@ This is the modification of the fork choice accompanying the Deneb upgrade.
 #### `validate_blob_sidecars`
 
 ```python
-def validate_blob_sidecars(slot: Slot,
-                           beacon_block_root: Root,
-                           expected_kzg_commitments: Sequence[KZGCommitment],
-                           blob_sidecars: Sequence[BlobSidecar]) -> None:
-    assert slot == blobs_sidecar.beacon_block_slot
-    assert beacon_block_root == blobs_sidecar.beacon_block_root
-    assert len(expected_kzg_commitments) == len(blob_sidecars)
-    # TODO validate commitments individually or aggregate first?
-    # assert verify_aggregate_kzg_proof(blobs, expected_kzg_commitments, kzg_aggregated_proof)
+def validate_blobs(expected_kzg_commitments: Sequence[KZGCommitment],
+                   blobs: Sequence[Blob],
+                   proofs: Sequence[KZGProof]) -> None:
+    assert len(expected_kzg_commitments) == len(blobs)
+    assert len(blobs) == len(proofs)
+
+    # Clients MAY use `verify_blob_kzg_proof_multi` for efficiency
+    for commitment, blob, proof in zip(expected_kzg_commitments, blobs, proofs):
+        assert verify_blob_kzg_proof(commitment, blob, proof)
 ```
 
 #### `is_data_available`
 
 The implementation of `is_data_available` will become more sophisticated during later scaling upgrades.
-Initially, verification requires every verifying actor to retrieve all matching `BlobSidecar`s,
-and validate the sidecar with `validate_blob_sidecars`.
+Initially, verification requires every verifying actor to retrieve all matching `Blob`s and `KZGProof`s, and validate them with `validate_blobs`.
 
-The block MUST NOT be considered valid until all valid `BlobSidecar`s have been downloaded. Blocks that have been previously validated as available SHOULD be considered available even if the associated `BlobSidecar`s have subsequently been pruned.
+The block MUST NOT be considered valid until all valid `Blob`s have been downloaded. Blocks that have been previously validated as available SHOULD be considered available even if the associated `Blob`s have subsequently been pruned.
 
 ```python
-def is_data_available(slot: Slot, beacon_block_root: Root, blob_kzg_commitments: Sequence[KZGCommitment]) -> bool:
-    # `retrieve_blobs_sidecar` is implementation and context dependent, raises an exception if not available.
+def is_data_available(beacon_block_root: Root, blob_kzg_commitments: Sequence[KZGCommitment]) -> bool:
+    # `retrieve_blobs_and_proofs` is implementation and context dependent, raises an exception if not available. It returns all the blobs for the given block root.
     # Note: the p2p network does not guarantee sidecar retrieval outside of `MIN_EPOCHS_FOR_BLOBS_SIDECARS_REQUESTS`
-    sidecars = retrieve_blob_sidecars(slot, beacon_block_root)
+    blobs, proofs = retrieve_blobs_and_proofs(beacon_block_root)
 
-    # For testing, `retrieve_blobs_sidecar` returns "TEST".
+    # For testing, `retrieve_blobs_and_proofs` returns "TEST".
     # TODO: Remove it once we have a way to inject `BlobSidecar` into tests.
     if isinstance(sidecar, str):
         return True
 
-    validate_blob_sidecars(slot, beacon_block_root, blob_kzg_commitments, sidecars)
+    validate_blobs(expected_kzg_commitments, blobs, proofs)
     return True
 ```
 
@@ -89,7 +88,7 @@ def on_block(store: Store, signed_block: SignedBeaconBlock) -> None:
     # [New in Deneb]
     # Check if blob data is available
     # If not, this block MAY be queued and subsequently considered when blob data becomes available
-    assert is_data_available(block.slot, hash_tree_root(block), block.body.blob_kzg_commitments)
+    assert is_data_available(hash_tree_root(block), block.body.blob_kzg_commitments)
 
     # Check the block is valid and compute the post-state
     state = pre_state.copy()

specs/deneb/p2p-interface.md

@@ -218,34 +218,24 @@ Request Content:
 ```
 
 Response Content:
-
-```python
-class BlobSidecars(Container):
-  block_root: Root
-  List[BlobSidecar, MAX_BLOBS_PER_BLOCK]
-```
-
 ```
 (
-  List[BlobSidecars, MAX_REQUEST_BLOB_SIDECARS]
+  List[BlobSidecar, MAX_REQUEST_BLOB_SIDECARS * MAX_BLOBS_PER_BLOCK]
 )
 ```
 
-Requests blob sidecars in the slot range `[start_slot, start_slot + count)`,
-leading up to the current head block as selected by fork choice.
+Requests blob sidecars in the slot range `[start_slot, start_slot + count)`, leading up to the current head block as selected by fork choice.
 
-The response is unsigned, i.e. `BlobSidecarsByRange`, as the signature of the beacon block proposer
-may not be available beyond the initial distribution via gossip.
+The response is unsigned, i.e. `BlobSidecarsByRange`, as the signature of the beacon block proposer may not be available beyond the initial distribution via gossip.
 
-Before consuming the next response chunk, the response reader SHOULD verify the blobs sidecar is well-formatted and
-correct w.r.t. the expected KZG commitments through `validate_blobs_sidecar`.
+Before consuming the next response chunk, the response reader SHOULD verify the blobs sidecar is well-formatted and correct w.r.t. the expected KZG commitments through `validate_blobs_sidecar`.
 
-`BlobsSidecarsByRange` is primarily used to sync blobs that may have been missed on gossip and to sync within the `MIN_EPOCHS_FOR_BLOBS_SIDECARS_REQUESTS` window.
+`BlobSidecarsByRange` is primarily used to sync blobs that may have been missed on gossip and to sync within the `MIN_EPOCHS_FOR_BLOBS_SIDECARS_REQUESTS` window.
 
 The request MUST be encoded as an SSZ-container.
 
 The response MUST consist of zero or more `response_chunk`.
-Each _successful_ `response_chunk` MUST contain a single `BlobsSidecar` payload.
+Each _successful_ `response_chunk` MUST contain a single `BlobSidecar` payload.
 
 Clients MUST keep a record of signed blobs sidecars seen on the epoch range
 `[max(current_epoch - MIN_EPOCHS_FOR_BLOBS_SIDECARS_REQUESTS, DENEB_FORK_EPOCH), current_epoch]`
@@ -265,26 +255,21 @@ to be fully compliant with `BlobsSidecarsByRange` requests.
 participating in the networking immediately, other peers MAY
 disconnect and/or temporarily ban such an un-synced or semi-synced client.
 
-Clients MUST respond with at least the first blobs sidecar that exists in the range, if they have it,
-and no more than `MAX_REQUEST_BLOCKS_DENEB` sidecars.
+Clients MUST respond with at least the first blobs sidecar that exists in the range, if they have it, and no more than `MAX_REQUEST_BLOB_SIDECARS * MAX_BLOBS_PER_BLOCK` sidecars.
 
-The following blobs sidecars, where they exist, MUST be sent in consecutive order.
+The following blobs sidecars, where they exist, MUST be sent in consecutive `(slot, index)` order.
 
 Clients MAY limit the number of blobs sidecars in the response.
 
-An empty `BlobSidecar` is one that does not contain any blobs, but contains non-zero `beacon_block_root`, `beacon_block_slot` and a valid `kzg_aggregated_proof`.
-Clients MAY NOT want to consider empty `BlobSidecar`s in rate limiting logic.
+The response MUST contain no more than `count * MAX_BLOBS_PER_BLOCK` blob sidecars.
 
-The response MUST contain no more than `count` blobs sidecars.
-
-Clients MUST respond with blobs sidecars from their view of the current fork choice
--- that is, blobs sidecars as included by blocks from the single chain defined by the current head.
+Clients MUST respond with blob sidecars from their view of the current fork choice
+-- that is, blob sidecars as included by blocks from the single chain defined by the current head.
 Of note, blocks from slots before the finalization MUST lead to the finalized block reported in the `Status` handshake.
 
-Clients MUST respond with blobs sidecars that are consistent from a single chain within the context of the request.
+Clients MUST respond with blob sidecars that are consistent from a single chain within the context of the request.
 
-After the initial blobs sidecar, clients MAY stop in the process of responding
-if their fork choice changes the view of the chain in the context of the request.
+After the initial blob sidecar, clients MAY stop in the process of responding if their fork choice changes the view of the chain in the context of the request.
 
 ## Design decision rationale