This PR reintroduces and further decouples blocks and blobs in EIP-4844,
so as to improve network and processing performance.
Block and blob processing, for the purpose of gossip validation, are
independent: they can both be propagated and gossip-validated
in parallel - the decoupled design allows 4 important optimizations
(or, if you are so inclined, removes 4 unnecessary pessimizations):
* Blocks and blobs travel on independent meshes allowing for better
parallelization and utilization of high-bandwidth peers
* Re-broadcasting after validation can start earlier allowing more
efficient use of upload bandwidth - blocks for example can be
rebroadcast to peers while blobs are still being downloaded
* bandwidth-reduction techniques such as per-peer deduplication are more
efficient because of the smaller message size
* gossip verification happens independently for blocks and blobs,
allowing better sharing / use of CPU and I/O resources in clients
With growing block sizes and additional blob data to stream, the network
streaming time becomes a dominant factor in propagation times - on a
100mbit line, streaming 1mb to 8 peers takes ~1s - this process is
repeated for each hop in both incoming and outgoing directions.
This design in particular sends each blob on a separate subnet, thus
maximising the potential for parallelisation and providing a natural
path for growing the number of blobs per block should the network be
judged to be able to handle it.
Changes compared to the current design include:
* `BlobsSidecar` is split into individual `BlobSidecar` containers -
each container is signed individually by the proposer
* the signature is used during gossip validation but later dropped.
* KZG commitment verification is moved out of the gossip pipeline and
instead done before fork choice addition, when both block and sidecars
have arrived
* clients may verify individual blob commitments earlier
* more generally and similar to block verification, gossip propagation
is performed solely based on trivial consistency checks and proposer
signature verification
* by-root blob requests are done per-blob, so as to retain the ability
to fill in blobs one-by-one assuming clients generally receive blobs
from gossip
* by-range blob requests are done per-block, so as to simplify
historical sync
* range and root requests are limited to `128` entries for both blocks
and blobs - practically, the current higher limit of `1024` for blocks
does not get used and keeping the limits consistent simplifies
implementation - with the merge, block sizes have grown significantly
and clients generally fetch smaller chunks.
In Altair, light client sync protocol exchanges `BeaconBlockHeader`
structures for tracking current progress. Wrapping `BeaconBlockHeader`
inside a `LightClientHeader` allows future extensions of this header,
e.g., to also track `ExecutionPayloadHeader`.
Note: This changes the JSON REST format by adding a `beacon` nesting.
For SSZ, the serialization format stays same (but overall root changes).
* Add @description decorator
* Unify test case naming style
* more clean ups
* Altair tests cleanup
* Clean up Altair and Bellatrix `process_deposit` tests
* Clean up Bellatrix tests
* Clean up Capella tests
* PR feedback from @ralexstokes
* Add comments on the deposit fork version tests
* Remove `test_incorrect_sig_other_version` since it is duplicate to `test_ineffective_deposit_with_bad_fork_version`
* Add `test_ineffective_deposit_with_current_fork_version`
While the light client sync protocol currently provides access to the
latest `BeaconBlockHeader`, obtaining the matching execution data needs
workarounds such as downloading the full block.
Having ready access to the EL state root simplifies use cases that need
a way to cross-check `eth_getProof` responses against LC data.
Access to `block_hash` unlocks scenarios where a CL light client drives
an EL without `engine_newPayload`. As of Altair, only the CL beacon
block root is available, but the EL block hash is needed for engine API.
Other fields in the `ExecutionPayloadHeader` such as `logs_bloom` may
allow light client applications to monitor blocks for local interest,
e.g. for transfers affecting a certain wallet. This enables to download
only the few relevant blocks instead of every single one.
A new `LightClientStore` is proposed into the Capella spec that may be
used to sync LC data that includes execution data. Existing pre-Capella
LC data will remain as is, but can be locally upgraded before feeding it
into the new `LightClientStore` so that light clients do not need to run
a potentially expensive fork transition at a specific time. This enables
the `LightClientStore` to be upgraded at a use case dependent timing at
any time before Capella hits. Smart contract and embedded deployments
benefit from reduced code size and do not need synchronization with the
beacon chain clock to perform the Capella fork.
Introduce `get_lc_beacon_slot` and `get_lc_beacon_root` accessors
similar to `get_current_slot(state)` to account for future extensions
to the light client header structure that may override how those fields
are accessed. Idea is to extend with execution accessors in the future.
Introduce `block_to_light_client_header` helper function to enable
future forks to override it with additional info (e.g., execution),
without having to change the general light client logic.
Likewise, update existing light client data creation flow to use
`block_to_light_client_header` and default-initialize empty fields.
Furthermore, generalize `create_update` helper to streamline test code
using `block_to_light_client_header`.
Note: In Altair spec, LC header is the same as `BeaconBlockHeader`.
however; future forks will extend it with more information.
* EIP4844: bytes_to_bls_field() must not accept values >= BLS_MODULUS
bytes_to_bls_field() will be used in the precompile and hence it should error out when provided with malicious inputs.
* EIP4844: Add hash_to_bls_field() for use in compute_challenges()
The previous commit made bytes_to_bls_field() be strict about its inputs. However in compute_challenges() we are
dealing with Fiat-Shamir and hash outputs that could be innocuously higher than the modulus. For this reason we add the
hash_to_bls_field() helper for use in compute_challenges().
* EIP4844: Further use of bytes_to_bls_field() // Fix executable spec
Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com>
Additionally, it makes the Fiat-Shamir hashing logic more robust by making the challenges independent of each other. It also makes it more efficient to implement by moving both challenge computations to a single function needing a single transcript hash.
Co-authored-by: George Kadianakis <desnacked@riseup.net>
Co-authored-by: Dankrad Feist <mail@dankradfeist.de>
- Implemented many of Alex's comments including reinsertion of the
withdrawal index in the BeaconState
- Implemented Sean's suggestion of separating the logic for block
production so that one matches the list in the payload with what
`get_expected_withdrawals` returns
- Changed `get_expected_wihdrawals` to match the current behavior and
moved it to `beacon-chain.md`
This commit changes the public API of the KZG library to the following high-level API:
```
- verify_kzg_proof()
- compute_aggregate_kzg_proof()
- verify_aggregate_kzg_proof()
- blob_to_kzg_commitment()
```
compared to the previous much more low-level API:
```
- compute_powers()
- matrix_lincomb()
- lincomb()
- bytes_to_bls_field()
- evaluate_polynomial_in_evaluation_form()
- verify_kzg_proof()
- compute_kzg_proof()
```
This means that all the cryptographic logic (including Fiat-Shamir) is now isolated and hidden in the KZG library and the `validator.md` file ends up being significantly simplified, only calling high-level KZG functions.
Some additional things that this commit does:
- Moves all EIP4844 cryptography into polynomial-commitments.md
- Improves the Fiat-Shamir stack by removing the need for SSZ and by introducing simple domain separators
Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com>
Co-authored-by: Dankrad Feist <mail@dankradfeist.de>
Future light client protocol extensions may include data from the block
in addition to data from the state, e.g., `ExecutionPayloadHeader`.
To prepare for this, also pass the block to the corresponding functions.
In practice, blocks access is easier than historic state access, meaning
there are no practical limitations due to this change.
Add more detailed LC object documentation to explain that the various
merkle proofs are relative to the beacon block's state root.
Likewise, clarify that sync committees relate to the finalized header
(not to the optimistic header, which can be a period ahead).
For LC gossip, the documentation did not specify what slot number to use
for deriving the gossip objects. This missing documentation is now added
to document using `attested_header.slot`.
This PR, a continuation of
replaces `historical_roots` with
`historical_block_roots`.
By keeping an accumulator of historical block roots in the state, it
becomes possible to validate the entire block history that led up to
that particular state without executing the transitions, and without
checking them one by one in backwards order using a parent chain.
This is interesting for archival purposes as well as when implementing
sync protocols that can verify chunks of blocks quickly, meaning they
can be downloaded in any order.
It's also useful as it provides a canonical hash by which such chunks of
blocks can be named, with a direct reference in the state.
In this PR, `historical_roots` is frozen at its current value and
`historical_batches` are computed from the merge epoch onwards.
After this PR, `block_batch_root` in the state can be used to verify an
era of blocks against the state with a simple root check.
The `historical_roots` values on the other hand can be used to verify
that a constant distributed with clients is valid for a particular
state, and therefore extends the block validation all the way back to
genesis without backfilling `block_batch_root` and without introducing
any new security assumptions in the client.
As far as naming goes, it's convenient to talk about an "era" being 8192
slots ~= 1.14 days. The 8192 number comes from the
SLOTS_PER_HISTORICAL_ROOT constant.
With multiple easily verifable blocks in a file, it becomes trivial to
offload block history to out-of-protocol transfer methods (bittorrent /
ftp / whatever) - including execution payloads, paving the way for a
future in which clients purge block history in p2p.
This PR can be applied along with the merge which simplifies payload
distribution from the get-go. Both execution and consensus clients
benefit because from the merge onwards, they both need to be able to
supply ranges of blocks in the sync protocol from what effectively is
"cold storage".
Another possibility is to include it in a future cleanup PR - this
complicates the "cold storage" mode above by not covering exection
payloads from start.
Jacek Sieka
Dankrad Feist
Potuz
Paul Harris
Jacek Sieka
Age Manning
Hsiao-Wei Wang
protolambda
George Kadianakis
kevaundray
Ben Edgington
Danny Ryan
Pawan Dhananjay
Emilia Hane
Etan Kissling
terence tsao
GeemoCandama
henridf
Mikhail Kalinin
dapplion
Justin Traglia
Stefan Bratanov
Alex Stokes
Tim Beiko
Roberto Bayardo
Dmitrii Shmatko
Ramana Kumar
Enrico Del Fante
realbigsean
Enrico Del Fante
inphi
Michael Sproul
Michael Sproul
ethDreamer
Mark Mackey