mirror of
https://github.com/status-im/eth2.0-specs.git
synced 2025-01-20 15:38:55 +00:00
WIP implementation of Danksharding (#2792)
* Rough structure * Most of the KZG checks in * Fixes suggested by Vitalik * Add low degree check for commitments * Remove -1 check from degree proof (not needed) * Require block builders to be validators to simplify things * remove verify_intermediate_block_bid_commitment * Rename back to process_block * Degree check formula corrections * Updated TOC, bid processing corrections * Link to latest sharding doc * Add shard samples + P2P * Add validator guide for attestations and reconstruction * Update specs/sharding/beacon-chain.md Co-authored-by: terence tsao <terence@prysmaticlabs.com> * Update specs/sharding/beacon-chain.md Co-authored-by: vbuterin <v@buterin.com> * Update specs/sharding/beacon-chain.md Co-authored-by: vbuterin <v@buterin.com> * Refactor polynomial operations into separate file * Add missing polynomial functions * Fix polynomial commitment file toc levels * Refactor the payload to make better use of unions * Add reverse bit order convention * Correct inequality in verify_degree_proof * Small fix * Fix polynomial evaluation * Update specs/sharding/beacon-chain.md Co-authored-by: George Kadianakis <desnacked@riseup.net> * MAX_BEACON_BLOCKS_BETWEEN_INTERMEDIATE_BLOCKS definition added * Sample reconstruction estimate * Update specs/sharding/beacon-chain.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Fix return value of roots_of_unity() * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/beacon-chain.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/beacon-chain.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/beacon-chain.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Intermediate block -> Builder block * Some small omissions in intermediate -> builder * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/polynomial-commitments.md Co-authored-by: George Kadianakis <desnacked@riseup.net> * Update specs/sharding/polynomial-commitments.md Co-authored-by: George Kadianakis <desnacked@riseup.net> * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Update specs/sharding/polynomial-commitments.md Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> * Z1 -> inf_G1 * Fix degree proof bound * SignedShardSample -> ShardSample * Typo * Don't allow 0 for `next_power_of_two` * Remove unused `ROOT_OF_UNITY` constant * Throwaway variable name * Fix function documentation of `bls_modular_inverse` * Builder block bid increase by at least 1%, no RANDAO processing in builder blocks * Fix tocs * Fix tocs * Fix typo * Update specs/sharding/polynomial-commitments.md Co-authored-by: George Kadianakis <desnacked@riseup.net> Co-authored-by: Danny Ryan <dannyjryan@gmail.com> Co-authored-by: Hsiao-Wei Wang <hsiaowei.eth@gmail.com> Co-authored-by: terence tsao <terence@prysmaticlabs.com> Co-authored-by: vbuterin <v@buterin.com> Co-authored-by: George Kadianakis <desnacked@riseup.net>
This commit is contained in:
dankrad
GitHub
parent
fed037fe89
commit
02a2b71d64
File diff suppressed because it is too large
Load Diff
@ -13,17 +13,14 @@
|
||||
- [Misc](#misc)
|
||||
- [Gossip domain](#gossip-domain)
|
||||
- [Topics and messages](#topics-and-messages)
|
||||
- [Shard blob subnets](#shard-blob-subnets)
|
||||
- [`shard_blob_{subnet_id}`](#shard_blob_subnet_id)
|
||||
- [Global topics](#global-topics)
|
||||
- [`shard_blob_header`](#shard_blob_header)
|
||||
- [`shard_blob_tx`](#shard_blob_tx)
|
||||
- [`shard_proposer_slashing`](#shard_proposer_slashing)
|
||||
- [Builder block bid](#builder-block-bid)
|
||||
- [`builder_block_bid`](#builder_block_bid)
|
||||
- [Shard sample subnets](#shard-sample-subnets)
|
||||
- [`shard_row_{subnet_id}`](#shard_row_subnet_id)
|
||||
|
||||
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
|
||||
<!-- /TOC -->
|
||||
|
||||
|
||||
## Introduction
|
||||
|
||||
The specification of these changes continues in the same format as the network specifications of previous upgrades, and assumes them as pre-requisite.
|
||||
@ -33,12 +30,10 @@ The adjustments and additions for Shards are outlined in this document.
|
||||
|
||||
### Misc
|
||||
|
||||
| Name | Value | Description |
|
||||
| ---- | ----- | ----------- |
|
||||
| `SHARD_BLOB_SUBNET_COUNT` | `64` | The number of `shard_blob_{subnet_id}` subnets used in the gossipsub protocol. |
|
||||
| `SHARD_TX_PROPAGATION_GRACE_SLOTS` | `4` | The number of slots for a late transaction to propagate |
|
||||
| `SHARD_TX_PROPAGATION_BUFFER_SLOTS` | `8` | The number of slots for an early transaction to propagate |
|
||||
|
||||
| Name | Value | Description |
|
||||
| --------------------------- | ----- | -------------------------------------------------------------------------------- |
|
||||
| `SHARD_ROW_SUBNET_COUNT` | `512` | The number of `shard_row_{subnet_id}` subnets used in the gossipsub protocol. |
|
||||
| `SHARD_COLUMN_SUBNET_COUNT` | `512` | The number of `shard_column_{subnet_id}` subnets used in the gossipsub protocol. |
|
||||
|
||||
## Gossip domain
|
||||
|
||||
@ -48,130 +43,49 @@ Following the same scheme as the [Phase0 gossip topics](../phase0/p2p-interface.
|
||||
|
||||
| Name | Message Type |
|
||||
|---------------------------------|--------------------------|
|
||||
| `shard_blob_{subnet_id}` | `SignedShardBlob` |
|
||||
| `shard_blob_header` | `SignedShardBlobHeader` |
|
||||
| `shard_blob_tx` | `SignedShardBlobHeader` |
|
||||
| `shard_proposer_slashing` | `ShardProposerSlashing` |
|
||||
| `shard_row_{subnet_id}` | `SignedShardSample` |
|
||||
| `shard_column_{subnet_id}` | `SignedShardSample` |
|
||||
| `builder_block_bid` | `BuilderBlockBid` |
|
||||
|
||||
The [DAS network specification](./das-p2p.md) defines additional topics.
|
||||
|
||||
#### Shard blob subnets
|
||||
#### Builder block bid
|
||||
|
||||
Shard blob subnets are used by builders to make their blobs available after selection by shard proposers.
|
||||
##### `builder_block_bid`
|
||||
|
||||
##### `shard_blob_{subnet_id}`
|
||||
- _[IGNORE]_ The `bid` is published 1 slot early or later (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
|
||||
i.e. validate that `bid.slot <= current_slot + 1`
|
||||
(a client MAY queue future samples for propagation at the appropriate slot).
|
||||
- _[IGNORE]_ The `bid` is for the current or next block
|
||||
i.e. validate that `bid.slot >= current_slot`
|
||||
- _[IGNORE]_ The `bid` is the first `bid` valid bid for `bid.slot`, or the bid is at least 1% higher than the previous known `bid`
|
||||
- _[REJECT]_ The validator defined by `bid.validator_index` exists and is slashable.
|
||||
- _[REJECT]_ The bid signature, which is an Eth1 signature, needs to be valid and the address needs to contain enough Ether to cover the bid and the data gas base fee.
|
||||
|
||||
Shard blob data, in the form of a `SignedShardBlob` is published to the `shard_blob_{subnet_id}` subnets.
|
||||
#### Shard sample subnets
|
||||
|
||||
```python
|
||||
def compute_subnet_for_shard_blob(state: BeaconState, slot: Slot, shard: Shard) -> uint64:
|
||||
"""
|
||||
Compute the correct subnet for a shard blob publication.
|
||||
Note, this mimics compute_subnet_for_attestation().
|
||||
"""
|
||||
committee_index = compute_committee_index_from_shard(state, slot, shard)
|
||||
committees_per_slot = get_committee_count_per_slot(state, compute_epoch_at_slot(slot))
|
||||
slots_since_epoch_start = Slot(slot % SLOTS_PER_EPOCH)
|
||||
committees_since_epoch_start = committees_per_slot * slots_since_epoch_start
|
||||
Shard sample (row/column) subnets are used by builders to make their samples available as part of their intermediate block release after selection by beacon block proposers.
|
||||
|
||||
return uint64((committees_since_epoch_start + committee_index) % SHARD_BLOB_SUBNET_COUNT)
|
||||
```
|
||||
##### `shard_row_{subnet_id}`
|
||||
|
||||
The following validations MUST pass before forwarding the `signed_blob`,
|
||||
on the horizontal subnet or creating samples for it. Alias `blob = signed_blob.message`.
|
||||
Shard sample data, in the form of a `SignedShardSample` is published to the `shard_row_{subnet_id}` and `shard_column_{subnet_id}` subnets.
|
||||
|
||||
- _[IGNORE]_ The `blob` is published 1 slot early or later (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
|
||||
i.e. validate that `blob.slot <= current_slot + 1`
|
||||
(a client MAY queue future blobs for propagation at the appropriate slot).
|
||||
- _[IGNORE]_ The `blob` is new enough to still be processed --
|
||||
i.e. validate that `compute_epoch_at_slot(blob.slot) >= get_previous_epoch(state)`
|
||||
- _[REJECT]_ The shard blob is for an active shard --
|
||||
i.e. `blob.shard < get_active_shard_count(state, compute_epoch_at_slot(blob.slot))`
|
||||
- _[REJECT]_ The `blob.shard` MUST have a committee at the `blob.slot` --
|
||||
i.e. validate that `compute_committee_index_from_shard(state, blob.slot, blob.shard)` doesn't raise an error
|
||||
- _[REJECT]_ The shard blob is for the correct subnet --
|
||||
i.e. `compute_subnet_for_shard_blob(state, blob.slot, blob.shard) == subnet_id`
|
||||
- _[IGNORE]_ The blob is the first blob with valid signature received for the `(blob.proposer_index, blob.slot, blob.shard)` combination.
|
||||
- _[REJECT]_ The blob is not too large -- the data MUST NOT be larger than the SSZ list-limit, and a client MAY apply stricter bounds.
|
||||
- _[REJECT]_ The `blob.body.data` MUST NOT contain any point `p >= MODULUS`. Although it is a `uint256`, not the full 256 bit range is valid.
|
||||
- _[REJECT]_ The blob builder defined by `blob.builder_index` exists and has sufficient balance to back the fee payment.
|
||||
- _[REJECT]_ The blob signature, `signed_blob.signature`, is valid for the aggregate of proposer and builder --
|
||||
i.e. `bls.FastAggregateVerify([builder_pubkey, proposer_pubkey], blob_signing_root, signed_blob.signature)`.
|
||||
- _[REJECT]_ The blob is proposed by the expected `proposer_index` for the blob's `slot` and `shard`,
|
||||
in the context of the current shuffling (defined by the current node head state and `blob.slot`).
|
||||
If the `proposer_index` cannot immediately be verified against the expected shuffling,
|
||||
the blob MAY be queued for later processing while proposers for the blob's branch are calculated --
|
||||
in such a case _do not_ `REJECT`, instead `IGNORE` this message.
|
||||
The following validations MUST pass before forwarding the `sample`.
|
||||
|
||||
#### Global topics
|
||||
- _[IGNORE]_ The `sample` is published 1 slot early or later (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
|
||||
i.e. validate that `sample.slot <= current_slot + 1`
|
||||
(a client MAY queue future samples for propagation at the appropriate slot).
|
||||
- _[IGNORE]_ The `sample` is new enough to still be processed --
|
||||
i.e. validate that `compute_epoch_at_slot(sample.slot) >= get_previous_epoch(state)`
|
||||
- _[REJECT]_ The shard sample is for the correct subnet --
|
||||
i.e. `sample.row == subnet_id` for `shard_row_{subnet_id}` and `sample.column == subnet_id` for `shard_column_{subnet_id}`
|
||||
- _[IGNORE]_ The sample is the first sample with valid signature received for the `(sample.builder, sample.slot, sample.row, sample.column)` combination.
|
||||
- _[REJECT]_ The `sample.data` MUST NOT contain any point `x >= BLS_MODULUS`. Although it is a `uint256`, not the full 256 bit range is valid.
|
||||
- _[REJECT]_ The validator defined by `sample.builder` exists and is slashable.
|
||||
- _[REJECT]_ The sample is proposed by the expected `builder` for the sample's `slot`.
|
||||
i.e., the beacon block at `sample.slot - 1` according to the node's fork choice contains an `IntermediateBlockBid`
|
||||
with `intermediate_block_bid.validator_index == sample.builder`
|
||||
- _[REJECT]_ The sample signature, `sample.signature`, is valid for the builder --
|
||||
i.e. `bls.Verify(builder_pubkey, sample_signing_root, sample.signature)` OR `sample.signature == Bytes96(b"\0" * 96)` AND
|
||||
the sample verification `verify_sample` passes
|
||||
|
||||
There are three additional global topics for Sharding.
|
||||
|
||||
- `shard_blob_header`: co-signed headers to be included on-chain and to serve as a signal to the builder to publish full data.
|
||||
- `shard_blob_tx`: builder-signed headers, also known as "data transaction".
|
||||
- `shard_proposer_slashing`: slashings of duplicate shard proposals.
|
||||
|
||||
##### `shard_blob_header`
|
||||
|
||||
Shard header data, in the form of a `SignedShardBlobHeader` is published to the global `shard_blob_header` subnet.
|
||||
Shard blob headers select shard blob bids by builders
|
||||
and should be timely to ensure builders can publish the full shard blob before subsequent attestations.
|
||||
|
||||
The following validations MUST pass before forwarding the `signed_blob_header` on the network. Alias `header = signed_blob_header.message`.
|
||||
|
||||
- _[IGNORE]_ The `header` is published 1 slot early or later (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
|
||||
i.e. validate that `header.slot <= current_slot + 1`
|
||||
(a client MAY queue future headers for propagation at the appropriate slot).
|
||||
- _[IGNORE]_ The header is new enough to still be processed --
|
||||
i.e. validate that `compute_epoch_at_slot(header.slot) >= get_previous_epoch(state)`
|
||||
- _[REJECT]_ The shard header is for an active shard --
|
||||
i.e. `header.shard < get_active_shard_count(state, compute_epoch_at_slot(header.slot))`
|
||||
- _[REJECT]_ The `header.shard` MUST have a committee at the `header.slot` --
|
||||
i.e. validate that `compute_committee_index_from_shard(state, header.slot, header.shard)` doesn't raise an error.
|
||||
- _[IGNORE]_ The header is the first header with valid signature received for the `(header.proposer_index, header.slot, header.shard)` combination.
|
||||
- _[REJECT]_ The blob builder defined by `blob.builder_index` exists and has sufficient balance to back the fee payment.
|
||||
- _[REJECT]_ The header signature, `signed_blob_header.signature`, is valid for the aggregate of proposer and builder --
|
||||
i.e. `bls.FastAggregateVerify([builder_pubkey, proposer_pubkey], blob_signing_root, signed_blob_header.signature)`.
|
||||
- _[REJECT]_ The header is proposed by the expected `proposer_index` for the blob's `header.slot` and `header.shard`
|
||||
in the context of the current shuffling (defined by the current node head state and `header.slot`).
|
||||
If the `proposer_index` cannot immediately be verified against the expected shuffling,
|
||||
the blob MAY be queued for later processing while proposers for the blob's branch are calculated --
|
||||
in such a case _do not_ `REJECT`, instead `IGNORE` this message.
|
||||
|
||||
##### `shard_blob_tx`
|
||||
|
||||
Shard data-transactions in the form of a `SignedShardBlobHeader` are published to the global `shard_blob_tx` subnet.
|
||||
These shard blob headers are signed solely by the blob-builder.
|
||||
|
||||
The following validations MUST pass before forwarding the `signed_blob_header` on the network. Alias `header = signed_blob_header.message`.
|
||||
|
||||
- _[IGNORE]_ The header is not propagating more than `SHARD_TX_PROPAGATION_BUFFER_SLOTS` slots ahead of time --
|
||||
i.e. validate that `header.slot <= current_slot + SHARD_TX_PROPAGATION_BUFFER_SLOTS`.
|
||||
- _[IGNORE]_ The header is not propagating later than `SHARD_TX_PROPAGATION_GRACE_SLOTS` slots too late --
|
||||
i.e. validate that `header.slot + SHARD_TX_PROPAGATION_GRACE_SLOTS >= current_slot`
|
||||
- _[REJECT]_ The shard header is for an active shard --
|
||||
i.e. `header.shard < get_active_shard_count(state, compute_epoch_at_slot(header.slot))`
|
||||
- _[REJECT]_ The `header.shard` MUST have a committee at the `header.slot` --
|
||||
i.e. validate that `compute_committee_index_from_shard(state, header.slot, header.shard)` doesn't raise an error.
|
||||
- _[IGNORE]_ The header is not stale -- i.e. the corresponding shard proposer has not already selected a header for `(header.slot, header.shard)`.
|
||||
- _[IGNORE]_ The header is the first header with valid signature received for the `(header.builder_index, header.slot, header.shard)` combination.
|
||||
- _[REJECT]_ The blob builder, define by `header.builder_index`, exists and has sufficient balance to back the fee payment.
|
||||
- _[IGNORE]_ The header fee SHOULD be higher than previously seen headers for `(header.slot, header.shard)`, from any builder.
|
||||
Propagating nodes MAY increase fee increments in case of spam.
|
||||
- _[REJECT]_ The header signature, `signed_blob_header.signature`, is valid for ONLY the builder --
|
||||
i.e. `bls.Verify(builder_pubkey, blob_signing_root, signed_blob_header.signature)`. The signature is not an aggregate with the proposer.
|
||||
- _[REJECT]_ The header is designated for proposal by the expected `proposer_index` for the blob's `header.slot` and `header.shard`
|
||||
in the context of the current shuffling (defined by the current node head state and `header.slot`).
|
||||
If the `proposer_index` cannot immediately be verified against the expected shuffling,
|
||||
the blob MAY be queued for later processing while proposers for the blob's branch are calculated --
|
||||
in such a case _do not_ `REJECT`, instead `IGNORE` this message.
|
||||
|
||||
##### `shard_proposer_slashing`
|
||||
|
||||
Shard proposer slashings, in the form of `ShardProposerSlashing`, are published to the global `shard_proposer_slashing` topic.
|
||||
|
||||
The following validations MUST pass before forwarding the `shard_proposer_slashing` on to the network.
|
||||
- _[IGNORE]_ The shard proposer slashing is the first valid shard proposer slashing received
|
||||
for the proposer with index `proposer_slashing.proposer_index`.
|
||||
The `proposer_slashing.slot` and `proposer_slashing.shard` are ignored, there are no repeated or per-shard slashings.
|
||||
- _[REJECT]_ All of the conditions within `process_shard_proposer_slashing` pass validation.
|
||||
|
||||
396
specs/sharding/polynomial-commitments.md
Normal file
396
specs/sharding/polynomial-commitments.md
Normal file
@ -0,0 +1,396 @@
|
||||
# Sharding -- Polynomial Commitments
|
||||
|
||||
**Notice**: This document is a work-in-progress for researchers and implementers.
|
||||
|
||||
## Table of contents
|
||||
|
||||
<!-- TOC -->
|
||||
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
|
||||
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
|
||||
|
||||
- [Introduction](#introduction)
|
||||
- [Constants](#constants)
|
||||
- [BLS Field](#bls-field)
|
||||
- [KZG Trusted setup](#kzg-trusted-setup)
|
||||
- [Custom types](#custom-types)
|
||||
- [Helper functions](#helper-functions)
|
||||
- [`next_power_of_two`](#next_power_of_two)
|
||||
- [`reverse_bit_order`](#reverse_bit_order)
|
||||
- [`list_to_reverse_bit_order`](#list_to_reverse_bit_order)
|
||||
- [Field operations](#field-operations)
|
||||
- [Generic field operations](#generic-field-operations)
|
||||
- [`bls_modular_inverse`](#bls_modular_inverse)
|
||||
- [`roots_of_unity`](#roots_of_unity)
|
||||
- [Field helper functions](#field-helper-functions)
|
||||
- [`compute_powers`](#compute_powers)
|
||||
- [`low_degree_check`](#low_degree_check)
|
||||
- [`vector_lincomb`](#vector_lincomb)
|
||||
- [`bytes_to_field_elements`](#bytes_to_field_elements)
|
||||
- [Polynomial operations](#polynomial-operations)
|
||||
- [`add_polynomials`](#add_polynomials)
|
||||
- [`multiply_polynomials`](#multiply_polynomials)
|
||||
- [`interpolate_polynomial`](#interpolate_polynomial)
|
||||
- [`evaluate_polynomial_in_evaluation_form`](#evaluate_polynomial_in_evaluation_form)
|
||||
- [KZG Operations](#kzg-operations)
|
||||
- [Elliptic curve helper functoins](#elliptic-curve-helper-functoins)
|
||||
- [`elliptic_curve_lincomb`](#elliptic_curve_lincomb)
|
||||
- [Hash to field](#hash-to-field)
|
||||
- [`hash_to_bls_field`](#hash_to_bls_field)
|
||||
- [KZG operations](#kzg-operations)
|
||||
- [`verify_kzg_proof`](#verify_kzg_proof)
|
||||
- [`verify_kzg_multiproof`](#verify_kzg_multiproof)
|
||||
- [`verify_degree_proof`](#verify_degree_proof)
|
||||
|
||||
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
|
||||
<!-- /TOC -->
|
||||
|
||||
|
||||
## Introduction
|
||||
|
||||
This document specifies basic polynomial operations and KZG polynomial commitment operations as they are needed for the sharding specification. The implementations are not optimized for performance, but readability. All practical implementations should optimize the polynomial operations, and hints what the best known algorithms for these implementations are are included below.
|
||||
|
||||
## Constants
|
||||
|
||||
### BLS Field
|
||||
|
||||
| Name | Value | Notes |
|
||||
| - | - | - |
|
||||
| `BLS_MODULUS` | `0x73eda753299d7d483339d80809a1d80553bda402fffe5bfeffffffff00000001` (curve order of BLS12_381) |
|
||||
| `PRIMITIVE_ROOT_OF_UNITY` | `7` | Primitive root of unity of the BLS12_381 (inner) BLS_MODULUS |
|
||||
|
||||
### KZG Trusted setup
|
||||
|
||||
| Name | Value |
|
||||
| - | - |
|
||||
| `G1_SETUP` | Type `List[G1]`. The G1-side trusted setup `[G, G*s, G*s**2....]`; note that the first point is the generator. |
|
||||
| `G2_SETUP` | Type `List[G2]`. The G2-side trusted setup `[G, G*s, G*s**2....]` |
|
||||
|
||||
## Custom types
|
||||
|
||||
We define the following Python custom types for type hinting and readability:
|
||||
|
||||
| Name | SSZ equivalent | Description |
|
||||
| - | - | - |
|
||||
| `KZGCommitment` | `Bytes48` | A G1 curve point |
|
||||
| `BLSFieldElement` | `uint256` | A number `x` in the range `0 <= x < BLS_MODULUS` |
|
||||
| `BLSPolynomialByCoefficients` | `List[BLSFieldElement]` | A polynomial over the BLS field, given in coefficient form |
|
||||
| `BLSPolynomialByEvaluations` | `List[BLSFieldElement]` | A polynomial over the BLS field, given in evaluation form |
|
||||
|
||||
## Helper functions
|
||||
|
||||
#### `next_power_of_two`
|
||||
|
||||
```python
|
||||
def next_power_of_two(x: int) -> int:
|
||||
assert x > 0
|
||||
return 2 ** ((x - 1).bit_length())
|
||||
```
|
||||
|
||||
#### `reverse_bit_order`
|
||||
|
||||
```python
|
||||
def reverse_bit_order(n: int, order: int) -> int:
|
||||
"""
|
||||
Reverse the bit order of an integer n
|
||||
"""
|
||||
assert is_power_of_two(order)
|
||||
# Convert n to binary with the same number of bits as "order" - 1, then reverse its bit order
|
||||
return int(('{:0' + str(order.bit_length() - 1) + 'b}').format(n)[::-1], 2)
|
||||
```
|
||||
|
||||
#### `list_to_reverse_bit_order`
|
||||
|
||||
```python
|
||||
def list_to_reverse_bit_order(l: List[int]) -> List[int]:
|
||||
"""
|
||||
Convert a list between normal and reverse bit order. This operation is idempotent.
|
||||
"""
|
||||
return [l[reverse_bit_order(i, len(l))] for i in range(len(l))]
|
||||
```
|
||||
|
||||
## Field operations
|
||||
|
||||
### Generic field operations
|
||||
|
||||
#### `bls_modular_inverse`
|
||||
|
||||
```python
|
||||
def bls_modular_inverse(x: BLSFieldElement) -> BLSFieldElement:
|
||||
"""
|
||||
Compute the modular inverse of x, i.e. y such that x * y % BLS_MODULUS == 1 and return 1 for x == 0
|
||||
"""
|
||||
lm, hm = 1, 0
|
||||
low, high = x % BLS_MODULUS, BLS_MODULUS
|
||||
while low > 1:
|
||||
r = high // low
|
||||
nm, new = hm - lm * r, high - low * r
|
||||
lm, low, hm, high = nm, new, lm, low
|
||||
return lm % BLS_MODULUS
|
||||
```
|
||||
|
||||
#### `roots_of_unity`
|
||||
|
||||
```python
|
||||
def roots_of_unity(order: uint64) -> List[BLSFieldElement]:
|
||||
"""
|
||||
Compute a list of roots of unity for a given order.
|
||||
The order must divide the BLS multiplicative group order, i.e. BLS_MODULUS - 1
|
||||
"""
|
||||
assert (BLS_MODULUS - 1) % order == 0
|
||||
roots = []
|
||||
root_of_unity = pow(PRIMITIVE_ROOT_OF_UNITY, (BLS_MODULUS - 1) // order, BLS_MODULUS)
|
||||
|
||||
current_root_of_unity = 1
|
||||
for i in range(SAMPLES_PER_BLOB * FIELD_ELEMENTS_PER_SAMPLE):
|
||||
roots.append(current_root_of_unity)
|
||||
current_root_of_unity = current_root_of_unity * root_of_unity % BLS_MODULUS
|
||||
return roots
|
||||
```
|
||||
|
||||
### Field helper functions
|
||||
|
||||
#### `compute_powers`
|
||||
|
||||
```python
|
||||
def compute_powers(x: BLSFieldElement, n: uint64) -> List[BLSFieldElement]:
|
||||
current_power = 1
|
||||
powers = []
|
||||
for _ in range(n):
|
||||
powers.append(BLSFieldElement(current_power))
|
||||
current_power = current_power * int(x) % BLS_MODULUS
|
||||
return powers
|
||||
```
|
||||
|
||||
#### `low_degree_check`
|
||||
|
||||
```python
|
||||
def low_degree_check(commitments: List[KZGCommitment]):
|
||||
"""
|
||||
Checks that the commitments are on a low-degree polynomial.
|
||||
If there are 2*N commitments, that means they should lie on a polynomial
|
||||
of degree d = K - N - 1, where K = next_power_of_two(2*N)
|
||||
(The remaining positions are filled with 0, this is to make FFTs usable)
|
||||
|
||||
For details see here: https://notes.ethereum.org/@dankrad/barycentric_low_degree_check
|
||||
"""
|
||||
assert len(commitments) % 2 == 0
|
||||
N = len(commitments) // 2
|
||||
r = hash_to_bls_field(commitments, 0)
|
||||
K = next_power_of_two(2 * N)
|
||||
d = K - N - 1
|
||||
r_to_K = pow(r, N, K)
|
||||
roots = list_to_reverse_bit_order(roots_of_unity(K))
|
||||
|
||||
# For an efficient implementation, B and Bprime should be precomputed
|
||||
def B(z):
|
||||
r = 1
|
||||
for w in roots[:d + 1]:
|
||||
r = r * (z - w) % BLS_MODULUS
|
||||
return r
|
||||
|
||||
def Bprime(z):
|
||||
r = 0
|
||||
for i in range(d + 1):
|
||||
m = 1
|
||||
for w in roots[:i] + roots[i + 1:d + 1]:
|
||||
m = m * (z - w) % BLS_MODULUS
|
||||
r = (r + m) % BLS_MODULUS
|
||||
return r
|
||||
|
||||
coefs = []
|
||||
for i in range(K):
|
||||
coefs.append( - (r_to_K - 1) * bls_modular_inverse(K * roots[i * (K - 1) % K] * (r - roots[i])) % BLS_MODULUS)
|
||||
for i in range(d + 1):
|
||||
coefs[i] = (coefs[i] + B(r) * bls_modular_inverse(Bprime(r) * (r - roots[i]))) % BLS_MODULUS
|
||||
|
||||
assert elliptic_curve_lincomb(commitments, coefs) == bls.inf_G1()
|
||||
```
|
||||
|
||||
#### `vector_lincomb`
|
||||
|
||||
```python
|
||||
def vector_lincomb(vectors: List[List[BLSFieldElement]], scalars: List[BLSFieldElement]) -> List[BLSFieldElement]:
|
||||
"""
|
||||
Compute a linear combination of field element vectors.
|
||||
"""
|
||||
r = [0]*len(vectors[0])
|
||||
for v, a in zip(vectors, scalars):
|
||||
for i, x in enumerate(v):
|
||||
r[i] = (r[i] + a * x) % BLS_MODULUS
|
||||
return [BLSFieldElement(x) for x in r]
|
||||
```
|
||||
|
||||
#### `bytes_to_field_elements`
|
||||
|
||||
```python
|
||||
def bytes_to_field_elements(block: bytes) -> List[BLSFieldElement]:
|
||||
"""
|
||||
Slices a block into 31-byte chunks that can fit into field elements.
|
||||
"""
|
||||
sliced_block = [block[i:i + 31] for i in range(0, len(bytes), 31)]
|
||||
return [BLSFieldElement(int.from_bytes(x, "little")) for x in sliced_block]
|
||||
```
|
||||
|
||||
## Polynomial operations
|
||||
|
||||
#### `add_polynomials`
|
||||
|
||||
```python
|
||||
def add_polynomials(a: BLSPolynomialByCoefficients, b: BLSPolynomialByCoefficients) -> BLSPolynomialByCoefficients:
|
||||
"""
|
||||
Sum the polynomials ``a`` and ``b`` given by their coefficients.
|
||||
"""
|
||||
a, b = (a, b) if len(a) >= len(b) else (b, a)
|
||||
return [(a[i] + (b[i] if i < len(b) else 0)) % BLS_MODULUS for i in range(len(a))]
|
||||
```
|
||||
|
||||
#### `multiply_polynomials`
|
||||
|
||||
```python
|
||||
def multiply_polynomials(a: BLSPolynomialByCoefficients, b: BLSPolynomialByCoefficients) -> BLSPolynomialByCoefficients:
|
||||
"""
|
||||
Multiplies the polynomials `a` and `b` given by their coefficients
|
||||
"""
|
||||
r = [0]
|
||||
for power, coef in enumerate(a):
|
||||
summand = [0] * power + [coef * x % BLS_MODULUS for x in b]
|
||||
r = add_polynomials(r, summand)
|
||||
return r
|
||||
```
|
||||
|
||||
|
||||
#### `interpolate_polynomial`
|
||||
|
||||
```python
|
||||
def interpolate_polynomial(xs: List[BLSFieldElement], ys: List[BLSFieldElement]) -> BLSPolynomialByCoefficients:
|
||||
"""
|
||||
Lagrange interpolation
|
||||
"""
|
||||
assert len(xs) == len(ys)
|
||||
r = [0]
|
||||
|
||||
for i in range(len(xs)):
|
||||
summand = [ys[i]]
|
||||
for j in range(len(ys)):
|
||||
if j != i:
|
||||
weight_adjustment = bls_modular_inverse(xs[j] - xs[i])
|
||||
summand = multiply_polynomials(
|
||||
summand, [weight_adjustment, ((BLS_MODULUS - weight_adjustment) * xs[i])]
|
||||
)
|
||||
r = add_polynomials(r, summand)
|
||||
|
||||
return r
|
||||
```
|
||||
|
||||
#### `evaluate_polynomial_in_evaluation_form`
|
||||
|
||||
```python
|
||||
def evaluate_polynomial_in_evaluation_form(poly: BLSPolynomialByEvaluations, x: BLSFieldElement) -> BLSFieldElement:
|
||||
"""
|
||||
Evaluates a polynomial (in evaluation form) at an arbitrary point
|
||||
"""
|
||||
field_elements_per_blob = SAMPLES_PER_BLOB * FIELD_ELEMENTS_PER_SAMPLE
|
||||
roots = roots_of_unity(field_elements_per_blob)
|
||||
|
||||
def A(z):
|
||||
r = 1
|
||||
for w in roots:
|
||||
r = r * (z - w) % BLS_MODULUS
|
||||
return r
|
||||
|
||||
def Aprime(z):
|
||||
return field_elements_per_blob * pow(z, field_elements_per_blob - 1, BLS_MODULUS)
|
||||
|
||||
r = 0
|
||||
inverses = [bls_modular_inverse(z - x) for z in roots]
|
||||
for i, x in enumerate(inverses):
|
||||
r += poly[i] * bls_modular_inverse(Aprime(roots[i])) * x % BLS_MODULUS
|
||||
r = r * A(x) % BLS_MODULUS
|
||||
return r
|
||||
```
|
||||
|
||||
## KZG Operations
|
||||
|
||||
We are using the KZG10 polynomial commitment scheme (Kate, Zaverucha and Goldberg, 2010: https://www.iacr.org/archive/asiacrypt2010/6477178/6477178.pdf).
|
||||
|
||||
### Elliptic curve helper functoins
|
||||
|
||||
#### `elliptic_curve_lincomb`
|
||||
|
||||
```python
|
||||
def elliptic_curve_lincomb(points: List[KZGCommitment], scalars: List[BLSFieldElement]) -> KZGCommitment:
|
||||
"""
|
||||
BLS multiscalar multiplication. This function can be optimized using Pippenger's algorithm and variants.
|
||||
This is a non-optimized implementation.
|
||||
"""
|
||||
r = bls.inf_G1()
|
||||
for x, a in zip(points, scalars):
|
||||
r = r.add(x.mult(a))
|
||||
return r
|
||||
```
|
||||
|
||||
### Hash to field
|
||||
|
||||
#### `hash_to_bls_field`
|
||||
|
||||
```python
|
||||
def hash_to_bls_field(x: Container, challenge_number: uint64) -> BLSFieldElement:
|
||||
"""
|
||||
This function is used to generate Fiat-Shamir challenges. The output is not uniform over the BLS field.
|
||||
"""
|
||||
return (
|
||||
(int.from_bytes(hash(hash_tree_root(x) + int.to_bytes(challenge_number, 32, "little")), "little"))
|
||||
% BLS_MODULUS
|
||||
)
|
||||
```
|
||||
|
||||
### KZG operations
|
||||
|
||||
#### `verify_kzg_proof`
|
||||
|
||||
```python
|
||||
def verify_kzg_proof(commitment: KZGCommitment, x: BLSFieldElement, y: BLSFieldElement, proof: KZGCommitment) -> None:
|
||||
"""
|
||||
Check that `proof` is a valid KZG proof for the polynomial committed to by `commitment` evaluated
|
||||
at `x` equals `y`.
|
||||
"""
|
||||
zero_poly = G2_SETUP[1].add(G2_SETUP[0].mult(x).neg())
|
||||
|
||||
assert (
|
||||
bls.Pairing(proof, zero_poly)
|
||||
== bls.Pairing(commitment.add(G1_SETUP[0].mult(y).neg), G2_SETUP[0])
|
||||
)
|
||||
```
|
||||
|
||||
#### `verify_kzg_multiproof`
|
||||
|
||||
```python
|
||||
def verify_kzg_multiproof(commitment: KZGCommitment,
|
||||
xs: List[BLSFieldElement],
|
||||
ys: List[BLSFieldElement],
|
||||
proof: KZGCommitment) -> None:
|
||||
"""
|
||||
Verify a KZG multiproof.
|
||||
"""
|
||||
zero_poly = elliptic_curve_lincomb(G2_SETUP[:len(xs)], interpolate_polynomial(xs, [0] * len(ys)))
|
||||
interpolated_poly = elliptic_curve_lincomb(G2_SETUP[:len(xs)], interpolate_polynomial(xs, ys))
|
||||
|
||||
assert (
|
||||
bls.Pairing(proof, zero_poly)
|
||||
== bls.Pairing(commitment.add(interpolated_poly.neg()), G2_SETUP[0])
|
||||
)
|
||||
```
|
||||
|
||||
#### `verify_degree_proof`
|
||||
|
||||
```python
|
||||
def verify_degree_proof(commitment: KZGCommitment, degree_bound: uint64, proof: KZGCommitment):
|
||||
"""
|
||||
Verifies that the commitment is of polynomial degree < degree_bound.
|
||||
"""
|
||||
|
||||
assert (
|
||||
bls.Pairing(proof, G2_SETUP[0])
|
||||
== bls.Pairing(commitment, G2_SETUP[-degree_bound])
|
||||
)
|
||||
```
|
||||
141
specs/sharding/validator.md
Normal file
141
specs/sharding/validator.md
Normal file
@ -0,0 +1,141 @@
|
||||
# Sharding -- Honest Validator
|
||||
|
||||
**Notice**: This document is a work-in-progress for researchers and implementers.
|
||||
|
||||
## Table of contents
|
||||
|
||||
<!-- TOC -->
|
||||
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
|
||||
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
|
||||
|
||||
- [Introduction](#introduction)
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Constants](#constants)
|
||||
- [Sample counts](#sample-counts)
|
||||
- [Helpers](#helpers)
|
||||
- [`get_validator_row_subnets`](#get_validator_row_subnets)
|
||||
- [`get_validator_column_subnets`](#get_validator_column_subnets)
|
||||
- [`reconstruct_polynomial`](#reconstruct_polynomial)
|
||||
- [Sample verification](#sample-verification)
|
||||
- [`verify_sample`](#verify_sample)
|
||||
- [Beacon chain responsibilities](#beacon-chain-responsibilities)
|
||||
- [Validator assignments](#validator-assignments)
|
||||
- [Attesting](#attesting)
|
||||
- [Sample reconstruction](#sample-reconstruction)
|
||||
- [Minimum online validator requirement](#minimum-online-validator-requirement)
|
||||
|
||||
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
|
||||
<!-- /TOC -->
|
||||
|
||||
## Introduction
|
||||
|
||||
This document represents the changes to be made in the code of an "honest validator" to implement executable beacon chain proposal.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
This document is an extension of the [Bellatrix -- Honest Validator](../bellatrix/validator.md) guide.
|
||||
All behaviors and definitions defined in this document, and documents it extends, carry over unless explicitly noted or overridden.
|
||||
|
||||
All terminology, constants, functions, and protocol mechanics defined in the updated Beacon Chain doc of [Sharding](./beacon-chain.md) are requisite for this document and used throughout.
|
||||
Please see related Beacon Chain doc before continuing and use them as a reference throughout.
|
||||
|
||||
## Constants
|
||||
|
||||
### Sample counts
|
||||
|
||||
| Name | Value |
|
||||
| - | - |
|
||||
| `VALIDATOR_SAMPLE_ROW_COUNT` | `2` |
|
||||
| `VALIDATOR_SAMPLE_COLUMN_COUNT` | `2` |
|
||||
|
||||
## Helpers
|
||||
|
||||
### `get_validator_row_subnets`
|
||||
|
||||
TODO: Currently the subnets are public (i.e. anyone can derive them.) This is good for a proof of custody with public verifiability, but bad for validator privacy.
|
||||
|
||||
```python
|
||||
def get_validator_row_subnets(validator: Validator, epoch: Epoch) -> List[uint64]:
|
||||
return [int.from_bytes(hash_tree_root([validator.pubkey, 0, i])) for i in range(VALIDATOR_SAMPLE_ROW_COUNT)]
|
||||
```
|
||||
|
||||
### `get_validator_column_subnets`
|
||||
|
||||
```python
|
||||
def get_validator_column_subnets(validator: Validator, epoch: Epoch) -> List[uint64]:
|
||||
return [int.from_bytes(hash_tree_root([validator.pubkey, 1, i])) for i in range(VALIDATOR_SAMPLE_COLUMN_COUNT)]
|
||||
```
|
||||
|
||||
### `reconstruct_polynomial`
|
||||
|
||||
```python
|
||||
def reconstruct_polynomial(samples: List[SignedShardSample]) -> List[SignedShardSample]:
|
||||
"""
|
||||
Reconstructs one full row/column from at least 1/2 of the samples
|
||||
"""
|
||||
|
||||
```
|
||||
|
||||
## Sample verification
|
||||
|
||||
### `verify_sample`
|
||||
|
||||
```python
|
||||
def verify_sample(state: BeaconState, block: BeaconBlock, sample: SignedShardSample):
|
||||
assert sample.row < 2 * get_active_shard_count(state, get_current_epoch(block.slot))
|
||||
assert sample.column < 2 * SAMPLES_PER_BLOB
|
||||
assert block.slot == sample.slot
|
||||
|
||||
# Verify builder signature.
|
||||
# TODO: We should probably not do this. This should only be done by p2p to verify samples *before* intermediate block is in
|
||||
# builder = state.validators[signed_block.message.proposer_index]
|
||||
# signing_root = compute_signing_root(sample, get_domain(state, DOMAIN_SHARD_SAMPLE))
|
||||
# assert bls.Verify(sample.builder, signing_root, sample.signature)
|
||||
|
||||
roots_in_rbo = list_to_reverse_bit_order(roots_of_unity(SAMPLES_PER_BLOB * FIELD_ELEMENTS_PER_SAMPLE))
|
||||
|
||||
# Verify KZG proof
|
||||
verify_kzg_multiproof(block.body.payload_data.value.sharded_commitments_container.sharded_commitments[sample.row],
|
||||
roots_in_rbo[sample.column * FIELD_ELEMENTS_PER_SAMPLE:(sample.column + 1) * FIELD_ELEMENTS_PER_SAMPLE]
|
||||
sample.data,
|
||||
sample.proof)
|
||||
```
|
||||
|
||||
# Beacon chain responsibilities
|
||||
|
||||
## Validator assignments
|
||||
|
||||
### Attesting
|
||||
|
||||
Every attester is assigned `VALIDATOR_SAMPLE_ROW_COUNT` rows and `VALIDATOR_SAMPLE_COLUMN_COUNT` columns of shard samples. As part of their validator duties, they should subscribe to the subnets given by `get_validator_row_subnets` and `get_validator_column_subnets`, for the whole epoch.
|
||||
|
||||
A row or column is *available* for a `slot` if at least half of the total number of samples were received on the subnet and passed `verify_sample`. Otherwise it is called unavailable.
|
||||
|
||||
If a validator is assigned to an attestation at slot `attestation_slot` and had his previous attestation duty at `previous_attestation_slot`, then they should only attest under the following conditions:
|
||||
|
||||
* For all intermediate blocks `block` with `previous_attestation_slot < block.slot <= attestation_slot`: All sample rows and columns assigned to the validator were available.
|
||||
|
||||
If this condition is not fulfilled, then the validator should instead attest to the last block for which the condition holds.
|
||||
|
||||
This leads to the security property that a chain that is not fully available cannot have more than 1/16th of all validators voting for it. TODO: This claim is for an "infinite number" of validators. Compute the concrete security due to sampling bias.
|
||||
|
||||
# Sample reconstruction
|
||||
|
||||
A validator that has received enough samples of a row or column to mark it as available, should reconstruct all samples in that row/column (if they aren't all available already.) The function `reconstruct_polynomial` gives an example implementation for this.
|
||||
|
||||
Once they have run the reconstruction function, they should distribute the samples that they reconstructed on all pubsub that
|
||||
the local node is subscribed to, if they have not already received that sample on that pubsub. As an example:
|
||||
|
||||
* The validator is subscribed to row `2` and column `5`
|
||||
* The sample `(row, column) = (2, 5)` is missing in the column `5` pubsub
|
||||
* After they have reconstruction of row `2`, the validator should send the sample `(2, 5)` on to the row `2` pubsub (if it was missing) as well as the column `5` pubsub.
|
||||
|
||||
TODO: We need to verify the total complexity of doing this and make sure this does not cause too much load on a validator
|
||||
|
||||
## Minimum online validator requirement
|
||||
|
||||
The data availability construction guarantees that reconstruction is possible if 75% of all samples are available. In this case, at least 50% of all rows and 50% of all columns are independently available. In practice, it is likely that some supernodes will centrally collect all samples and fill in any gaps. However, we want to build a system that reliably reconstructs even absent all supernodes. Any row or column with 50% of samples will easily be reconstructed even with only 100s of validators online; so the only question is how we get to 50% of samples for all rows and columns, when some of them might be completely unseeded.
|
||||
|
||||
Each validator will transfer 4 samples between rows and columns where there is overlap. Without loss of generality, look at row 0. Each validator has 1/128 chance of having a sample in this row, and we need 256 samples to reconstruct it. So we expect that we need ~256 * 128 = 32,768 validators to have a fair chance of reconstructing it if it was completely unseeded.
|
||||
|
||||
A more elaborate estimate [here](https://notes.ethereum.org/@dankrad/minimum-reconstruction-validators) needs about 55,000 validators to be online for high safety that each row and column will be reconstructed.
|
||||
Loading…
x
Reference in New Issue
Block a user