From 96352726bbe3267a0287d6ad187d44fd70a6bf7f Mon Sep 17 00:00:00 2001 From: terence tsao Date: Sat, 22 May 2021 10:31:10 -0700 Subject: [PATCH 01/22] Sharding p2p: minor typo fixes --- specs/sharding/p2p-interface.md | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/specs/sharding/p2p-interface.md b/specs/sharding/p2p-interface.md index 47ed52970..1fef7c60f 100644 --- a/specs/sharding/p2p-interface.md +++ b/specs/sharding/p2p-interface.md @@ -17,9 +17,11 @@ - [SignedShardBlob](#signedshardblob) - [Gossip domain](#gossip-domain) - [Topics and messages](#topics-and-messages) - - [Shard blobs: `shard_blob_{subnet_id}`](#shard-blobs-shard_blob_subnet_id) - - [Shard header: `shard_header`](#shard-header-shard_header) - - [Shard proposer slashing: `shard_proposer_slashing`](#shard-proposer-slashing-shard_proposer_slashing) + - [Shard blob subnets](#shard-blob-subnets) + - [`shard_blob_{subnet_id}`](#shard_blob_subnet_id) + - [Global topics](#global-topics) + - [`shard_blob_header`](#shard_blob_header) + - [`shard_proposer_slashing`](#shard_proposer_slashing) @@ -88,12 +90,16 @@ Following the same scheme as the [Phase0 gossip topics](../phase0/p2p-interface. | Name | Message Type | |----------------------------------|---------------------------| | `shard_blob_{subnet_id}` | `SignedShardBlob` | -| `shard_header` | `SignedShardHeader` | +| `shard_blob_header` | `SignedShardBlobHeader` | | `shard_proposer_slashing` | `ShardProposerSlashing` | The [DAS network specification](./das-p2p.md) defines additional topics. -#### Shard blobs: `shard_blob_{subnet_id}` +#### Shard blob subnets + +Shard blob subnets are used to propagate shard blobs to subsections of the network. + +##### `shard_blob_{subnet_id}` Shard block data, in the form of a `SignedShardBlob` is published to the `shard_blob_{subnet_id}` subnets. @@ -129,19 +135,23 @@ The following validations MUST pass before forwarding the `signed_blob` (with in the block MAY be queued for later processing while proposers for the blob's branch are calculated -- in such a case _do not_ `REJECT`, instead `IGNORE` this message. +#### Global topics -#### Shard header: `shard_header` +There are two additional global topics for Sharding, one is used to propagate shard blob headers (`shard_blob_header`) to +all nodes on the network. Another one is used to propagate validator message (`shard_proposer_slashing`). -Shard header data, in the form of a `SignedShardBlobHeader` is published to the global `shard_header` subnet. +##### `shard_blob_header` -The following validations MUST pass before forwarding the `signed_shard_header` (with inner `message` as `header`) on the network. +Shard header data, in the form of a `SignedShardBlobHeader` is published to the global `shard_blob_header` subnet. + +The following validations MUST pass before forwarding the `signed_shard_blob_header` (with inner `message` as `header`) on the network. - _[IGNORE]_ The `header` is not from a future slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- i.e. validate that `header.slot <= current_slot` (a client MAY queue future headers for processing at the appropriate slot). - _[IGNORE]_ The `header` is new enough to be still be processed -- i.e. validate that `compute_epoch_at_slot(header.slot) >= get_previous_epoch(state)` - _[IGNORE]_ The header is the first header with valid signature received for the `(header.proposer_index, header.slot, header.shard)` combination. -- _[REJECT]_ The proposer signature, `signed_shard_header.signature`, is valid with respect to the `proposer_index` pubkey. +- _[REJECT]_ The proposer signature, `signed_shard_blob_header.signature`, is valid with respect to the `proposer_index` pubkey. - _[REJECT]_ The header is proposed by the expected `proposer_index` for the block's slot in the context of the current shuffling (defined by `header.body_summary.beacon_block_root`/`slot`). If the `proposer_index` cannot immediately be verified against the expected shuffling, @@ -149,7 +159,7 @@ The following validations MUST pass before forwarding the `signed_shard_header` in such a case _do not_ `REJECT`, instead `IGNORE` this message. -#### Shard proposer slashing: `shard_proposer_slashing` +##### `shard_proposer_slashing` Shard proposer slashings, in the form of `ShardProposerSlashing`, are published to the global `shard_proposer_slashing` topic. From 4a5947d49e5f952b5b7b8952125bcb944978310b Mon Sep 17 00:00:00 2001 From: terence tsao Date: Mon, 24 May 2021 06:49:20 -0700 Subject: [PATCH 02/22] Proto's suggestion --- specs/sharding/p2p-interface.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/specs/sharding/p2p-interface.md b/specs/sharding/p2p-interface.md index 1fef7c60f..65393bc0e 100644 --- a/specs/sharding/p2p-interface.md +++ b/specs/sharding/p2p-interface.md @@ -20,7 +20,7 @@ - [Shard blob subnets](#shard-blob-subnets) - [`shard_blob_{subnet_id}`](#shard_blob_subnet_id) - [Global topics](#global-topics) - - [`shard_blob_header`](#shard_blob_header) + - [`shard_header`](#shard_header) - [`shard_proposer_slashing`](#shard_proposer_slashing) @@ -30,7 +30,7 @@ ## Introduction The specification of these changes continues in the same format as the [Phase0](../phase0/p2p-interface.md) and -[Altair](../altair/p2p-interface.md) network specifications, and assumes them as pre-requisite. +[Altair](../altair/p2p-interface.md) network specifications, and assumes them as pre-requisite. The adjustments and additions for Shards are outlined in this document. ## Constants @@ -90,7 +90,7 @@ Following the same scheme as the [Phase0 gossip topics](../phase0/p2p-interface. | Name | Message Type | |----------------------------------|---------------------------| | `shard_blob_{subnet_id}` | `SignedShardBlob` | -| `shard_blob_header` | `SignedShardBlobHeader` | +| `shard_header` | `SignedShardBlobHeader` | | `shard_proposer_slashing` | `ShardProposerSlashing` | The [DAS network specification](./das-p2p.md) defines additional topics. @@ -124,7 +124,7 @@ The following validations MUST pass before forwarding the `signed_blob` (with in - _[IGNORE]_ The `blob` is new enough to be still be processed -- i.e. validate that `compute_epoch_at_slot(blob.slot) >= get_previous_epoch(state)` - _[REJECT]_ The shard blob is for the correct subnet -- - i.e. `compute_subnet_for_shard_blob(state, blob.slot, blob.shard) == subnet_id` + i.e. `compute_subnet_for_shard_blob(state, blob.slot, blob.shard) == subnet_id` - _[IGNORE]_ The blob is the first blob with valid signature received for the `(blob.proposer_index, blob.slot, blob.shard)` combination. - _[REJECT]_ As already limited by the SSZ list-limit, it is important the blob is well-formatted and not too large. - _[REJECT]_ The `blob.body.data` MUST NOT contain any point `p >= MODULUS`. Although it is a `uint256`, not the full 256 bit range is valid. @@ -137,12 +137,12 @@ The following validations MUST pass before forwarding the `signed_blob` (with in #### Global topics -There are two additional global topics for Sharding, one is used to propagate shard blob headers (`shard_blob_header`) to +There are two additional global topics for Sharding, one is used to propagate shard blob headers (`shard_header`) to all nodes on the network. Another one is used to propagate validator message (`shard_proposer_slashing`). -##### `shard_blob_header` +##### `shard_header` -Shard header data, in the form of a `SignedShardBlobHeader` is published to the global `shard_blob_header` subnet. +Shard header data, in the form of a `SignedShardBlobHeader` is published to the global `shard_header` subnet. The following validations MUST pass before forwarding the `signed_shard_blob_header` (with inner `message` as `header`) on the network. - _[IGNORE]_ The `header` is not from a future slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- @@ -168,3 +168,4 @@ The following validations MUST pass before forwarding the `shard_proposer_slashi for the proposer with index `proposer_slashing.signed_header_1.message.proposer_index`. The `slot` and `shard` are ignored, there are no per-shard slashings. - _[REJECT]_ All of the conditions within `process_shard_proposer_slashing` pass validation. +- From 814c7696d96bab5e62aaea15cee8cdb39e0de6a3 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Mon, 24 May 2021 06:50:03 -0700 Subject: [PATCH 03/22] shard_blob_header -> shard_header --- specs/sharding/p2p-interface.md | 1 - 1 file changed, 1 deletion(-) diff --git a/specs/sharding/p2p-interface.md b/specs/sharding/p2p-interface.md index 65393bc0e..5acd9735d 100644 --- a/specs/sharding/p2p-interface.md +++ b/specs/sharding/p2p-interface.md @@ -168,4 +168,3 @@ The following validations MUST pass before forwarding the `shard_proposer_slashi for the proposer with index `proposer_slashing.signed_header_1.message.proposer_index`. The `slot` and `shard` are ignored, there are no per-shard slashings. - _[REJECT]_ All of the conditions within `process_shard_proposer_slashing` pass validation. -- From 71d0d453431b21c61e44170d1865d01fc1c21482 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Mon, 24 May 2021 06:57:09 -0700 Subject: [PATCH 04/22] Align table --- specs/sharding/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/p2p-interface.md b/specs/sharding/p2p-interface.md index 5acd9735d..b7e229e87 100644 --- a/specs/sharding/p2p-interface.md +++ b/specs/sharding/p2p-interface.md @@ -90,7 +90,7 @@ Following the same scheme as the [Phase0 gossip topics](../phase0/p2p-interface. | Name | Message Type | |----------------------------------|---------------------------| | `shard_blob_{subnet_id}` | `SignedShardBlob` | -| `shard_header` | `SignedShardBlobHeader` | +| `shard_header` | `SignedShardBlobHeader` | | `shard_proposer_slashing` | `ShardProposerSlashing` | The [DAS network specification](./das-p2p.md) defines additional topics. From c9f37805656cbae68ba05f6ca054d85ce8410ee5 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Mon, 24 May 2021 11:22:30 -0700 Subject: [PATCH 05/22] Clean up outdated attestation helper --- .../test_process_attestation.py | 7 ++- .../test_process_chunk_challenge.py | 46 +++++++++---------- .../test_process_custody_slashing.py | 6 +-- .../test_process_challenge_deadlines.py | 6 +-- .../test_process_custody_final_updates.py | 10 ++-- .../test/custody_game/sanity/test_blocks.py | 6 +-- .../eth2spec/test/helpers/attestations.py | 43 ++--------------- .../test_process_attestation.py | 34 +++++++------- .../test/phase0/sanity/test_blocks.py | 2 +- 9 files changed, 62 insertions(+), 98 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_attestation.py b/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_attestation.py index 707ac0b2e..4ed3f5088 100644 --- a/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_attestation.py +++ b/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_attestation.py @@ -7,8 +7,7 @@ from eth2spec.test.helpers.constants import CUSTODY_GAME from eth2spec.test.helpers.state import transition_to from eth2spec.test.helpers.attestations import ( run_attestation_processing, - get_valid_late_attestation, - get_valid_on_time_attestation, + get_valid_attestation, ) @@ -16,7 +15,7 @@ from eth2spec.test.helpers.attestations import ( @spec_state_test @always_bls def test_on_time_success(spec, state): - attestation = get_valid_on_time_attestation(spec, state, signed=True) + attestation = get_valid_attestation(spec, state, signed=True) transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) @@ -27,7 +26,7 @@ def test_on_time_success(spec, state): @spec_state_test @always_bls def test_late_success(spec, state): - attestation = get_valid_late_attestation(spec, state, signed=True) + attestation = get_valid_attestation(spec, state, signed=True) transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY + 1) diff --git a/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_chunk_challenge.py b/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_chunk_challenge.py index 87f0238fb..cc12b66f5 100644 --- a/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_chunk_challenge.py +++ b/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_chunk_challenge.py @@ -4,7 +4,7 @@ from eth2spec.test.helpers.custody import ( get_sample_shard_transition, ) from eth2spec.test.helpers.attestations import ( - get_valid_on_time_attestation, + get_valid_attestation, ) from eth2spec.test.helpers.constants import ( CUSTODY_GAME, @@ -80,8 +80,8 @@ def test_challenge_appended(spec, state): shard = 0 offset_slots = spec.get_offset_slots(state, shard) shard_transition = get_sample_shard_transition(spec, state.slot, [2**15 // 3] * len(offset_slots)) - attestation = get_valid_on_time_attestation(spec, state, index=shard, signed=True, - shard_transition=shard_transition) + attestation = get_valid_attestation(spec, state, index=shard, signed=True, + shard_transition=shard_transition) transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) @@ -104,8 +104,8 @@ def test_challenge_empty_element_replaced(spec, state): shard = 0 offset_slots = spec.get_offset_slots(state, shard) shard_transition = get_sample_shard_transition(spec, state.slot, [2**15 // 3] * len(offset_slots)) - attestation = get_valid_on_time_attestation(spec, state, index=shard, signed=True, - shard_transition=shard_transition) + attestation = get_valid_attestation(spec, state, index=shard, signed=True, + shard_transition=shard_transition) transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) @@ -130,8 +130,8 @@ def test_duplicate_challenge(spec, state): shard = 0 offset_slots = spec.get_offset_slots(state, shard) shard_transition = get_sample_shard_transition(spec, state.slot, [2**15 // 3] * len(offset_slots)) - attestation = get_valid_on_time_attestation(spec, state, index=shard, signed=True, - shard_transition=shard_transition) + attestation = get_valid_attestation(spec, state, index=shard, signed=True, + shard_transition=shard_transition) transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) @@ -156,8 +156,8 @@ def test_second_challenge(spec, state): shard = 0 offset_slots = spec.get_offset_slots(state, shard) shard_transition = get_sample_shard_transition(spec, state.slot, [2**15 // 3] * len(offset_slots)) - attestation = get_valid_on_time_attestation(spec, state, index=shard, signed=True, - shard_transition=shard_transition) + attestation = get_valid_attestation(spec, state, index=shard, signed=True, + shard_transition=shard_transition) transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) @@ -185,8 +185,8 @@ def test_multiple_epochs_custody(spec, state): shard = 0 offset_slots = spec.get_offset_slots(state, shard) shard_transition = get_sample_shard_transition(spec, state.slot, [2**15 // 3] * len(offset_slots)) - attestation = get_valid_on_time_attestation(spec, state, index=shard, signed=True, - shard_transition=shard_transition) + attestation = get_valid_attestation(spec, state, index=shard, signed=True, + shard_transition=shard_transition) transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) @@ -210,8 +210,8 @@ def test_many_epochs_custody(spec, state): shard = 0 offset_slots = spec.get_offset_slots(state, shard) shard_transition = get_sample_shard_transition(spec, state.slot, [2**15 // 3] * len(offset_slots)) - attestation = get_valid_on_time_attestation(spec, state, index=shard, signed=True, - shard_transition=shard_transition) + attestation = get_valid_attestation(spec, state, index=shard, signed=True, + shard_transition=shard_transition) transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) @@ -235,8 +235,8 @@ def test_off_chain_attestation(spec, state): shard = 0 offset_slots = spec.get_offset_slots(state, shard) shard_transition = get_sample_shard_transition(spec, state.slot, [2**15 // 3] * len(offset_slots)) - attestation = get_valid_on_time_attestation(spec, state, index=shard, signed=True, - shard_transition=shard_transition) + attestation = get_valid_attestation(spec, state, index=shard, signed=True, + shard_transition=shard_transition) transition_to(spec, state, state.slot + spec.SLOTS_PER_EPOCH * (spec.EPOCHS_PER_CUSTODY_PERIOD - 1)) @@ -256,8 +256,8 @@ def test_custody_response(spec, state): shard = 0 offset_slots = spec.get_offset_slots(state, shard) shard_transition = get_sample_shard_transition(spec, state.slot, [2**15 // 3] * len(offset_slots)) - attestation = get_valid_on_time_attestation(spec, state, index=shard, signed=True, - shard_transition=shard_transition) + attestation = get_valid_attestation(spec, state, index=shard, signed=True, + shard_transition=shard_transition) transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) @@ -287,8 +287,8 @@ def test_custody_response_chunk_index_2(spec, state): shard = 0 offset_slots = spec.get_offset_slots(state, shard) shard_transition = get_sample_shard_transition(spec, state.slot, [2**15 // 3] * len(offset_slots)) - attestation = get_valid_on_time_attestation(spec, state, index=shard, signed=True, - shard_transition=shard_transition) + attestation = get_valid_attestation(spec, state, index=shard, signed=True, + shard_transition=shard_transition) transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) @@ -319,8 +319,8 @@ def test_custody_response_multiple_epochs(spec, state): shard = 0 offset_slots = spec.get_offset_slots(state, shard) shard_transition = get_sample_shard_transition(spec, state.slot, [2**15 // 3] * len(offset_slots)) - attestation = get_valid_on_time_attestation(spec, state, index=shard, signed=True, - shard_transition=shard_transition) + attestation = get_valid_attestation(spec, state, index=shard, signed=True, + shard_transition=shard_transition) transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) @@ -351,8 +351,8 @@ def test_custody_response_many_epochs(spec, state): shard = 0 offset_slots = spec.get_offset_slots(state, shard) shard_transition = get_sample_shard_transition(spec, state.slot, [2**15 // 3] * len(offset_slots)) - attestation = get_valid_on_time_attestation(spec, state, index=shard, signed=True, - shard_transition=shard_transition) + attestation = get_valid_attestation(spec, state, index=shard, signed=True, + shard_transition=shard_transition) transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) diff --git a/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_custody_slashing.py b/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_custody_slashing.py index 7ee5cd394..4891c7b23 100644 --- a/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_custody_slashing.py +++ b/tests/core/pyspec/eth2spec/test/custody_game/block_processing/test_process_custody_slashing.py @@ -3,7 +3,7 @@ from eth2spec.test.helpers.custody import ( get_custody_slashable_shard_transition, ) from eth2spec.test.helpers.attestations import ( - get_valid_on_time_attestation, + get_valid_attestation, ) from eth2spec.test.helpers.constants import ( CUSTODY_GAME, @@ -96,8 +96,8 @@ def run_standard_custody_slashing_test(spec, slashable=correct, ) - attestation = get_valid_on_time_attestation(spec, state, index=shard, signed=True, - shard_transition=shard_transition) + attestation = get_valid_attestation(spec, state, index=shard, signed=True, + shard_transition=shard_transition) transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) diff --git a/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_challenge_deadlines.py b/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_challenge_deadlines.py index 7332dcc80..144ea0213 100644 --- a/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_challenge_deadlines.py +++ b/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_challenge_deadlines.py @@ -3,7 +3,7 @@ from eth2spec.test.helpers.custody import ( get_sample_shard_transition, ) from eth2spec.test.helpers.attestations import ( - get_valid_on_time_attestation, + get_valid_attestation, ) from eth2spec.test.helpers.state import transition_to, transition_to_valid_shard_slot from eth2spec.test.context import ( @@ -36,8 +36,8 @@ def test_validator_slashed_after_chunk_challenge(spec, state): shard = 0 offset_slots = spec.get_offset_slots(state, shard) shard_transition = get_sample_shard_transition(spec, state.slot, [2**15 // 3] * len(offset_slots)) - attestation = get_valid_on_time_attestation(spec, state, index=shard, signed=True, - shard_transition=shard_transition) + attestation = get_valid_attestation(spec, state, index=shard, signed=True, + shard_transition=shard_transition) transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) diff --git a/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_custody_final_updates.py b/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_custody_final_updates.py index 92c311a29..d8dd3d19e 100644 --- a/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_custody_final_updates.py +++ b/tests/core/pyspec/eth2spec/test/custody_game/epoch_processing/test_process_custody_final_updates.py @@ -8,7 +8,7 @@ from eth2spec.test.helpers.custody import ( get_sample_shard_transition ) from eth2spec.test.helpers.attestations import ( - get_valid_on_time_attestation, + get_valid_attestation, ) from eth2spec.test.helpers.state import next_epoch_via_block, transition_to, transition_to_valid_shard_slot from eth2spec.test.context import ( @@ -77,8 +77,8 @@ def test_validator_withdrawal_suspend_after_chunk_challenge(spec, state): shard = 0 offset_slots = spec.get_offset_slots(state, shard) shard_transition = get_sample_shard_transition(spec, state.slot, [2**15 // 3] * len(offset_slots)) - attestation = get_valid_on_time_attestation(spec, state, index=shard, signed=True, - shard_transition=shard_transition) + attestation = get_valid_attestation(spec, state, index=shard, signed=True, + shard_transition=shard_transition) transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) @@ -126,8 +126,8 @@ def test_validator_withdrawal_resume_after_chunk_challenge_response(spec, state) shard = 0 offset_slots = spec.get_offset_slots(state, shard) shard_transition = get_sample_shard_transition(spec, state.slot, [2**15 // 3] * len(offset_slots)) - attestation = get_valid_on_time_attestation(spec, state, index=shard, signed=True, - shard_transition=shard_transition) + attestation = get_valid_attestation(spec, state, index=shard, signed=True, + shard_transition=shard_transition) transition_to(spec, state, state.slot + spec.MIN_ATTESTATION_INCLUSION_DELAY) diff --git a/tests/core/pyspec/eth2spec/test/custody_game/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/custody_game/sanity/test_blocks.py index f242c361b..77ce3c5ad 100644 --- a/tests/core/pyspec/eth2spec/test/custody_game/sanity/test_blocks.py +++ b/tests/core/pyspec/eth2spec/test/custody_game/sanity/test_blocks.py @@ -5,7 +5,7 @@ from eth2spec.test.context import ( spec_state_test, with_presets, ) -from eth2spec.test.helpers.attestations import get_valid_on_time_attestation +from eth2spec.test.helpers.attestations import get_valid_attestation from eth2spec.test.helpers.block import build_empty_block from eth2spec.test.helpers.constants import ( CUSTODY_GAME, @@ -60,7 +60,7 @@ def test_with_shard_transition_with_custody_challenge_and_response(spec, state): shard_block = build_shard_block(spec, state, shard, body=body, slot=state.slot, signed=True) shard_block_dict: Dict[spec.Shard, Sequence[spec.SignedShardBlock]] = {shard: [shard_block]} shard_transitions = get_shard_transitions(spec, state, shard_block_dict) - attestation = get_valid_on_time_attestation( + attestation = get_valid_attestation( spec, state, index=committee_index, shard_transition=shard_transitions[shard], signed=True, ) @@ -127,7 +127,7 @@ def test_custody_slashing(spec, state): shard_block_dict: Dict[spec.Shard, Sequence[spec.SignedShardBlock]] = {shard: [shard_block]} shard_transitions = get_shard_transitions(spec, state, shard_block_dict) - attestation = get_valid_on_time_attestation( + attestation = get_valid_attestation( spec, state, index=committee_index, shard_transition=shard_transitions[shard], signed=True, ) diff --git a/tests/core/pyspec/eth2spec/test/helpers/attestations.py b/tests/core/pyspec/eth2spec/test/helpers/attestations.py index b55aff9e5..c92860ffa 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/attestations.py +++ b/tests/core/pyspec/eth2spec/test/helpers/attestations.py @@ -50,7 +50,7 @@ def run_attestation_processing(spec, state, attestation, valid=True): yield 'post', state -def build_attestation_data(spec, state, slot, index, shard=None, on_time=True): +def build_attestation_data(spec, state, slot, index, shard=None): assert state.slot >= slot if slot == state.slot: @@ -85,45 +85,12 @@ def build_attestation_data(spec, state, slot, index, shard=None, on_time=True): return data -def get_valid_on_time_attestation(spec, state, slot=None, index=None, signed=False): - ''' - Construct on-time attestation for next slot - ''' - if slot is None: - slot = state.slot - if index is None: - index = 0 - - return get_valid_attestation( - spec, - state, - slot=slot, - index=index, - signed=signed, - on_time=True, - ) - - -def get_valid_late_attestation(spec, state, slot=None, index=None, signed=False): - ''' - Construct on-time attestation for next slot - ''' - if slot is None: - slot = state.slot - if index is None: - index = 0 - - return get_valid_attestation(spec, state, slot=slot, index=index, - signed=signed, on_time=False) - - def get_valid_attestation(spec, state, slot=None, index=None, filter_participant_set=None, - signed=False, - on_time=True): + signed=False): # If filter_participant_set filters everything, the attestation has 0 participants, and cannot be signed. # Thus strictly speaking invalid when no participant is added later. if slot is None: @@ -132,7 +99,7 @@ def get_valid_attestation(spec, index = 0 attestation_data = build_attestation_data( - spec, state, slot=slot, index=index, on_time=on_time + spec, state, slot=slot, index=index ) beacon_committee = spec.get_beacon_committee( @@ -219,7 +186,7 @@ def add_attestations_to_state(spec, state, attestations, slot): spec.process_attestation(state, attestation) -def _get_valid_attestation_at_slot(state, spec, slot_to_attest, participation_fn=None, on_time=True): +def _get_valid_attestation_at_slot(state, spec, slot_to_attest, participation_fn=None): committees_per_slot = spec.get_committee_count_per_slot(state, spec.compute_epoch_at_slot(slot_to_attest)) for index in range(committees_per_slot): def participants_filter(comm): @@ -234,7 +201,6 @@ def _get_valid_attestation_at_slot(state, spec, slot_to_attest, participation_fn slot_to_attest, index=index, signed=True, - on_time=on_time, filter_participant_set=participants_filter ) @@ -269,7 +235,6 @@ def next_slots_with_attestations(spec, post_state, spec, slot_to_attest, - on_time=False, participation_fn=participation_fn ) for attestation in attestations: diff --git a/tests/core/pyspec/eth2spec/test/phase0/block_processing/test_process_attestation.py b/tests/core/pyspec/eth2spec/test/phase0/block_processing/test_process_attestation.py index 38a050ebc..c30320066 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/block_processing/test_process_attestation.py +++ b/tests/core/pyspec/eth2spec/test/phase0/block_processing/test_process_attestation.py @@ -45,7 +45,7 @@ def test_success_multi_proposer_index_iterations(spec, state): @with_all_phases @spec_state_test def test_success_previous_epoch(spec, state): - attestation = get_valid_attestation(spec, state, signed=True, on_time=False) + attestation = get_valid_attestation(spec, state, signed=True) next_epoch_via_block(spec, state) yield from run_attestation_processing(spec, state, attestation) @@ -96,7 +96,7 @@ def test_before_inclusion_delay(spec, state): @with_all_phases @spec_state_test def test_after_epoch_slots(spec, state): - attestation = get_valid_attestation(spec, state, signed=True, on_time=False) + attestation = get_valid_attestation(spec, state, signed=True) # increment past latest inclusion slot transition_to_slot_via_block(spec, state, state.slot + spec.SLOTS_PER_EPOCH + 1) @@ -197,7 +197,7 @@ def test_mismatched_target_and_slot(spec, state): next_epoch_via_block(spec, state) next_epoch_via_block(spec, state) - attestation = get_valid_attestation(spec, state, on_time=False) + attestation = get_valid_attestation(spec, state) attestation.data.slot = attestation.data.slot - spec.SLOTS_PER_EPOCH sign_attestation(spec, state, attestation) @@ -210,7 +210,7 @@ def test_mismatched_target_and_slot(spec, state): def test_old_target_epoch(spec, state): assert spec.MIN_ATTESTATION_INCLUSION_DELAY < spec.SLOTS_PER_EPOCH * 2 - attestation = get_valid_attestation(spec, state, signed=True, on_time=False) + attestation = get_valid_attestation(spec, state, signed=True) next_slots(spec, state, spec.SLOTS_PER_EPOCH * 2) # target epoch will be too old to handle @@ -275,7 +275,7 @@ def test_invalid_current_source_root(spec, state): state.previous_justified_checkpoint = spec.Checkpoint(epoch=3, root=b'\x01' * 32) state.current_justified_checkpoint = spec.Checkpoint(epoch=4, root=b'\x32' * 32) - attestation = get_valid_attestation(spec, state, slot=(spec.SLOTS_PER_EPOCH * 3) + 1, on_time=False) + attestation = get_valid_attestation(spec, state, slot=(spec.SLOTS_PER_EPOCH * 3) + 1) next_slots(spec, state, spec.MIN_ATTESTATION_INCLUSION_DELAY) # Test logic sanity checks: @@ -348,7 +348,7 @@ def test_correct_min_inclusion_delay(spec, state): @with_all_phases @spec_state_test def test_correct_sqrt_epoch_delay(spec, state): - attestation = get_valid_attestation(spec, state, signed=True, on_time=False) + attestation = get_valid_attestation(spec, state, signed=True) next_slots(spec, state, spec.integer_squareroot(spec.SLOTS_PER_EPOCH)) yield from run_attestation_processing(spec, state, attestation) @@ -357,7 +357,7 @@ def test_correct_sqrt_epoch_delay(spec, state): @with_all_phases @spec_state_test def test_correct_epoch_delay(spec, state): - attestation = get_valid_attestation(spec, state, signed=True, on_time=False) + attestation = get_valid_attestation(spec, state, signed=True) next_slots(spec, state, spec.SLOTS_PER_EPOCH) yield from run_attestation_processing(spec, state, attestation) @@ -366,7 +366,7 @@ def test_correct_epoch_delay(spec, state): @with_all_phases @spec_state_test def test_correct_after_epoch_delay(spec, state): - attestation = get_valid_attestation(spec, state, signed=True, on_time=False) + attestation = get_valid_attestation(spec, state, signed=True) # increment past latest inclusion slot next_slots(spec, state, spec.SLOTS_PER_EPOCH + 1) @@ -393,7 +393,7 @@ def test_incorrect_head_min_inclusion_delay(spec, state): @with_all_phases @spec_state_test def test_incorrect_head_sqrt_epoch_delay(spec, state): - attestation = get_valid_attestation(spec, state, signed=False, on_time=False) + attestation = get_valid_attestation(spec, state, signed=False) next_slots(spec, state, spec.integer_squareroot(spec.SLOTS_PER_EPOCH)) attestation.data.beacon_block_root = b'\x42' * 32 @@ -405,7 +405,7 @@ def test_incorrect_head_sqrt_epoch_delay(spec, state): @with_all_phases @spec_state_test def test_incorrect_head_epoch_delay(spec, state): - attestation = get_valid_attestation(spec, state, signed=False, on_time=False) + attestation = get_valid_attestation(spec, state, signed=False) next_slots(spec, state, spec.SLOTS_PER_EPOCH) attestation.data.beacon_block_root = b'\x42' * 32 @@ -417,7 +417,7 @@ def test_incorrect_head_epoch_delay(spec, state): @with_all_phases @spec_state_test def test_incorrect_head_after_epoch_delay(spec, state): - attestation = get_valid_attestation(spec, state, signed=False, on_time=False) + attestation = get_valid_attestation(spec, state, signed=False) # increment past latest inclusion slot next_slots(spec, state, spec.SLOTS_PER_EPOCH + 1) @@ -448,7 +448,7 @@ def test_incorrect_head_and_target_min_inclusion_delay(spec, state): @with_all_phases @spec_state_test def test_incorrect_head_and_target_sqrt_epoch_delay(spec, state): - attestation = get_valid_attestation(spec, state, signed=False, on_time=False) + attestation = get_valid_attestation(spec, state, signed=False) next_slots(spec, state, spec.integer_squareroot(spec.SLOTS_PER_EPOCH)) attestation.data.beacon_block_root = b'\x42' * 32 @@ -461,7 +461,7 @@ def test_incorrect_head_and_target_sqrt_epoch_delay(spec, state): @with_all_phases @spec_state_test def test_incorrect_head_and_target_epoch_delay(spec, state): - attestation = get_valid_attestation(spec, state, signed=False, on_time=False) + attestation = get_valid_attestation(spec, state, signed=False) next_slots(spec, state, spec.SLOTS_PER_EPOCH) attestation.data.beacon_block_root = b'\x42' * 32 @@ -474,7 +474,7 @@ def test_incorrect_head_and_target_epoch_delay(spec, state): @with_all_phases @spec_state_test def test_incorrect_head_and_target_after_epoch_delay(spec, state): - attestation = get_valid_attestation(spec, state, signed=False, on_time=False) + attestation = get_valid_attestation(spec, state, signed=False) # increment past latest inclusion slot next_slots(spec, state, spec.SLOTS_PER_EPOCH + 1) @@ -504,7 +504,7 @@ def test_incorrect_target_min_inclusion_delay(spec, state): @with_all_phases @spec_state_test def test_incorrect_target_sqrt_epoch_delay(spec, state): - attestation = get_valid_attestation(spec, state, signed=False, on_time=False) + attestation = get_valid_attestation(spec, state, signed=False) next_slots(spec, state, spec.integer_squareroot(spec.SLOTS_PER_EPOCH)) attestation.data.target.root = b'\x42' * 32 @@ -516,7 +516,7 @@ def test_incorrect_target_sqrt_epoch_delay(spec, state): @with_all_phases @spec_state_test def test_incorrect_target_epoch_delay(spec, state): - attestation = get_valid_attestation(spec, state, signed=False, on_time=False) + attestation = get_valid_attestation(spec, state, signed=False) next_slots(spec, state, spec.SLOTS_PER_EPOCH) attestation.data.target.root = b'\x42' * 32 @@ -528,7 +528,7 @@ def test_incorrect_target_epoch_delay(spec, state): @with_all_phases @spec_state_test def test_incorrect_target_after_epoch_delay(spec, state): - attestation = get_valid_attestation(spec, state, signed=False, on_time=False) + attestation = get_valid_attestation(spec, state, signed=False) # increment past latest inclusion slot next_slots(spec, state, spec.SLOTS_PER_EPOCH + 1) diff --git a/tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py index 0e22e75b8..dba623855 100644 --- a/tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py +++ b/tests/core/pyspec/eth2spec/test/phase0/sanity/test_blocks.py @@ -771,7 +771,7 @@ def test_attestation(spec, state): # if spec.fork == SHARDING: # TODO add shard data to block to vote on - attestation = get_valid_attestation(spec, state, index=index, signed=True, on_time=True) + attestation = get_valid_attestation(spec, state, index=index, signed=True) if not is_post_altair(spec): pre_current_attestations_len = len(state.current_epoch_attestations) From fc1af1cff377ce68a5d2c54a415084632b4e53ab Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 25 May 2021 21:13:12 +0800 Subject: [PATCH 06/22] [pyspec] Use mainnet.py as the default spec --- setup.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 8095e3b7f..77b38b640 100644 --- a/setup.py +++ b/setup.py @@ -912,7 +912,8 @@ class PySpecCommand(Command): if not self.dry_run: with open(os.path.join(self.out_dir, '__init__.py'), 'w') as out: - out.write("") + # `mainnet` is the default spec. + out.write("from . import mainnet as spec\n") class BuildPyCommand(build_py): From 74761db7a38a0eb81fc5b5b246f4ea2f2065fe9d Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 25 May 2021 21:40:10 +0800 Subject: [PATCH 07/22] Fix lint error --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 77b38b640..99e501d9f 100644 --- a/setup.py +++ b/setup.py @@ -913,7 +913,7 @@ class PySpecCommand(Command): if not self.dry_run: with open(os.path.join(self.out_dir, '__init__.py'), 'w') as out: # `mainnet` is the default spec. - out.write("from . import mainnet as spec\n") + out.write("from . import mainnet as spec # noqa:F401\n") class BuildPyCommand(build_py): From 715e450e0b7e8e78c2edbe0ee56babb30557ca56 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 26 May 2021 00:18:59 +0800 Subject: [PATCH 08/22] Generate coverage report on `minimal` config spec by default --- Makefile | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 007f49f5b..d637d26cf 100644 --- a/Makefile +++ b/Makefile @@ -94,13 +94,15 @@ pyspec: install_test: python3 -m venv venv; . venv/bin/activate; python3 -m pip install -e .[lint]; python3 -m pip install -e .[test] +# Testing against `minimal` config by default test: pyspec . venv/bin/activate; cd $(PY_SPEC_DIR); \ - python3 -m pytest -n 4 --disable-bls --cov=eth2spec.phase0.mainnet --cov=eth2spec.altair.mainnet --cov-report="html:$(COV_HTML_OUT)" --cov-branch eth2spec + python3 -m pytest -n 4 --disable-bls --cov=eth2spec.phase0.minimal --cov=eth2spec.altair.minimal --cov-report="html:$(COV_HTML_OUT)" --cov-branch eth2spec +# Testing against `minimal` config by default find_test: pyspec . venv/bin/activate; cd $(PY_SPEC_DIR); \ - python3 -m pytest -k=$(K) --disable-bls --cov=eth2spec.phase0.mainnet --cov=eth2spec.altair.mainnet --cov-report="html:$(COV_HTML_OUT)" --cov-branch eth2spec + python3 -m pytest -k=$(K) --disable-bls --cov=eth2spec.phase0.minimal --cov=eth2spec.altair.minimal --cov-report="html:$(COV_HTML_OUT)" --cov-branch eth2spec citest: pyspec mkdir -p tests/core/pyspec/test-reports/eth2spec; . venv/bin/activate; cd $(PY_SPEC_DIR); \ From 4664ccbc4276ee1613e598a654dd3948a90b316b Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Tue, 25 May 2021 09:30:38 -0700 Subject: [PATCH 09/22] Fix bug in Altair transition tests with missing state root --- .../test/altair/transition/test_transition.py | 10 +++++----- tests/core/pyspec/eth2spec/test/helpers/state.py | 12 ++++++++++-- 2 files changed, 15 insertions(+), 7 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py index 20e13b9ba..62740df4e 100644 --- a/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py +++ b/tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py @@ -1,7 +1,7 @@ import random from eth2spec.test.context import fork_transition_test from eth2spec.test.helpers.constants import PHASE0, ALTAIR -from eth2spec.test.helpers.state import state_transition_and_sign_block, next_slot, next_epoch_via_block +from eth2spec.test.helpers.state import state_transition_and_sign_block, next_slot, next_epoch_via_signed_block from eth2spec.test.helpers.block import build_empty_block_for_next_slot, build_empty_block, sign_block from eth2spec.test.helpers.attestations import next_slots_with_attestations @@ -261,12 +261,12 @@ def _run_transition_test_with_attestations(state, assert current_epoch == spec.GENESIS_EPOCH # skip genesis epoch to avoid dealing with some edge cases... - block = next_epoch_via_block(spec, state) + block = next_epoch_via_signed_block(spec, state) # regular state transition until fork: fill_cur_epoch = False fill_prev_epoch = True - blocks = [pre_tag(sign_block(spec, state, block))] + blocks = [pre_tag(block)] current_epoch = spec.get_current_epoch(state) for _ in range(current_epoch, fork_epoch - 1): _, blocks_in_epoch, state = next_slots_with_attestations( @@ -414,8 +414,8 @@ def test_transition_with_no_attestations_until_after_fork(state, fork_epoch, spe # continue regular state transition but add attestations # for enough epochs to finalize the ``fork_epoch`` - block = next_epoch_via_block(post_spec, state) - blocks.append(post_tag(sign_block(post_spec, state, block))) + block = next_epoch_via_signed_block(post_spec, state) + blocks.append(post_tag(block)) for _ in range(4): _, blocks_in_epoch, state = next_slots_with_attestations( post_spec, diff --git a/tests/core/pyspec/eth2spec/test/helpers/state.py b/tests/core/pyspec/eth2spec/test/helpers/state.py index ef09c6e07..05f0e9013 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/state.py +++ b/tests/core/pyspec/eth2spec/test/helpers/state.py @@ -58,11 +58,19 @@ def next_epoch(spec, state): spec.process_slots(state, slot) -def next_epoch_via_block(spec, state): +def next_epoch_via_block(spec, state, insert_state_root=False): """ Transition to the start slot of the next epoch via a full block transition """ - return apply_empty_block(spec, state, state.slot + spec.SLOTS_PER_EPOCH - state.slot % spec.SLOTS_PER_EPOCH) + block = apply_empty_block(spec, state, state.slot + spec.SLOTS_PER_EPOCH - state.slot % spec.SLOTS_PER_EPOCH) + if insert_state_root: + block.state_root = state.hash_tree_root() + return block + + +def next_epoch_via_signed_block(spec, state): + block = next_epoch_via_block(spec, state, insert_state_root=True) + return sign_block(spec, state, block) def get_state_root(spec, state, slot) -> bytes: From d71c50f6564c1f7a73f09a20557084df9195679c Mon Sep 17 00:00:00 2001 From: protolambda Date: Tue, 25 May 2021 19:46:23 +0200 Subject: [PATCH 10/22] Union type update --- ssz/simple-serialize.md | 39 +++++++++++++++++++++++++++------------ 1 file changed, 27 insertions(+), 12 deletions(-) diff --git a/ssz/simple-serialize.md b/ssz/simple-serialize.md index d97b8ea1c..89a1ebc0b 100644 --- a/ssz/simple-serialize.md +++ b/ssz/simple-serialize.md @@ -20,7 +20,8 @@ - [`null`](#null) - [`Bitvector[N]`](#bitvectorn) - [`Bitlist[N]`](#bitlistn) - - [Vectors, containers, lists, unions](#vectors-containers-lists-unions) + - [Vectors, containers, lists](#vectors-containers-lists) + - [Union](#union) - [Deserialization](#deserialization) - [Merkleization](#merkleization) - [Summaries and expansions](#summaries-and-expansions) @@ -61,7 +62,7 @@ * **bitlist**: ordered variable-length collection of `boolean` values, limited to `N` bits * notation `Bitlist[N]` * **union**: union type containing one of the given subtypes - * notation `Union[type_0, type_1, ...]`, e.g. `union[null, uint64]` + * notation `Union[type_0, type_1, ...]`, e.g. `union[None, uint64, uint32]` *Note*: Both `Vector[boolean, N]` and `Bitvector[N]` are valid, yet distinct due to their different serialization requirements. Similarly, both `List[boolean, N]` and `Bitlist[N]` are valid, yet distinct. Generally `Bitvector[N]`/`Bitlist[N]` are preferred because of their serialization efficiencies. @@ -77,7 +78,6 @@ For convenience we alias: * `byte` to `uint8` (this is a basic type) * `BytesN` and `ByteVector[N]` to `Vector[byte, N]` (this is *not* a basic type) * `ByteList[N]` to `List[byte, N]` -* `null`: `{}` ### Default values Assuming a helper function `default(type)` which returns the default value for `type`, we can recursively define the default value for all types. @@ -101,7 +101,7 @@ An SSZ object is called zeroed (and thus, `is_zero(object)` returns true) if it - Empty vector types (`Vector[type, 0]`, `Bitvector[0]`) are illegal. - Containers with no fields are illegal. -- The `null` type is only legal as the first type in a union subtype (i.e. with type index zero). +- The `None` type option in a `Union` type is only legal as the first option (i.e. with index zero). ## Serialization @@ -150,7 +150,7 @@ array[len(value) // 8] |= 1 << (len(value) % 8) return bytes(array) ``` -### Vectors, containers, lists, unions +### Vectors, containers, lists ```python # Recursively serialize @@ -170,14 +170,26 @@ fixed_parts = [part if part != None else variable_offsets[i] for i, part in enum return b"".join(fixed_parts + variable_parts) ``` -If `value` is a union type: +### Union -Define value as an object that has properties `value.value` with the contained value, and `value.type_index` which indexes the type. +A `value` as `Union[T...]` type has properties `value.value` with the contained value, and `value.selector` which indexes the selected `Union` type option `T`. + +A `Union`: +- May have multiple selectors with the same type. +- Should not use selectors above 127 (i.e. highest bit is set), these are reserved for backwards compatible extensions. +- Must have at least 1 type option. +- May have `None` as first type option, i.e. `selector == 0` +- Must have at least 2 type options if the first is `None` +- Is always considered a variable-length type, even if all type options have an equal fixed-length. ```python -serialized_bytes = serialize(value.value) -serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_OFFSET, "little") -return serialized_type_index + serialized_bytes +if value.value is None: + assert value.selector == 0 + return b"\x00" +else: + serialized_bytes = serialize(value.value) + serialized_selector_index = value.selector.to_bytes(1, "little") + return serialized_selector_index + serialized_bytes ``` ## Deserialization @@ -191,12 +203,14 @@ Deserialization can be implemented using a recursive algorithm. The deserializat * The size of each object in the vector/list can be inferred from the difference of two offsets. To get the size of the last object, the total number of bytes has to be known (it is not generally possible to deserialize an SSZ object of unknown length) * Containers follow the same principles as vectors, with the difference that there may be fixed-size objects in a container as well. This means the `fixed_parts` data will contain offsets as well as fixed-size objects. * In the case of bitlists, the length in bits cannot be uniquely inferred from the number of bytes in the object. Because of this, they have a bit at the end that is always set. This bit has to be used to infer the size of the bitlist in bits. +* The first byte of the deserialization scope is deserialized as type selector, the remainder of the scope is deserialized as the selected type. Note that deserialization requires hardening against invalid inputs. A non-exhaustive list: - Offsets: out of order, out of range, mismatching minimum element size. - Scope: Extra unused bytes, not aligned with element size. - More elements than a list limit allows. Part of enforcing consensus. +- An out-of-bounds selected index in an `Union` Efficient algorithms for computing this object can be found in [the implementations](#implementations). @@ -227,7 +241,7 @@ We first define helper functions: - If `1` chunk: the root is the chunk itself. - If `> 1` chunks: merkleize as binary tree. * `mix_in_length`: Given a Merkle root `root` and a length `length` (`"uint256"` little-endian serialization) return `hash(root + length)`. -* `mix_in_type`: Given a Merkle root `root` and a type_index `type_index` (`"uint256"` little-endian serialization) return `hash(root + type_index)`. +* `mix_in_selector`: Given a Merkle root `root` and a type selector `selector` (`"uint256"` little-endian serialization) return `hash(root + selector)`. We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: @@ -237,7 +251,8 @@ We now define Merkleization `hash_tree_root(value)` of an object `value` recursi * `mix_in_length(merkleize(pack_bits(value), limit=chunk_count(type)), len(value))` if `value` is a bitlist. * `merkleize([hash_tree_root(element) for element in value])` if `value` is a vector of composite objects or a container. * `mix_in_length(merkleize([hash_tree_root(element) for element in value], limit=chunk_count(type)), len(value))` if `value` is a list of composite objects. -* `mix_in_type(merkleize(value.value), value.type_index)` if `value` is of union type. +* `mix_in_selector(hash_tree_root(value.value), value.selector)` if `value` is of union type, and `value.value` is not `None` +* `mix_in_selector(Bytes32(), 0)` if `value` is of union type, and `value.value` is `None` ## Summaries and expansions From cb008f2c1a26b5b1ed13925b65763b921cd127a3 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 26 May 2021 02:40:22 +0800 Subject: [PATCH 11/22] Fix _get_sync_committee_signature helper. Should have used `target_slot` --- .../test/altair/unittests/validator/test_validator.py | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py index dfe90b5b5..048e5f43d 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py @@ -4,7 +4,7 @@ from eth2spec.utils.ssz.ssz_typing import Bitvector from eth2spec.test.helpers.block import build_empty_block from eth2spec.test.helpers.keys import pubkey_to_privkey from eth2spec.test.helpers.state import transition_to -from eth2spec.utils import bls +from eth2spec.test.helpers.sync_committee import compute_sync_committee_signature from eth2spec.utils.bls import only_with_bls from eth2spec.test.context import ( with_altair_and_later, @@ -85,12 +85,9 @@ def _get_sync_committee_signature( pubkey = state.current_sync_committee.pubkeys[sync_committee_index] privkey = pubkey_to_privkey[pubkey] - domain = spec.get_domain( - state, - spec.DOMAIN_SYNC_COMMITTEE, + return compute_sync_committee_signature( + spec, state, target_slot, privkey, block_root=target_block_root ) - signing_data = spec.compute_signing_root(target_block_root, domain) - return bls.Sign(privkey, spec.hash_tree_root(signing_data)) @only_with_bls() From 1360860d1e7b2701d6c413186496dc43eecf32b6 Mon Sep 17 00:00:00 2001 From: protolambda Date: Thu, 27 May 2021 03:57:59 +0200 Subject: [PATCH 12/22] add/update comments on shard blob/header/reference body field --- specs/sharding/beacon-chain.md | 3 ++- specs/sharding/p2p-interface.md | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 5522e044d..a15a002e4 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -216,6 +216,7 @@ class ShardBlobHeader(Container): # Slot and shard that this header is intended for slot: Slot shard: Shard + # SSZ-summary of ShardBlobBody body_summary: ShardBlobBodySummary # Proposer of the shard-blob proposer_index: ValidatorIndex @@ -253,7 +254,7 @@ class ShardBlobReference(Container): # Slot and shard that this reference is intended for slot: Slot shard: Shard - # Hash-tree-root of commitment data + # Hash-tree-root of ShardBlobBody body_root: Root # Proposer of the shard-blob proposer_index: ValidatorIndex diff --git a/specs/sharding/p2p-interface.md b/specs/sharding/p2p-interface.md index 47ed52970..39da8166d 100644 --- a/specs/sharding/p2p-interface.md +++ b/specs/sharding/p2p-interface.md @@ -64,6 +64,7 @@ class ShardBlob(Container): # Slot and shard that this blob is intended for slot: Slot shard: Shard + # Shard data with related commitments and beacon anchor body: ShardBlobBody # Proposer of the shard-blob proposer_index: ValidatorIndex From 103d029a1abf054ead8eaa9c7d4c77facc10ace2 Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Thu, 27 May 2021 15:28:51 +1000 Subject: [PATCH 13/22] Add clock disparity tolerance for sync subnets --- specs/altair/p2p-interface.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index 6f250b57e..c66ed015c 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -120,7 +120,7 @@ def get_sync_subcommittee_pubkeys(state: BeaconState, subcommittee_index: uint64 return sync_committee.pubkeys[i:i + sync_subcommittee_size] ``` -- _[IGNORE]_ The contribution's slot is for the current slot, i.e. `contribution.slot == current_slot`. +- _[IGNORE]_ The contribution's slot is for the current slot (with a MAXIMUM_GOSSIP_CLOCK_DISPARITY allowance), i.e. `contribution.slot == current_slot` . - _[IGNORE]_ The block being signed over (`contribution.beacon_block_root`) has been seen (via both gossip and non-gossip sources). - _[REJECT]_ The subcommittee index is in the allowed range, i.e. `contribution.subcommittee_index < SYNC_COMMITTEE_SUBNET_COUNT`. - _[IGNORE]_ The sync committee contribution is the first valid contribution received for the aggregator with index `contribution_and_proof.aggregator_index` for the slot `contribution.slot` and subcommittee index `contribution.subcommittee_index`. @@ -141,7 +141,7 @@ The `sync_committee_{subnet_id}` topics are used to propagate unaggregated sync The following validations MUST pass before forwarding the `sync_committee_signature` on the network: -- _[IGNORE]_ The signature's slot is for the current slot, i.e. `sync_committee_signature.slot == current_slot`. +- _[IGNORE]_ The signature's slot is for the current slot (with a MAXIMUM_GOSSIP_CLOCK_DISPARITY allowance), i.e. `sync_committee_signature.slot == current_slot`. - _[IGNORE]_ The block being signed over (`sync_committee_signature.beacon_block_root`) has been seen (via both gossip and non-gossip sources). - _[IGNORE]_ There has been no other valid sync committee signature for the declared `slot` for the validator referenced by `sync_committee_signature.validator_index`. - _[REJECT]_ The `subnet_id` is valid for the given validator, i.e. `subnet_id in compute_subnets_for_sync_committee(state, sync_committee_signature.validator_index)`. From 48f989070db324bf8d37ff231cfb38766de353b8 Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Thu, 27 May 2021 15:30:44 +1000 Subject: [PATCH 14/22] Remove naughty space --- specs/altair/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index c66ed015c..1f7d0f76c 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -120,7 +120,7 @@ def get_sync_subcommittee_pubkeys(state: BeaconState, subcommittee_index: uint64 return sync_committee.pubkeys[i:i + sync_subcommittee_size] ``` -- _[IGNORE]_ The contribution's slot is for the current slot (with a MAXIMUM_GOSSIP_CLOCK_DISPARITY allowance), i.e. `contribution.slot == current_slot` . +- _[IGNORE]_ The contribution's slot is for the current slot (with a MAXIMUM_GOSSIP_CLOCK_DISPARITY allowance), i.e. `contribution.slot == current_slot`. - _[IGNORE]_ The block being signed over (`contribution.beacon_block_root`) has been seen (via both gossip and non-gossip sources). - _[REJECT]_ The subcommittee index is in the allowed range, i.e. `contribution.subcommittee_index < SYNC_COMMITTEE_SUBNET_COUNT`. - _[IGNORE]_ The sync committee contribution is the first valid contribution received for the aggregator with index `contribution_and_proof.aggregator_index` for the slot `contribution.slot` and subcommittee index `contribution.subcommittee_index`. From d11586122fea6ba00b67392191b9344a474c3749 Mon Sep 17 00:00:00 2001 From: protolambda Date: Fri, 28 May 2021 01:02:08 +0200 Subject: [PATCH 15/22] update remerkleable, union support --- setup.py | 2 +- tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 99e501d9f..cd2006f71 100644 --- a/setup.py +++ b/setup.py @@ -1017,7 +1017,7 @@ setup( "py_ecc==5.2.0", "milagro_bls_binding==1.6.3", "dataclasses==0.6", - "remerkleable==0.1.19", + "remerkleable==0.1.20", RUAMEL_YAML_VERSION, "lru-dict==1.1.6", MARKO_VERSION, diff --git a/tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py b/tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py index 9b18f8bda..5a1b61d0b 100644 --- a/tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py +++ b/tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py @@ -2,6 +2,7 @@ # Ignore linter: This module makes importing SSZ types easy, and hides away the underlying library from the spec. from remerkleable.complex import Container, Vector, List +from remerkleable.union import Union from remerkleable.basic import boolean, bit, uint, byte, uint8, uint16, uint32, uint64, uint128, uint256 from remerkleable.bitfields import Bitvector, Bitlist from remerkleable.byte_arrays import ByteVector, Bytes1, Bytes4, Bytes8, Bytes32, Bytes48, Bytes96, ByteList From ef9b7125c2efe9d786d12501422bbfad357637b4 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 28 May 2021 12:27:19 -0700 Subject: [PATCH 16/22] whitespace --- ssz/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ssz/simple-serialize.md b/ssz/simple-serialize.md index 89a1ebc0b..2ae8d9bdd 100644 --- a/ssz/simple-serialize.md +++ b/ssz/simple-serialize.md @@ -175,7 +175,7 @@ return b"".join(fixed_parts + variable_parts) A `value` as `Union[T...]` type has properties `value.value` with the contained value, and `value.selector` which indexes the selected `Union` type option `T`. A `Union`: -- May have multiple selectors with the same type. +- May have multiple selectors with the same type. - Should not use selectors above 127 (i.e. highest bit is set), these are reserved for backwards compatible extensions. - Must have at least 1 type option. - May have `None` as first type option, i.e. `selector == 0` @@ -194,7 +194,7 @@ else: ## Deserialization -Because serialization is an injective function (i.e. two distinct objects of the same type will serialize to different values) any bytestring has at most one object it could deserialize to. +Because serialization is an injective function (i.e. two distinct objects of the same type will serialize to different values) any bytestring has at most one object it could deserialize to. Deserialization can be implemented using a recursive algorithm. The deserialization of basic objects is easy, and from there we can find a simple recursive algorithm for all fixed-size objects. For variable-size objects we have to do one of the following depending on what kind of object it is: From b763416a6b486b4fe0c82c9a9ceb07b0047d3034 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 28 May 2021 12:27:29 -0700 Subject: [PATCH 17/22] remove unnecessary defn of `null` --- ssz/simple-serialize.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/ssz/simple-serialize.md b/ssz/simple-serialize.md index 2ae8d9bdd..cc03cec09 100644 --- a/ssz/simple-serialize.md +++ b/ssz/simple-serialize.md @@ -17,7 +17,6 @@ - [Serialization](#serialization) - [`uintN`](#uintn) - [`boolean`](#boolean) - - [`null`](#null) - [`Bitvector[N]`](#bitvectorn) - [`Bitlist[N]`](#bitlistn) - [Vectors, containers, lists](#vectors-containers-lists) @@ -123,12 +122,6 @@ assert value in (True, False) return b"\x01" if value is True else b"\x00" ``` -### `null` - -```python -return b"" -``` - ### `Bitvector[N]` ```python From fa09d896484bbe240334fa21ffaa454bafe5842e Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 28 May 2021 18:13:22 -0700 Subject: [PATCH 18/22] Update simple-serialize.md --- ssz/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ssz/simple-serialize.md b/ssz/simple-serialize.md index cc03cec09..4ef64f2f2 100644 --- a/ssz/simple-serialize.md +++ b/ssz/simple-serialize.md @@ -196,7 +196,7 @@ Deserialization can be implemented using a recursive algorithm. The deserializat * The size of each object in the vector/list can be inferred from the difference of two offsets. To get the size of the last object, the total number of bytes has to be known (it is not generally possible to deserialize an SSZ object of unknown length) * Containers follow the same principles as vectors, with the difference that there may be fixed-size objects in a container as well. This means the `fixed_parts` data will contain offsets as well as fixed-size objects. * In the case of bitlists, the length in bits cannot be uniquely inferred from the number of bytes in the object. Because of this, they have a bit at the end that is always set. This bit has to be used to infer the size of the bitlist in bits. -* The first byte of the deserialization scope is deserialized as type selector, the remainder of the scope is deserialized as the selected type. +* In the case of unions, the first byte of the deserialization scope is deserialized as type selector, the remainder of the scope is deserialized as the selected type. Note that deserialization requires hardening against invalid inputs. A non-exhaustive list: From 8f005c18e59a749eca83aa16ff0b2a36fd4e13ea Mon Sep 17 00:00:00 2001 From: protolambda Date: Mon, 31 May 2021 15:20:50 +0200 Subject: [PATCH 19/22] when an untyped var is not the last config var, it needs a comma --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index cd2006f71..6e6349b6b 100644 --- a/setup.py +++ b/setup.py @@ -596,7 +596,7 @@ def objects_to_spec(preset_name: str, def format_config_var(name: str, vardef: VariableDefinition) -> str: if vardef.type_name is None: - out = f'{name}={vardef.value}' + out = f'{name}={vardef.value},' else: out = f'{name}={vardef.type_name}({vardef.value}),' if vardef.comment is not None: From d87e076ce3e73da00531893ed22c73ca25d3b7f7 Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 1 Jun 2021 17:05:12 +0100 Subject: [PATCH 20/22] Minor Altair cosmetic polishing --- specs/altair/beacon-chain.md | 41 ++++++++++++------------------------ 1 file changed, 14 insertions(+), 27 deletions(-) diff --git a/specs/altair/beacon-chain.md b/specs/altair/beacon-chain.md index 7412a8490..6597751da 100644 --- a/specs/altair/beacon-chain.md +++ b/specs/altair/beacon-chain.md @@ -137,8 +137,8 @@ This patch updates a few configuration values to move penalty parameters closer | Name | Value | Description | | - | - | - | -| `INACTIVITY_SCORE_BIAS` | `uint64(4)` | score points per inactive epoch | -| `INACTIVITY_SCORE_RECOVERY_RATE` | `uint64(16)` | score points per recovering epoch | +| `INACTIVITY_SCORE_BIAS` | `uint64(2**2)` (= 4) | score points per inactive epoch | +| `INACTIVITY_SCORE_RECOVERY_RATE` | `uint64(2**4)` (= 16) | score points per leak-free epoch | ## Containers @@ -157,8 +157,7 @@ class BeaconBlockBody(Container): attestations: List[Attestation, MAX_ATTESTATIONS] deposits: List[Deposit, MAX_DEPOSITS] voluntary_exits: List[SignedVoluntaryExit, MAX_VOLUNTARY_EXITS] - # [New in Altair] - sync_aggregate: SyncAggregate + sync_aggregate: SyncAggregate # [New in Altair] ``` #### `BeaconState` @@ -266,10 +265,7 @@ def has_flag(flags: ParticipationFlags, flag_index: int) -> bool: ```python def get_next_sync_committee_indices(state: BeaconState) -> Sequence[ValidatorIndex]: """ - Return the sequence of sync committee indices (which may include duplicate indices) - for the next sync committee, given a ``state`` at a sync committee period boundary. - - Note: Committee can contain duplicate indices for small validator sets (< SYNC_COMMITTEE_SIZE + 128) + Return the sync committee indices, with possible duplicates, for the next sync committee. """ epoch = Epoch(get_current_epoch(state) + 1) @@ -292,21 +288,12 @@ def get_next_sync_committee_indices(state: BeaconState) -> Sequence[ValidatorInd #### `get_next_sync_committee` +*Note*: The function `get_next_sync_committee` should only be called at sync committee period boundaries. + ```python def get_next_sync_committee(state: BeaconState) -> SyncCommittee: """ - Return the *next* sync committee for a given ``state``. - - ``SyncCommittee`` contains an aggregate pubkey that enables - resource-constrained clients to save some computation when verifying - the sync committee's signature. - - ``SyncCommittee`` can also contain duplicate pubkeys, when ``get_next_sync_committee_indices`` - returns duplicate indices. Implementations must take care when handling - optimizations relating to aggregation and verification in the presence of duplicates. - - Note: This function should only be called at sync committee period boundaries by ``process_sync_committee_updates`` - as ``get_next_sync_committee_indices`` is not stable within a given period. + Return the next sync committee, with possible pubkey duplicates. """ indices = get_next_sync_committee_indices(state) pubkeys = [state.validators[index].pubkey for index in indices] @@ -325,14 +312,12 @@ def get_base_reward_per_increment(state: BeaconState) -> Gwei: *Note*: The function `get_base_reward` is modified with the removal of `BASE_REWARDS_PER_EPOCH` and the use of increment based accounting. +*Note*: On average an optimally performing validator earns one base reward per epoch. + ```python def get_base_reward(state: BeaconState, index: ValidatorIndex) -> Gwei: """ Return the base reward for the validator defined by ``index`` with respect to the current ``state``. - - Note: An optimally performing validator can earn one base reward per epoch over a long time horizon. - This takes into account both per-epoch (e.g. attestation) and intermittent duties (e.g. block proposal - and sync committees). """ increments = state.validators[index].effective_balance // EFFECTIVE_BALANCE_INCREMENT return Gwei(increments * get_base_reward_per_increment(state)) @@ -559,6 +544,8 @@ def process_deposit(state: BeaconState, deposit: Deposit) -> None: #### Sync committee processing +*Note*: The function `process_sync_committee` is new. + ```python def process_sync_committee(state: BeaconState, aggregate: SyncAggregate) -> None: # Verify sync committee aggregate signature signing over the previous slot block root @@ -627,17 +614,17 @@ def process_justification_and_finalization(state: BeaconState) -> None: ```python def process_inactivity_updates(state: BeaconState) -> None: - # Score updates based on previous epoch participation, skip genesis epoch + # Skip the genesis epoch as score updates are based on the previous epoch participation if get_current_epoch(state) == GENESIS_EPOCH: return for index in get_eligible_validator_indices(state): - # Increase inactivity score of inactive validators + # Increase the inactivity score of inactive validators if index in get_unslashed_participating_indices(state, TIMELY_TARGET_FLAG_INDEX, get_previous_epoch(state)): state.inactivity_scores[index] -= min(1, state.inactivity_scores[index]) else: state.inactivity_scores[index] += INACTIVITY_SCORE_BIAS - # Decrease the score of all validators for forgiveness when not during a leak + # Decrease the inactivity score of all eligible validators during a leak-free epoch if not is_in_inactivity_leak(state): state.inactivity_scores[index] -= min(INACTIVITY_SCORE_RECOVERY_RATE, state.inactivity_scores[index]) ``` From 65f48178b7f2848d0dedea90d3a028b8d22b255d Mon Sep 17 00:00:00 2001 From: terence tsao Date: Wed, 2 Jun 2021 14:41:24 -0700 Subject: [PATCH 21/22] Include block_root in SyncCommitteeSignature --- specs/altair/validator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/validator.md b/specs/altair/validator.md index 3b3362b22..7469552fb 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -282,7 +282,7 @@ def get_sync_committee_signature(state: BeaconState, signing_root = compute_signing_root(block_root, domain) signature = bls.Sign(privkey, signing_root) - return SyncCommitteeSignature(slot=state.slot, validator_index=validator_index, signature=signature) + return SyncCommitteeSignature(slot=state.slot, beacon_block_root=block_root, validator_index=validator_index, signature=signature) ``` ##### Broadcast sync committee signature From 45a71eb2670e3ba74ca48586e397240ff0a14c10 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 2 Jun 2021 15:58:51 -0600 Subject: [PATCH 22/22] line length lint --- specs/altair/validator.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/specs/altair/validator.md b/specs/altair/validator.md index 7469552fb..0ba776d5e 100644 --- a/specs/altair/validator.md +++ b/specs/altair/validator.md @@ -282,7 +282,12 @@ def get_sync_committee_signature(state: BeaconState, signing_root = compute_signing_root(block_root, domain) signature = bls.Sign(privkey, signing_root) - return SyncCommitteeSignature(slot=state.slot, beacon_block_root=block_root, validator_index=validator_index, signature=signature) + return SyncCommitteeSignature( + slot=state.slot, + beacon_block_root=block_root, + validator_index=validator_index, + signature=signature, + ) ``` ##### Broadcast sync committee signature