From 96352726bbe3267a0287d6ad187d44fd70a6bf7f Mon Sep 17 00:00:00 2001 From: terence tsao Date: Sat, 22 May 2021 10:31:10 -0700 Subject: [PATCH 01/14] Sharding p2p: minor typo fixes --- specs/sharding/p2p-interface.md | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/specs/sharding/p2p-interface.md b/specs/sharding/p2p-interface.md index 47ed52970..1fef7c60f 100644 --- a/specs/sharding/p2p-interface.md +++ b/specs/sharding/p2p-interface.md @@ -17,9 +17,11 @@ - [SignedShardBlob](#signedshardblob) - [Gossip domain](#gossip-domain) - [Topics and messages](#topics-and-messages) - - [Shard blobs: `shard_blob_{subnet_id}`](#shard-blobs-shard_blob_subnet_id) - - [Shard header: `shard_header`](#shard-header-shard_header) - - [Shard proposer slashing: `shard_proposer_slashing`](#shard-proposer-slashing-shard_proposer_slashing) + - [Shard blob subnets](#shard-blob-subnets) + - [`shard_blob_{subnet_id}`](#shard_blob_subnet_id) + - [Global topics](#global-topics) + - [`shard_blob_header`](#shard_blob_header) + - [`shard_proposer_slashing`](#shard_proposer_slashing) @@ -88,12 +90,16 @@ Following the same scheme as the [Phase0 gossip topics](../phase0/p2p-interface. | Name | Message Type | |----------------------------------|---------------------------| | `shard_blob_{subnet_id}` | `SignedShardBlob` | -| `shard_header` | `SignedShardHeader` | +| `shard_blob_header` | `SignedShardBlobHeader` | | `shard_proposer_slashing` | `ShardProposerSlashing` | The [DAS network specification](./das-p2p.md) defines additional topics. -#### Shard blobs: `shard_blob_{subnet_id}` +#### Shard blob subnets + +Shard blob subnets are used to propagate shard blobs to subsections of the network. + +##### `shard_blob_{subnet_id}` Shard block data, in the form of a `SignedShardBlob` is published to the `shard_blob_{subnet_id}` subnets. @@ -129,19 +135,23 @@ The following validations MUST pass before forwarding the `signed_blob` (with in the block MAY be queued for later processing while proposers for the blob's branch are calculated -- in such a case _do not_ `REJECT`, instead `IGNORE` this message. +#### Global topics -#### Shard header: `shard_header` +There are two additional global topics for Sharding, one is used to propagate shard blob headers (`shard_blob_header`) to +all nodes on the network. Another one is used to propagate validator message (`shard_proposer_slashing`). -Shard header data, in the form of a `SignedShardBlobHeader` is published to the global `shard_header` subnet. +##### `shard_blob_header` -The following validations MUST pass before forwarding the `signed_shard_header` (with inner `message` as `header`) on the network. +Shard header data, in the form of a `SignedShardBlobHeader` is published to the global `shard_blob_header` subnet. + +The following validations MUST pass before forwarding the `signed_shard_blob_header` (with inner `message` as `header`) on the network. - _[IGNORE]_ The `header` is not from a future slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- i.e. validate that `header.slot <= current_slot` (a client MAY queue future headers for processing at the appropriate slot). - _[IGNORE]_ The `header` is new enough to be still be processed -- i.e. validate that `compute_epoch_at_slot(header.slot) >= get_previous_epoch(state)` - _[IGNORE]_ The header is the first header with valid signature received for the `(header.proposer_index, header.slot, header.shard)` combination. -- _[REJECT]_ The proposer signature, `signed_shard_header.signature`, is valid with respect to the `proposer_index` pubkey. +- _[REJECT]_ The proposer signature, `signed_shard_blob_header.signature`, is valid with respect to the `proposer_index` pubkey. - _[REJECT]_ The header is proposed by the expected `proposer_index` for the block's slot in the context of the current shuffling (defined by `header.body_summary.beacon_block_root`/`slot`). If the `proposer_index` cannot immediately be verified against the expected shuffling, @@ -149,7 +159,7 @@ The following validations MUST pass before forwarding the `signed_shard_header` in such a case _do not_ `REJECT`, instead `IGNORE` this message. -#### Shard proposer slashing: `shard_proposer_slashing` +##### `shard_proposer_slashing` Shard proposer slashings, in the form of `ShardProposerSlashing`, are published to the global `shard_proposer_slashing` topic. From 4a5947d49e5f952b5b7b8952125bcb944978310b Mon Sep 17 00:00:00 2001 From: terence tsao Date: Mon, 24 May 2021 06:49:20 -0700 Subject: [PATCH 02/14] Proto's suggestion --- specs/sharding/p2p-interface.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/specs/sharding/p2p-interface.md b/specs/sharding/p2p-interface.md index 1fef7c60f..65393bc0e 100644 --- a/specs/sharding/p2p-interface.md +++ b/specs/sharding/p2p-interface.md @@ -20,7 +20,7 @@ - [Shard blob subnets](#shard-blob-subnets) - [`shard_blob_{subnet_id}`](#shard_blob_subnet_id) - [Global topics](#global-topics) - - [`shard_blob_header`](#shard_blob_header) + - [`shard_header`](#shard_header) - [`shard_proposer_slashing`](#shard_proposer_slashing) @@ -30,7 +30,7 @@ ## Introduction The specification of these changes continues in the same format as the [Phase0](../phase0/p2p-interface.md) and -[Altair](../altair/p2p-interface.md) network specifications, and assumes them as pre-requisite. +[Altair](../altair/p2p-interface.md) network specifications, and assumes them as pre-requisite. The adjustments and additions for Shards are outlined in this document. ## Constants @@ -90,7 +90,7 @@ Following the same scheme as the [Phase0 gossip topics](../phase0/p2p-interface. | Name | Message Type | |----------------------------------|---------------------------| | `shard_blob_{subnet_id}` | `SignedShardBlob` | -| `shard_blob_header` | `SignedShardBlobHeader` | +| `shard_header` | `SignedShardBlobHeader` | | `shard_proposer_slashing` | `ShardProposerSlashing` | The [DAS network specification](./das-p2p.md) defines additional topics. @@ -124,7 +124,7 @@ The following validations MUST pass before forwarding the `signed_blob` (with in - _[IGNORE]_ The `blob` is new enough to be still be processed -- i.e. validate that `compute_epoch_at_slot(blob.slot) >= get_previous_epoch(state)` - _[REJECT]_ The shard blob is for the correct subnet -- - i.e. `compute_subnet_for_shard_blob(state, blob.slot, blob.shard) == subnet_id` + i.e. `compute_subnet_for_shard_blob(state, blob.slot, blob.shard) == subnet_id` - _[IGNORE]_ The blob is the first blob with valid signature received for the `(blob.proposer_index, blob.slot, blob.shard)` combination. - _[REJECT]_ As already limited by the SSZ list-limit, it is important the blob is well-formatted and not too large. - _[REJECT]_ The `blob.body.data` MUST NOT contain any point `p >= MODULUS`. Although it is a `uint256`, not the full 256 bit range is valid. @@ -137,12 +137,12 @@ The following validations MUST pass before forwarding the `signed_blob` (with in #### Global topics -There are two additional global topics for Sharding, one is used to propagate shard blob headers (`shard_blob_header`) to +There are two additional global topics for Sharding, one is used to propagate shard blob headers (`shard_header`) to all nodes on the network. Another one is used to propagate validator message (`shard_proposer_slashing`). -##### `shard_blob_header` +##### `shard_header` -Shard header data, in the form of a `SignedShardBlobHeader` is published to the global `shard_blob_header` subnet. +Shard header data, in the form of a `SignedShardBlobHeader` is published to the global `shard_header` subnet. The following validations MUST pass before forwarding the `signed_shard_blob_header` (with inner `message` as `header`) on the network. - _[IGNORE]_ The `header` is not from a future slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- @@ -168,3 +168,4 @@ The following validations MUST pass before forwarding the `shard_proposer_slashi for the proposer with index `proposer_slashing.signed_header_1.message.proposer_index`. The `slot` and `shard` are ignored, there are no per-shard slashings. - _[REJECT]_ All of the conditions within `process_shard_proposer_slashing` pass validation. +- From 814c7696d96bab5e62aaea15cee8cdb39e0de6a3 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Mon, 24 May 2021 06:50:03 -0700 Subject: [PATCH 03/14] shard_blob_header -> shard_header --- specs/sharding/p2p-interface.md | 1 - 1 file changed, 1 deletion(-) diff --git a/specs/sharding/p2p-interface.md b/specs/sharding/p2p-interface.md index 65393bc0e..5acd9735d 100644 --- a/specs/sharding/p2p-interface.md +++ b/specs/sharding/p2p-interface.md @@ -168,4 +168,3 @@ The following validations MUST pass before forwarding the `shard_proposer_slashi for the proposer with index `proposer_slashing.signed_header_1.message.proposer_index`. The `slot` and `shard` are ignored, there are no per-shard slashings. - _[REJECT]_ All of the conditions within `process_shard_proposer_slashing` pass validation. -- From 71d0d453431b21c61e44170d1865d01fc1c21482 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Mon, 24 May 2021 06:57:09 -0700 Subject: [PATCH 04/14] Align table --- specs/sharding/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/sharding/p2p-interface.md b/specs/sharding/p2p-interface.md index 5acd9735d..b7e229e87 100644 --- a/specs/sharding/p2p-interface.md +++ b/specs/sharding/p2p-interface.md @@ -90,7 +90,7 @@ Following the same scheme as the [Phase0 gossip topics](../phase0/p2p-interface. | Name | Message Type | |----------------------------------|---------------------------| | `shard_blob_{subnet_id}` | `SignedShardBlob` | -| `shard_header` | `SignedShardBlobHeader` | +| `shard_header` | `SignedShardBlobHeader` | | `shard_proposer_slashing` | `ShardProposerSlashing` | The [DAS network specification](./das-p2p.md) defines additional topics. From d71c50f6564c1f7a73f09a20557084df9195679c Mon Sep 17 00:00:00 2001 From: protolambda Date: Tue, 25 May 2021 19:46:23 +0200 Subject: [PATCH 05/14] Union type update --- ssz/simple-serialize.md | 39 +++++++++++++++++++++++++++------------ 1 file changed, 27 insertions(+), 12 deletions(-) diff --git a/ssz/simple-serialize.md b/ssz/simple-serialize.md index d97b8ea1c..89a1ebc0b 100644 --- a/ssz/simple-serialize.md +++ b/ssz/simple-serialize.md @@ -20,7 +20,8 @@ - [`null`](#null) - [`Bitvector[N]`](#bitvectorn) - [`Bitlist[N]`](#bitlistn) - - [Vectors, containers, lists, unions](#vectors-containers-lists-unions) + - [Vectors, containers, lists](#vectors-containers-lists) + - [Union](#union) - [Deserialization](#deserialization) - [Merkleization](#merkleization) - [Summaries and expansions](#summaries-and-expansions) @@ -61,7 +62,7 @@ * **bitlist**: ordered variable-length collection of `boolean` values, limited to `N` bits * notation `Bitlist[N]` * **union**: union type containing one of the given subtypes - * notation `Union[type_0, type_1, ...]`, e.g. `union[null, uint64]` + * notation `Union[type_0, type_1, ...]`, e.g. `union[None, uint64, uint32]` *Note*: Both `Vector[boolean, N]` and `Bitvector[N]` are valid, yet distinct due to their different serialization requirements. Similarly, both `List[boolean, N]` and `Bitlist[N]` are valid, yet distinct. Generally `Bitvector[N]`/`Bitlist[N]` are preferred because of their serialization efficiencies. @@ -77,7 +78,6 @@ For convenience we alias: * `byte` to `uint8` (this is a basic type) * `BytesN` and `ByteVector[N]` to `Vector[byte, N]` (this is *not* a basic type) * `ByteList[N]` to `List[byte, N]` -* `null`: `{}` ### Default values Assuming a helper function `default(type)` which returns the default value for `type`, we can recursively define the default value for all types. @@ -101,7 +101,7 @@ An SSZ object is called zeroed (and thus, `is_zero(object)` returns true) if it - Empty vector types (`Vector[type, 0]`, `Bitvector[0]`) are illegal. - Containers with no fields are illegal. -- The `null` type is only legal as the first type in a union subtype (i.e. with type index zero). +- The `None` type option in a `Union` type is only legal as the first option (i.e. with index zero). ## Serialization @@ -150,7 +150,7 @@ array[len(value) // 8] |= 1 << (len(value) % 8) return bytes(array) ``` -### Vectors, containers, lists, unions +### Vectors, containers, lists ```python # Recursively serialize @@ -170,14 +170,26 @@ fixed_parts = [part if part != None else variable_offsets[i] for i, part in enum return b"".join(fixed_parts + variable_parts) ``` -If `value` is a union type: +### Union -Define value as an object that has properties `value.value` with the contained value, and `value.type_index` which indexes the type. +A `value` as `Union[T...]` type has properties `value.value` with the contained value, and `value.selector` which indexes the selected `Union` type option `T`. + +A `Union`: +- May have multiple selectors with the same type. +- Should not use selectors above 127 (i.e. highest bit is set), these are reserved for backwards compatible extensions. +- Must have at least 1 type option. +- May have `None` as first type option, i.e. `selector == 0` +- Must have at least 2 type options if the first is `None` +- Is always considered a variable-length type, even if all type options have an equal fixed-length. ```python -serialized_bytes = serialize(value.value) -serialized_type_index = value.type_index.to_bytes(BYTES_PER_LENGTH_OFFSET, "little") -return serialized_type_index + serialized_bytes +if value.value is None: + assert value.selector == 0 + return b"\x00" +else: + serialized_bytes = serialize(value.value) + serialized_selector_index = value.selector.to_bytes(1, "little") + return serialized_selector_index + serialized_bytes ``` ## Deserialization @@ -191,12 +203,14 @@ Deserialization can be implemented using a recursive algorithm. The deserializat * The size of each object in the vector/list can be inferred from the difference of two offsets. To get the size of the last object, the total number of bytes has to be known (it is not generally possible to deserialize an SSZ object of unknown length) * Containers follow the same principles as vectors, with the difference that there may be fixed-size objects in a container as well. This means the `fixed_parts` data will contain offsets as well as fixed-size objects. * In the case of bitlists, the length in bits cannot be uniquely inferred from the number of bytes in the object. Because of this, they have a bit at the end that is always set. This bit has to be used to infer the size of the bitlist in bits. +* The first byte of the deserialization scope is deserialized as type selector, the remainder of the scope is deserialized as the selected type. Note that deserialization requires hardening against invalid inputs. A non-exhaustive list: - Offsets: out of order, out of range, mismatching minimum element size. - Scope: Extra unused bytes, not aligned with element size. - More elements than a list limit allows. Part of enforcing consensus. +- An out-of-bounds selected index in an `Union` Efficient algorithms for computing this object can be found in [the implementations](#implementations). @@ -227,7 +241,7 @@ We first define helper functions: - If `1` chunk: the root is the chunk itself. - If `> 1` chunks: merkleize as binary tree. * `mix_in_length`: Given a Merkle root `root` and a length `length` (`"uint256"` little-endian serialization) return `hash(root + length)`. -* `mix_in_type`: Given a Merkle root `root` and a type_index `type_index` (`"uint256"` little-endian serialization) return `hash(root + type_index)`. +* `mix_in_selector`: Given a Merkle root `root` and a type selector `selector` (`"uint256"` little-endian serialization) return `hash(root + selector)`. We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: @@ -237,7 +251,8 @@ We now define Merkleization `hash_tree_root(value)` of an object `value` recursi * `mix_in_length(merkleize(pack_bits(value), limit=chunk_count(type)), len(value))` if `value` is a bitlist. * `merkleize([hash_tree_root(element) for element in value])` if `value` is a vector of composite objects or a container. * `mix_in_length(merkleize([hash_tree_root(element) for element in value], limit=chunk_count(type)), len(value))` if `value` is a list of composite objects. -* `mix_in_type(merkleize(value.value), value.type_index)` if `value` is of union type. +* `mix_in_selector(hash_tree_root(value.value), value.selector)` if `value` is of union type, and `value.value` is not `None` +* `mix_in_selector(Bytes32(), 0)` if `value` is of union type, and `value.value` is `None` ## Summaries and expansions From cb008f2c1a26b5b1ed13925b65763b921cd127a3 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 26 May 2021 02:40:22 +0800 Subject: [PATCH 06/14] Fix _get_sync_committee_signature helper. Should have used `target_slot` --- .../test/altair/unittests/validator/test_validator.py | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py index dfe90b5b5..048e5f43d 100644 --- a/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py +++ b/tests/core/pyspec/eth2spec/test/altair/unittests/validator/test_validator.py @@ -4,7 +4,7 @@ from eth2spec.utils.ssz.ssz_typing import Bitvector from eth2spec.test.helpers.block import build_empty_block from eth2spec.test.helpers.keys import pubkey_to_privkey from eth2spec.test.helpers.state import transition_to -from eth2spec.utils import bls +from eth2spec.test.helpers.sync_committee import compute_sync_committee_signature from eth2spec.utils.bls import only_with_bls from eth2spec.test.context import ( with_altair_and_later, @@ -85,12 +85,9 @@ def _get_sync_committee_signature( pubkey = state.current_sync_committee.pubkeys[sync_committee_index] privkey = pubkey_to_privkey[pubkey] - domain = spec.get_domain( - state, - spec.DOMAIN_SYNC_COMMITTEE, + return compute_sync_committee_signature( + spec, state, target_slot, privkey, block_root=target_block_root ) - signing_data = spec.compute_signing_root(target_block_root, domain) - return bls.Sign(privkey, spec.hash_tree_root(signing_data)) @only_with_bls() From 1360860d1e7b2701d6c413186496dc43eecf32b6 Mon Sep 17 00:00:00 2001 From: protolambda Date: Thu, 27 May 2021 03:57:59 +0200 Subject: [PATCH 07/14] add/update comments on shard blob/header/reference body field --- specs/sharding/beacon-chain.md | 3 ++- specs/sharding/p2p-interface.md | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/specs/sharding/beacon-chain.md b/specs/sharding/beacon-chain.md index 5522e044d..a15a002e4 100644 --- a/specs/sharding/beacon-chain.md +++ b/specs/sharding/beacon-chain.md @@ -216,6 +216,7 @@ class ShardBlobHeader(Container): # Slot and shard that this header is intended for slot: Slot shard: Shard + # SSZ-summary of ShardBlobBody body_summary: ShardBlobBodySummary # Proposer of the shard-blob proposer_index: ValidatorIndex @@ -253,7 +254,7 @@ class ShardBlobReference(Container): # Slot and shard that this reference is intended for slot: Slot shard: Shard - # Hash-tree-root of commitment data + # Hash-tree-root of ShardBlobBody body_root: Root # Proposer of the shard-blob proposer_index: ValidatorIndex diff --git a/specs/sharding/p2p-interface.md b/specs/sharding/p2p-interface.md index 47ed52970..39da8166d 100644 --- a/specs/sharding/p2p-interface.md +++ b/specs/sharding/p2p-interface.md @@ -64,6 +64,7 @@ class ShardBlob(Container): # Slot and shard that this blob is intended for slot: Slot shard: Shard + # Shard data with related commitments and beacon anchor body: ShardBlobBody # Proposer of the shard-blob proposer_index: ValidatorIndex From 103d029a1abf054ead8eaa9c7d4c77facc10ace2 Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Thu, 27 May 2021 15:28:51 +1000 Subject: [PATCH 08/14] Add clock disparity tolerance for sync subnets --- specs/altair/p2p-interface.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index 6f250b57e..c66ed015c 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -120,7 +120,7 @@ def get_sync_subcommittee_pubkeys(state: BeaconState, subcommittee_index: uint64 return sync_committee.pubkeys[i:i + sync_subcommittee_size] ``` -- _[IGNORE]_ The contribution's slot is for the current slot, i.e. `contribution.slot == current_slot`. +- _[IGNORE]_ The contribution's slot is for the current slot (with a MAXIMUM_GOSSIP_CLOCK_DISPARITY allowance), i.e. `contribution.slot == current_slot` . - _[IGNORE]_ The block being signed over (`contribution.beacon_block_root`) has been seen (via both gossip and non-gossip sources). - _[REJECT]_ The subcommittee index is in the allowed range, i.e. `contribution.subcommittee_index < SYNC_COMMITTEE_SUBNET_COUNT`. - _[IGNORE]_ The sync committee contribution is the first valid contribution received for the aggregator with index `contribution_and_proof.aggregator_index` for the slot `contribution.slot` and subcommittee index `contribution.subcommittee_index`. @@ -141,7 +141,7 @@ The `sync_committee_{subnet_id}` topics are used to propagate unaggregated sync The following validations MUST pass before forwarding the `sync_committee_signature` on the network: -- _[IGNORE]_ The signature's slot is for the current slot, i.e. `sync_committee_signature.slot == current_slot`. +- _[IGNORE]_ The signature's slot is for the current slot (with a MAXIMUM_GOSSIP_CLOCK_DISPARITY allowance), i.e. `sync_committee_signature.slot == current_slot`. - _[IGNORE]_ The block being signed over (`sync_committee_signature.beacon_block_root`) has been seen (via both gossip and non-gossip sources). - _[IGNORE]_ There has been no other valid sync committee signature for the declared `slot` for the validator referenced by `sync_committee_signature.validator_index`. - _[REJECT]_ The `subnet_id` is valid for the given validator, i.e. `subnet_id in compute_subnets_for_sync_committee(state, sync_committee_signature.validator_index)`. From 48f989070db324bf8d37ff231cfb38766de353b8 Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Thu, 27 May 2021 15:30:44 +1000 Subject: [PATCH 09/14] Remove naughty space --- specs/altair/p2p-interface.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/altair/p2p-interface.md b/specs/altair/p2p-interface.md index c66ed015c..1f7d0f76c 100644 --- a/specs/altair/p2p-interface.md +++ b/specs/altair/p2p-interface.md @@ -120,7 +120,7 @@ def get_sync_subcommittee_pubkeys(state: BeaconState, subcommittee_index: uint64 return sync_committee.pubkeys[i:i + sync_subcommittee_size] ``` -- _[IGNORE]_ The contribution's slot is for the current slot (with a MAXIMUM_GOSSIP_CLOCK_DISPARITY allowance), i.e. `contribution.slot == current_slot` . +- _[IGNORE]_ The contribution's slot is for the current slot (with a MAXIMUM_GOSSIP_CLOCK_DISPARITY allowance), i.e. `contribution.slot == current_slot`. - _[IGNORE]_ The block being signed over (`contribution.beacon_block_root`) has been seen (via both gossip and non-gossip sources). - _[REJECT]_ The subcommittee index is in the allowed range, i.e. `contribution.subcommittee_index < SYNC_COMMITTEE_SUBNET_COUNT`. - _[IGNORE]_ The sync committee contribution is the first valid contribution received for the aggregator with index `contribution_and_proof.aggregator_index` for the slot `contribution.slot` and subcommittee index `contribution.subcommittee_index`. From d11586122fea6ba00b67392191b9344a474c3749 Mon Sep 17 00:00:00 2001 From: protolambda Date: Fri, 28 May 2021 01:02:08 +0200 Subject: [PATCH 10/14] update remerkleable, union support --- setup.py | 2 +- tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 99e501d9f..cd2006f71 100644 --- a/setup.py +++ b/setup.py @@ -1017,7 +1017,7 @@ setup( "py_ecc==5.2.0", "milagro_bls_binding==1.6.3", "dataclasses==0.6", - "remerkleable==0.1.19", + "remerkleable==0.1.20", RUAMEL_YAML_VERSION, "lru-dict==1.1.6", MARKO_VERSION, diff --git a/tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py b/tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py index 9b18f8bda..5a1b61d0b 100644 --- a/tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py +++ b/tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py @@ -2,6 +2,7 @@ # Ignore linter: This module makes importing SSZ types easy, and hides away the underlying library from the spec. from remerkleable.complex import Container, Vector, List +from remerkleable.union import Union from remerkleable.basic import boolean, bit, uint, byte, uint8, uint16, uint32, uint64, uint128, uint256 from remerkleable.bitfields import Bitvector, Bitlist from remerkleable.byte_arrays import ByteVector, Bytes1, Bytes4, Bytes8, Bytes32, Bytes48, Bytes96, ByteList From ef9b7125c2efe9d786d12501422bbfad357637b4 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 28 May 2021 12:27:19 -0700 Subject: [PATCH 11/14] whitespace --- ssz/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ssz/simple-serialize.md b/ssz/simple-serialize.md index 89a1ebc0b..2ae8d9bdd 100644 --- a/ssz/simple-serialize.md +++ b/ssz/simple-serialize.md @@ -175,7 +175,7 @@ return b"".join(fixed_parts + variable_parts) A `value` as `Union[T...]` type has properties `value.value` with the contained value, and `value.selector` which indexes the selected `Union` type option `T`. A `Union`: -- May have multiple selectors with the same type. +- May have multiple selectors with the same type. - Should not use selectors above 127 (i.e. highest bit is set), these are reserved for backwards compatible extensions. - Must have at least 1 type option. - May have `None` as first type option, i.e. `selector == 0` @@ -194,7 +194,7 @@ else: ## Deserialization -Because serialization is an injective function (i.e. two distinct objects of the same type will serialize to different values) any bytestring has at most one object it could deserialize to. +Because serialization is an injective function (i.e. two distinct objects of the same type will serialize to different values) any bytestring has at most one object it could deserialize to. Deserialization can be implemented using a recursive algorithm. The deserialization of basic objects is easy, and from there we can find a simple recursive algorithm for all fixed-size objects. For variable-size objects we have to do one of the following depending on what kind of object it is: From b763416a6b486b4fe0c82c9a9ceb07b0047d3034 Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 28 May 2021 12:27:29 -0700 Subject: [PATCH 12/14] remove unnecessary defn of `null` --- ssz/simple-serialize.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/ssz/simple-serialize.md b/ssz/simple-serialize.md index 2ae8d9bdd..cc03cec09 100644 --- a/ssz/simple-serialize.md +++ b/ssz/simple-serialize.md @@ -17,7 +17,6 @@ - [Serialization](#serialization) - [`uintN`](#uintn) - [`boolean`](#boolean) - - [`null`](#null) - [`Bitvector[N]`](#bitvectorn) - [`Bitlist[N]`](#bitlistn) - [Vectors, containers, lists](#vectors-containers-lists) @@ -123,12 +122,6 @@ assert value in (True, False) return b"\x01" if value is True else b"\x00" ``` -### `null` - -```python -return b"" -``` - ### `Bitvector[N]` ```python From fa09d896484bbe240334fa21ffaa454bafe5842e Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Fri, 28 May 2021 18:13:22 -0700 Subject: [PATCH 13/14] Update simple-serialize.md --- ssz/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ssz/simple-serialize.md b/ssz/simple-serialize.md index cc03cec09..4ef64f2f2 100644 --- a/ssz/simple-serialize.md +++ b/ssz/simple-serialize.md @@ -196,7 +196,7 @@ Deserialization can be implemented using a recursive algorithm. The deserializat * The size of each object in the vector/list can be inferred from the difference of two offsets. To get the size of the last object, the total number of bytes has to be known (it is not generally possible to deserialize an SSZ object of unknown length) * Containers follow the same principles as vectors, with the difference that there may be fixed-size objects in a container as well. This means the `fixed_parts` data will contain offsets as well as fixed-size objects. * In the case of bitlists, the length in bits cannot be uniquely inferred from the number of bytes in the object. Because of this, they have a bit at the end that is always set. This bit has to be used to infer the size of the bitlist in bits. -* The first byte of the deserialization scope is deserialized as type selector, the remainder of the scope is deserialized as the selected type. +* In the case of unions, the first byte of the deserialization scope is deserialized as type selector, the remainder of the scope is deserialized as the selected type. Note that deserialization requires hardening against invalid inputs. A non-exhaustive list: From 8f005c18e59a749eca83aa16ff0b2a36fd4e13ea Mon Sep 17 00:00:00 2001 From: protolambda Date: Mon, 31 May 2021 15:20:50 +0200 Subject: [PATCH 14/14] when an untyped var is not the last config var, it needs a comma --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index cd2006f71..6e6349b6b 100644 --- a/setup.py +++ b/setup.py @@ -596,7 +596,7 @@ def objects_to_spec(preset_name: str, def format_config_var(name: str, vardef: VariableDefinition) -> str: if vardef.type_name is None: - out = f'{name}={vardef.value}' + out = f'{name}={vardef.value},' else: out = f'{name}={vardef.type_name}({vardef.value}),' if vardef.comment is not None: