From ad07649ec3aa56737191052385aca1c2a482747c Mon Sep 17 00:00:00 2001 From: Piper Merriam Date: Mon, 18 Mar 2019 08:57:02 -0600 Subject: [PATCH 01/28] Update spec to use SOS style offset layout for variable size values. --- specs/simple-serialize.md | 61 ++++++++++++++++++++++++++++++--------- 1 file changed, 47 insertions(+), 14 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 378a1a7cb..ef442042b 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -33,6 +33,8 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ### Composite types +Composite types are limited to a maximum of `2**32 - 1` elements. + * **container**: ordered heterogenous collection of values * key-pair curly bracket notation `{}`, e.g. `{"foo": "uint64", "bar": "bool"}` * **vector**: ordered fixed-length homogeneous collection of values @@ -54,43 +56,74 @@ For convenience we alias: We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a bytestring of type `"bytes"`. -*Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signed_root`, etc.) objects implicitly carry their type. +> *Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signed_root`, etc.) objects implicitly carry their type. -### `"uintN"` +### Basic Types + +For basic types the `serialize` function is defined as follows. + +#### `"uintN"` ```python assert N in [8, 16, 32, 64, 128, 256] return value.to_bytes(N // 8, "little") ``` -### `"bool"` +#### `"bool"` ```python assert value in (True, False) return b"\x01" if value is True else b"\x00" ``` -### Vectors, containers, lists +### Composite Types (Vectors, Containers and Lists) + +The serialized representation of composite types is comprised of three binary segments. + +* The first segment contains the concatenation of the serialized representation of **only** the *fixed size* types: + - This section is empty in the case of a purely *variable-size* type. +* The second segment contains the concatenation of the `uint32` serialized offsets where the serialized representation of the *variable sized* types can be found in the third section. + - This section is empty in the case of a purely *fixed size* type. + - The first offset will always be the length of the serialized *fixed size* elements + the length of all of the serialized offsets. + - Subsequent offsets are the first offset + the combined lengths of the serialized representations for all of the previous *variable size* elements. +* The third segment contains the concatenation of the serialized representations of **only** the *variable size* types. + - This section is empty in the case of a purely *fixed size* type. + + +#### `"vector"`, `"container"` and `"list"` + +For conmposite types the `serialize` function is defined as follows. -If `value` is fixed-size: ```python -return "".join([serialize(element) for element in value]) +# section 1 +section_1 = ''.join([serialize(element) for element in value if is_fixed_size(element)]) + +# section 3 +section_3_parts = [serialize(element) for element in value if is_variable_size(element)] +section_3 ''.join(section_3_parts) + +# section 2 +section_1_length = len(section_1) +section_2_length = 4 * len(section_3_parts) +section_3_base_offset = section_1_length + section_2_length +section_3_lengths = [len(part) for part in section_3_parts] +section_3_offsets = [ + (section_3_base_offset + sum(section_3_lengths[:index])) + for index in range(len(section_3_parts)) +] +assert all(offset < 2**32 for offset in section_3_offsets) +section_2 = ''.join([serialize(offset) for offset in section_3_offsets]) + +return ''.join([section_1, section_2, section_3]) ``` -If `value` is variable-size: - -```python -serialized_bytes = "".join([serialize(element) for element in value]) -assert len(serialized_bytes) < 2**(8 * BYTES_PER_LENGTH_PREFIX) -serialized_length = len(serialized_bytes).to_bytes(BYTES_PER_LENGTH_PREFIX, "little") -return serialized_length + serialized_bytes -``` ## Deserialization Because serialization is an injective function (i.e. two distinct objects of the same type will serialize to different values) any bytestring has at most one object it could deserialize to. Efficient algorithms for computing this object can be found in [the implementations](#implementations). + ## Merkleization We first define helper functions: From 4a0459a087ecd3cef8ba819a82bcb2c86188ca76 Mon Sep 17 00:00:00 2001 From: Piper Merriam Date: Mon, 18 Mar 2019 10:20:58 -0600 Subject: [PATCH 02/28] PR feedback --- specs/simple-serialize.md | 55 ++++++++++++++++++--------------------- 1 file changed, 26 insertions(+), 29 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index ef442042b..b695956d2 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -12,7 +12,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Serialization](#serialization) - [`"uintN"`](#uintn) - [`"bool"`](#bool) - - [Vectors, containers, lists](#vectors-containers-lists) + - [Vectors, containers, lists](#composite-types-vectors-containers-and-lists) - [Deserialization](#deserialization) - [Merkleization](#merkleization) - [Self-signed containers](#self-signed-containers) @@ -33,8 +33,6 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ### Composite types -Composite types are limited to a maximum of `2**32 - 1` elements. - * **container**: ordered heterogenous collection of values * key-pair curly bracket notation `{}`, e.g. `{"foo": "uint64", "bar": "bool"}` * **vector**: ordered fixed-length homogeneous collection of values @@ -78,15 +76,12 @@ return b"\x01" if value is True else b"\x00" ### Composite Types (Vectors, Containers and Lists) -The serialized representation of composite types is comprised of three binary segments. +The serialized representation of composite types is comprised of two binary segments. -* The first segment contains the concatenation of the serialized representation of **only** the *fixed size* types: - - This section is empty in the case of a purely *variable-size* type. -* The second segment contains the concatenation of the `uint32` serialized offsets where the serialized representation of the *variable sized* types can be found in the third section. - - This section is empty in the case of a purely *fixed size* type. - - The first offset will always be the length of the serialized *fixed size* elements + the length of all of the serialized offsets. - - Subsequent offsets are the first offset + the combined lengths of the serialized representations for all of the previous *variable size* elements. -* The third segment contains the concatenation of the serialized representations of **only** the *variable size* types. +* The first segment is *fixed size* for all types, containing the concatenation of *either* + - The serialized representation of value for each of the *fixed size* types + - The `"uint32"` serialized offset where the serialized representation of the *variable sized* type is located in the second section. +* The second segment contains the concatenation of the serialized representations of **only** the *variable size* types. - This section is empty in the case of a purely *fixed size* type. @@ -94,28 +89,30 @@ The serialized representation of composite types is comprised of three binary se For conmposite types the `serialize` function is defined as follows. +> *Note*: The `collate` function combines the serialized *fixed size* values +> and the serialized offsets into a single array correctly ordered with respect +> to the original element types. The implementation of this logic is not +> included in this example for simplicity. + ```python -# section 1 -section_1 = ''.join([serialize(element) for element in value if is_fixed_size(element)]) - -# section 3 -section_3_parts = [serialize(element) for element in value if is_variable_size(element)] -section_3 ''.join(section_3_parts) - # section 2 -section_1_length = len(section_1) -section_2_length = 4 * len(section_3_parts) -section_3_base_offset = section_1_length + section_2_length -section_3_lengths = [len(part) for part in section_3_parts] -section_3_offsets = [ - (section_3_base_offset + sum(section_3_lengths[:index])) - for index in range(len(section_3_parts)) -] -assert all(offset < 2**32 for offset in section_3_offsets) -section_2 = ''.join([serialize(offset) for offset in section_3_offsets]) +section_2_parts = [serialize(element) for element in value if is_variable_size(element)] +section_2_lengths = [len(part) for part in section_2_parts] +section_2 ''.join(section_2_parts) -return ''.join([section_1, section_2, section_3]) +# section 1 +section_1_fixed_parts = [serialize(element) for element in value if is_fixed_size(element)] +section_1_length = sum(len(part) for part in section_1_fixed_parts) + 4 * len(section_2_parts) +section_1_offsets = [ + section_1_length + sum(section_2_lengths[:index]) + for index in range(len(section_2_parts)) +] +assert all(offset < 2**32 for offset in section_1_offsets) +section_1_parts = collate(section_1_fixed_parts, section_1_offsets) +section_1 = ''.join(section_1_parts) + +return ''.join([section_1, section_2]) ``` From 605028bbda38b7af7191cad1018ba121f240cbd7 Mon Sep 17 00:00:00 2001 From: Piper Merriam Date: Mon, 18 Mar 2019 16:28:33 -0600 Subject: [PATCH 03/28] more precise definitions for and and expand code example for how sections are created --- specs/simple-serialize.md | 42 +++++++++++++++++++++++++++++---------- 1 file changed, 32 insertions(+), 10 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index b695956d2..b85a20656 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -62,6 +62,8 @@ For basic types the `serialize` function is defined as follows. #### `"uintN"` +A byte string of width `N // 8` containing the little-endian encode integer. + ```python assert N in [8, 16, 32, 64, 128, 256] return value.to_bytes(N // 8, "little") @@ -69,6 +71,9 @@ return value.to_bytes(N // 8, "little") #### `"bool"` +* The byte `\x00` **if** the value is `False` +* The byte `\x01` **if** the value is `True` + ```python assert value in (True, False) return b"\x01" if value is True else b"\x00" @@ -87,29 +92,46 @@ The serialized representation of composite types is comprised of two binary segm #### `"vector"`, `"container"` and `"list"` -For conmposite types the `serialize` function is defined as follows. - -> *Note*: The `collate` function combines the serialized *fixed size* values -> and the serialized offsets into a single array correctly ordered with respect -> to the original element types. The implementation of this logic is not -> included in this example for simplicity. - +An implementation of the `serialize` function for `"Vector"`, `"Container"` and +`"List"` types would take the following form. ```python -# section 2 +# The second section is just the concatenation of the serialized *variable size* elements section_2_parts = [serialize(element) for element in value if is_variable_size(element)] section_2_lengths = [len(part) for part in section_2_parts] section_2 ''.join(section_2_parts) -# section 1 +# Serialize the *fixed size* elements section_1_fixed_parts = [serialize(element) for element in value if is_fixed_size(element)] + +# Compute the length of the first section section_1_length = sum(len(part) for part in section_1_fixed_parts) + 4 * len(section_2_parts) + +# Compute the offset values for each part of the second section section_1_offsets = [ section_1_length + sum(section_2_lengths[:index]) for index in range(len(section_2_parts)) ] assert all(offset < 2**32 for offset in section_1_offsets) -section_1_parts = collate(section_1_fixed_parts, section_1_offsets) + +# compute the appropriate indices for *fixed size* elements for the first section +fixed_size_element_indices = [index for index, element in enumerate(value) if is_fixed_size(element)] + +# compute the appropriate indices for the offsets of the *variable size* elements +variable_size_element_indices = [index for index, element in enumerate(value) if is_variable_size(element)] + +# create a list with placeholders for all values +section_1_parts = [None] * len(value) + +# populate all of the serialized *fixed size* elements +for index, data in zip(fixed_size_element_indices, section_1_fixed_parts): + section_1_parts[index] = data + +# populate all of the serialized offsets for the *variable size* elements +for index, offset in zip(variable_size_element_indices, section_1_offsets): + section_1_parts[index] = serialize(offset) + +assert not any(part is None for part in section_1_parts) section_1 = ''.join(section_1_parts) return ''.join([section_1, section_2]) From fa66640a00aed4fae5f8878d00ff02b6e50b6705 Mon Sep 17 00:00:00 2001 From: jannikluhn Date: Wed, 20 Mar 2019 09:01:27 -0600 Subject: [PATCH 04/28] Update specs/simple-serialize.md Co-Authored-By: pipermerriam --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index b85a20656..f2ca7c0cd 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -86,7 +86,7 @@ The serialized representation of composite types is comprised of two binary segm * The first segment is *fixed size* for all types, containing the concatenation of *either* - The serialized representation of value for each of the *fixed size* types - The `"uint32"` serialized offset where the serialized representation of the *variable sized* type is located in the second section. -* The second segment contains the concatenation of the serialized representations of **only** the *variable size* types. +* The second section contains the concatenation of the serialized representations of **only** the *variable size* types. - This section is empty in the case of a purely *fixed size* type. From 32684d582a14f59ea3d02490316ae84496a099ea Mon Sep 17 00:00:00 2001 From: jannikluhn Date: Wed, 20 Mar 2019 09:01:51 -0600 Subject: [PATCH 05/28] Update specs/simple-serialize.md Co-Authored-By: pipermerriam --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index f2ca7c0cd..8577dcd2b 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -85,7 +85,7 @@ The serialized representation of composite types is comprised of two binary segm * The first segment is *fixed size* for all types, containing the concatenation of *either* - The serialized representation of value for each of the *fixed size* types - - The `"uint32"` serialized offset where the serialized representation of the *variable sized* type is located in the second section. + - The `"uint32"` serialized offset where the serialized representation of the *variable sized* type is located in the second section relative to the beginning of the first section. * The second section contains the concatenation of the serialized representations of **only** the *variable size* types. - This section is empty in the case of a purely *fixed size* type. From 3741b7517b2facfd4beb50a8fd8f1d021d2cb7b1 Mon Sep 17 00:00:00 2001 From: jannikluhn Date: Wed, 20 Mar 2019 09:02:08 -0600 Subject: [PATCH 06/28] Update specs/simple-serialize.md Co-Authored-By: pipermerriam --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 8577dcd2b..cf4dd55ba 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -83,7 +83,7 @@ return b"\x01" if value is True else b"\x00" The serialized representation of composite types is comprised of two binary segments. -* The first segment is *fixed size* for all types, containing the concatenation of *either* +* The first section is *fixed size* for all types, containing the concatenation of *either* - The serialized representation of value for each of the *fixed size* types - The `"uint32"` serialized offset where the serialized representation of the *variable sized* type is located in the second section relative to the beginning of the first section. * The second section contains the concatenation of the serialized representations of **only** the *variable size* types. From 1ab501975cc9f615fec6d4770b60319b68672c52 Mon Sep 17 00:00:00 2001 From: jannikluhn Date: Wed, 20 Mar 2019 09:02:20 -0600 Subject: [PATCH 07/28] Update specs/simple-serialize.md Co-Authored-By: pipermerriam --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index cf4dd55ba..9d30f4f3d 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -84,7 +84,7 @@ return b"\x01" if value is True else b"\x00" The serialized representation of composite types is comprised of two binary segments. * The first section is *fixed size* for all types, containing the concatenation of *either* - - The serialized representation of value for each of the *fixed size* types + - The serialized representation for each of the *fixed size* elements of value - The `"uint32"` serialized offset where the serialized representation of the *variable sized* type is located in the second section relative to the beginning of the first section. * The second section contains the concatenation of the serialized representations of **only** the *variable size* types. - This section is empty in the case of a purely *fixed size* type. From ca98d752d2586968aee22cd73402dc5cff3d2b38 Mon Sep 17 00:00:00 2001 From: Piper Merriam Date: Wed, 20 Mar 2019 09:04:55 -0600 Subject: [PATCH 08/28] d --- specs/simple-serialize.md | 52 +++++++++++++++++---------------------- 1 file changed, 23 insertions(+), 29 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 9d30f4f3d..1669c00ea 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -81,7 +81,7 @@ return b"\x01" if value is True else b"\x00" ### Composite Types (Vectors, Containers and Lists) -The serialized representation of composite types is comprised of two binary segments. +The serialized representation of composite types is comprised of two binary sections. * The first section is *fixed size* for all types, containing the concatenation of *either* - The serialized representation for each of the *fixed size* elements of value @@ -97,41 +97,35 @@ An implementation of the `serialize` function for `"Vector"`, `"Container"` and ```python # The second section is just the concatenation of the serialized *variable size* elements -section_2_parts = [serialize(element) for element in value if is_variable_size(element)] +section_2_parts = [ + serialize(element) if is_variable_size(element) + else '' + for element in value +] section_2_lengths = [len(part) for part in section_2_parts] -section_2 ''.join(section_2_parts) +section_2 = ''.join(section_2_parts) -# Serialize the *fixed size* elements -section_1_fixed_parts = [serialize(element) for element in value if is_fixed_size(element)] - -# Compute the length of the first section -section_1_length = sum(len(part) for part in section_1_fixed_parts) + 4 * len(section_2_parts) +# Compute the length of the first section (can also be extracted from the type directly) +section_1_length = sum( + len(serialize(element)) if is_fixed_size(element) + else 4 + for element in value +) # Compute the offset values for each part of the second section section_1_offsets = [ - section_1_length + sum(section_2_lengths[:index]) - for index in range(len(section_2_parts)) + section_1_length + sum(section_2_lengths[:element_index]) if is_variable_size(element) + else None + for element_index, element in enumerate(value) ] -assert all(offset < 2**32 for offset in section_1_offsets) +assert all(offset is None or offset < 2**32 for offset in section_1_offsets) -# compute the appropriate indices for *fixed size* elements for the first section -fixed_size_element_indices = [index for index, element in enumerate(value) if is_fixed_size(element)] - -# compute the appropriate indices for the offsets of the *variable size* elements -variable_size_element_indices = [index for index, element in enumerate(value) if is_variable_size(element)] - -# create a list with placeholders for all values -section_1_parts = [None] * len(value) - -# populate all of the serialized *fixed size* elements -for index, data in zip(fixed_size_element_indices, section_1_fixed_parts): - section_1_parts[index] = data - -# populate all of the serialized offsets for the *variable size* elements -for index, offset in zip(variable_size_element_indices, section_1_offsets): - section_1_parts[index] = serialize(offset) - -assert not any(part is None for part in section_1_parts) +# The first section is the concatenation of the serialized static size elements and offsets +section_1_parts = [ + serialize(element) if is_fixed_size(element) + else serialize(section_1_offsets[element_index]) + for element_index, element in enumerate(value) +] section_1 = ''.join(section_1_parts) return ''.join([section_1, section_2]) From 5f465842a4173bb50693d67cdae69ed5c2cfc5ce Mon Sep 17 00:00:00 2001 From: Piper Merriam Date: Wed, 20 Mar 2019 09:12:49 -0600 Subject: [PATCH 09/28] more language updates --- specs/simple-serialize.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 1669c00ea..4fcefb86f 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -54,7 +54,7 @@ For convenience we alias: We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a bytestring of type `"bytes"`. -> *Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signed_root`, etc.) objects implicitly carry their type. +> *Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signed_root`, `is_fixed_size`, `is_variable_size` etc.) objects implicitly carry their type. ### Basic Types @@ -83,17 +83,19 @@ return b"\x01" if value is True else b"\x00" The serialized representation of composite types is comprised of two binary sections. -* The first section is *fixed size* for all types, containing the concatenation of *either* - - The serialized representation for each of the *fixed size* elements of value - - The `"uint32"` serialized offset where the serialized representation of the *variable sized* type is located in the second section relative to the beginning of the first section. +* The first section is *fixed size* for all types, containing the concatenation of + - The serialized representation for each of the *fixed size* elements from the value + - The `"uint32"` serialized offset where the serialized representation of the *variable sized* elements from the value are located in the second section. * The second section contains the concatenation of the serialized representations of **only** the *variable size* types. - This section is empty in the case of a purely *fixed size* type. +> **NOTE**: Offsets are relative to the beginning of the beginning of the entire serialized representation (the start of the first section) + #### `"vector"`, `"container"` and `"list"` -An implementation of the `serialize` function for `"Vector"`, `"Container"` and -`"List"` types would take the following form. +Below is an illustrative implementation of the `serialize` function for `"Vector"`, +`"Container"` and `"List"` types. ```python # The second section is just the concatenation of the serialized *variable size* elements From 66173b8ba3869b9c130627a3d45e1d9199a8dfe5 Mon Sep 17 00:00:00 2001 From: Piper Merriam Date: Wed, 20 Mar 2019 09:14:02 -0600 Subject: [PATCH 10/28] static > fixed --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 4fcefb86f..52bd52587 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -122,7 +122,7 @@ section_1_offsets = [ ] assert all(offset is None or offset < 2**32 for offset in section_1_offsets) -# The first section is the concatenation of the serialized static size elements and offsets +# The first section is the concatenation of the serialized fixed size elements and offsets section_1_parts = [ serialize(element) if is_fixed_size(element) else serialize(section_1_offsets[element_index]) From 92f002c501dac0ff6bd0da8122dc99322450e33e Mon Sep 17 00:00:00 2001 From: Piper Merriam Date: Thu, 28 Mar 2019 08:26:18 -0600 Subject: [PATCH 11/28] specify offsets better --- specs/simple-serialize.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 52bd52587..6bd9e3dd7 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -83,13 +83,21 @@ return b"\x01" if value is True else b"\x00" The serialized representation of composite types is comprised of two binary sections. -* The first section is *fixed size* for all types, containing the concatenation of - - The serialized representation for each of the *fixed size* elements from the value - - The `"uint32"` serialized offset where the serialized representation of the *variable sized* elements from the value are located in the second section. -* The second section contains the concatenation of the serialized representations of **only** the *variable size* types. +* The first section is the concatenation of a mixture of either the serialized representation for *fixed size* elements **or** a serialized offset value for *variable size* elements. + - All *fixed size* elements are represented in this section as their serialized representation. + - All *variable size* elements are represented in this section with a `"uint32"` serialized offset where the serialized representation of the element is located in the second section. + - offsets are relative to the beginning of the beginning of the entire serialized representation (the start of the first section) +* The second section is the concatenation of the serialized representations of **only** the *variable size* types. - This section is empty in the case of a purely *fixed size* type. -> **NOTE**: Offsets are relative to the beginning of the beginning of the entire serialized representation (the start of the first section) + +Offset values are subject to the following validity rules: + +- For Vector and Container types: + - The first offset **must** be equal to the length of the first section. +- For all types: + - Offsets **MAY NOT** be less than any previous offset. + - Offsets **MUST** be less than `2**32` #### `"vector"`, `"container"` and `"list"` From 4d2bdf8628eb430c301043e9e4d91b8f883b0ca4 Mon Sep 17 00:00:00 2001 From: Justin Date: Sat, 13 Apr 2019 23:44:14 +1000 Subject: [PATCH 12/28] Cleanup spec --- specs/simple-serialize.md | 88 +++++++++------------------------------ 1 file changed, 19 insertions(+), 69 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 6bd9e3dd7..ff2e207f0 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -12,7 +12,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Serialization](#serialization) - [`"uintN"`](#uintn) - [`"bool"`](#bool) - - [Vectors, containers, lists](#composite-types-vectors-containers-and-lists) + - [Vectors, containers, lists](#vectors-containers-lists) - [Deserialization](#deserialization) - [Merkleization](#merkleization) - [Self-signed containers](#self-signed-containers) @@ -22,8 +22,9 @@ This is a **work in progress** describing typing, serialization and Merkleizatio | Name | Value | Description | |-|-|-| -| `BYTES_PER_CHUNK` | `32` | Number of bytes per chunk. +| `BYTES_PER_CHUNK` | `32` | Number of bytes per chunk. | | `BYTES_PER_LENGTH_PREFIX` | `4` | Number of bytes per serialized length prefix. | +| `BITS_PER_BYTE` | `8` | Number of bits per byte. | ## Typing ### Basic types @@ -56,97 +57,46 @@ We recursively define the `serialize` function which consumes an object `value` > *Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signed_root`, `is_fixed_size`, `is_variable_size` etc.) objects implicitly carry their type. -### Basic Types - -For basic types the `serialize` function is defined as follows. - -#### `"uintN"` - -A byte string of width `N // 8` containing the little-endian encode integer. +### `"uintN"` ```python assert N in [8, 16, 32, 64, 128, 256] return value.to_bytes(N // 8, "little") ``` -#### `"bool"` - -* The byte `\x00` **if** the value is `False` -* The byte `\x01` **if** the value is `True` +### `"bool"` ```python assert value in (True, False) return b"\x01" if value is True else b"\x00" ``` -### Composite Types (Vectors, Containers and Lists) - -The serialized representation of composite types is comprised of two binary sections. - -* The first section is the concatenation of a mixture of either the serialized representation for *fixed size* elements **or** a serialized offset value for *variable size* elements. - - All *fixed size* elements are represented in this section as their serialized representation. - - All *variable size* elements are represented in this section with a `"uint32"` serialized offset where the serialized representation of the element is located in the second section. - - offsets are relative to the beginning of the beginning of the entire serialized representation (the start of the first section) -* The second section is the concatenation of the serialized representations of **only** the *variable size* types. - - This section is empty in the case of a purely *fixed size* type. - - -Offset values are subject to the following validity rules: - -- For Vector and Container types: - - The first offset **must** be equal to the length of the first section. -- For all types: - - Offsets **MAY NOT** be less than any previous offset. - - Offsets **MUST** be less than `2**32` - - -#### `"vector"`, `"container"` and `"list"` - -Below is an illustrative implementation of the `serialize` function for `"Vector"`, -`"Container"` and `"List"` types. +### Vectors, containers, lists ```python -# The second section is just the concatenation of the serialized *variable size* elements -section_2_parts = [ - serialize(element) if is_variable_size(element) - else '' - for element in value -] -section_2_lengths = [len(part) for part in section_2_parts] -section_2 = ''.join(section_2_parts) +# Reccursively serialize fixed-size elements +fixed_parts = [serialize(element) if is_fixed_size(element) else None for element in value] +fixed_lengths = [len(part) if part != None else BYTES_PER_LENGTH_PREFIX for part in fixed_parts] -# Compute the length of the first section (can also be extracted from the type directly) -section_1_length = sum( - len(serialize(element)) if is_fixed_size(element) - else 4 - for element in value -) +# Reccursively serialize variable-size elements +variable_parts = [serialize(element) if is_variable_size(element) else None for element in value] +variable_lengths = [len(part) if part != None else 0 for part in variable_parts] -# Compute the offset values for each part of the second section -section_1_offsets = [ - section_1_length + sum(section_2_lengths[:element_index]) if is_variable_size(element) - else None - for element_index, element in enumerate(value) -] -assert all(offset is None or offset < 2**32 for offset in section_1_offsets) +# Compute offsets of variable-size elements +assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_PREFIX * BITS_PER_BYTE) +offsets = [sum(fixed_lengths) + sum(variable_lengths[:i]) for i in range(len(value))] -# The first section is the concatenation of the serialized fixed size elements and offsets -section_1_parts = [ - serialize(element) if is_fixed_size(element) - else serialize(section_1_offsets[element_index]) - for element_index, element in enumerate(value) -] -section_1 = ''.join(section_1_parts) +# Interleave offsets in fixed parts +fixed_parts = [part if part != None else offsets[i] for i, part in enumerate(fixed_parts)] -return ''.join([section_1, section_2]) +# Return the of fixed parts (with interleaved offsets) followed by variable parts +return "".join(fixed_parts + variable_parts) ``` - ## Deserialization Because serialization is an injective function (i.e. two distinct objects of the same type will serialize to different values) any bytestring has at most one object it could deserialize to. Efficient algorithms for computing this object can be found in [the implementations](#implementations). - ## Merkleization We first define helper functions: From aaa5a1676572819befe3ca937449e261474e51bb Mon Sep 17 00:00:00 2001 From: Justin Date: Sat, 13 Apr 2019 23:45:18 +1000 Subject: [PATCH 13/28] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index ff2e207f0..150cf2d54 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -55,7 +55,7 @@ For convenience we alias: We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a bytestring of type `"bytes"`. -> *Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signed_root`, `is_fixed_size`, `is_variable_size` etc.) objects implicitly carry their type. +> *Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signed_root`, `is_fixed_size`, `is_variable_size`, etc.) objects implicitly carry their type. ### `"uintN"` From 0695d0ad1c4725733ae0916a56340eb5ccf7c722 Mon Sep 17 00:00:00 2001 From: Justin Date: Sat, 13 Apr 2019 23:48:47 +1000 Subject: [PATCH 14/28] Update simple-serialize.md --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 150cf2d54..bc1fa6ccb 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -86,10 +86,10 @@ variable_lengths = [len(part) if part != None else 0 for part in variable_parts] assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_PREFIX * BITS_PER_BYTE) offsets = [sum(fixed_lengths) + sum(variable_lengths[:i]) for i in range(len(value))] -# Interleave offsets in fixed parts +# Interleave offsets with fixed parts fixed_parts = [part if part != None else offsets[i] for i, part in enumerate(fixed_parts)] -# Return the of fixed parts (with interleaved offsets) followed by variable parts +# Return the fixed parts (with offsets interleaved) followed by variable parts return "".join(fixed_parts + variable_parts) ``` From 35a6311208a61bf2b09850bb7731aa0c968066e4 Mon Sep 17 00:00:00 2001 From: Justin Date: Sat, 13 Apr 2019 23:50:23 +1000 Subject: [PATCH 15/28] Update simple-serialize.md --- specs/simple-serialize.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index bc1fa6ccb..5aba9aff5 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -74,19 +74,17 @@ return b"\x01" if value is True else b"\x00" ### Vectors, containers, lists ```python -# Reccursively serialize fixed-size elements +# Reccursively serialize fixed_parts = [serialize(element) if is_fixed_size(element) else None for element in value] -fixed_lengths = [len(part) if part != None else BYTES_PER_LENGTH_PREFIX for part in fixed_parts] - -# Reccursively serialize variable-size elements variable_parts = [serialize(element) if is_variable_size(element) else None for element in value] + +# Compute and check lengths +fixed_lengths = [len(part) if part != None else BYTES_PER_LENGTH_PREFIX for part in fixed_parts] variable_lengths = [len(part) if part != None else 0 for part in variable_parts] - -# Compute offsets of variable-size elements assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_PREFIX * BITS_PER_BYTE) -offsets = [sum(fixed_lengths) + sum(variable_lengths[:i]) for i in range(len(value))] -# Interleave offsets with fixed parts +# Compute offsets of variable-size elements, and interleave with fixed parts +offsets = [sum(fixed_lengths) + sum(variable_lengths[:i]) for i in range(len(value))] fixed_parts = [part if part != None else offsets[i] for i, part in enumerate(fixed_parts)] # Return the fixed parts (with offsets interleaved) followed by variable parts From 80bd4a381b02dd7b226cab59742402990c7a7479 Mon Sep 17 00:00:00 2001 From: Justin Date: Sat, 13 Apr 2019 23:55:08 +1000 Subject: [PATCH 16/28] Update simple-serialize.md --- specs/simple-serialize.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 5aba9aff5..417d70b95 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -23,7 +23,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio | Name | Value | Description | |-|-|-| | `BYTES_PER_CHUNK` | `32` | Number of bytes per chunk. | -| `BYTES_PER_LENGTH_PREFIX` | `4` | Number of bytes per serialized length prefix. | +| `BYTES_PER_LENGTH_OFFSET` | `4` | Number of bytes per serialized length prefix. | | `BITS_PER_BYTE` | `8` | Number of bits per byte. | ## Typing @@ -79,9 +79,9 @@ fixed_parts = [serialize(element) if is_fixed_size(element) else None for elemen variable_parts = [serialize(element) if is_variable_size(element) else None for element in value] # Compute and check lengths -fixed_lengths = [len(part) if part != None else BYTES_PER_LENGTH_PREFIX for part in fixed_parts] +fixed_lengths = [len(part) if part != None else BYTES_PER_LENGTH_OFFSET for part in fixed_parts] variable_lengths = [len(part) if part != None else 0 for part in variable_parts] -assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_PREFIX * BITS_PER_BYTE) +assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_OFFSET * BITS_PER_BYTE) # Compute offsets of variable-size elements, and interleave with fixed parts offsets = [sum(fixed_lengths) + sum(variable_lengths[:i]) for i in range(len(value))] From 10f3db977dc4f5e9e2ae7a1eb92034b1e4547f3e Mon Sep 17 00:00:00 2001 From: Justin Date: Sat, 13 Apr 2019 23:56:06 +1000 Subject: [PATCH 17/28] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 417d70b95..a8f0d9add 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -23,7 +23,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio | Name | Value | Description | |-|-|-| | `BYTES_PER_CHUNK` | `32` | Number of bytes per chunk. | -| `BYTES_PER_LENGTH_OFFSET` | `4` | Number of bytes per serialized length prefix. | +| `BYTES_PER_LENGTH_OFFSET` | `4` | Number of bytes per serialized length offset. | | `BITS_PER_BYTE` | `8` | Number of bits per byte. | ## Typing From 27cf02a9b0a1914a92a8b5aac47c44ebe1699996 Mon Sep 17 00:00:00 2001 From: Justin Date: Sat, 13 Apr 2019 23:59:03 +1000 Subject: [PATCH 18/28] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index a8f0d9add..f57945749 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -84,7 +84,7 @@ variable_lengths = [len(part) if part != None else 0 for part in variable_parts] assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_OFFSET * BITS_PER_BYTE) # Compute offsets of variable-size elements, and interleave with fixed parts -offsets = [sum(fixed_lengths) + sum(variable_lengths[:i]) for i in range(len(value))] +offsets = [sum(fixed_lengths + variable_lengths[:i]) for i in range(len(value))] fixed_parts = [part if part != None else offsets[i] for i, part in enumerate(fixed_parts)] # Return the fixed parts (with offsets interleaved) followed by variable parts From a90bcc0cd4fb186beb0ef166ce7de87f526dad6a Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 14 Apr 2019 00:00:46 +1000 Subject: [PATCH 19/28] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index f57945749..a7d80e264 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -87,7 +87,7 @@ assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_OFFSET * BIT offsets = [sum(fixed_lengths + variable_lengths[:i]) for i in range(len(value))] fixed_parts = [part if part != None else offsets[i] for i, part in enumerate(fixed_parts)] -# Return the fixed parts (with offsets interleaved) followed by variable parts +# Return the fixed parts (with offsets interleaved) followed by the variable parts return "".join(fixed_parts + variable_parts) ``` From 23c09541e2a44fe4a56641ef354d24eb24c3e461 Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 14 Apr 2019 00:05:43 +1000 Subject: [PATCH 20/28] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index a7d80e264..41999d1dc 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -83,7 +83,7 @@ fixed_lengths = [len(part) if part != None else BYTES_PER_LENGTH_OFFSET for part variable_lengths = [len(part) if part != None else 0 for part in variable_parts] assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_OFFSET * BITS_PER_BYTE) -# Compute offsets of variable-size elements, and interleave with fixed parts +# Compute offsets of variable-size parts and interleave offsets with fixed parts offsets = [sum(fixed_lengths + variable_lengths[:i]) for i in range(len(value))] fixed_parts = [part if part != None else offsets[i] for i, part in enumerate(fixed_parts)] From f6ed1df62bd09758aebbaaeadf51041df1d62936 Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 14 Apr 2019 00:10:02 +1000 Subject: [PATCH 21/28] Update simple-serialize.md --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 41999d1dc..ceb3652f3 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -76,11 +76,11 @@ return b"\x01" if value is True else b"\x00" ```python # Reccursively serialize fixed_parts = [serialize(element) if is_fixed_size(element) else None for element in value] -variable_parts = [serialize(element) if is_variable_size(element) else None for element in value] +variable_parts = [serialize(element) if is_variable_size(element) else "" for element in value] # Compute and check lengths fixed_lengths = [len(part) if part != None else BYTES_PER_LENGTH_OFFSET for part in fixed_parts] -variable_lengths = [len(part) if part != None else 0 for part in variable_parts] +variable_lengths = [len(part) for part in variable_parts] assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_OFFSET * BITS_PER_BYTE) # Compute offsets of variable-size parts and interleave offsets with fixed parts From 9adbaba96e01c756b474f48b13630ec0c339f396 Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 14 Apr 2019 00:14:30 +1000 Subject: [PATCH 22/28] Update simple-serialize.md --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index ceb3652f3..6626b9cf4 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -83,11 +83,11 @@ fixed_lengths = [len(part) if part != None else BYTES_PER_LENGTH_OFFSET for part variable_lengths = [len(part) for part in variable_parts] assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_OFFSET * BITS_PER_BYTE) -# Compute offsets of variable-size parts and interleave offsets with fixed parts +# Compute offsets of variable-size parts and interleave offsets with fixed-size parts offsets = [sum(fixed_lengths + variable_lengths[:i]) for i in range(len(value))] fixed_parts = [part if part != None else offsets[i] for i, part in enumerate(fixed_parts)] -# Return the fixed parts (with offsets interleaved) followed by the variable parts +# Return the fixed-size parts (with offsets interleaved) followed by the variable-size parts return "".join(fixed_parts + variable_parts) ``` From 97ca6721056723193d8b8fc69aa8dcadb6c38098 Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 14 Apr 2019 00:18:44 +1000 Subject: [PATCH 23/28] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 6626b9cf4..b956df4d1 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -85,7 +85,7 @@ assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_OFFSET * BIT # Compute offsets of variable-size parts and interleave offsets with fixed-size parts offsets = [sum(fixed_lengths + variable_lengths[:i]) for i in range(len(value))] -fixed_parts = [part if part != None else offsets[i] for i, part in enumerate(fixed_parts)] +fixed_parts = [part if part != None else serialize(offsets[i]) for i, part in enumerate(fixed_parts)] # Return the fixed-size parts (with offsets interleaved) followed by the variable-size parts return "".join(fixed_parts + variable_parts) From 09d927405c9020da2ed466d8e2782929589b9ab1 Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 14 Apr 2019 00:22:41 +1000 Subject: [PATCH 24/28] Update simple-serialize.md --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index b956df4d1..1af409c58 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -83,11 +83,11 @@ fixed_lengths = [len(part) if part != None else BYTES_PER_LENGTH_OFFSET for part variable_lengths = [len(part) for part in variable_parts] assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_OFFSET * BITS_PER_BYTE) -# Compute offsets of variable-size parts and interleave offsets with fixed-size parts +# Interleave offsets of variable-size parts with fixed-size parts offsets = [sum(fixed_lengths + variable_lengths[:i]) for i in range(len(value))] fixed_parts = [part if part != None else serialize(offsets[i]) for i, part in enumerate(fixed_parts)] -# Return the fixed-size parts (with offsets interleaved) followed by the variable-size parts +# Return the concatenation of the fixed-size parts (offsets interleaved) with the variable-size parts return "".join(fixed_parts + variable_parts) ``` From 7255b0fc0d63ae891910228cfaf635ea66258f7d Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 14 Apr 2019 00:25:47 +1000 Subject: [PATCH 25/28] Update simple-serialize.md --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 1af409c58..cf482c39d 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -84,8 +84,8 @@ variable_lengths = [len(part) for part in variable_parts] assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_OFFSET * BITS_PER_BYTE) # Interleave offsets of variable-size parts with fixed-size parts -offsets = [sum(fixed_lengths + variable_lengths[:i]) for i in range(len(value))] -fixed_parts = [part if part != None else serialize(offsets[i]) for i, part in enumerate(fixed_parts)] +offsets = [serialize(sum(fixed_lengths + variable_lengths[:i])) for i in range(len(value))] +fixed_parts = [part if part != None else offsets[i] for i, part in enumerate(fixed_parts)] # Return the concatenation of the fixed-size parts (offsets interleaved) with the variable-size parts return "".join(fixed_parts + variable_parts) From 59f568073afeb7723ba33fc8fd601375adc5d205 Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 14 Apr 2019 00:26:44 +1000 Subject: [PATCH 26/28] Update simple-serialize.md --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index cf482c39d..03e47deb2 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -84,8 +84,8 @@ variable_lengths = [len(part) for part in variable_parts] assert sum(fixed_lengths + variable_lengths) < 2**(BYTES_PER_LENGTH_OFFSET * BITS_PER_BYTE) # Interleave offsets of variable-size parts with fixed-size parts -offsets = [serialize(sum(fixed_lengths + variable_lengths[:i])) for i in range(len(value))] -fixed_parts = [part if part != None else offsets[i] for i, part in enumerate(fixed_parts)] +variable_offsets = [serialize(sum(fixed_lengths + variable_lengths[:i])) for i in range(len(value))] +fixed_parts = [part if part != None else variable_offsets[i] for i, part in enumerate(fixed_parts)] # Return the concatenation of the fixed-size parts (offsets interleaved) with the variable-size parts return "".join(fixed_parts + variable_parts) From 62ffb897ae6949a0acb372be0e31e36dd3f16dd0 Mon Sep 17 00:00:00 2001 From: Justin Date: Sun, 14 Apr 2019 00:41:48 +1000 Subject: [PATCH 27/28] Update simple-serialize.md --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 03e47deb2..3f02335db 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -12,7 +12,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Serialization](#serialization) - [`"uintN"`](#uintn) - [`"bool"`](#bool) - - [Vectors, containers, lists](#vectors-containers-lists) + - [Containers, vectors, lists](#containers-vectors-lists) - [Deserialization](#deserialization) - [Merkleization](#merkleization) - [Self-signed containers](#self-signed-containers) @@ -71,7 +71,7 @@ assert value in (True, False) return b"\x01" if value is True else b"\x00" ``` -### Vectors, containers, lists +### Containers, vectors, lists ```python # Reccursively serialize From 1284b93416cebb0fd4b58059ed72aa7332a78814 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 24 Apr 2019 15:53:28 +1000 Subject: [PATCH 28/28] Update simple-serialize.md --- specs/simple-serialize.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 3f02335db..882147f0f 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -55,7 +55,7 @@ For convenience we alias: We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a bytestring of type `"bytes"`. -> *Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signed_root`, `is_fixed_size`, `is_variable_size`, etc.) objects implicitly carry their type. +> *Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signing_root`, `is_variable_size`, etc.) objects implicitly carry their type. ### `"uintN"` @@ -75,8 +75,8 @@ return b"\x01" if value is True else b"\x00" ```python # Reccursively serialize -fixed_parts = [serialize(element) if is_fixed_size(element) else None for element in value] -variable_parts = [serialize(element) if is_variable_size(element) else "" for element in value] +fixed_parts = [serialize(element) if not is_variable_size(element) else None for element in value] +variable_parts = [serialize(element) if is_variable_size(element) else b"" for element in value] # Compute and check lengths fixed_lengths = [len(part) if part != None else BYTES_PER_LENGTH_OFFSET for part in fixed_parts] @@ -88,7 +88,7 @@ variable_offsets = [serialize(sum(fixed_lengths + variable_lengths[:i])) for i i fixed_parts = [part if part != None else variable_offsets[i] for i, part in enumerate(fixed_parts)] # Return the concatenation of the fixed-size parts (offsets interleaved) with the variable-size parts -return "".join(fixed_parts + variable_parts) +return b"".join(fixed_parts + variable_parts) ``` ## Deserialization @@ -112,7 +112,7 @@ We now define Merkleization `hash_tree_root(value)` of an object `value` recursi ## Self-signed containers -Let `value` be a self-signed container object. The convention is that the signature (e.g. a `"bytes96"` BLS12-381 signature) be the last field of `value`. Further, the signed message for `value` is `signed_root(value) = hash_tree_root(truncate_last(value))` where `truncate_last` truncates the last element of `value`. +Let `value` be a self-signed container object. The convention is that the signature (e.g. a `"bytes96"` BLS12-381 signature) be the last field of `value`. Further, the signed message for `value` is `signing_root(value) = hash_tree_root(truncate_last(value))` where `truncate_last` truncates the last element of `value`. ## Implementations