eth2.0-specs/specs/simple-serialize.md

127 lines
5.8 KiB
Markdown
Raw Normal View History

2019-04-15 18:02:23 +00:00
# SimpleSerialize (SSZ)
2019-02-27 11:06:06 +00:00
This is a **work in progress** describing typing, serialization and Merkleization of Ethereum 2.0 objects.
## Table of contents
2019-02-27 21:33:36 +00:00
- [Constants](#constants)
2019-02-27 11:53:24 +00:00
- [Typing](#typing)
2019-02-27 12:04:36 +00:00
- [Basic types](#basic-types)
- [Composite types](#composite-types)
- [Aliases](#aliases)
- [Serialization](#serialization)
- [`"uintN"`](#uintn)
- [`"bool"`](#bool)
- [Vectors, containers, lists](#vectors-containers-lists)
- [Deserialization](#deserialization)
- [Merkleization](#merkleization)
2019-02-27 11:40:08 +00:00
- [Self-signed containers](#self-signed-containers)
- [Implementations](#implementations)
2018-09-23 23:52:38 +00:00
2019-02-27 21:33:36 +00:00
## Constants
| Name | Value | Description |
|-|-|-|
| `BYTES_PER_CHUNK` | `32` | Number of bytes per chunk.
| `BYTES_PER_LENGTH_PREFIX` | `4` | Number of bytes per serialized length prefix. |
2018-09-23 23:52:38 +00:00
2019-02-27 21:33:36 +00:00
## Typing
2019-02-27 16:54:19 +00:00
### Basic types
2018-09-23 23:52:38 +00:00
* `"uintN"`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`)
* `"bool"`: `True` or `False`
2018-11-27 07:45:04 +00:00
2019-02-27 16:54:19 +00:00
### Composite types
2018-09-23 23:52:38 +00:00
2019-02-27 16:59:49 +00:00
* **container**: ordered heterogenous collection of values
* key-pair curly bracket notation `{}`, e.g. `{"foo": "uint64", "bar": "bool"}`
* **vector**: ordered fixed-length homogeneous collection of values
* angle bracket notation `[type, N]`, e.g. `["uint64", N]`
2019-02-27 16:59:49 +00:00
* **list**: ordered variable-length homogenous collection of values
* angle bracket notation `[type]`, e.g. `["uint64"]`
2018-09-23 23:52:38 +00:00
We recursively define "variable-size" types to be lists and all types that contains a variable-size type. All other types are said to be "fixed-size".
2019-02-27 17:00:49 +00:00
### Aliases
For convenience we alias:
2019-03-06 22:56:35 +00:00
* `"byte"` to `"uint8"` (this is a basic type)
* `"bytes"` to `["byte"]` (this is *not* a basic type)
* `"bytesN"` to `["byte", N]` (this is *not* a basic type)
## Serialization
We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a bytestring of type `"bytes"`.
2019-02-28 10:32:54 +00:00
2019-04-08 01:51:13 +00:00
*Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signing_root`, etc.) objects implicitly carry their type.
2019-02-11 14:49:11 +00:00
### `"uintN"`
2018-09-23 23:52:38 +00:00
```python
assert N in [8, 16, 32, 64, 128, 256]
return value.to_bytes(N // 8, "little")
2018-09-23 23:52:38 +00:00
```
### `"bool"`
```python
2019-02-27 16:35:26 +00:00
assert value in (True, False)
return b"\x01" if value is True else b"\x00"
2018-09-23 23:52:38 +00:00
```
### Vectors, containers, lists
2019-03-05 14:20:36 +00:00
If `value` is fixed-size:
2019-03-01 11:02:29 +00:00
```python
return "".join([serialize(element) for element in value])
2019-03-01 11:02:29 +00:00
```
If `value` is variable-size:
```python
serialized_bytes = "".join([serialize(element) for element in value])
2019-02-27 21:33:36 +00:00
assert len(serialized_bytes) < 2**(8 * BYTES_PER_LENGTH_PREFIX)
serialized_length = len(serialized_bytes).to_bytes(BYTES_PER_LENGTH_PREFIX, "little")
return serialized_length + serialized_bytes
2018-10-03 05:08:20 +00:00
```
## Deserialization
2019-02-28 10:32:54 +00:00
Because serialization is an injective function (i.e. two distinct objects of the same type will serialize to different values) any bytestring has at most one object it could deserialize to. Efficient algorithms for computing this object can be found in [the implementations](#implementations).
## Merkleization
We first define helper functions:
2019-02-27 21:44:20 +00:00
* `pack`: Given ordered objects of the same basic type, serialize them, pack them into `BYTES_PER_CHUNK`-byte chunks, right-pad the last chunk with zero bytes, and return the chunks.
2019-02-28 10:32:54 +00:00
* `merkleize`: Given ordered `BYTES_PER_CHUNK`-byte chunks, if necessary append zero chunks so that the number of chunks is a power of two, Merkleize the chunks, and return the root.
* `mix_in_length`: Given a Merkle root `root` and a length `length` (`"uint256"` little-endian serialization) return `hash(root + length)`.
2019-02-27 16:56:51 +00:00
We now define Merkleization `hash_tree_root(value)` of an object `value` recursively:
* `merkleize(pack(value))` if `value` is a basic object or a vector of basic objects
2019-02-27 16:35:26 +00:00
* `mix_in_length(merkleize(pack(value)), len(value))` if `value` is a list of basic objects
* `merkleize([hash_tree_root(element) for element in value])` if `value` is a vector of composite objects or a container
2019-02-27 16:35:26 +00:00
* `mix_in_length(merkleize([hash_tree_root(element) for element in value]), len(value))` if `value` is a list of composite objects
2019-02-27 11:40:08 +00:00
## Self-signed containers
2019-04-08 01:51:13 +00:00
Let `value` be a self-signed container object. The convention is that the signature (e.g. a `"bytes96"` BLS12-381 signature) be the last field of `value`. Further, the signed message for `value` is `signing_root(value) = hash_tree_root(truncate_last(value))` where `truncate_last` truncates the last element of `value`.
2018-09-23 23:52:38 +00:00
## Implementations
2019-02-27 11:40:08 +00:00
| Language | Project | Maintainer | Implementation |
2019-02-27 11:54:56 +00:00
|-|-|-|-|
2019-02-27 11:40:08 +00:00
| Python | Ethereum 2.0 | Ethereum Foundation | [https://github.com/ethereum/py-ssz](https://github.com/ethereum/py-ssz) |
2019-04-10 18:28:24 +00:00
| Rust | Lighthouse | Sigma Prime | [https://github.com/sigp/lighthouse/tree/master/eth2/utils/ssz](https://github.com/sigp/lighthouse/tree/master/eth2/utils/ssz) |
2019-02-27 11:40:08 +00:00
| Nim | Nimbus | Status | [https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) |
| Rust | Shasper | ParityTech | [https://github.com/paritytech/shasper/tree/master/util/ssz](https://github.com/paritytech/shasper/tree/master/util/ssz) |
2019-04-10 18:28:24 +00:00
| TypeScript | Lodestar | ChainSafe Systems | [https://github.com/ChainSafe/ssz-js](https://github.com/ChainSafe/ssz-js) |
2019-02-27 11:40:08 +00:00
| Java | Cava | ConsenSys | [https://www.github.com/ConsenSys/cava/tree/master/ssz](https://www.github.com/ConsenSys/cava/tree/master/ssz) |
| Go | Prysm | Prysmatic Labs | [https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz](https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz) |
| Swift | Yeeth | Dean Eigenmann | [https://github.com/yeeth/SimpleSerialize.swift](https://github.com/yeeth/SimpleSerialize.swift) |
| C# | | Jordan Andrews | [https://github.com/codingupastorm/csharp-ssz](https://github.com/codingupastorm/csharp-ssz) |
| C++ | | | [https://github.com/NAKsir-melody/cpp_ssz](https://github.com/NAKsir-melody/cpp_ssz) |