eth2.0-specs/specs/simpleserialize.md

# SimpleSerialize (SSZ) Spec

***Work In Progress***

This is the work in progress document to describe `simpleserialize`, the
current selected serialization method for Ethereum 2.0 using the Beacon Chain.

This document specifies the general information for serializing and
deserializing objects and data types.

## ToC

* [About](#about)
* [Terminology](#terminology)
* [Constants](#constants)
* [Overview](#overview)
   + [Serialize/Encode](#serializeencode)
      - [uint: 8/16/24/32/64/256](#uint-816243264256)
      - [Address](#address)
      - [Hash](#hash)
         * [Hash32](#hash32)
         * [Hash96](#hash96)
         * [Hash97](#hash97)
      - [Bytes](#bytes)
      - [List/Vectors](#listvectors)
   + [Deserialize/Decode](#deserializedecode)
      - [uint: 8/16/24/32/64/256](#uint-816243264256-1)
      - [Address](#address-1)
      - [Hash](#hash-1)
         * [Hash32](#hash32-1)
         * [Hash96](#hash96-1)
         * [Hash97](#hash97-1)
      - [Bytes](#bytes-1)
      - [List/Vectors](#listvectors-1)
* [Implementations](#implementations)

## About

`SimpleSerialize` was first proposed by Vitalik Buterin as the serialization
protocol for use in the Ethereum 2.0 Beacon Chain.

The core feature of `ssz` is the simplicity of the serialization with low
overhead.

## Terminology

| Term         | Definition                                                                                     |
|:-------------|:-----------------------------------------------------------------------------------------------|
| `big`        | Big Endian                                                                                     |
| `byte_order` | Specifies [endianness:](https://en.wikipedia.org/wiki/Endianness) Big Endian or Little Endian. |
| `len`        | Length/Number of Bytes.                                                                        |
| `to_bytes`   | Convert to bytes. Should take parameters ``size`` and ``byte_order``.                          |
| `from_bytes` | Convert form bytes to object. Should take ``bytes`` and ``byte_order``.                        |
| `value`      | The value to serialize.                                                                        |
| `rawbytes`   | Raw serialized bytes.                                                                          |

## Constants

| Constant       | Value | Definition                                                              |
|:---------------|:-----:|:------------------------------------------------------------------------|
| `LENGTH_BYTES` |   4   | Number of bytes used for the length added before the serialized object. |


## Overview

### Serialize/Encode

#### uint: 8/16/24/32/64/256

Convert directly to bytes the size of the int. (e.g. ``uint16 = 2 bytes``)

All integers are serialized as **big endian**.

| Check to perform                 | Code                    |
|:---------------------------------|:------------------------|
| Size is a byte integer           | ``int_size % 8 == 0``   |
| Value is less than max           | ``2**int_size > value`` |

```python
buffer_size = int_size / 8
return value.to_bytes(buffer_size, 'big')
```

#### Address

The address should already come as a hash/byte format. Ensure that length is
**20**.

| Check to perform       | Code                 |
|:-----------------------|:---------------------|
| Length is correct (20) | ``len(value) == 20`` |

```python
assert( len(value) == 20 )
return value
```

#### Hash

| Hash Type | Usage                                           |
|:---------:|:------------------------------------------------|
|  `hash32` | Hash size of ``keccak`` or `blake2b[0.. < 32]`. |
|  `hash96` | BLS Public Key Size.                            |
|  `hash97` | BLS Public Key Size with recovery bit.          |


| Checks to perform                   | Code                 |
|:-----------------------------------|:---------------------|
| Length is correct (32) if `hash32` | ``len(value) == 32`` |
| Length is correct (96) if `hash96` | ``len(value) == 96`` |
| Length is correct (97) if `hash97` | ``len(value) == 97`` |


**Example all together**

```python
if (type(value) == 'hash32'):
   assert(len(value) == 32)
elif (type(value) == 'hash96'):
   assert(len(value) == 96)
elif (type(value) == 'hash97'):
   assert(len(value) == 97)
else:
   raise TypeError('Invalid hash type supplied')

return value
```

##### Hash32

Ensure 32 byte length and return the bytes.

```python
assert(len(value) == 32)
return value
```

##### Hash96

Ensure 96 byte length and return the bytes.

```python
assert(len(value) == 96)
return value
```

##### Hash97

Ensure 97 byte length and return the bytes.

```python
assert(len(value) == 97)
return value
```

#### Bytes

For general `byte` type:
1. Get the length/number of bytes; Encode into a 4 byte integer.
2. Append the value to the length and return: ``[ length_bytes ] + [
   value_bytes ]``

| Check to perform                     | Code                   |
|:-------------------------------------|:-----------------------|
| Length of bytes can fit into 4 bytes | ``len(value) < 2**32`` |

```python
byte_length = (len(value)).to_bytes(LENGTH_BYTES, 'big')
return byte_length + value
```

#### List/Vectors

1. Get the number of raw bytes to serialize: it is `len(list) * sizeof(element)`.
   * Encode that as a `4-byte` **big endian** `uint32`.
2. Append your elements in a packed manner.

* *Note on efficiency*: consider using a container that does not need to iterate over all elements to get its length. For example Python lists, C++ vectors or Rust Vec.

**Example in Python**

```python
serialized_list_string = ''

for item in value:
   serialized_list_string += serialize(item)

serialized_len = (len(serialized_list_string).to_bytes(LENGTH_BYTES, 'big'))

return serialized_len + serialized_list_string
```

### Deserialize/Decode

The decoding requires knowledge of the type of the item to be decoded. When
performing decoding on an entire serialized string, it also requires knowledge
of what order the objects have been serialized in.

Note: Each return will provide ``deserialized_object, new_index`` keeping track
of the new index.

At each step, the following checks should be made:

| Check Type               | Check                                                     |
|:-------------------------|:----------------------------------------------------------|
| Ensure sufficient length | ``length(rawbytes) > current_index + deserialize_length`` |

#### uint: 8/16/24/32/64/256

Convert directly from bytes into integer utilising the number of bytes the same
size as the integer length. (e.g. ``uint16 == 2 bytes``)

All integers are interpreted as **big endian**.

```python
byte_length = int_size / 8
new_index = current_index + int_size
return int.from_bytes(rawbytes[current_index:current_index+int_size], 'big'), new_index
```

#### Address

Return the 20 bytes.

```python
new_index = current_index + 20
return rawbytes[current_index:current_index+20], new_index
```

#### Hash

##### Hash32

Return the 32 bytes.

```python
new_index = current_index + 32
return rawbytes[current_index:current_index+32], new_index
```

##### Hash96

Return the 96 bytes.

```python
new_index = current_index + 96
return rawbytes[current_index:current_index+96], new_index
```

##### Hash97

Return the 97 bytes.

```python
new_index = current_index + 97
return rawbytes[current_index:current_index+97], new_index
```


#### Bytes

Get the length of the bytes, return the bytes.

```python
bytes_length = int.from_bytes(rawbytes[current_index:current_index + LENGTH_BYTES], 'big')
new_index = current_index + LENGTH_BYTES + bytes_lenth
return rawbytes[current_index + LENGTH_BYTES:current_index+ LENGTH_BYTES +bytes_length], new_index
```

#### List/Vectors

Deserialize each object in the list.
1. Get the length of the serialized list.
2. Loop through deserializing each item in the list until you reach the
entire length of the list.


| Check type                          | code                                  |
|:------------------------------------|:--------------------------------------|
| rawbytes has enough left for length | ``len(rawbytes) > current_index + 4`` |

```python
total_length = int.from_bytes(rawbytes[current_index:current_index + LENGTH_BYTES], 'big')
new_index = current_index + LENGTH_BYTES + total_length
item_index = current_index + LENGTH_BYTES
deserialized_list = []

while item_index < new_index:
   object, item_index = deserialize(rawbytes, item_index, item_type)
   deserialized_list.append(object)

return deserialized_list, new_index
```

## Implementations

| Language | Implementation                                                                                                                                                     | Description                                              |
|:--------:|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------|
|  Python  | [ https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py ](https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py)                               | Beacon chain reference implementation written in Python. |
|   Rust   | [ https://github.com/sigp/lighthouse/tree/master/ssz ](https://github.com/sigp/lighthouse/tree/master/ssz)                                                         | Lighthouse (Rust Ethereum 2.0 Node) maintained SSZ.      |
|    Nim   | [ https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim ](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) | Nim Implementation maintained SSZ.                       |
|   Rust   | [ https://github.com/paritytech/shasper/tree/master/util/ssz ](https://github.com/paritytech/shasper/tree/master/util/ssz)                                         | Shasper implementation of SSZ maintained by ParityTech.  |