226 lines
7.4 KiB
Markdown
226 lines
7.4 KiB
Markdown
|
# SimpleSerialize (SSZ) Spec
|
||
|
|
||
|
***Work In Progress***
|
||
|
|
||
|
This is the work in progress document to describe `simpleserialize`, the
|
||
|
current selected serialization method for Ethereum 2.0 using the Beacon Chain.
|
||
|
|
||
|
This document specifies the general information for serializing and
|
||
|
deserializing objects and data types.
|
||
|
|
||
|
## ToC
|
||
|
|
||
|
* [About](#about)
|
||
|
* [Terminology](#terminology)
|
||
|
* [Constants](#constants)
|
||
|
* [Overview](#overview)
|
||
|
+ [Serialize/Encode](#serializeencode)
|
||
|
- [int/uint: 8/16/24/32/64/256](#intuint-816243264256)
|
||
|
- [Address](#address)
|
||
|
- [Hash32](#hash32)
|
||
|
- [Bytes](#bytes)
|
||
|
- [List](#list)
|
||
|
+ [Deserialize/Decode](#deserializedecode)
|
||
|
- [int/uint: 8/16/24/32/64/256](#intuint-816243264256-1)
|
||
|
- [Address](#address-1)
|
||
|
- [Hash32](#hash32-1)
|
||
|
- [Bytes](#bytes-1)
|
||
|
- [List](#list-1)
|
||
|
* [Implementations](#implementations)
|
||
|
|
||
|
## About
|
||
|
|
||
|
`SimpleSerialize` was first proposed by Vitalik Buterin as the serializaiton
|
||
|
protocol for use in the Ethereum 2.0 Beacon Chain.
|
||
|
|
||
|
The core feature of `ssz` is the simplicity of the serialization with low
|
||
|
overhead.
|
||
|
|
||
|
## Terminology
|
||
|
|
||
|
| Term | Definition |
|
||
|
|:-------------|:-----------------------------------------------------------------------------------------------|
|
||
|
| `big` | Big Endian |
|
||
|
| `byte_order` | Specifies [endianness:](https://en.wikipedia.org/wiki/Endianness) Big Endian or Little Endian. |
|
||
|
| `len` | Length/Number of Bytes. |
|
||
|
| `to_bytes` | Convert to bytes. Should take parameters ``size`` and ``byte_order``. |
|
||
|
| `from_bytes` | Convert form bytes to object. Should take ``bytes`` and ``byte_order``. |
|
||
|
| `value` | The value to serialize. |
|
||
|
| `rawbytes` | Raw serialized bytes. |
|
||
|
|
||
|
## Constants
|
||
|
|
||
|
| Constant | Value | Definition |
|
||
|
|:---------------|:-----:|:------------------------------------------------------------------------|
|
||
|
| `LENGTH_BYTES` | 4 | Number of bytes used for the length added before the serialized object. |
|
||
|
|
||
|
|
||
|
## Overview
|
||
|
|
||
|
### Serialize/Encode
|
||
|
|
||
|
#### int/uint: 8/16/24/32/64/256
|
||
|
|
||
|
Convert directly to bytes the size of the int. (e.g. ``uint16 = 2 bytes``)
|
||
|
|
||
|
All integers are serialized as **big endian**.
|
||
|
|
||
|
| Check to perform | Code |
|
||
|
|:---------------------------------|:------------------------|
|
||
|
| Size is a byte integer | ``int_size % 8 == 0`` |
|
||
|
| Value is less than max | ``2**int_size > value`` |
|
||
|
|
||
|
```python
|
||
|
buffer_size = int_size / 8
|
||
|
return value.to_bytes(buffer_size, 'big')
|
||
|
```
|
||
|
|
||
|
#### Address
|
||
|
|
||
|
The address should already come as a hash/byte format. Ensure that length is
|
||
|
**20**.
|
||
|
|
||
|
| Check to perform | Code |
|
||
|
|:-----------------------|:---------------------|
|
||
|
| Length is correct (20) | ``len(value) == 20`` |
|
||
|
|
||
|
```python
|
||
|
assert( len(value) == 20 )
|
||
|
return value
|
||
|
```
|
||
|
|
||
|
#### Hash32
|
||
|
|
||
|
The hash32 should already be a 32 byte length serialized data format. The safety
|
||
|
check ensures the 32 byte length is satisfied.
|
||
|
|
||
|
| Check to perform | Code |
|
||
|
|:-----------------------|:---------------------|
|
||
|
| Length is correct (32) | ``len(value) == 32`` |
|
||
|
|
||
|
```python
|
||
|
assert( len(value) == 32 )
|
||
|
return value
|
||
|
```
|
||
|
|
||
|
#### Bytes
|
||
|
|
||
|
For general `byte` type:
|
||
|
1. Get the length/number of bytes; Encode into a 4 byte integer.
|
||
|
2. Append the value to the length and return: ``[ length_bytes ] + [
|
||
|
value_bytes ]``
|
||
|
|
||
|
```python
|
||
|
byte_length = (len(value)).to_bytes(4, 'big')
|
||
|
return byte_length + value
|
||
|
```
|
||
|
|
||
|
#### List
|
||
|
|
||
|
For lists of values, get the length of the list and then serialize the value
|
||
|
of each item in the list:
|
||
|
1. For each item in list:
|
||
|
1. serialize.
|
||
|
2. append to string.
|
||
|
2. Get size of serialized string. Encode into a 4 byte integer.
|
||
|
|
||
|
```python
|
||
|
serialized_list_string = ''
|
||
|
|
||
|
for item in value:
|
||
|
serialized_list_string += serialize(item)
|
||
|
|
||
|
serialized_len = len(serialized_list_string)
|
||
|
|
||
|
return serialized_len + serialized_list_string
|
||
|
```
|
||
|
|
||
|
### Deserialize/Decode
|
||
|
|
||
|
The decoding requires knowledge of the type of the item to be decoded. When
|
||
|
performing decoding on an entire serialized string, it also requires knowledge
|
||
|
of what order the objects have been serialized in.
|
||
|
|
||
|
Note: Each return will provide ``deserialized_object, new_index`` keeping track
|
||
|
of the new index.
|
||
|
|
||
|
At each step, the following checks should be made:
|
||
|
|
||
|
| Check Type | Check |
|
||
|
|:-------------------------|:----------------------------------------------------------|
|
||
|
| Ensure sufficient length | ``length(rawbytes) > current_index + deserialize_length`` |
|
||
|
|
||
|
#### int/uint: 8/16/24/32/64/256
|
||
|
|
||
|
Convert directly from bytes into integer utilising the number of bytes the same
|
||
|
size as the integer length. (e.g. ``uint16 == 2 bytes``)
|
||
|
|
||
|
All integers are interpreted as **big endian**.
|
||
|
|
||
|
```python
|
||
|
byte_length = int_size / 8
|
||
|
new_index = current_index + int_size
|
||
|
return int.from_bytes(rawbytes[current_index:current_index+int_size], 'big'), new_index
|
||
|
```
|
||
|
|
||
|
#### Address
|
||
|
|
||
|
Return the 20 bytes.
|
||
|
|
||
|
```python
|
||
|
new_index = current_index + 20
|
||
|
return rawbytes[current_index:current_index+20], new_index
|
||
|
```
|
||
|
|
||
|
#### Hash32
|
||
|
|
||
|
Return the 32 bytes.
|
||
|
|
||
|
```python
|
||
|
new_index = current_index + 32
|
||
|
return rawbytes[current_index:current_index+32], new_index
|
||
|
```
|
||
|
|
||
|
#### Bytes
|
||
|
|
||
|
Get the length of the bytes, return the bytes.
|
||
|
|
||
|
```python
|
||
|
bytes_length = int.from_bytes(rawbytes[current_index:current_index+4], 'big')
|
||
|
new_index = current_index + 4 + bytes_lenth
|
||
|
return rawbytes[current_index+4:current_index+4+bytes_length], new_index
|
||
|
```
|
||
|
|
||
|
#### List
|
||
|
|
||
|
Deserailize each object in the list.
|
||
|
1. Get the length of the serialized list.
|
||
|
2. Loop through deseralizing each item in the list until you reach the
|
||
|
entire length of the list.
|
||
|
|
||
|
|
||
|
| Check type | code |
|
||
|
|:------------------------------------|:--------------------------------------|
|
||
|
| rawbytes has enough left for length | ``len(rawbytes) > current_index + 4`` |
|
||
|
|
||
|
```python
|
||
|
total_length = int.from_bytes(rawbytes[current_index:current_index+4], 'big')
|
||
|
new_index = current_index + 4 + total_length
|
||
|
item_index = current_index + 4
|
||
|
deserialized_list = []
|
||
|
|
||
|
while item_index < new_index:
|
||
|
object, item_index = deserialize(rawbytes, item_index, item_type)
|
||
|
deserialized_list.append(object)
|
||
|
|
||
|
return deserialized_list, new_index
|
||
|
```
|
||
|
|
||
|
## Implementations
|
||
|
|
||
|
| Language | Implementation | Description |
|
||
|
|:--------:|--------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------|
|
||
|
| Python | [ https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py ](https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py) | Beacon chain reference implementation written in Python. |
|
||
|
| Rust | [ https://github.com/sigp/lighthouse/tree/master/ssz ](https://github.com/sigp/lighthouse/tree/master/ssz) | Lighthouse (Rust Ethereum 2.0 Node) maintained SimpleSerialize. |
|
||
|
|