From 4c75cd0db29ec81ed685316c150f1cf49c03349c Mon Sep 17 00:00:00 2001 From: NatoliChris Date: Mon, 24 Sep 2018 09:52:38 +1000 Subject: [PATCH] Initial SimpleSerialize spec --- specs/simpleserialize.md | 225 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 225 insertions(+) create mode 100644 specs/simpleserialize.md diff --git a/specs/simpleserialize.md b/specs/simpleserialize.md new file mode 100644 index 000000000..ef2bcf33f --- /dev/null +++ b/specs/simpleserialize.md @@ -0,0 +1,225 @@ +# SimpleSerialize (SSZ) Spec + +***Work In Progress*** + +This is the work in progress document to describe `simpleserialize`, the +current selected serialization method for Ethereum 2.0 using the Beacon Chain. + +This document specifies the general information for serializing and +deserializing objects and data types. + +## ToC + +* [About](#about) +* [Terminology](#terminology) +* [Constants](#constants) +* [Overview](#overview) + + [Serialize/Encode](#serializeencode) + - [int/uint: 8/16/24/32/64/256](#intuint-816243264256) + - [Address](#address) + - [Hash32](#hash32) + - [Bytes](#bytes) + - [List](#list) + + [Deserialize/Decode](#deserializedecode) + - [int/uint: 8/16/24/32/64/256](#intuint-816243264256-1) + - [Address](#address-1) + - [Hash32](#hash32-1) + - [Bytes](#bytes-1) + - [List](#list-1) +* [Implementations](#implementations) + +## About + +`SimpleSerialize` was first proposed by Vitalik Buterin as the serializaiton +protocol for use in the Ethereum 2.0 Beacon Chain. + +The core feature of `ssz` is the simplicity of the serialization with low +overhead. + +## Terminology + +| Term | Definition | +|:-------------|:-----------------------------------------------------------------------------------------------| +| `big` | Big Endian | +| `byte_order` | Specifies [endianness:](https://en.wikipedia.org/wiki/Endianness) Big Endian or Little Endian. | +| `len` | Length/Number of Bytes. | +| `to_bytes` | Convert to bytes. Should take parameters ``size`` and ``byte_order``. | +| `from_bytes` | Convert form bytes to object. Should take ``bytes`` and ``byte_order``. | +| `value` | The value to serialize. | +| `rawbytes` | Raw serialized bytes. | + +## Constants + +| Constant | Value | Definition | +|:---------------|:-----:|:------------------------------------------------------------------------| +| `LENGTH_BYTES` | 4 | Number of bytes used for the length added before the serialized object. | + + +## Overview + +### Serialize/Encode + +#### int/uint: 8/16/24/32/64/256 + +Convert directly to bytes the size of the int. (e.g. ``uint16 = 2 bytes``) + +All integers are serialized as **big endian**. + +| Check to perform | Code | +|:---------------------------------|:------------------------| +| Size is a byte integer | ``int_size % 8 == 0`` | +| Value is less than max | ``2**int_size > value`` | + +```python +buffer_size = int_size / 8 +return value.to_bytes(buffer_size, 'big') +``` + +#### Address + +The address should already come as a hash/byte format. Ensure that length is +**20**. + +| Check to perform | Code | +|:-----------------------|:---------------------| +| Length is correct (20) | ``len(value) == 20`` | + +```python +assert( len(value) == 20 ) +return value +``` + +#### Hash32 + +The hash32 should already be a 32 byte length serialized data format. The safety +check ensures the 32 byte length is satisfied. + +| Check to perform | Code | +|:-----------------------|:---------------------| +| Length is correct (32) | ``len(value) == 32`` | + +```python +assert( len(value) == 32 ) +return value +``` + +#### Bytes + +For general `byte` type: +1. Get the length/number of bytes; Encode into a 4 byte integer. +2. Append the value to the length and return: ``[ length_bytes ] + [ + value_bytes ]`` + +```python +byte_length = (len(value)).to_bytes(4, 'big') +return byte_length + value +``` + +#### List + +For lists of values, get the length of the list and then serialize the value +of each item in the list: +1. For each item in list: + 1. serialize. + 2. append to string. +2. Get size of serialized string. Encode into a 4 byte integer. + +```python +serialized_list_string = '' + +for item in value: + serialized_list_string += serialize(item) + +serialized_len = len(serialized_list_string) + +return serialized_len + serialized_list_string +``` + +### Deserialize/Decode + +The decoding requires knowledge of the type of the item to be decoded. When +performing decoding on an entire serialized string, it also requires knowledge +of what order the objects have been serialized in. + +Note: Each return will provide ``deserialized_object, new_index`` keeping track +of the new index. + +At each step, the following checks should be made: + +| Check Type | Check | +|:-------------------------|:----------------------------------------------------------| +| Ensure sufficient length | ``length(rawbytes) > current_index + deserialize_length`` | + +#### int/uint: 8/16/24/32/64/256 + +Convert directly from bytes into integer utilising the number of bytes the same +size as the integer length. (e.g. ``uint16 == 2 bytes``) + +All integers are interpreted as **big endian**. + +```python +byte_length = int_size / 8 +new_index = current_index + int_size +return int.from_bytes(rawbytes[current_index:current_index+int_size], 'big'), new_index +``` + +#### Address + +Return the 20 bytes. + +```python +new_index = current_index + 20 +return rawbytes[current_index:current_index+20], new_index +``` + +#### Hash32 + +Return the 32 bytes. + +```python +new_index = current_index + 32 +return rawbytes[current_index:current_index+32], new_index +``` + +#### Bytes + +Get the length of the bytes, return the bytes. + +```python +bytes_length = int.from_bytes(rawbytes[current_index:current_index+4], 'big') +new_index = current_index + 4 + bytes_lenth +return rawbytes[current_index+4:current_index+4+bytes_length], new_index +``` + +#### List + +Deserailize each object in the list. +1. Get the length of the serialized list. +2. Loop through deseralizing each item in the list until you reach the +entire length of the list. + + +| Check type | code | +|:------------------------------------|:--------------------------------------| +| rawbytes has enough left for length | ``len(rawbytes) > current_index + 4`` | + +```python +total_length = int.from_bytes(rawbytes[current_index:current_index+4], 'big') +new_index = current_index + 4 + total_length +item_index = current_index + 4 +deserialized_list = [] + +while item_index < new_index: + object, item_index = deserialize(rawbytes, item_index, item_type) + deserialized_list.append(object) + +return deserialized_list, new_index +``` + +## Implementations + +| Language | Implementation | Description | +|:--------:|--------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------| +| Python | [ https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py ](https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py) | Beacon chain reference implementation written in Python. | +| Rust | [ https://github.com/sigp/lighthouse/tree/master/ssz ](https://github.com/sigp/lighthouse/tree/master/ssz) | Lighthouse (Rust Ethereum 2.0 Node) maintained SimpleSerialize. | +