status-im
/
eth2.0-specs
mirror of https://github.com/status-im/eth2.0-specs.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge pull request #755 from ethereum/dev 

v0.5.0 release
This commit is contained in:
Danny Ryan 2019-03-14 15:49:49 -06:00 committed by GitHub
parent 34cd96be54 15bf3c4258
commit aeb5bb9b11
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 1842 additions and 1239 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

116
LICENSE Normal file
View File

@ -0,0 +1,116 @@
CC0 1.0 Universal
Statement of Purpose
The laws of most jurisdictions throughout the world automatically confer
exclusive Copyright and Related Rights (defined below) upon the creator and
subsequent owner(s) (each and all, an "owner") of an original work of
authorship and/or a database (each, a "Work").
Certain owners wish to permanently relinquish those rights to a Work for the
purpose of contributing to a commons of creative, cultural and scientific
works ("Commons") that the public can reliably and without fear of later
claims of infringement build upon, modify, incorporate in other works, reuse
and redistribute as freely as possible in any form whatsoever and for any
purposes, including without limitation commercial purposes. These owners may
contribute to the Commons to promote the ideal of a free culture and the
further production of creative, cultural and scientific works, or to gain
reputation or greater distribution for their Work in part through the use and
efforts of others.
For these and/or other purposes and motivations, and without any expectation
of additional consideration or compensation, the person associating CC0 with a
Work (the "Affirmer"), to the extent that he or she is an owner of Copyright
and Related Rights in the Work, voluntarily elects to apply CC0 to the Work
and publicly distribute the Work under its terms, with knowledge of his or her
Copyright and Related Rights in the Work and the meaning and intended legal
effect of CC0 on those rights.
1. Copyright and Related Rights. A Work made available under CC0 may be
protected by copyright and related or neighboring rights ("Copyright and
Related Rights"). Copyright and Related Rights include, but are not limited
to, the following:
i. the right to reproduce, adapt, distribute, perform, display, communicate,
and translate a Work;
ii. moral rights retained by the original author(s) and/or performer(s);
iii. publicity and privacy rights pertaining to a person's image or likeness
depicted in a Work;
iv. rights protecting against unfair competition in regards to a Work,
subject to the limitations in paragraph 4(a), below;
v. rights protecting the extraction, dissemination, use and reuse of data in
a Work;
vi. database rights (such as those arising under Directive 96/9/EC of the
European Parliament and of the Council of 11 March 1996 on the legal
protection of databases, and under any national implementation thereof,
including any amended or successor version of such directive); and
vii. other similar, equivalent or corresponding rights throughout the world
based on applicable law or treaty, and any national implementations thereof.
2. Waiver. To the greatest extent permitted by, but not in contravention of,
applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and
unconditionally waives, abandons, and surrenders all of Affirmer's Copyright
and Related Rights and associated claims and causes of action, whether now
known or unknown (including existing as well as future claims and causes of
action), in the Work (i) in all territories worldwide, (ii) for the maximum
duration provided by applicable law or treaty (including future time
extensions), (iii) in any current or future medium and for any number of
copies, and (iv) for any purpose whatsoever, including without limitation
commercial, advertising or promotional purposes (the "Waiver"). Affirmer makes
the Waiver for the benefit of each member of the public at large and to the
detriment of Affirmer's heirs and successors, fully intending that such Waiver
shall not be subject to revocation, rescission, cancellation, termination, or
any other legal or equitable action to disrupt the quiet enjoyment of the Work
by the public as contemplated by Affirmer's express Statement of Purpose.
3. Public License Fallback. Should any part of the Waiver for any reason be
judged legally invalid or ineffective under applicable law, then the Waiver
shall be preserved to the maximum extent permitted taking into account
Affirmer's express Statement of Purpose. In addition, to the extent the Waiver
is so judged Affirmer hereby grants to each affected person a royalty-free,
non transferable, non sublicensable, non exclusive, irrevocable and
unconditional license to exercise Affirmer's Copyright and Related Rights in
the Work (i) in all territories worldwide, (ii) for the maximum duration
provided by applicable law or treaty (including future time extensions), (iii)
in any current or future medium and for any number of copies, and (iv) for any
purpose whatsoever, including without limitation commercial, advertising or
promotional purposes (the "License"). The License shall be deemed effective as
of the date CC0 was applied by Affirmer to the Work. Should any part of the
License for any reason be judged legally invalid or ineffective under
applicable law, such partial invalidity or ineffectiveness shall not
invalidate the remainder of the License, and in such case Affirmer hereby
affirms that he or she will not (i) exercise any of his or her remaining
Copyright and Related Rights in the Work or (ii) assert any associated claims
and causes of action with respect to the Work, in either case contrary to
Affirmer's express Statement of Purpose.
4. Limitations and Disclaimers.
a. No trademark or patent rights held by Affirmer are waived, abandoned,
surrendered, licensed or otherwise affected by this document.
b. Affirmer offers the Work as-is and makes no representations or warranties
of any kind concerning the Work, express, implied, statutory or otherwise,
including without limitation warranties of title, merchantability, fitness
for a particular purpose, non infringement, or the absence of latent or
other defects, accuracy, or the present or absence of errors, whether or not
discoverable, all to the greatest extent permissible under applicable law.
c. Affirmer disclaims responsibility for clearing rights of other persons
that may apply to the Work or any use thereof, including without limitation
any person's Copyright and Related Rights in the Work. Further, Affirmer
disclaims responsibility for obtaining any necessary consents, permissions
or other rights required for any use of the Work.
d. Affirmer understands and acknowledges that Creative Commons is not a
party to this document and has no duty or obligation with respect to this
CC0 or use of the Work.
For more information, please see
<http://creativecommons.org/publicdomain/zero/1.0/>

2006
specs/core/0_beacon-chain.md
View File

File diff suppressed because it is too large Load Diff

402
specs/core/1_shard-data-chains.md
View File

@ -2,6 +2,8 @@
**NOTICE**: This document is a work-in-progress for researchers and implementers. It reflects recent spec changes and takes precedence over the [Python proof-of-concept implementation](https://github.com/ethereum/beacon_chain).
At the current stage, Phase 1, while fundamentally feature-complete, is still subject to change. Development teams with spare resources may consider starting on the "Shard chains and crosslink data" section; at least basic properties, such as the fact that a shard block can get created every slot and is dependent on both a parent block in the same shard and a beacon chain block at or before that same slot, are unlikely to change, though details are likely to undergo similar kinds of changes to what Phase 0 has undergone since the start of the year.
## Table of contents
<!-- TOC -->
@ -15,19 +17,20 @@
- [Time parameters](#time-parameters)
- [Max operations per block](#max-operations-per-block)
- [Signature domains](#signature-domains)
- [Helper functions](#helper-functions)
- [Shard chains and crosslink data](#shard-chains-and-crosslink-data)
- [Helper functions](#helper-functions)
- [`get_split_offset`](#get_split_offset)
- [`get_shuffled_committee`](#get_shuffled_committee)
- [`get_persistent_committee`](#get_persistent_committee)
- [`get_shard_proposer_index`](#get_shard_proposer_index)
- [Data Structures](#data-structures)
- [Data Structures](#data-structures)
- [Shard chain blocks](#shard-chain-blocks)
- [Shard block processing](#shard-block-processing)
- [Shard block processing](#shard-block-processing)
- [Verifying shard block data](#verifying-shard-block-data)
- [Verifying a crosslink](#verifying-a-crosslink)
- [Shard block fork choice rule](#shard-block-fork-choice-rule)
- [Updates to the beacon chain](#updates-to-the-beacon-chain)
- [Data structures](#data-structures)
- [Updates to the beacon chain](#updates-to-the-beacon-chain)
- [Data structures](#data-structures)
- [`Validator`](#validator)
- [`BeaconBlockBody`](#beaconblockbody)
- [`BranchChallenge`](#branchchallenge)
@ -35,20 +38,20 @@
- [`BranchChallengeRecord`](#branchchallengerecord)
- [`SubkeyReveal`](#subkeyreveal)
- [Helpers](#helpers)
- [`get_attestation_merkle_depth`](#get_attestation_merkle_depth)
- [`get_attestation_data_merkle_depth`](#get_attestation_data_merkle_depth)
- [`epoch_to_custody_period`](#epoch_to_custody_period)
- [`slot_to_custody_period`](#slot_to_custody_period)
- [`get_current_custody_period`](#get_current_custody_period)
- [`verify_custody_subkey_reveal`](#verify_custody_subkey_reveal)
- [`prepare_validator_for_withdrawal`](#prepare_validator_for_withdrawal)
- [`penalize_validator`](#penalize_validator)
- [Per-slot processing](#per-slot-processing)
- [Per-slot processing](#per-slot-processing)
- [Operations](#operations)
- [Branch challenges](#branch-challenges)
- [Branch responses](#branch-responses)
- [Subkey reveals](#subkey-reveals)
- [Per-epoch processing](#per-epoch-processing)
- [One-time phase 1 initiation transition](#one-time-phase-1-initiation-transition)
- [Per-epoch processing](#per-epoch-processing)
- [One-time phase 1 initiation transition](#one-time-phase-1-initiation-transition)
<!-- /TOC -->
@ -71,6 +74,9 @@ Phase 1 depends upon all of the constants defined in [Phase 0](0_beacon-chain.md
| `SHARD_CHUNK_SIZE` | 2**5 (= 32) | bytes |
| `SHARD_BLOCK_SIZE` | 2**14 (= 16,384) | bytes |
| `MINOR_REWARD_QUOTIENT` | 2**8 (= 256) | |
| `MAX_POC_RESPONSE_DEPTH` | 5 | |
| `ZERO_PUBKEY` | int_to_bytes48(0)| |
| `VALIDATOR_NULL` | 2**64 - 1 | |
#### Time parameters
@ -84,19 +90,25 @@ Phase 1 depends upon all of the constants defined in [Phase 0](0_beacon-chain.md
#### Max operations per block
| Name | Value |
|-------------------------------|---------------|
| `MAX_BRANCH_CHALLENGES` | 2**2 (= 4) |
| `MAX_BRANCH_RESPONSES` | 2**4 (= 16) |
| `MAX_EARLY_SUBKEY_REVEALS` | 2**4 (= 16) |
| Name | Value |
|----------------------------------------------------|---------------|
| `MAX_BRANCH_CHALLENGES` | 2**2 (= 4) |
| `MAX_BRANCH_RESPONSES` | 2**4 (= 16) |
| `MAX_EARLY_SUBKEY_REVEALS` | 2**4 (= 16) |
| `MAX_INTERACTIVE_CUSTODY_CHALLENGE_INITIATIONS` | 2 |
| `MAX_INTERACTIVE_CUSTODY_CHALLENGE_RESPONSES` | 16 |
| `MAX_INTERACTIVE_CUSTODY_CHALLENGE_CONTINUTATIONS` | 16 |
#### Signature domains
| Name | Value |
|------------------------|-----------------|
| `DOMAIN_SHARD_PROPOSER`| 129 |
| `DOMAIN_SHARD_ATTESTER`| 130 |
| `DOMAIN_CUSTODY_SUBKEY`| 131 |
| Name | Value |
|------------------------------|-----------------|
| `DOMAIN_SHARD_PROPOSER` | 129 |
| `DOMAIN_SHARD_ATTESTER` | 130 |
| `DOMAIN_CUSTODY_SUBKEY` | 131 |
| `DOMAIN_CUSTODY_INTERACTIVE` | 132 |
# Shard chains and crosslink data
## Helper functions
@ -158,7 +170,6 @@ def get_persistent_committee(state: BeaconState,
[i for i in later_committee if epoch % PERSISTENT_COMMITTEE_PERIOD >= get_switchover_epoch(i)]
)))
```
#### `get_shard_proposer_index`
```python
@ -290,7 +301,7 @@ The `shard_chain_commitment` is only valid if it equals `compute_commitment(head
### Shard block fork choice rule
The fork choice rule for any shard is LMD GHOST using the shard chain attestations of the persistent committee and the beacon chain attestations of the crosslink committee currently assigned to that shard, but instead of being rooted in the genesis it is rooted in the latest block referenced in the most recent accepted crosslink (ie. `state.crosslinks[shard].crosslink_data_root`). Only blocks whose `beacon_chain_ref` is the block in the main beacon chain at the specified `slot` should be considered (if the beacon chain skips a slot, then the block at that slot is considered to be the block in the beacon chain at the highest slot lower than a slot).
The fork choice rule for any shard is LMD GHOST using the shard chain attestations of the persistent committee and the beacon chain attestations of the crosslink committee currently assigned to that shard, but instead of being rooted in the genesis it is rooted in the block referenced in the most recent accepted crosslink (ie. `state.crosslinks[shard].shard_block_root`). Only blocks whose `beacon_chain_ref` is the block in the main beacon chain at the specified `slot` should be considered (if the beacon chain skips a slot, then the block at that slot is considered to be the block in the beacon chain at the highest slot lower than a slot).
# Updates to the beacon chain
@ -301,7 +312,6 @@ The fork choice rule for any shard is LMD GHOST using the shard chain attestatio
Add member values to the end of the `Validator` object:
```python
'open_branch_challenges': [BranchChallengeRecord],
'next_subkey_to_reveal': 'uint64',
'reveal_max_periods_late': 'uint64',
```
@ -309,7 +319,6 @@ Add member values to the end of the `Validator` object:
And the initializers:
```python
'open_branch_challenges': [],
'next_subkey_to_reveal': get_current_custody_period(state),
'reveal_max_periods_late': 0,
```
@ -322,6 +331,10 @@ Add member values to the `BeaconBlockBody` structure:
'branch_challenges': [BranchChallenge],
'branch_responses': [BranchResponse],
'subkey_reveals': [SubkeyReveal],
'interactive_custody_challenge_initiations': [InteractiveCustodyChallengeInitiation],
'interactive_custody_challenge_responses': [InteractiveCustodyChallengeResponse],
'interactive_custody_challenge_continuations': [InteractiveCustodyChallengeContinuation],
```
And initialize to the following:
@ -332,6 +345,17 @@ And initialize to the following:
'subkey_reveals': [],
```
### `BeaconState`
Add member values to the `BeaconState` structure:
```python
'branch_challenge_records': [BranchChallengeRecord],
'next_branch_challenge_id': 'uint64',
'custody_challenge_records': [InteractiveCustodyChallengeRecord],
'next_custody_challenge_id': 'uint64',
```
### `BranchChallenge`
Define a `BranchChallenge` as follows:
@ -350,11 +374,10 @@ Define a `BranchResponse` as follows:
```python
{
'responder_index': 'uint64',
'challenge_id': 'uint64',
'responding_to_custody_challenge': 'bool',
'data': 'bytes32',
'branch': ['bytes32'],
'data_index': 'uint64',
'root': 'bytes32',
}
```
@ -364,14 +387,75 @@ Define a `BranchChallengeRecord` as follows:
```python
{
'challenge_id': 'uint64',
'challenger_index': 'uint64',
'responder_index': 'uint64',
'root': 'bytes32',
'depth': 'uint64',
'inclusion_epoch': 'uint64',
'deadline': 'uint64',
'data_index': 'uint64',
}
```
### `InteractiveCustodyChallengeRecord`
```python
{
'challenge_id': 'uint64',
'challenger_index': 'uint64',
'responder_index': 'uint64',
# Initial data root
'data_root': 'bytes32',
# Initial custody bit
'custody_bit': 'bool',
# Responder subkey
'responder_subkey': 'bytes96',
# The hash in the PoC tree in the position that we are currently at
'current_custody_tree_node': 'bytes32',
# The position in the tree, in terms of depth and position offset
'depth': 'uint64',
'offset': 'uint64',
# Max depth of the branch
'max_depth': 'uint64',
# Deadline to respond (as an epoch)
'deadline': 'uint64',
}
```
### `InteractiveCustodyChallengeInitiation`
```python
{
'attestation': SlashableAttestation,
'responder_index': 'uint64',
'challenger_index': 'uint64',
'responder_subkey': 'bytes96',
'signature': 'bytes96',
}
```
### `InteractiveCustodyChallengeResponse`
```python
{
'challenge_id': 'uint64',
'hashes': ['bytes32'],
'signature': 'bytes96',
}
```
### `InteractiveCustodyChallengeContinuation`
```python
{
'challenge_id': 'uint64',
'sub_index': 'uint64',
'new_custody_tree_node': 'bytes32',
'proof': ['bytes32'],
'signature': 'bytes96',
}
```
### `SubkeyReveal`
Define a `SubkeyReveal` as follows:
@ -388,6 +472,20 @@ Define a `SubkeyReveal` as follows:
## Helpers
### `get_branch_challenge_record_by_id`
```python
def get_branch_challenge_record_by_id(state: BeaconState, id: int) -> BranchChallengeRecord:
return [c for c in state.branch_challenges if c.challenge_id == id][0]
```
### `get_custody_challenge_record_by_id`
```python
def get_custody_challenge_record_by_id(state: BeaconState, id: int) -> BranchChallengeRecord:
return [c for c in state.branch_challenges if c.challenge_id == id][0]
```
### `get_attestation_merkle_depth`
```python
@ -453,6 +551,19 @@ def verify_custody_subkey_reveal(pubkey: bytes48,
)
```
### `verify_signed_challenge_message`
```python
def verify_signed_challenge_message(message: Any, pubkey: bytes48) -> bool:
return bls_verify(
message_hash=signed_root(message),
pubkey=pubkey,
signature=message.signature,
domain=get_domain(state, get_current_epoch(state), DOMAIN_CUSTODY_INTERACTIVE)
)
```
### `penalize_validator`
Change the definition of `penalize_validator` as follows:
@ -493,29 +604,88 @@ Add the following operations to the per-slot processing, in order the given belo
Verify that `len(block.body.branch_challenges) <= MAX_BRANCH_CHALLENGES`.
For each `challenge` in `block.body.branch_challenges`:
For each `challenge` in `block.body.branch_challenges`, run:
* Verify that `slot_to_epoch(challenge.attestation.data.slot) >= get_current_epoch(state) - MAX_BRANCH_CHALLENGE_DELAY`.
* Verify that `state.validator_registry[responder_index].exit_epoch >= get_current_epoch(state) - MAX_BRANCH_CHALLENGE_DELAY`.
* Verify that `verify_slashable_attestation(state, challenge.attestation)` returns `True`.
* Verify that `challenge.responder_index` is in `challenge.attestation.validator_indices`.
* Let `depth = get_attestation_merkle_depth(challenge.attestation)`. Verify that `challenge.data_index < 2**depth`.
* Verify that there does not exist a `BranchChallengeRecord` in `state.validator_registry[challenge.responder_index].open_branch_challenges` with `root == challenge.attestation.data.shard_chain_commitment` and `data_index == data_index`.
* Append to `state.validator_registry[challenge.responder_index].open_branch_challenges` the object `BranchChallengeRecord(challenger_index=get_beacon_proposer_index(state, state.slot), root=challenge.attestation.data.shard_chain_commitment, depth=depth, inclusion_epoch=get_current_epoch(state), data_index=data_index)`.
**Invariant**: the `open_branch_challenges` array will always stay sorted in order of `inclusion_epoch`.
```python
def process_branch_challenge(state: BeaconState,
challenge: BranchChallenge) -> None:
# Check that it's not too late to challenge
assert slot_to_epoch(challenge.attestation.data.slot) >= get_current_epoch(state) - MAX_BRANCH_CHALLENGE_DELAY
assert state.validator_registry[responder_index].exit_epoch >= get_current_epoch(state) - MAX_BRANCH_CHALLENGE_DELAY
# Check the attestation is valid
assert verify_slashable_attestation(state, challenge.attestation)
# Check that the responder participated
assert challenger.responder_index in challenge.attestation.validator_indices
# Check the challenge is not a duplicate
assert [
c for c in state.branch_challenge_records if c.root == challenge.attestation.data.crosslink_data_root and
c.data_index == challenge.data_index
] == []
# Check validity of depth
depth = get_attestation_merkle_depth(challenge.attestation)
assert c.data_index < 2**depth
# Add new challenge
state.branch_challenge_records.append(BranchChallengeRecord(
challenge_id=state.next_branch_challenge_id,
challenger_index=get_beacon_proposer_index(state, state.slot),
root=challenge.attestation.data.shard_chain_commitment,
depth=depth,
deadline=get_current_epoch(state) + CHALLENGE_RESPONSE_DEADLINE,
data_index=challenge.data_index
))
state.next_branch_challenge_id += 1
```
#### Branch responses
Verify that `len(block.body.branch_responses) <= MAX_BRANCH_RESPONSES`.
For each `response` in `block.body.branch_responses`:
For each `response` in `block.body.branch_responses`, if `response.responding_to_custody_challenge == False`, run:
* Find the `BranchChallengeRecord` in `state.validator_registry[response.responder_index].open_branch_challenges` whose (`root`, `data_index`) match the (`root`, `data_index`) of the `response`. Verify that one such record exists (it is not possible for there to be more than one), call it `record`.
* Verify that `verify_merkle_branch(leaf=response.data, branch=response.branch, depth=record.depth, index=record.data_index, root=record.root)` is True.
* Verify that `get_current_epoch(state) >= record.inclusion_epoch + ENTRY_EXIT_DELAY`.
* Remove the `record` from `state.validator_registry[response.responder_index].open_branch_challenges`
* Determine the proposer `proposer_index = get_beacon_proposer_index(state, state.slot)` and set `state.validator_balances[proposer_index] += base_reward(state, index) // MINOR_REWARD_QUOTIENT`.
```python
def process_branch_exploration_response(state: BeaconState,
response: BranchResponse) -> None:
challenge = get_branch_challenge_record_by_id(response.challenge_id)
assert verify_merkle_branch(
leaf=response.data,
branch=response.branch,
depth=challenge.depth,
index=challenge.data_index,
root=challenge.root
)
# Must wait at least ENTRY_EXIT_DELAY before responding to a branch challenge
assert get_current_epoch(state) >= challenge.inclusion_epoch + ENTRY_EXIT_DELAY
state.branch_challenge_records.pop(challenge)
# Reward the proposer
proposer_index = get_beacon_proposer_index(state, state.slot)
state.validator_balances[proposer_index] += base_reward(state, index) // MINOR_REWARD_QUOTIENT
```
If `response.responding_to_custody_challenge == True`, run:
```python
def process_branch_custody_response(state: BeaconState,
response: BranchResponse) -> None:
challenge = get_custody_challenge_record_by_id(response.challenge_id)
responder = state.validator_registry[challenge.responder_index]
# Verify we're not too late
assert get_current_epoch(state) < responder.withdrawable_epoch
# Verify the Merkle branch *of the data tree*
assert verify_merkle_branch(
leaf=response.data,
branch=response.branch,
depth=challenge.max_depth,
index=challenge.offset,
root=challenge.data_root
)
# Responder wins
if hash(challenge.responder_subkey + response.data) == challenge.current_custody_tree_node:
penalize_validator(state, challenge.challenger_index, challenge.responder_index)
# Challenger wins
else:
penalize_validator(state, challenge.responder_index, challenge.challenger_index)
state.custody_challenge_records.pop(challenge)
```
#### Subkey reveals
@ -541,6 +711,126 @@ In case (ii):
* Set `state.validator_registry[reveal.validator_index].next_subkey_to_reveal += 1`
* Set `state.validator_registry[reveal.validator_index].reveal_max_periods_late = max(state.validator_registry[reveal.validator_index].reveal_max_periods_late, get_current_period(state) - reveal.period)`.
#### Interactive custody challenge initiations
Verify that `len(block.body.interactive_custody_challenge_initiations) <= MAX_INTERACTIVE_CUSTODY_CHALLENGE_INITIATIONS`.
For each `initiation` in `block.body.interactive_custody_challenge_initiations`, use the following function to process it:
```python
def process_initiation(state: BeaconState,
initiation: InteractiveCustodyChallengeInitiation) -> None:
challenger = state.validator_registry[initiation.challenger_index]
responder = state.validator_registry[initiation.responder_index]
# Verify the signature
assert verify_signed_challenge_message(initiation, challenger.pubkey)
# Verify the attestation
assert verify_slashable_attestation(initiation.attestation, state)
# Check that the responder actually participated in the attestation
assert initiation.responder_index in attestation.validator_indices
# Any validator can be a challenger or responder of max 1 challenge at a time
for c in state.custody_challenge_records:
assert c.challenger_index != initiation.challenger_index
assert c.responder_index != initiation.responder_index
# Can't challenge if you've been penalized
assert challenger.penalized_epoch == FAR_FUTURE_EPOCH
# Make sure the revealed subkey is valid
assert verify_custody_subkey_reveal(
pubkey=state.validator_registry[responder_index].pubkey,
subkey=initiation.responder_subkey,
period=slot_to_custody_period(attestation.data.slot)
)
# Verify that the attestation is still eligible for challenging
min_challengeable_epoch = responder.exit_epoch - CUSTODY_PERIOD_LENGTH * (1 + responder.reveal_max_periods_late)
assert min_challengeable_epoch <= slot_to_epoch(initiation.attestation.data.slot)
# Create a new challenge object
state.branch_challenge_records.append(InteractiveCustodyChallengeRecord(
challenge_id=state.next_branch_challenge_id,
challenger_index=initiation.challenger_index,
responder_index=initiation.responder_index,
data_root=attestation.custody_commitment,
custody_bit=get_bitfield_bit(attestation.custody_bitfield, attestation.validator_indices.index(responder_index)),
responder_subkey=responder_subkey,
current_custody_tree_node=ZERO_HASH,
depth=0,
offset=0,
max_depth=get_attestation_data_merkle_depth(initiation.attestation.data),
deadline=get_current_epoch(state) + CHALLENGE_RESPONSE_DEADLINE
))
state.next_branch_challenge_id += 1
# Responder can't withdraw yet!
state.validator_registry[responder_index].withdrawable_epoch = FAR_FUTURE_EPOCH
```
#### Interactive custody challenge responses
A response provides 32 hashes that are under current known proof of custody tree node. Note that at the beginning the tree node is just one bit of the custody root, so we ask the responder to sign to commit to the top 5 levels of the tree and therefore the root hash; at all other stages in the game responses are self-verifying.
Verify that `len(block.body.interactive_custody_challenge_responses) <= MAX_INTERACTIVE_CUSTODY_CHALLENGE_RESPONSES`.
For each `response` in `block.body.interactive_custody_challenge_responses`, use the following function to process it:
```python
def process_response(state: BeaconState,
response: InteractiveCustodyChallengeResponse) -> None:
challenge = get_custody_challenge_record_by_id(state, response.challenge_id)
responder = state.validator_registry[challenge.responder_index]
# Check that the right number of hashes was provided
expected_depth = min(challenge.max_depth - challenge.depth, MAX_POC_RESPONSE_DEPTH)
assert 2**expected_depth == len(response.hashes)
# Must make some progress!
assert expected_depth > 0
# Check the hashes match the previously provided root
root = merkle_root(response.hashes)
# If this is the first response check the bit and the signature and set the root
if challenge.depth == 0:
assert get_bitfield_bit(root, 0) == challenge.custody_bit
assert verify_signed_challenge_message(response, responder.pubkey)
challenge.current_custody_tree_node = root
# Otherwise just check the response against the root
else:
assert root == challenge_data.current_custody_tree_node
# Update challenge data
challenge.deadline=FAR_FUTURE_EPOCH
responder.withdrawable_epoch = get_current_epoch(state) + MAX_POC_RESPONSE_DEPTH
```
#### Interactive custody challenge continuations
Once a response provides 32 hashes, the challenger has the right to choose any one of them that they feel is constructed incorrectly to continue the game. Note that eventually, the game will get to the point where the `new_custody_tree_node` is a leaf node.
Verify that `len(block.body.interactive_custody_challenge_continuations) <= MAX_INTERACTIVE_CUSTODY_CHALLENGE_CONTINUATIONS`.
For each `continuation` in `block.body.interactive_custody_challenge_continuations`, use the following function to process it:
```python
def process_continuation(state: BeaconState,
continuation: InteractiveCustodyChallengeContinuation) -> None:
challenge = get_custody_challenge_record_by_id(state, continuation.challenge_id)
challenger = state.validator_registry[challenge.challenger_index]
responder = state.validator_registry[challenge.responder_index]
expected_depth = min(challenge_data.max_depth - challenge_data.depth, MAX_POC_RESPONSE_DEPTH)
# Verify we're not too late
assert get_current_epoch(state) < responder.withdrawable_epoch
# Verify the Merkle branch (the previous custody response provided the next level of hashes so the
# challenger has the info to make any Merkle branch)
assert verify_merkle_branch(
leaf=new_custody_tree_node,
branch=continuation.proof,
depth=expected_depth,
index=sub_index,
root=challenge_data.current_custody_tree_node
)
# Verify signature
assert verify_signed_challenge_message(continuation, challenger.pubkey)
# Update the challenge data
challenge.current_custody_tree_node = continuation.new_custody_tree_node
challenge.depth += expected_depth
challenge.deadline = get_current_epoch(state) + MAX_POC_RESPONSE_DEPTH
responder.withdrawable_epoch = FAR_FUTURE_EPOCH
challenge.offset = challenge_data.offset * 2**expected_depth + sub_index
```
## Per-epoch processing
Add the following loop immediately below the `process_ejections` loop:
@ -548,12 +838,18 @@ Add the following loop immediately below the `process_ejections` loop:
```python
def process_challenge_absences(state: BeaconState) -> None:
"""
Iterate through the validator registry
Iterate through the challenge list
and penalize validators with balance that did not answer challenges.
"""
for index, validator in enumerate(state.validator_registry):
if len(validator.open_branch_challenges) > 0 and get_current_epoch(state) > validator.open_branch_challenges[0].inclusion_epoch + CHALLENGE_RESPONSE_DEADLINE:
penalize_validator(state, index, validator.open_branch_challenges[0].challenger_index)
for c in state.branch_challenge_records:
if get_current_epoch(state) > c.deadline:
penalize_validator(state, c.responder_index, c.challenger_index)
for c in state.custody_challenge_records:
if get_current_epoch(state) > c.deadline:
penalize_validator(state, c.responder_index, c.challenger_index)
if get_current_epoch(state) > state.validator_registry[c.responder_index].withdrawable_epoch:
penalize_validator(state, c.challenger_index, c.responder_index)
```
In `process_penalties_and_exits`, change the definition of `eligible` to the following (note that it is not a pure function because `state` is declared in the surrounding scope):
@ -562,7 +858,7 @@ In `process_penalties_and_exits`, change the definition of `eligible` to the fol
def eligible(index):
validator = state.validator_registry[index]
# Cannot exit if there are still open branch challenges
if len(validator.open_branch_challenges) > 0:
if [c for c in state.branch_challenge_records if c.responder_index == index] != []:
return False
# Cannot exit if you have not revealed all of your subkeys
elif validator.next_subkey_to_reveal <= epoch_to_custody_period(validator.exit_epoch):
@ -582,7 +878,15 @@ Run the following on the fork block after per-slot processing and before per-blo
For all `validator` in `ValidatorRegistry`, update it to the new format and fill the new member values with:
```python
'open_branch_challenges': [],
'next_subkey_to_reveal': get_current_custody_period(state),
'reveal_max_periods_late': 0,
```
Update the `BeaconState` to the new format and fill the new member values with:
```python
'branch_challenge_records': [],
'next_branch_challenge_id': 0,
'custody_challenge_records': [],
'next_custody_challenge_id': 0,
```

464
specs/simple-serialize.md
View File

@ -1,424 +1,124 @@
# [WIP] SimpleSerialize (SSZ) Spec
# SimpleSerialiZe (SSZ)
This is the **work in progress** document to describe `SimpleSerialize`, the
current selected serialization method for Ethereum 2.0 using the Beacon Chain.
This is a **work in progress** describing typing, serialization and Merkleization of Ethereum 2.0 objects.
This document specifies the general information for serializing and
deserializing objects and data types.
## Table of contents
## ToC
* [About](#about)
* [Variables and Functions](#variables-and-functions)
* [Constants](#constants)
* [Overview](#overview)
+ [Serialize/Encode](#serializeencode)
- [uintN](#uintn)
- [bool](#bool)
- [bytesN](#bytesn)
- [List/Vectors](#listvectors)
- [Container](#container)
+ [Deserialize/Decode](#deserializedecode)
- [uintN](#uintn-1)
- [bool](#bool-1)
- [bytesN](#bytesn-1)
- [List/Vectors](#listvectors-1)
- [Container](#container-1)
+ [Tree Hash](#tree-hash)
- [`uint8`..`uint256`, `bool`, `bytes1`..`bytes32`](#uint8uint256-bool-bytes1bytes32)
- [`uint264`..`uintN`, `bytes33`..`bytesN`](#uint264uintn-bytes33bytesn)
- [List/Vectors](#listvectors-2)
- [Container](#container-2)
+ [Signed Roots](#signed-roots)
* [Implementations](#implementations)
## About
`SimpleSerialize` was first proposed by Vitalik Buterin as the serialization
protocol for use in the Ethereum 2.0 Beacon Chain.
The core feature of `ssz` is the simplicity of the serialization with low
overhead.
## Variables and Functions
| Term | Definition |
|:-------------|:-----------------------------------------------------------------------------------------------|
| `little` | Little endian. |
| `byteorder` | Specifies [endianness](https://en.wikipedia.org/wiki/Endianness): big endian or little endian. |
| `len` | Length/number of bytes. |
| `to_bytes` | Convert to bytes. Should take parameters ``size`` and ``byteorder``. |
| `from_bytes` | Convert from bytes to object. Should take ``bytes`` and ``byteorder``. |
| `value` | The value to serialize. |
| `rawbytes` | Raw serialized bytes. |
| `deserialized_object` | The deserialized data in the data structure of your programming language. |
| `new_index` | An index to keep track the latest position where the `rawbytes` have been deserialized. |
- [Constants](#constants)
- [Typing](#typing)
- [Basic types](#basic-types)
- [Composite types](#composite-types)
- [Aliases](#aliases)
- [Serialization](#serialization)
- [`"uintN"`](#uintn)
- [`"bool"`](#bool)
- [Tuples, containers, lists](#tuples-containers-lists)
- [Deserialization](#deserialization)
- [Merkleization](#merkleization)
- [Self-signed containers](#self-signed-containers)
- [Implementations](#implementations)
## Constants
| Constant | Value | Definition |
|:------------------|:-----:|:--------------------------------------------------------------------------------------|
| `LENGTH_BYTES` | 4 | Number of bytes used for the length added before a variable-length serialized object. |
| `SSZ_CHUNK_SIZE` | 128 | Number of bytes for the chunk size of the Merkle tree leaf. |
| Name | Value | Description |
|-|-|-|
| `BYTES_PER_CHUNK` | `32` | Number of bytes per chunk.
| `BYTES_PER_LENGTH_PREFIX` | `4` | Number of bytes per serialized length prefix. |
## Overview
## Typing
### Basic types
### Serialize/Encode
* `"uintN"`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`)
* `"bool"`: `True` or `False`
#### uintN
### Composite types
| uint Type | Usage |
|:---------:|:-----------------------------------------------------------|
| `uintN` | Type of `N` bits unsigned integer, where ``N % 8 == 0``. |
* **container**: ordered heterogenous collection of values
* key-pair curly bracket notation `{}`, e.g. `{'foo': "uint64", 'bar': "bool"}`
* **tuple**: ordered fixed-length homogeneous collection of values
* angle bracket notation `[type, N]`, e.g. `["uint64", N]`
* **list**: ordered variable-length homogenous collection of values
* angle bracket notation `[type]`, e.g. `["uint64"]`
Convert directly to bytes the size of the int. (e.g. ``uint16 = 2 bytes``)
### Aliases
All integers are serialized as **little endian**.
For convenience we alias:
| Check to perform | Code |
|:-----------------------|:----------------------|
| Size is a byte integer | ``int_size % 8 == 0`` |
* `"byte"` to `"uint8"` (this is a basic type)
* `"bytes"` to `["byte"]` (this is *not* a basic type)
* `"bytesN"` to `["byte", N]` (this is *not* a basic type)
## Serialization
We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a bytestring of type `"bytes"`.
*Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signed_root`, etc.) objects implicitly carry their type.
### `uintN`
```python
assert(int_size % 8 == 0)
buffer_size = int_size / 8
return value.to_bytes(buffer_size, 'little')
assert N in [8, 16, 32, 64, 128, 256]
return value.to_bytes(N // 8, 'little')
```
#### bool
Convert directly to a single 0x00 or 0x01 byte.
| Check to perform | Code |
|:------------------|:---------------------------|
| Value is boolean | ``value in (True, False)`` |
### `bool`
```python
assert(value in (True, False))
assert value in (True, False)
return b'\x01' if value is True else b'\x00'
```
#### bytesN
### Tuples, containers, lists
A fixed-size byte array.
| Checks to perform | Code |
|:---------------------------------------|:---------------------|
| Length in bytes is correct for `bytesN` | ``len(value) == N`` |
If `value` is fixed-length (i.e. does not embed a list):
```python
assert(len(value) == N)
return value
return ''.join([serialize(element) for element in value])
```
#### List/Vectors
Lists are a collection of elements of the same homogeneous type.
| Check to perform | Code |
|:--------------------------------------------|:----------------------------|
| Length of serialized list fits into 4 bytes | ``len(serialized) < 2**32`` |
1. Serialize all list elements individually and concatenate them.
2. Prefix the concatenation with its length encoded as a `4-byte` **little-endian** unsigned integer.
We define `bytes` to be a synonym of `List[bytes1]`.
**Example in Python**
If `value` is variable-length (i.e. embeds a list):
```python
serialized_list_string = b''
for item in value:
serialized_list_string += serialize(item)
assert(len(serialized_list_string) < 2**32)
serialized_len = (len(serialized_list_string).to_bytes(LENGTH_BYTES, 'little'))
return serialized_len + serialized_list_string
serialized_bytes = ''.join([serialize(element) for element in value])
assert len(serialized_bytes) < 2**(8 * BYTES_PER_LENGTH_PREFIX)
serialized_length = len(serialized_bytes).to_bytes(BYTES_PER_LENGTH_PREFIX, 'little')
return serialized_length + serialized_bytes
```
#### Container
## Deserialization
A container represents a heterogenous, associative collection of key-value pairs. Each pair is referred to as a `field`. To get the value for a given field, you supply the key which is a symbol unique to the container referred to as the field's `name`. The container data type is analogous to the `struct` type found in many languages like C or Go.
Because serialization is an injective function (i.e. two distinct objects of the same type will serialize to different values) any bytestring has at most one object it could deserialize to. Efficient algorithms for computing this object can be found in [the implementations](#implementations).
To serialize a container, obtain the list of its field's names in the specified order. For each field name in this list, obtain the corresponding value and serialize it. Tightly pack the complete set of serialized values in the same order as the field names into a buffer. Calculate the size of this buffer of serialized bytes and encode as a `4-byte` **little endian** `uint32`. Prepend the encoded length to the buffer. The result of this concatenation is the final serialized value of the container.
## Merkleization
| Check to perform | Code |
|:----------------------------------------------|:----------------------------|
| Length of serialized fields fits into 4 bytes | ``len(serialized) < 2**32`` |
We first define helper functions:
To serialize:
* `pack`: Given ordered objects of the same basic type, serialize them, pack them into `BYTES_PER_CHUNK`-byte chunks, right-pad the last chunk with zero bytes, and return the chunks.
* `merkleize`: Given ordered `BYTES_PER_CHUNK`-byte chunks, if necessary append zero chunks so that the number of chunks is a power of two, Merkleize the chunks, and return the root.
* `mix_in_length`: Given a Merkle root `root` and a length `length` (`"uint256"` little-endian serialization) return `hash(root + length)`.
1. Get the list of the container's fields.
We now define Merkleization `hash_tree_root(value)` of an object `value` recursively:
2. For each name in the list, obtain the corresponding value from the container and serialize it. Place this serialized value into a buffer. The serialized values should be tightly packed.
* `merkleize(pack(value))` if `value` is a basic object or a tuple of basic objects
* `mix_in_length(merkleize(pack(value)), len(value))` if `value` is a list of basic objects
* `merkleize([hash_tree_root(element) for element in value])` if `value` is a tuple of composite objects or a container
* `mix_in_length(merkleize([hash_tree_root(element) for element in value]), len(value))` if `value` is a list of composite objects
3. Get the number of raw bytes in the serialized buffer. Encode that number as a `4-byte` **little endian** `uint32`.
## Self-signed containers
4. Prepend the length to the serialized buffer.
**Example in Python**
```python
def get_field_names(typ):
return typ.fields.keys()
def get_value_for_field_name(value, field_name):
return getattr(value, field_name)
def get_type_for_field_name(typ, field_name):
return typ.fields[field_name]
serialized_buffer = b''
typ = type(value)
for field_name in get_field_names(typ):
field_value = get_value_for_field_name(value, field_name)
field_type = get_type_for_field_name(typ, field_name)
serialized_buffer += serialize(field_value, field_type)
assert(len(serialized_buffer) < 2**32)
serialized_len = (len(serialized_buffer).to_bytes(LENGTH_BYTES, 'little'))
return serialized_len + serialized_buffer
```
### Deserialize/Decode
The decoding requires knowledge of the type of the item to be decoded. When
performing decoding on an entire serialized string, it also requires knowledge
of the order in which the objects have been serialized.
Note: Each return will provide:
- `deserialized_object`
- `new_index`
At each step, the following checks should be made:
| Check to perform | Check |
|:-------------------------|:-----------------------------------------------------------|
| Ensure sufficient length | ``len(rawbytes) >= current_index + deserialize_length`` |
At the final step, the following checks should be made:
| Check to perform | Check |
|:-------------------------|:-------------------------------------|
| Ensure no extra length | `new_index == len(rawbytes)` |
#### uintN
Convert directly from bytes into integer utilising the number of bytes the same
size as the integer length. (e.g. ``uint16 == 2 bytes``)
All integers are interpreted as **little endian**.
```python
byte_length = int_size / 8
new_index = current_index + byte_length
assert(len(rawbytes) >= new_index)
return int.from_bytes(rawbytes[current_index:current_index+byte_length], 'little'), new_index
```
#### bool
Return True if 0x01, False if 0x00.
```python
assert rawbytes in (b'\x00', b'\x01')
return True if rawbytes == b'\x01' else False
```
#### bytesN
Return the `N` bytes.
```python
assert(len(rawbytes) >= current_index + N)
new_index = current_index + N
return rawbytes[current_index:current_index+N], new_index
```
#### List/Vectors
Deserialize each element in the list.
1. Get the length of the serialized list.
2. Loop through deserializing each item in the list until you reach the
entire length of the list.
| Check to perform | code |
|:------------------------------------------|:----------------------------------------------------------------|
| ``rawbytes`` has enough left for length | ``len(rawbytes) > current_index + LENGTH_BYTES`` |
| list is not greater than serialized bytes | ``len(rawbytes) > current_index + LENGTH_BYTES + total_length`` |
```python
assert(len(rawbytes) > current_index + LENGTH_BYTES)
total_length = int.from_bytes(rawbytes[current_index:current_index + LENGTH_BYTES], 'little')
new_index = current_index + LENGTH_BYTES + total_length
assert(len(rawbytes) >= new_index)
item_index = current_index + LENGTH_BYTES
deserialized_list = []
while item_index < new_index:
object, item_index = deserialize(rawbytes, item_index, item_type)
deserialized_list.append(object)
return deserialized_list, new_index
```
#### Container
Refer to the section on container encoding for some definitions.
To deserialize a container, loop over each field in the container and use the type of that field to know what kind of deserialization to perform. Consume successive elements of the data stream for each successful deserialization.
Instantiate a container with the full set of deserialized data, matching each member with the corresponding field.
| Check to perform | code |
|:------------------------------------------|:----------------------------------------------------------------|
| ``rawbytes`` has enough left for length | ``len(rawbytes) > current_index + LENGTH_BYTES`` |
| list is not greater than serialized bytes | ``len(rawbytes) > current_index + LENGTH_BYTES + total_length`` |
To deserialize:
1. Get the list of the container's fields.
2. For each name in the list, attempt to deserialize a value for that type. Collect these values as they will be used to construct an instance of the container.
3. Construct a container instance after successfully consuming the entire subset of the stream for the serialized container.
**Example in Python**
```python
def get_field_names(typ):
return typ.fields.keys()
def get_value_for_field_name(value, field_name):
return getattr(value, field_name)
def get_type_for_field_name(typ, field_name):
return typ.fields[field_name]
class Container:
# this is the container; here we will define an empty class for demonstration
pass
# get a reference to the type in some way...
container = Container()
typ = type(container)
assert(len(rawbytes) > current_index + LENGTH_BYTES)
total_length = int.from_bytes(rawbytes[current_index:current_index + LENGTH_BYTES], 'little')
new_index = current_index + LENGTH_BYTES + total_length
assert(len(rawbytes) >= new_index)
item_index = current_index + LENGTH_BYTES
values = {}
for field_name in get_field_names(typ):
field_name_type = get_type_for_field_name(typ, field_name)
values[field_name], item_index = deserialize(data, item_index, field_name_type)
assert item_index == new_index
return typ(**values), item_index
```
### Tree Hash
The below `hash_tree_root_internal` algorithm is defined recursively in the case of lists and containers, and it outputs a value equal to or less than 32 bytes in size. For use as a "final output" (eg. for signing), use `hash_tree_root(x) = zpad(hash_tree_root_internal(x), 32)`, where `zpad` is a helper that extends the given `bytes` value to the desired `length` by adding zero bytes on the right:
```python
def zpad(input: bytes, length: int) -> bytes:
return input + b'\x00' * (length - len(input))
```
Refer to [the helper function `hash`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#hash) of Phase 0 of the [Eth2.0 specs](https://github.com/ethereum/eth2.0-specs) for a definition of the hash function used below, `hash(x)`.
#### `uint8`..`uint256`, `bool`, `bytes1`..`bytes32`
Return the serialization of the value.
#### `uint264`..`uintN`, `bytes33`..`bytesN`
Return the hash of the serialization of the value.
#### List/Vectors
First, we define the Merkle tree function.
```python
# Merkle tree hash of a list of homogenous, non-empty items
def merkle_hash(lst):
# Store length of list (to compensate for non-bijectiveness of padding)
datalen = len(lst).to_bytes(32, 'little')
if len(lst) == 0:
# Handle empty list case
chunkz = [b'\x00' * SSZ_CHUNK_SIZE]
elif len(lst[0]) < SSZ_CHUNK_SIZE:
# See how many items fit in a chunk
items_per_chunk = SSZ_CHUNK_SIZE // len(lst[0])
# Build a list of chunks based on the number of items in the chunk
chunkz = [
zpad(b''.join(lst[i:i + items_per_chunk]), SSZ_CHUNK_SIZE)
for i in range(0, len(lst), items_per_chunk)
]
else:
# Leave large items alone
chunkz = lst
# Merkleise
def next_power_of_2(x):
return 1 if x == 0 else 2**(x - 1).bit_length()
for i in range(len(chunkz), next_power_of_2(len(chunkz))):
chunkz.append(b'\x00' * SSZ_CHUNK_SIZE)
while len(chunkz) > 1:
chunkz = [hash(chunkz[i] + chunkz[i+1]) for i in range(0, len(chunkz), 2)]
# Return hash of root and data length
return hash(chunkz[0] + datalen)
```
To `hash_tree_root_internal` a list, we simply do:
```python
return merkle_hash([hash_tree_root_internal(item) for item in value])
```
Where the inner `hash_tree_root_internal` is a recursive application of the tree-hashing function (returning less than 32 bytes for short single values).
#### Container
Recursively tree hash the values in the container in the same order as the fields, and Merkle hash the results.
```python
return merkle_hash([hash_tree_root_internal(getattr(x, field)) for field in value.fields])
```
### Signed roots
Let `field_name` be a field name in an SSZ container `container`. We define `truncate(container, field_name)` to be the `container` with the fields from `field_name` onwards truncated away. That is, `truncate(container, field_name) = [getattr(container, field)) for field in value.fields[:i]]` where `i = value.fields.index(field_name)`.
When `field_name` maps to a signature (e.g. a BLS12-381 signature of type `Bytes96`) the convention is that the corresponding signed message be `signed_root(container, field_name) = hash_tree_root(truncate(container, field_name))`. For example if `container = {"foo": sub_object_1, "bar": sub_object_2, "signature": bytes96, "baz": sub_object_3}` then `signed_root(container, "signature") = merkle_hash([hash_tree_root(sub_object_1), hash_tree_root(sub_object_2)])`.
Note that this convention means that fields after the signature are _not_ signed over. If there are multiple signatures in `container` then those are expected to be signing over the fields in the order specified. If multiple signatures of the same value are expected the convention is that the signature field be an array of signatures.
Let `value` be a self-signed container object. The convention is that the signature (e.g. a `"bytes96"` BLS12-381 signature) be the last field of `value`. Further, the signed message for `value` is `signed_root(value) = hash_tree_root(truncate_last(value))` where `truncate_last` truncates the last element of `value`.
## Implementations
| Language | Implementation | Description |
|:--------:|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------|
| Python | [ https://github.com/ethereum/py-ssz ](https://github.com/ethereum/py-ssz) | Python implementation of SSZ |
| Rust | [ https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz ](https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz) | Lighthouse (Rust Ethereum 2.0 Node) maintained SSZ. |
| Nim | [ https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim ](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) | Nim Implementation maintained SSZ. |
| Rust | [ https://github.com/paritytech/shasper/tree/master/util/ssz ](https://github.com/paritytech/shasper/tree/master/util/ssz) | Shasper implementation of SSZ maintained by ParityTech. |
| Javascript | [ https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js ](https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js) | Javascript Implementation maintained SSZ |
| Java | [ https://www.github.com/ConsenSys/cava/tree/master/ssz ](https://www.github.com/ConsenSys/cava/tree/master/ssz) | SSZ Java library part of the Cava suite |
| Go | [ https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz ](https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz) | Go implementation of SSZ mantained by Prysmatic Labs |
| Swift | [ https://github.com/yeeth/SimpleSerialize.swift ](https://github.com/yeeth/SimpleSerialize.swift) | Swift implementation maintained SSZ |
| C# | [ https://github.com/codingupastorm/csharp-ssz ](https://github.com/codingupastorm/csharp-ssz) | C# implementation maintained SSZ |
| C++ | [ https://github.com/NAKsir-melody/cpp_ssz](https://github.com/NAKsir-melody/cpp_ssz) | C++ implementation maintained SSZ |
## Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
| Language | Project | Maintainer | Implementation |
|-|-|-|-|
| Python | Ethereum 2.0 | Ethereum Foundation | [https://github.com/ethereum/py-ssz](https://github.com/ethereum/py-ssz) |
| Rust | Lighthouse | Sigma Prime | [https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz](https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz) |
| Nim | Nimbus | Status | [https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) |
| Rust | Shasper | ParityTech | [https://github.com/paritytech/shasper/tree/master/util/ssz](https://github.com/paritytech/shasper/tree/master/util/ssz) |
| Javascript | Lodestart | Chain Safe Systems | [https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js](https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js) |
| Java | Cava | ConsenSys | [https://www.github.com/ConsenSys/cava/tree/master/ssz](https://www.github.com/ConsenSys/cava/tree/master/ssz) |
| Go | Prysm | Prysmatic Labs | [https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz](https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz) |
| Swift | Yeeth | Dean Eigenmann | [https://github.com/yeeth/SimpleSerialize.swift](https://github.com/yeeth/SimpleSerialize.swift) |
| C# | | Jordan Andrews | [https://github.com/codingupastorm/csharp-ssz](https://github.com/codingupastorm/csharp-ssz) |
| C++ | | | [https://github.com/NAKsir-melody/cpp_ssz](https://github.com/NAKsir-melody/cpp_ssz) |

93
specs/validator/0_beacon-chain-validator.md
View File

@ -40,17 +40,17 @@ __NOTICE__: This document is a work-in-progress for researchers and implementers
- [Slot](#slot-1)
- [Shard](#shard)
- [Beacon block root](#beacon-block-root)
- [Epoch boundary root](#epoch-boundary-root)
- [Target root](#target-root)
- [Crosslink data root](#crosslink-data-root)
- [Latest crosslink](#latest-crosslink)
- [Justified epoch](#justified-epoch)
- [Justified block root](#justified-block-root)
- [Source epoch](#source-epoch)
- [Source root](#source-root)
- [Construct attestation](#construct-attestation)
- [Data](#data)
- [Aggregation bitfield](#aggregation-bitfield)
- [Custody bitfield](#custody-bitfield)
- [Aggregate signature](#aggregate-signature)
- [Validator assigments](#validator-assignments)
- [Validator assignments](#validator-assignments)
- [Lookahead](#lookahead)
- [How to avoid slashing](#how-to-avoid-slashing)
- [Proposer slashing](#proposer-slashing)
@ -101,8 +101,7 @@ In phase 0, all incoming validator deposits originate from the Ethereum 1.0 PoW
To submit a deposit:
* Pack the validator's [initialization parameters](#initialization) into `deposit_input`, a [`DepositInput`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#depositinput) SSZ object.
* Set `deposit_input.proof_of_possession = EMPTY_SIGNATURE`.
* Let `proof_of_possession` be the result of `bls_sign` of the `hash_tree_root(deposit_input)` with `domain=DOMAIN_DEPOSIT`.
* Let `proof_of_possession` be the result of `bls_sign` of the `signed_root(deposit_input)` with `domain=DOMAIN_DEPOSIT`.
* Set `deposit_input.proof_of_possession = proof_of_possession`.
* Let `amount` be the amount in Gwei to be deposited by the validator where `MIN_DEPOSIT_AMOUNT <= amount <= MAX_DEPOSIT_AMOUNT`.
* Send a transaction on the Ethereum 1.0 chain to `DEPOSIT_CONTRACT_ADDRESS` executing `deposit` along with `serialize(deposit_input)` as the singular `bytes` input along with a deposit `amount` in Gwei.
@ -121,11 +120,12 @@ Once a validator has been processed and added to the beacon state's `validator_r
In normal operation, the validator is quickly activated at which point the validator is added to the shuffling and begins validation after an additional `ACTIVATION_EXIT_DELAY` epochs (25.6 minutes).
The function [`is_active_validator`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#is_active_validator) can be used to check if a validator is active during a given epoch. Usage is as follows:
The function [`is_active_validator`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#is_active_validator) can be used to check if a validator is active during a given shuffling epoch. Note that the `BeaconState` contains a field `current_shuffling_epoch` which dictates from which epoch the current active validators are taken. Usage is as follows:
```python
shuffling_epoch = state.current_shuffling_epoch
validator = state.validator_registry[validator_index]
is_active = is_active_validator(validator, epoch)
is_active = is_active_validator(validator, shuffling_epoch)
```
Once a validator is activated, the validator is assigned [responsibilities](#beacon-chain-responsibilities) until exited.
@ -138,7 +138,7 @@ A validator has two primary responsibilities to the beacon chain -- [proposing b
### Block proposal
A validator is expected to propose a [`BeaconBlock`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#beaconblock) at the beginning of any slot during which `get_beacon_proposer_index(state, slot)` returns the validator's `validator_index`. To propose, the validator selects the `BeaconBlock`, `parent`, that in their view of the fork choice is the head of the chain during `slot`. The validator is to create, sign, and broadcast a `block` that is a child of `parent` and that executes a valid [beacon chain state transition](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#beacon-chain-state-transition-function).
A validator is expected to propose a [`BeaconBlock`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#beaconblock) at the beginning of any slot during which `get_beacon_proposer_index(state, slot)` returns the validator's `validator_index`. To propose, the validator selects the `BeaconBlock`, `parent`, that in their view of the fork choice is the head of the chain during `slot - 1`. The validator is to create, sign, and broadcast a `block` that is a child of `parent` and that executes a valid [beacon chain state transition](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#beacon-chain-state-transition-function).
There is one proposer per slot, so if there are N active validators any individual validator will on average be assigned to propose once per N slots (eg. at 312500 validators = 10 million ETH, that's once per ~3 weeks).
@ -152,13 +152,13 @@ _Note:_ there might be "skipped" slots between the `parent` and `block`. These s
##### Parent root
Set `block.parent_root = hash_tree_root(parent)`.
Set `block.previous_block_root = hash_tree_root(parent)`.
##### State root
Set `block.state_root = hash_tree_root(state)` of the resulting `state` of the `parent -> block` state transition.
_Note_: To calculate `state_root`, the validator should first run the state transition function on an unsigned `block` containing a stub for the `state_root`. It is useful to be able to run a state transition function that does _not_ validate signatures for this purpose.
_Note_: To calculate `state_root`, the validator should first run the state transition function on an unsigned `block` containing a stub for the `state_root`. It is useful to be able to run a state transition function that does _not_ validate signatures or state root for this purpose.
##### Randao reveal
@ -166,8 +166,8 @@ Set `block.randao_reveal = epoch_signature` where `epoch_signature` is defined a
```python
epoch_signature = bls_sign(
privkey=validator.privkey, # privkey store locally, not in state
message_hash=int_to_bytes32(slot_to_epoch(block.slot)),
privkey=validator.privkey, # privkey stored locally, not in state
message_hash=hash_tree_root(slot_to_epoch(block.slot)),
domain=get_domain(
fork=fork, # `fork` is the fork object at the slot `block.slot`
epoch=slot_to_epoch(block.slot),
@ -194,23 +194,16 @@ epoch_signature = bls_sign(
##### Signature
Set `block.signature = signed_proposal_data` where `signed_proposal_data` is defined as:
Set `block.signature = block_signature` where `block_signature` is defined as:
```python
proposal_data = ProposalSignedData(
slot=slot,
shard=BEACON_CHAIN_SHARD_NUMBER,
block_root=hash_tree_root(block), # where `block.sigature == EMPTY_SIGNATURE
)
proposal_root = hash_tree_root(proposal_data)
signed_proposal_data = bls_sign(
block_signature = bls_sign(
privkey=validator.privkey, # privkey store locally, not in state
message_hash=proposal_root,
message_hash=signed_root(block),
domain=get_domain(
fork=fork, # `fork` is the fork object at the slot `block.slot`
epoch=slot_to_epoch(block.slot),
domain_type=DOMAIN_PROPOSAL,
domain_type=DOMAIN_BEACON_BLOCK,
)
)
```
@ -227,12 +220,14 @@ Up to `MAX_ATTESTER_SLASHINGS` [`AttesterSlashing`](https://github.com/ethereum/
##### Attestations
Up to `MAX_ATTESTATIONS` aggregate attestations can be included in the `block`. The attestations added must satisfy the verification conditions found in [attestation processing](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#attestations-1). To maximize profit, the validator should attempt to create aggregate attestations that include singular attestations from the largest number of validators whose signatures from the same epoch have not previously been added on chain.
Up to `MAX_ATTESTATIONS` aggregate attestations can be included in the `block`. The attestations added must satisfy the verification conditions found in [attestation processing](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#attestations-1). To maximize profit, the validator should attempt to gather aggregate attestations that include singular attestations from the largest number of validators whose signatures from the same epoch have not previously been added on chain.
##### Deposits
Up to `MAX_DEPOSITS` [`Deposit`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#deposit) objects can be included in the `block`. These deposits are constructed from the `Deposit` logs from the [Eth1.0 deposit contract](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#ethereum-10-deposit-contract) and must be processed in sequential order. The deposits included in the `block` must satisfy the verification conditions found in [deposits processing](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#deposits-1).
The `proof` for each deposit must be constructed against the deposit root contained in `state.latest_eth1_data` rather than the deposit root at the time the deposit was initially logged from the 1.0 chain. This entails storing a full deposit merkle tree locally and computing updated proofs against the `latest_eth1_data.deposit_root` as needed. See [`minimal_merkle.py`](https://github.com/ethereum/research/blob/master/spec_pythonizer/utils/merkle_minimal.py) for a sample implementation.
##### Voluntary exits
Up to `MAX_VOLUNTARY_EXITS` [`VoluntaryExit`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#voluntaryexit) objects can be included in the `block`. The exits must satisfy the verification conditions found in [exits processing](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#exits-1).
@ -247,9 +242,12 @@ A validator should create and broadcast the attestation halfway through the `slo
First the validator should construct `attestation_data`, an [`AttestationData`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#attestationdata) object based upon the state at the assigned slot.
* Let `head_block` be the result of running the fork choice during the assigned slot.
* Let `head_state` be the state of `head_block` processed through any empty slots up to the assigned slot.
##### Slot
Set `attestation_data.slot = slot` where `slot` is the current slot of which the validator is a member of a committee.
Set `attestation_data.slot = head_state.slot`.
##### Shard
@ -257,15 +255,15 @@ Set `attestation_data.shard = shard` where `shard` is the shard associated with
##### Beacon block root
Set `attestation_data.beacon_block_root = hash_tree_root(head)` where `head` is the validator's view of the `head` block of the beacon chain during `slot`.
Set `attestation_data.beacon_block_root = hash_tree_root(head_block)`.
##### Epoch boundary root
##### Target root
Set `attestation_data.epoch_boundary_root = hash_tree_root(epoch_boundary)` where `epoch_boundary` is the block at the most recent epoch boundary in the chain defined by `head` -- i.e. the `BeaconBlock` where `block.slot == get_epoch_start_slot(slot_to_epoch(head.slot))`.
Set `attestation_data.target_root = hash_tree_root(epoch_boundary)` where `epoch_boundary` is the block at the most recent epoch boundary.
_Note:_ This can be looked up in the state using:
* Let `epoch_start_slot = get_epoch_start_slot(slot_to_epoch(head.slot))`.
* Set `epoch_boundary_root = hash_tree_root(head) if epoch_start_slot == head.slot else get_block_root(state, epoch_start_slot)`.
* Let `epoch_start_slot = get_epoch_start_slot(get_current_epoch(head_state))`.
* Set `epoch_boundary = head if epoch_start_slot == head_state.slot else get_block_root(state, epoch_start_slot)`.
##### Crosslink data root
@ -275,17 +273,15 @@ _Note:_ This is a stub for phase 0.
##### Latest crosslink
Set `attestation_data.latest_crosslink = state.latest_crosslinks[shard]` where `state` is the beacon state at `head` and `shard` is the validator's assigned shard.
Set `attestation_data.previous_crosslink = head_state.latest_crosslinks[shard]`.
##### Justified epoch
##### Source epoch
Set `attestation_data.justified_epoch = state.justified_epoch` where `state` is the beacon state at `head`.
Set `attestation_data.source_epoch = head_state.justified_epoch`.
##### Justified block root
##### Source root
Set `attestation_data.justified_block_root = hash_tree_root(justified_block)` where `justified_block` is the block at the slot `get_epoch_start_slot(state.justified_epoch)` in the chain defined by `head`.
_Note:_ This can be looked up in the state using `get_block_root(state, get_epoch_start_slot(state.justified_epoch))`.
Set `attestation_data.source_root = head_state.current_justified_root`.
#### Construct attestation
@ -320,11 +316,11 @@ attestation_data_and_custody_bit = AttestationDataAndCustodyBit(
data=attestation.data,
custody_bit=0b0,
)
attestation_message_to_sign = hash_tree_root(attestation_data_and_custody_bit)
attestation_message = hash_tree_root(attestation_data_and_custody_bit)
signed_attestation_data = bls_sign(
privkey=validator.privkey, # privkey store locally, not in state
message_hash=attestation_message_to_sign,
privkey=validator.privkey, # privkey stored locally, not in state
message_hash=attestation_message,
domain=get_domain(
fork=fork, # `fork` is the fork object at the slot, `attestation_data.slot`
epoch=slot_to_epoch(attestation_data.slot),
@ -353,7 +349,7 @@ def get_committee_assignment(
a beacon block at the assigned slot.
"""
previous_epoch = get_previous_epoch(state)
next_epoch = get_current_epoch(state)
next_epoch = get_current_epoch(state) + 1
assert previous_epoch <= epoch <= next_epoch
epoch_start_slot = get_epoch_start_slot(epoch)
@ -371,8 +367,7 @@ def get_committee_assignment(
if len(selected_committees) > 0:
validators = selected_committees[0][0]
shard = selected_committees[0][1]
first_committee_at_slot = crosslink_committees[0][0] # List[ValidatorIndex]
is_proposer = first_committee_at_slot[slot % len(first_committee_at_slot)] == validator_index
is_proposer = validator_index == get_beacon_proposer_index(state, slot, registry_change=registry_change)
assignment = (validators, shard, slot, is_proposer)
return assignment
@ -380,7 +375,7 @@ def get_committee_assignment(
### Lookahead
The beacon chain shufflings are designed to provide a minimum of 1 epoch lookahead on the validator's upcoming assignemnts of proposing and attesting dictated by the shuffling and slot.
The beacon chain shufflings are designed to provide a minimum of 1 epoch lookahead on the validator's upcoming assignments of proposing and attesting dictated by the shuffling and slot.
There are three possibilities for the shuffling at the next epoch:
1. The shuffling changes due to a "validator registry change".
@ -403,12 +398,12 @@ _Note_: Signed data must be within a sequential `Fork` context to conflict. Mess
### Proposer slashing
To avoid "proposer slashings", a validator must not sign two conflicting [`ProposalSignedData`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#proposalsigneddata) where conflicting is defined as having the same `slot` and `shard` but a different `block_root`. In phase 0, proposals are only made for the beacon chain (`shard == BEACON_CHAIN_SHARD_NUMBER`).
To avoid "proposer slashings", a validator must not sign two conflicting [`BeaconBlock`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#proposalsigneddata) where conflicting is defined as two distinct blocks within the same epoch.
_In phase 0, as long as the validator does not sign two different beacon chain proposals for the same slot, the validator is safe against proposer slashings._
_In phase 0, as long as the validator does not sign two different beacon blocks for the same epoch, the validator is safe against proposer slashings._
Specifically, when signing an `BeaconBlock`, a validator should perform the following steps in the following order:
1. Save a record to hard disk that an beacon block has been signed for the `slot=slot` and `shard=BEACON_CHAIN_SHARD_NUMBER`.
1. Save a record to hard disk that an beacon block has been signed for the `epoch=slot_to_epoch(block.slot)`.
2. Generate and broadcast the block.
If the software crashes at some point within this routine, then when the validator comes back online the hard disk has the record of the _potentially_ signed/broadcast block and can effectively avoid slashing.
@ -418,7 +413,7 @@ If the software crashes at some point within this routine, then when the validat
To avoid "attester slashings", a validator must not sign two conflicting [`AttestationData`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#attestationdata) objects where conflicting is defined as a set of two attestations that satisfy either [`is_double_vote`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#is_double_vote) or [`is_surround_vote`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#is_surround_vote).
Specifically, when signing an `Attestation`, a validator should perform the following steps in the following order:
1. Save a record to hard disk that an attestation has been signed for source -- `attestation_data.justified_epoch` -- and target -- `slot_to_epoch(attestation_data.slot)`.
1. Save a record to hard disk that an attestation has been signed for source -- `attestation_data.source_epoch` -- and target -- `slot_to_epoch(attestation_data.slot)`.
2. Generate and broadcast attestation.
If the software crashes at some point within this routine, then when the validator comes back online the hard disk has the record of the _potentially_ signed/broadcast attestation and can effectively avoid slashing.