From 850d45dae0753d935a29d6fd153bb84468d55d5c Mon Sep 17 00:00:00 2001 From: Jacek Sieka <jacek@status.im> Date: Thu, 19 Dec 2019 09:25:05 +0100 Subject: [PATCH 01/12] Allow empty lists in streamed responses It's possible that block request responses end up not containing any blocks, so we need a way to encode this. --- specs/networking/p2p-interface.md | 37 ++++++++++++++++++++++++------- 1 file changed, 29 insertions(+), 8 deletions(-) diff --git a/specs/networking/p2p-interface.md b/specs/networking/p2p-interface.md index 84539713d..948ed5ba9 100644 --- a/specs/networking/p2p-interface.md +++ b/specs/networking/p2p-interface.md @@ -92,6 +92,7 @@ It consists of four main sections: - [Why are messages length-prefixed with a protobuf varint in the SSZ-encoding?](#why-are-messages-length-prefixed-with-a-protobuf-varint-in-the-ssz-encoding) - [Why do we version protocol strings with ordinals instead of semver?](#why-do-we-version-protocol-strings-with-ordinals-instead-of-semver) - [Why is it called Req/Resp and not RPC?](#why-is-it-called-reqresp-and-not-rpc) + - [Why do we allow empty responses in block requests?](#why-do-we-allow-empty-responses-in-block-requests) - [Discovery](#discovery) - [Why are we using discv5 and not libp2p Kademlia DHT?](#why-are-we-using-discv5-and-not-libp2p-kademlia-dht) - [What is the difference between an ENR and a multiaddr, and why are we using ENRs?](#what-is-the-difference-between-an-enr-and-a-multiaddr-and-why-are-we-using-enrs) @@ -314,14 +315,14 @@ Request/response messages MUST adhere to the encoding specified in the protocol ``` request ::= <encoding-dependent-header> | <encoded-payload> -response ::= <response_chunk>+ +response ::= <response_chunk>* response_chunk ::= <result> | <encoding-dependent-header> | <encoded-payload> result ::= “0” | “1” | “2” | [“128” ... ”255”] ``` The encoding-dependent header may carry metadata or assertions such as the encoded payload length, for integrity and attack proofing purposes. Because req/resp streams are single-use and stream closures implicitly delimit the boundaries, it is not strictly necessary to length-prefix payloads; however, certain encodings like SSZ do, for added security. -A `response` is formed by one or more `response_chunk`s. The exact request determines whether a response consists of a single `response_chunk` or possibly many. Responses that consist of a single SSZ-list (such as `BlocksByRange` and `BlocksByRoot`) send each list item as a `response_chunk`. All other response types (non-Lists) send a single `response_chunk`. The encoded-payload of a `response_chunk` has a maximum uncompressed byte size of `MAX_CHUNK_SIZE`. +A `response` is formed by zero or more `response_chunk`s. Responses that consist of a single SSZ-list (such as `BlocksByRange` and `BlocksByRoot`) send each list item as a `response_chunk`. All other response types (non-Lists) send a single `response_chunk`. The encoded-payload of a `response_chunk` has a maximum uncompressed byte size of `MAX_CHUNK_SIZE`. Clients MUST ensure the each encoded payload of a `response_chunk` is less than or equal to `MAX_CHUNK_SIZE`; if not, they SHOULD reset the stream immediately. Clients tracking peer reputation MAY decrement the score of the misbehaving peer under this circumstance. @@ -344,7 +345,7 @@ The responder MUST: 1. Use the encoding strategy to read the optional header. 2. If there are any length assertions for length `N`, it should read exactly `N` bytes from the stream, at which point an EOF should arise (no more bytes). Should this not be the case, it should be treated as a failure. 3. Deserialize the expected type, and process the request. -4. Write the response which may consist of one or more `response_chunk`s (result, optional header, payload). +4. Write the response which may consist of zero or more `response_chunk`s (result, optional header, payload). 5. Close their write side of the stream. At this point, the stream will be fully closed. If steps (1), (2), or (3) fail due to invalid, malformed, or inconsistent data, the responder MUST respond in error. Clients tracking peer reputation MAY record such failures, as well as unexpected events, e.g. early stream resets. @@ -398,7 +399,7 @@ All messages that contain only a single field MUST be encoded directly as the ty Responses that are SSZ-lists (for example `[]SignedBeaconBlock`) send their constituents individually as `response_chunk`s. For example, the -`[]SignedBeaconBlock` response type sends one or more `response_chunk`s. Each _successful_ `response_chunk` contains a single `SignedBeaconBlock` payload. +`[]SignedBeaconBlock` response type sends zero or more `response_chunk`s. Each _successful_ `response_chunk` contains a single `SignedBeaconBlock` payload. ### Messages @@ -486,7 +487,7 @@ Requests count beacon blocks from the peer starting from `start_slot` on the cha The request MUST be encoded as an SSZ-container. -The response MUST consist of at least one `response_chunk` and MAY consist of many. Each _successful_ `response_chunk` MUST contain a single `SignedBeaconBlock` payload. +The response MUST consist of zero or more `response_chunk`. Each _successful_ `response_chunk` MUST contain a single `SignedBeaconBlock` payload. `BeaconBlocksByRange` is primarily used to sync historical blocks. @@ -494,7 +495,7 @@ Clients MUST support requesting blocks since the start of the weak subjectivity Clients MUST support `head_block_root` values since the latest finalized epoch. -Clients MUST respond with at least one block, if they have it. +Clients MUST respond with at least one block, if they have it and it exists in the range. Clients MAY limit the number of blocks in the response. Clients MUST order blocks by increasing slot number. @@ -524,11 +525,11 @@ Requests blocks by block root (= `hash_tree_root(SignedBeaconBlock.message)`). T The request MUST be encoded as an SSZ-field. -The response MUST consist of at least one `response_chunk` and MAY consist of many. Each _successful_ `response_chunk` MUST contain a single `SignedBeaconBlock` payload. +The response MUST consist of zero or more `response_chunk`. Each _successful_ `response_chunk` MUST contain a single `SignedBeaconBlock` payload. Clients MUST support requesting blocks since the latest finalized epoch. -Clients MUST respond with at least one block, if they have it. +Clients MUST respond with at least one block, if they have it. Clients MAY limit the number of blocks in the response. ## The discovery domain: discv5 @@ -827,6 +828,26 @@ For this reason, we remove and replace semver with ordinals that require explici Req/Resp is used to avoid confusion with JSON-RPC and similar user-client interaction mechanisms. +### Why do we allow empty responses in block requests? + +When requesting blocks by range or root, it may happen that there are no blocks in the selected range or the responding node does not have the requested blocks. + +Thus, it may happen that we need to transmit an empty list - there are several ways to encode this: + +0) Close the stream without sending any data +1) Add a `null` option to the `success` response, for example by introducing an additional byte +2) Respond with an error result, using a specific error code for "No data" + +Semantically, it is not an error that a block is missing during a slot making option 2 unnatural. + +Option 1 allows allows the responder to signal "no block", but this information may be wrong - for example in the case of a malicious node. + +Under option 0, there is no way for a client to distinguish between a slot without a block and an incomplete response, but given that it already must contain logic to handle the uncertainty of a malicious peer, option 0 was chosen. Clients should mark any slots missing blocks as unknown until they can be verified as not containing a block by successive blocks. + +Assuming option 0 with no special `null` encoding, consider a request for slots `2, 3, 4` - if there was no block produced at slot 4, the response would be `2, 3, EOF`. Now consider the same situation, but where only `4` is requested - closing the stream with only `EOF` (without any `response_chunk`) is consistent. + +Failing to provide blocks that nodes "should" have is reason to trust a peer less - for example, if a particular peer gossips a block, it should have access to its parent. If a request for the parent fails, it's indicative of poor peer quality since peers should validate blocks before gossiping them. + ## Discovery ### Why are we using discv5 and not libp2p Kademlia DHT? From 261b6c0d239ebd96005a7e704c2053cb3fc6af96 Mon Sep 17 00:00:00 2001 From: Danny Ryan <dannyjryan@gmail.com> Date: Fri, 3 Jan 2020 09:37:37 -0700 Subject: [PATCH 02/12] modify get_eth1_data to use timestamp instead of block height --- configs/mainnet.yaml | 2 + configs/minimal.yaml | 2 + specs/validator/0_beacon-chain-validator.md | 49 ++++++++++++++++----- 3 files changed, 41 insertions(+), 12 deletions(-) diff --git a/configs/mainnet.yaml b/configs/mainnet.yaml index 0c3c058d5..d2d219562 100644 --- a/configs/mainnet.yaml +++ b/configs/mainnet.yaml @@ -39,6 +39,8 @@ TARGET_AGGREGATORS_PER_COMMITTEE: 16 RANDOM_SUBNETS_PER_VALIDATOR: 1 # 2**8 (= 256) EPOCHS_PER_RANDOM_SUBNET_SUBSCRIPTION: 256 +# 14 (estimate from Eth1 mainnet) +SECONDS_PER_ETH1_BLOCK: 14 # Deposit contract diff --git a/configs/minimal.yaml b/configs/minimal.yaml index 7adc82eae..aa3756fd3 100644 --- a/configs/minimal.yaml +++ b/configs/minimal.yaml @@ -39,6 +39,8 @@ TARGET_AGGREGATORS_PER_COMMITTEE: 16 RANDOM_SUBNETS_PER_VALIDATOR: 1 # 2**8 (= 256) EPOCHS_PER_RANDOM_SUBNET_SUBSCRIPTION: 256 +# 14 (estimate from Eth1 mainnet) +SECONDS_PER_ETH1_BLOCK: 14 # Deposit contract diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 341fb8e8c..99b23c3fe 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -31,6 +31,8 @@ - [Constructing the `BeaconBlockBody`](#constructing-the-beaconblockbody) - [Randao reveal](#randao-reveal) - [Eth1 Data](#eth1-data) + - [`Eth1Block`](#eth1block) + - [`get_eth1_data`](#get_eth1_data) - [Proposer slashings](#proposer-slashings) - [Attester slashings](#attester-slashings) - [Attestations](#attestations) @@ -85,6 +87,7 @@ All terminology, constants, functions, and protocol mechanics defined in the [Ph | `TARGET_AGGREGATORS_PER_COMMITTEE` | `2**4` (= 16) | validators | | | `RANDOM_SUBNETS_PER_VALIDATOR` | `2**0` (= 1) | subnets | | | `EPOCHS_PER_RANDOM_SUBNET_SUBSCRIPTION` | `2**8` (= 256) | epochs | ~27 hours | +| `SECONDS_PER_ETH1_BLOCK` | `14` | seconds | | ## Becoming a validator @@ -239,28 +242,50 @@ def get_epoch_signature(state: BeaconState, block: BeaconBlock, privkey: int) -> ##### Eth1 Data -The `block.body.eth1_data` field is for block proposers to vote on recent Eth1 data. This recent data contains an Eth1 block hash as well as the associated deposit root (as calculated by the `get_deposit_root()` method of the deposit contract) and deposit count after execution of the corresponding Eth1 block. If over half of the block proposers in the current Eth1 voting period vote for the same `eth1_data` then `state.eth1_data` updates at the end of the voting period. Each deposit in `block.body.deposits` must verify against `state.eth1_data.eth1_deposit_root`. +The `block.body.eth1_data` field is for block proposers to vote on recent Eth1 data. This recent data contains an Eth1 block hash as well as the associated deposit root (as calculated by the `get_deposit_root()` method of the deposit contract) and deposit count after execution of the corresponding Eth1 block. If over half of the block proposers in the current Eth1 voting period vote for the same `eth1_data` then `state.eth1_data` updates immediately allowing new deposits to be processed. Each deposit in `block.body.deposits` must verify against `state.eth1_data.eth1_deposit_root`. -Let `get_eth1_data(distance: uint64) -> Eth1Data` be the (subjective) function that returns the Eth1 data at distance `distance` relative to the Eth1 head at the start of the current Eth1 voting period. Let `previous_eth1_distance` be the distance relative to the Eth1 block corresponding to `eth1_data.block_hash` found in the state at the _start_ of the current Eth1 voting period. Note that `eth1_data` can be updated in the middle of a voting period and thus the starting `eth1_data.block_hash` must be stored separately. +###### `Eth1Block` -An honest block proposer sets `block.body.eth1_data = get_eth1_vote(state, previous_eth1_distance)` where: +Let `Eth1Block` be an abstract object representing Eth1 blocks with the `timestamp` field available. ```python -def get_eth1_vote(state: BeaconState, previous_eth1_distance: uint64) -> Eth1Data: - new_eth1_data = [get_eth1_data(distance) for distance in range(ETH1_FOLLOW_DISTANCE, 2 * ETH1_FOLLOW_DISTANCE)] - all_eth1_data = [get_eth1_data(distance) for distance in range(ETH1_FOLLOW_DISTANCE, previous_eth1_distance)] +class Eth1Block(Container): + timestamp: uint64 + # All other eth1 block fields +``` - period_tail = state.slot % SLOTS_PER_ETH1_VOTING_PERIOD >= integer_squareroot(SLOTS_PER_ETH1_VOTING_PERIOD) - if period_tail: - votes_to_consider = all_eth1_data - else: - votes_to_consider = new_eth1_data +###### `get_eth1_data` + +Let `get_eth1_data(block: Eth1Block) -> Eth1Data` be the function that returns the Eth1 data for a given Eth1 block. + +An honest block proposer sets `block.body.eth1_data = get_eth1_vote(state)` where: + +```python +def voting_period_start_time(state: BeaconState) -> uint64: + eth1_voting_period_start_slot = state.slot % SLOTS_PER_ETH1_VOTING_PERIOD + return state.genesis_time + eth1_voting_period_start_slot * SECONDS_PER_SLOT +``` + +```python +def is_candidate_block(block: Eth1Block, period_start: uint64) -> bool: + return ( + block.timestamp <= period_start - SECONDS_PER_ETH1_BLOCK * ETH1_FOLLOW_DISTANCE + and block.timestamp >= period_start - SECONDS_PER_ETH1_BLOCK * ETH1_FOLLOW_DISTANCE * 2 + ) +``` + +```python +def get_eth1_vote(state: BeaconState, eth1_chain: Sequence[Eth1Block]) -> Eth1Data: + period_start = voting_period_start_time(state) + # `eth1_chain` abstractly represents all blocks in the eth1 chain. + votes_to_consider = [get_eth1_data(block) for block in eth1_chain if + is_candidate_block(block, period_start)] valid_votes = [vote for vote in state.eth1_data_votes if vote in votes_to_consider] return max( valid_votes, - key=lambda v: (valid_votes.count(v), -all_eth1_data.index(v)), # Tiebreak by smallest distance + key=lambda v: (valid_votes.count(v), -valid_votes.index(v)), # Tiebreak by smallest distance default=get_eth1_data(ETH1_FOLLOW_DISTANCE), ) ``` From 6dbc02031db930daf4f8f9d64793048f43aee857 Mon Sep 17 00:00:00 2001 From: Danny Ryan <dannyjryan@gmail.com> Date: Fri, 3 Jan 2020 17:48:03 -0700 Subject: [PATCH 03/12] add GENESIS_FORK_VERSION and make associated modifications to support configuration of this variable --- configs/mainnet.yaml | 2 ++ configs/minimal.yaml | 2 ++ specs/core/0_beacon-chain.md | 9 ++++++++- specs/validator/0_beacon-chain-validator.md | 2 +- 4 files changed, 13 insertions(+), 2 deletions(-) diff --git a/configs/mainnet.yaml b/configs/mainnet.yaml index 0c3c058d5..b61c25e6d 100644 --- a/configs/mainnet.yaml +++ b/configs/mainnet.yaml @@ -63,6 +63,8 @@ EFFECTIVE_BALANCE_INCREMENT: 1000000000 # --------------------------------------------------------------- # 0, GENESIS_EPOCH is derived from this constant GENESIS_SLOT: 0 +# Mainnet initial fork version, recommend altering for testnets +GENESIS_FORK_VERSION: 0x00000000 BLS_WITHDRAWAL_PREFIX: 0x00 diff --git a/configs/minimal.yaml b/configs/minimal.yaml index 7adc82eae..1be5d85ee 100644 --- a/configs/minimal.yaml +++ b/configs/minimal.yaml @@ -63,6 +63,8 @@ EFFECTIVE_BALANCE_INCREMENT: 1000000000 # --------------------------------------------------------------- # 0, GENESIS_EPOCH is derived from this constant GENESIS_SLOT: 0 +# Highest byte set to 0x01 to avoid collisions with mainnet versioning +GENESIS_FORK_VERSION: 0x00000001 BLS_WITHDRAWAL_PREFIX: 0x00 diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 647d1c9bd..ffc292059 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -199,6 +199,7 @@ The following values are (non-configurable) constants used throughout the specif | - | - | | `GENESIS_SLOT` | `Slot(0)` | | `GENESIS_EPOCH` | `Epoch(0)` | +| `GENESIS_FORK_VERSION` | `Version('0x00000000')` | | `BLS_WITHDRAWAL_PREFIX` | `Bytes1('0x00')` | ### Time parameters @@ -780,7 +781,7 @@ def compute_activation_exit_epoch(epoch: Epoch) -> Epoch: #### `compute_domain` ```python -def compute_domain(domain_type: DomainType, fork_version: Version=Version()) -> Domain: +def compute_domain(domain_type: DomainType, fork_version: Version=GENESIS_FORK_VERSION) -> Domain: """ Return the domain for the ``domain_type`` and ``fork_version``. """ @@ -1063,8 +1064,14 @@ Before the Ethereum 2.0 genesis has been triggered, and for every Ethereum 1.0 b def initialize_beacon_state_from_eth1(eth1_block_hash: Bytes32, eth1_timestamp: uint64, deposits: Sequence[Deposit]) -> BeaconState: + fork = Fork( + previous_version=GENESIS_FORK_VERSION, + current_version=GENESIS_FORK_VERSION, + epoch=GENESIS_EPOCH, + ) state = BeaconState( genesis_time=eth1_timestamp - eth1_timestamp % SECONDS_PER_DAY + 2 * SECONDS_PER_DAY, + fork=fork, eth1_data=Eth1Data(block_hash=eth1_block_hash, deposit_count=len(deposits)), latest_block_header=BeaconBlockHeader(body_root=hash_tree_root(BeaconBlockBody())), randao_mixes=[eth1_block_hash] * EPOCHS_PER_HISTORICAL_VECTOR, # Seed RANDAO with Eth1 entropy diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 341fb8e8c..f52bdc495 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -117,7 +117,7 @@ To submit a deposit: - Set `deposit_data.withdrawal_credentials` to `withdrawal_credentials`. - Set `deposit_data.amount` to `amount`. - Let `deposit_message` be a `DepositMessage` with all the `DepositData` contents except the `signature`. -- Let `signature` be the result of `bls_sign` of the `hash_tree_root(deposit_message)` with `domain=compute_domain(DOMAIN_DEPOSIT)`. (Deposits are valid regardless of fork version, `compute_domain` will default to zeroes there). +- Let `signature` be the result of `bls_sign` of the `hash_tree_root(deposit_message)` with `domain=compute_domain(DOMAIN_DEPOSIT)`. (_Warning_: Deposits _must_ be signed with `GENESIS_FORK_VERSION`, calling `compute_domain` without a second argument defaults to the correct version). - Let `deposit_data_root` be `hash_tree_root(deposit_data)`. - Send a transaction on the Ethereum 1.0 chain to `DEPOSIT_CONTRACT_ADDRESS` executing `def deposit(pubkey: bytes[48], withdrawal_credentials: bytes[32], signature: bytes[96], deposit_data_root: bytes32)` along with a deposit of `amount` Gwei. From 43d095214c653c8a5ed6243ddb69863034ce1945 Mon Sep 17 00:00:00 2001 From: Danny Ryan <dannyjryan@gmail.com> Date: Sun, 5 Jan 2020 14:10:06 -0700 Subject: [PATCH 04/12] add note that validator should broadcast attestation immediately when receiving block from expected proposer --- specs/validator/0_beacon-chain-validator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index f52bdc495..5d27eee47 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -318,7 +318,7 @@ def get_block_signature(state: BeaconState, header: BeaconBlockHeader, privkey: A validator is expected to create, sign, and broadcast an attestation during each epoch. The `committee`, assigned `index`, and assigned `slot` for which the validator performs this role during an epoch are defined by `get_committee_assignment(state, epoch, validator_index)`. -A validator should create and broadcast the `attestation` to the associated attestation subnet one-third of the way through the `slot` during which the validator is assigned―that is, `SECONDS_PER_SLOT / 3` seconds after the start of `slot`. +A validator should create and broadcast the `attestation` to the associated attestation subnet when either (a) the validator has received a valid block from the expected block proposer for the assigned `slot` or (b) one-third of the `slot` hash transpired (`SECONDS_PER_SLOT / 3` seconds after the start of `slot`) -- whichever comes _first_. *Note*: Although attestations during `GENESIS_EPOCH` do not count toward FFG finality, these initial attestations do give weight to the fork choice, are rewarded fork, and should be made. From 8515aec7aaeb465a61ccdef83fad672bfc69a642 Mon Sep 17 00:00:00 2001 From: Danny Ryan <dannyjryan@gmail.com> Date: Sun, 5 Jan 2020 15:03:13 -0700 Subject: [PATCH 05/12] move GENESIS_SLOT/EPOCH to constants as they are not truly configurable --- configs/mainnet.yaml | 2 -- configs/minimal.yaml | 2 -- specs/core/0_beacon-chain.md | 4 ++-- specs/validator/0_beacon-chain-validator.md | 3 ++- 4 files changed, 4 insertions(+), 7 deletions(-) diff --git a/configs/mainnet.yaml b/configs/mainnet.yaml index c211f8b43..866b6520b 100644 --- a/configs/mainnet.yaml +++ b/configs/mainnet.yaml @@ -63,8 +63,6 @@ EFFECTIVE_BALANCE_INCREMENT: 1000000000 # Initial values # --------------------------------------------------------------- -# 0, GENESIS_EPOCH is derived from this constant -GENESIS_SLOT: 0 # Mainnet initial fork version, recommend altering for testnets GENESIS_FORK_VERSION: 0x00000000 BLS_WITHDRAWAL_PREFIX: 0x00 diff --git a/configs/minimal.yaml b/configs/minimal.yaml index 7f26879d5..327cadbf3 100644 --- a/configs/minimal.yaml +++ b/configs/minimal.yaml @@ -63,8 +63,6 @@ EFFECTIVE_BALANCE_INCREMENT: 1000000000 # Initial values # --------------------------------------------------------------- -# 0, GENESIS_EPOCH is derived from this constant -GENESIS_SLOT: 0 # Highest byte set to 0x01 to avoid collisions with mainnet versioning GENESIS_FORK_VERSION: 0x00000001 BLS_WITHDRAWAL_PREFIX: 0x00 diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index ffc292059..614002056 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -158,6 +158,8 @@ The following values are (non-configurable) constants used throughout the specif | Name | Value | | - | - | +| `GENESIS_SLOT` | `Slot(0)` | +| `GENESIS_EPOCH` | `Epoch(0)` | | `FAR_FUTURE_EPOCH` | `Epoch(2**64 - 1)` | | `BASE_REWARDS_PER_EPOCH` | `4` | | `DEPOSIT_CONTRACT_TREE_DEPTH` | `2**5` (= 32) | @@ -197,8 +199,6 @@ The following values are (non-configurable) constants used throughout the specif | Name | Value | | - | - | -| `GENESIS_SLOT` | `Slot(0)` | -| `GENESIS_EPOCH` | `Epoch(0)` | | `GENESIS_FORK_VERSION` | `Version('0x00000000')` | | `BLS_WITHDRAWAL_PREFIX` | `Bytes1('0x00')` | diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 62e80fe03..492da1aa3 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -263,7 +263,8 @@ An honest block proposer sets `block.body.eth1_data = get_eth1_vote(state)` wher ```python def voting_period_start_time(state: BeaconState) -> uint64: eth1_voting_period_start_slot = state.slot % SLOTS_PER_ETH1_VOTING_PERIOD - return state.genesis_time + eth1_voting_period_start_slot * SECONDS_PER_SLOT + time_since_genesis = (eth1_voting_period_start_slot - GENESIS_SLOT) * SECONDS_PER_SLOT + return state.genesis_time + time_since_genesis ``` ```python From 50c8727ae7f520274864a232417d013aecaf17b5 Mon Sep 17 00:00:00 2001 From: Danny Ryan <dannyjryan@gmail.com> Date: Sun, 5 Jan 2020 15:29:23 -0700 Subject: [PATCH 06/12] make genesis delay configurable --- configs/mainnet.yaml | 2 ++ configs/minimal.yaml | 2 ++ specs/core/0_beacon-chain.md | 4 ++-- test_libs/pyspec/eth2spec/test/genesis/test_initialization.py | 4 ++-- 4 files changed, 8 insertions(+), 4 deletions(-) diff --git a/configs/mainnet.yaml b/configs/mainnet.yaml index c211f8b43..7713f9863 100644 --- a/configs/mainnet.yaml +++ b/configs/mainnet.yaml @@ -72,6 +72,8 @@ BLS_WITHDRAWAL_PREFIX: 0x00 # Time parameters # --------------------------------------------------------------- +# 86400 seconds (1 day) +MIN_GENESIS_DELAY: 86400 # 12 seconds SECONDS_PER_SLOT: 12 # 2**0 (= 1) slots 12 seconds diff --git a/configs/minimal.yaml b/configs/minimal.yaml index 7f26879d5..70bb6ca34 100644 --- a/configs/minimal.yaml +++ b/configs/minimal.yaml @@ -72,6 +72,8 @@ BLS_WITHDRAWAL_PREFIX: 0x00 # Time parameters # --------------------------------------------------------------- +# [customized] Faster to spin up testnets, but does not give validator reasonable warning time for genesis +MIN_GENESIS_DELAY: 300 # [customized] Faster for testing purposes SECONDS_PER_SLOT: 6 # 2**0 (= 1) slots 6 seconds diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index ffc292059..9ffce3317 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -161,7 +161,6 @@ The following values are (non-configurable) constants used throughout the specif | `FAR_FUTURE_EPOCH` | `Epoch(2**64 - 1)` | | `BASE_REWARDS_PER_EPOCH` | `4` | | `DEPOSIT_CONTRACT_TREE_DEPTH` | `2**5` (= 32) | -| `SECONDS_PER_DAY` | `86400` | | `JUSTIFICATION_BITS_LENGTH` | `4` | | `ENDIANNESS` | `'little'` | @@ -206,6 +205,7 @@ The following values are (non-configurable) constants used throughout the specif | Name | Value | Unit | Duration | | - | - | :-: | :-: | +| `MIN_GENESIS_DELAY` | `86400` | seconds | 1 day | | `SECONDS_PER_SLOT` | `12` | seconds | 12 seconds | | `MIN_ATTESTATION_INCLUSION_DELAY` | `2**0` (= 1) | slots | 12 seconds | | `SLOTS_PER_EPOCH` | `2**5` (= 32) | slots | 6.4 minutes | @@ -1070,7 +1070,7 @@ def initialize_beacon_state_from_eth1(eth1_block_hash: Bytes32, epoch=GENESIS_EPOCH, ) state = BeaconState( - genesis_time=eth1_timestamp - eth1_timestamp % SECONDS_PER_DAY + 2 * SECONDS_PER_DAY, + genesis_time=eth1_timestamp - eth1_timestamp % MIN_GENESIS_DELAY + 2 * MIN_GENESIS_DELAY, fork=fork, eth1_data=Eth1Data(block_hash=eth1_block_hash, deposit_count=len(deposits)), latest_block_header=BeaconBlockHeader(body_root=hash_tree_root(BeaconBlockBody())), diff --git a/test_libs/pyspec/eth2spec/test/genesis/test_initialization.py b/test_libs/pyspec/eth2spec/test/genesis/test_initialization.py index 462065bb9..9b326590f 100644 --- a/test_libs/pyspec/eth2spec/test/genesis/test_initialization.py +++ b/test_libs/pyspec/eth2spec/test/genesis/test_initialization.py @@ -20,7 +20,7 @@ def test_initialize_beacon_state_from_eth1(spec): # initialize beacon_state state = spec.initialize_beacon_state_from_eth1(eth1_block_hash, eth1_timestamp, deposits) - assert state.genesis_time == eth1_timestamp - eth1_timestamp % spec.SECONDS_PER_DAY + 2 * spec.SECONDS_PER_DAY + assert state.genesis_time == eth1_timestamp - eth1_timestamp % spec.MIN_GENESIS_DELAY + 2 * spec.MIN_GENESIS_DELAY assert len(state.validators) == deposit_count assert state.eth1_data.deposit_root == deposit_root assert state.eth1_data.deposit_count == deposit_count @@ -55,7 +55,7 @@ def test_initialize_beacon_state_some_small_balances(spec): # initialize beacon_state state = spec.initialize_beacon_state_from_eth1(eth1_block_hash, eth1_timestamp, deposits) - assert state.genesis_time == eth1_timestamp - eth1_timestamp % spec.SECONDS_PER_DAY + 2 * spec.SECONDS_PER_DAY + assert state.genesis_time == eth1_timestamp - eth1_timestamp % spec.MIN_GENESIS_DELAY + 2 * spec.MIN_GENESIS_DELAY assert len(state.validators) == small_deposit_count assert state.eth1_data.deposit_root == deposit_root assert state.eth1_data.deposit_count == len(deposits) From 71206c9a261aaab60366beaf371095c3406cd2a8 Mon Sep 17 00:00:00 2001 From: protolambda <proto@protolambda.com> Date: Mon, 6 Jan 2020 16:04:34 +0100 Subject: [PATCH 07/12] deposit with other fork version --- .../pyspec/eth2spec/test/helpers/deposits.py | 5 +++ .../block_processing/test_process_deposit.py | 32 ++++++++++++++++++- 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/test_libs/pyspec/eth2spec/test/helpers/deposits.py b/test_libs/pyspec/eth2spec/test/helpers/deposits.py index fdab01ca9..125a9e73c 100644 --- a/test_libs/pyspec/eth2spec/test/helpers/deposits.py +++ b/test_libs/pyspec/eth2spec/test/helpers/deposits.py @@ -49,6 +49,11 @@ def build_deposit(spec, deposit_data = build_deposit_data(spec, pubkey, privkey, amount, withdrawal_credentials, state=state, signed=signed) index = len(deposit_data_list) deposit_data_list.append(deposit_data) + return deposit_from_context(spec, deposit_data_list, index) + + +def deposit_from_context(spec, deposit_data_list, index): + deposit_data = deposit_data_list[index] root = hash_tree_root(List[spec.DepositData, 2**spec.DEPOSIT_CONTRACT_TREE_DEPTH](*deposit_data_list)) tree = calc_merkle_tree_from_leaves(tuple([d.hash_tree_root() for d in deposit_data_list])) proof = list(get_merkle_proof(tree, item_index=index, tree_len=32)) + [(index + 1).to_bytes(32, 'little')] diff --git a/test_libs/pyspec/eth2spec/test/phase_0/block_processing/test_process_deposit.py b/test_libs/pyspec/eth2spec/test/phase_0/block_processing/test_process_deposit.py index 1cef99394..71d23dcba 100644 --- a/test_libs/pyspec/eth2spec/test/phase_0/block_processing/test_process_deposit.py +++ b/test_libs/pyspec/eth2spec/test/phase_0/block_processing/test_process_deposit.py @@ -3,9 +3,10 @@ from eth2spec.test.helpers.deposits import ( build_deposit, prepare_state_and_deposit, sign_deposit_data, -) + deposit_from_context) from eth2spec.test.helpers.state import get_balance from eth2spec.test.helpers.keys import privkeys, pubkeys +from eth2spec.utils.bls import bls_sign def run_deposit_processing(spec, state, deposit, validator_index, valid=True, effective=True): @@ -93,6 +94,35 @@ def test_new_deposit_over_max(spec, state): yield from run_deposit_processing(spec, state, deposit, validator_index) +@with_all_phases +@spec_state_test +@always_bls +def test_invalid_sig_other_version(spec, state): + validator_index = len(state.validators) + amount = spec.MAX_EFFECTIVE_BALANCE + + pubkey = pubkeys[validator_index] + privkey = privkeys[validator_index] + withdrawal_credentials = spec.BLS_WITHDRAWAL_PREFIX + spec.hash(pubkey)[1:] + + deposit_data = spec.DepositData( + pubkey=pubkey, withdrawal_credentials=withdrawal_credentials, amount=amount, + signature=bls_sign( + message_hash=spec.hash_tree_root( + spec.DepositMessage(pubkey=pubkey, withdrawal_credentials=withdrawal_credentials, amount=amount)), + privkey=privkey, + domain=spec.compute_domain(domain_type=spec.DOMAIN_DEPOSIT, fork_version=spec.Version('0xaabbccdd')), + ) + ) + deposit, root, _ = deposit_from_context(spec, [deposit_data], 0) + + state.eth1_deposit_index = 0 + state.eth1_data.deposit_root = root + state.eth1_data.deposit_count = 1 + + yield from run_deposit_processing(spec, state, deposit, validator_index, valid=True, effective=False) + + @with_all_phases @spec_state_test @always_bls From 60954286f9e41f2870721889628ec91494964ae9 Mon Sep 17 00:00:00 2001 From: protolambda <proto@protolambda.com> Date: Mon, 6 Jan 2020 16:16:38 +0100 Subject: [PATCH 08/12] make tests correctly sign for general genesis-domain --- .../pyspec/eth2spec/test/helpers/deposits.py | 22 +++++-------------- .../block_processing/test_process_deposit.py | 15 +++++++++++-- 2 files changed, 18 insertions(+), 19 deletions(-) diff --git a/test_libs/pyspec/eth2spec/test/helpers/deposits.py b/test_libs/pyspec/eth2spec/test/helpers/deposits.py index 125a9e73c..071e177fd 100644 --- a/test_libs/pyspec/eth2spec/test/helpers/deposits.py +++ b/test_libs/pyspec/eth2spec/test/helpers/deposits.py @@ -5,27 +5,18 @@ from eth2spec.utils.ssz.ssz_impl import hash_tree_root from eth2spec.utils.ssz.ssz_typing import List -def build_deposit_data(spec, pubkey, privkey, amount, withdrawal_credentials, state=None, signed=False): +def build_deposit_data(spec, pubkey, privkey, amount, withdrawal_credentials, signed=False): deposit_data = spec.DepositData( pubkey=pubkey, withdrawal_credentials=withdrawal_credentials, amount=amount, ) if signed: - sign_deposit_data(spec, deposit_data, privkey, state) + sign_deposit_data(spec, deposit_data, privkey) return deposit_data -def sign_deposit_data(spec, deposit_data, privkey, state=None): - if state is None: - # Genesis - domain = spec.compute_domain(spec.DOMAIN_DEPOSIT) - else: - domain = spec.get_domain( - state, - spec.DOMAIN_DEPOSIT, - ) - +def sign_deposit_data(spec, deposit_data, privkey): deposit_message = spec.DepositMessage( pubkey=deposit_data.pubkey, withdrawal_credentials=deposit_data.withdrawal_credentials, @@ -33,20 +24,19 @@ def sign_deposit_data(spec, deposit_data, privkey, state=None): signature = bls_sign( message_hash=hash_tree_root(deposit_message), privkey=privkey, - domain=domain, + domain=spec.compute_domain(spec.DOMAIN_DEPOSIT), ) deposit_data.signature = signature def build_deposit(spec, - state, deposit_data_list, pubkey, privkey, amount, withdrawal_credentials, signed): - deposit_data = build_deposit_data(spec, pubkey, privkey, amount, withdrawal_credentials, state=state, signed=signed) + deposit_data = build_deposit_data(spec, pubkey, privkey, amount, withdrawal_credentials, signed=signed) index = len(deposit_data_list) deposit_data_list.append(deposit_data) return deposit_from_context(spec, deposit_data_list, index) @@ -75,7 +65,6 @@ def prepare_genesis_deposits(spec, genesis_validator_count, amount, signed=False withdrawal_credentials = spec.BLS_WITHDRAWAL_PREFIX + spec.hash(pubkey)[1:] deposit, root, deposit_data_list = build_deposit( spec, - None, deposit_data_list, pubkey, privkey, @@ -103,7 +92,6 @@ def prepare_state_and_deposit(spec, state, validator_index, amount, withdrawal_c deposit, root, deposit_data_list = build_deposit( spec, - state, deposit_data_list, pubkey, privkey, diff --git a/test_libs/pyspec/eth2spec/test/phase_0/block_processing/test_process_deposit.py b/test_libs/pyspec/eth2spec/test/phase_0/block_processing/test_process_deposit.py index 71d23dcba..25222664d 100644 --- a/test_libs/pyspec/eth2spec/test/phase_0/block_processing/test_process_deposit.py +++ b/test_libs/pyspec/eth2spec/test/phase_0/block_processing/test_process_deposit.py @@ -105,6 +105,7 @@ def test_invalid_sig_other_version(spec, state): privkey = privkeys[validator_index] withdrawal_credentials = spec.BLS_WITHDRAWAL_PREFIX + spec.hash(pubkey)[1:] + # Go through the effort of manually signing, not something normally done. This sig domain will be invalid. deposit_data = spec.DepositData( pubkey=pubkey, withdrawal_credentials=withdrawal_credentials, amount=amount, signature=bls_sign( @@ -123,6 +124,18 @@ def test_invalid_sig_other_version(spec, state): yield from run_deposit_processing(spec, state, deposit, validator_index, valid=True, effective=False) +@with_all_phases +@spec_state_test +@always_bls +def test_valid_sig_but_forked_state(spec, state): + validator_index = len(state.validators) + amount = spec.MAX_EFFECTIVE_BALANCE + # deposits will always be valid, regardless of the current fork + state.fork.current_version = spec.Version('0x1234abcd') + deposit = prepare_state_and_deposit(spec, state, validator_index, amount, signed=True) + yield from run_deposit_processing(spec, state, deposit, validator_index, valid=True, effective=True) + + @with_all_phases @spec_state_test @always_bls @@ -185,7 +198,6 @@ def test_wrong_deposit_for_deposit_count(spec, state): privkey_1 = privkeys[index_1] _, _, deposit_data_leaves = build_deposit( spec, - state, deposit_data_leaves, pubkey_1, privkey_1, @@ -201,7 +213,6 @@ def test_wrong_deposit_for_deposit_count(spec, state): privkey_2 = privkeys[index_2] deposit_2, root_2, deposit_data_leaves = build_deposit( spec, - state, deposit_data_leaves, pubkey_2, privkey_2, From 8391d8ee5b847f45914d86bd073c473172f72fb1 Mon Sep 17 00:00:00 2001 From: protolambda <proto@protolambda.com> Date: Mon, 6 Jan 2020 16:22:18 +0100 Subject: [PATCH 09/12] missed deposit case, also fix signing here --- .../test/phase_0/block_processing/test_process_deposit.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test_libs/pyspec/eth2spec/test/phase_0/block_processing/test_process_deposit.py b/test_libs/pyspec/eth2spec/test/phase_0/block_processing/test_process_deposit.py index 25222664d..05a40407b 100644 --- a/test_libs/pyspec/eth2spec/test/phase_0/block_processing/test_process_deposit.py +++ b/test_libs/pyspec/eth2spec/test/phase_0/block_processing/test_process_deposit.py @@ -238,6 +238,6 @@ def test_bad_merkle_proof(spec, state): # mess up merkle branch deposit.proof[5] = spec.Bytes32() - sign_deposit_data(spec, deposit.data, privkeys[validator_index], state=state) + sign_deposit_data(spec, deposit.data, privkeys[validator_index]) yield from run_deposit_processing(spec, state, deposit, validator_index, valid=False) From 9ea03dce6045ed515fdd03d9a66472271c75afe6 Mon Sep 17 00:00:00 2001 From: Danny Ryan <dannyjryan@gmail.com> Date: Mon, 6 Jan 2020 12:13:15 -0700 Subject: [PATCH 10/12] add compute_time_at_slot helper in validator doc --- specs/validator/0_beacon-chain-validator.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 492da1aa3..49261f21a 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -260,11 +260,15 @@ Let `get_eth1_data(block: Eth1Block) -> Eth1Data` be the function that returns t An honest block proposer sets `block.body.eth1_data = get_eth1_vote(state)` where: +```python +def compute_time_at_slot(state: BeaconState, slot: Slot) -> uint64: + return state.genesis_time + slot * SECONDS_PER_SLOT +``` + ```python def voting_period_start_time(state: BeaconState) -> uint64: - eth1_voting_period_start_slot = state.slot % SLOTS_PER_ETH1_VOTING_PERIOD - time_since_genesis = (eth1_voting_period_start_slot - GENESIS_SLOT) * SECONDS_PER_SLOT - return state.genesis_time + time_since_genesis + eth1_voting_period_start_slot = Slot(state.slot % SLOTS_PER_ETH1_VOTING_PERIOD) + return compute_time_at_slot(state, eth1_voting_period_start_slot) ``` ```python From b637b9ad72cd18619aaf8a7cd7422e31f7b190c6 Mon Sep 17 00:00:00 2001 From: Danny Ryan <dannyjryan@gmail.com> Date: Mon, 6 Jan 2020 17:19:31 -0700 Subject: [PATCH 11/12] fix mod bug for voting period --- specs/validator/0_beacon-chain-validator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/validator/0_beacon-chain-validator.md b/specs/validator/0_beacon-chain-validator.md index 49261f21a..36b4f4ac0 100644 --- a/specs/validator/0_beacon-chain-validator.md +++ b/specs/validator/0_beacon-chain-validator.md @@ -267,7 +267,7 @@ def compute_time_at_slot(state: BeaconState, slot: Slot) -> uint64: ```python def voting_period_start_time(state: BeaconState) -> uint64: - eth1_voting_period_start_slot = Slot(state.slot % SLOTS_PER_ETH1_VOTING_PERIOD) + eth1_voting_period_start_slot = Slot(state.slot - state.slot % SLOTS_PER_ETH1_VOTING_PERIOD) return compute_time_at_slot(state, eth1_voting_period_start_slot) ``` From fbfe024e7ad13b62efd5c2d4c56a34c5b15b45a9 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang <hwwang156@gmail.com> Date: Tue, 7 Jan 2020 17:20:17 +0800 Subject: [PATCH 12/12] Rename vyper file from `.v.py` to `.vy` --- .gitattributes | 1 + Makefile | 2 +- .../{validator_registration.v.py => validator_registration.vy} | 0 deposit_contract/tests/contracts/utils.py | 2 +- specs/core/0_deposit-contract.md | 2 +- 5 files changed, 4 insertions(+), 3 deletions(-) create mode 100644 .gitattributes rename deposit_contract/contracts/{validator_registration.v.py => validator_registration.vy} (100%) diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 000000000..c2b17bf1a --- /dev/null +++ b/.gitattributes @@ -0,0 +1 @@ +*.vy linguist-language=Python diff --git a/Makefile b/Makefile index 2b165ad7d..51bff90d0 100644 --- a/Makefile +++ b/Makefile @@ -91,7 +91,7 @@ install_deposit_contract_test: $(PY_SPEC_ALL_TARGETS) compile_deposit_contract: cd $(DEPOSIT_CONTRACT_DIR); . venv/bin/activate; \ - python tool/compile_deposit_contract.py contracts/validator_registration.v.py; + python tool/compile_deposit_contract.py contracts/validator_registration.vy; test_deposit_contract: cd $(DEPOSIT_CONTRACT_DIR); . venv/bin/activate; \ diff --git a/deposit_contract/contracts/validator_registration.v.py b/deposit_contract/contracts/validator_registration.vy similarity index 100% rename from deposit_contract/contracts/validator_registration.v.py rename to deposit_contract/contracts/validator_registration.vy diff --git a/deposit_contract/tests/contracts/utils.py b/deposit_contract/tests/contracts/utils.py index de7c54489..12eac5832 100644 --- a/deposit_contract/tests/contracts/utils.py +++ b/deposit_contract/tests/contracts/utils.py @@ -5,7 +5,7 @@ DIR = os.path.dirname(__file__) def get_deposit_contract_code(): - file_path = os.path.join(DIR, './../../contracts/validator_registration.v.py') + file_path = os.path.join(DIR, './../../contracts/validator_registration.vy') deposit_contract_code = open(file_path).read() return deposit_contract_code diff --git a/specs/core/0_deposit-contract.md b/specs/core/0_deposit-contract.md index c9f366330..11be41b86 100644 --- a/specs/core/0_deposit-contract.md +++ b/specs/core/0_deposit-contract.md @@ -61,6 +61,6 @@ Every Ethereum 1.0 deposit emits a `DepositEvent` log for consumption by the bea ## Vyper code -The deposit contract source code, written in Vyper, is available [here](../../deposit_contract/contracts/validator_registration.v.py). +The deposit contract source code, written in Vyper, is available [here](../../deposit_contract/contracts/validator_registration.vy). *Note*: To save on gas, the deposit contract uses a progressive Merkle root calculation algorithm that requires only O(log(n)) storage. See [here](https://github.com/ethereum/research/blob/master/beacon_chain_impl/progressive_merkle_tree.py) for a Python implementation, and [here](https://github.com/runtimeverification/verified-smart-contracts/blob/master/deposit/formal-incremental-merkle-tree-algorithm.pdf) for a formal correctness proof.