From 2fead870ad311b3e498eeea92d386c6280a646aa Mon Sep 17 00:00:00 2001 From: vbuterin Date: Mon, 19 Nov 2018 11:07:41 -0500 Subject: [PATCH 01/36] Replace cycles with epochs Also adjusted constants to keep interest rates and the quadratic leak period the same. Also, did some simplifications of the cycle calculation procedure. Make the decision to remove automatic registration of a validator as a proposer when they join, because that can just happen on its own due to the reshuffling procedure. --- specs/core/0_beacon-chain.md | 115 ++++++++++++++++------------------- 1 file changed, 53 insertions(+), 62 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 3769f11d3..6ee2f1fbb 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -41,15 +41,14 @@ The primary source of load on the beacon chain are "attestations". Attestations | `GENESIS_TIME` | **TBD** | seconds | | `SLOT_DURATION` | 2**4 (= 16) | seconds | | `CYCLE_LENGTH` | 2**6 (= 64) | slots | ~17 minutes | -| `MIN_VALIDATOR_SET_CHANGE_INTERVAL` | 2**8 (= 256) | slots | ~1.1 hours | | `RANDAO_SLOTS_PER_LAYER` | 2**12 (= 4096) | slots | ~18 hours | -| `SQRT_E_DROP_TIME` | 2**16 (= 65,536) | slots | ~12 days | +| `SQRT_E_DROP_TIME` | 2**10 (= 1,024) | cycles | ~12 days | | `MIN_WITHDRAWAL_PERIOD` | 2**12 (= 4096) | slots | ~18 hours | | `WITHDRAWALS_PER_CYCLE` | 8 | - | 4.3m ETH in ~6 months | | `COLLECTIVE_PENALTY_CALCULATION_PERIOD` | 2**19 (= 524,288) | slots | ~3 months | | `DELETION_PERIOD` | 2**21 (= 2,097,152) | slots | ~1.06 years | | `SHARD_PERSISTENT_COMMITTEE_CHANGE_PERIOD` | 2**16 (= 65,536) | slots | ~12 days | -| `BASE_REWARD_QUOTIENT` | 2**15 (= 32,768) | — | +| `BASE_REWARD_QUOTIENT` | 2**9 (= 512) | — | | `MAX_VALIDATOR_CHURN_QUOTIENT` | 2**5 (= 32) | — | | `POW_HASH_VOTING_PERIOD` | 2**10 (=1024) | - | | `POW_CONTRACT_MERKLE_TREE_DEPTH` | 2**5 (=32) | - | @@ -61,7 +60,7 @@ The primary source of load on the beacon chain are "attestations". Attestations * See a recommended min committee size of 111 here https://vitalik.ca/files/Ithaca201807_Sharding.pdf); our algorithm will generally ensure the committee size is at least half the target. * The `SQRT_E_DROP_TIME` constant is the amount of time it takes for the quadratic leak to cut deposits of non-participating validators by ~39.4%. -* The `BASE_REWARD_QUOTIENT` constant is the per-slot interest rate assuming all validators are participating, assuming total deposits of 1 ETH. It corresponds to ~3.88% annual interest assuming 10 million participating ETH. +* The `BASE_REWARD_QUOTIENT` constant is the per-cycle interest rate assuming all validators are participating, assuming total deposits of 1 ETH. It corresponds to ~3.88% annual interest assuming 10 million participating ETH. * At most `1/MAX_VALIDATOR_CHURN_QUOTIENT` of the validators can change during each validator set change. **Validator status codes** @@ -200,8 +199,6 @@ The `BeaconState` has the following fields: 'last_finalized_slot': 'uint64', # Last justified slot 'last_justified_slot': 'uint64', - # Number of consecutive justified slots - 'justified_streak': 'uint64', # Committee members and their assigned shard, per slot 'shard_and_committee_for_slots': [[ShardAndCommittee]], # Persistent shard committees @@ -692,13 +689,14 @@ def add_validator(validators: List[ValidatorRecord], exit_slot=0, exit_seq=0 ) + # Add the validator index = min_empty_validator(validators) if index is None: validators.append(rec) - return len(validators) - 1 + index = len(validators) - 1 else: validators[index] = rec - return index + return index ``` ### Routine for removing a validator @@ -709,6 +707,11 @@ def exit_validator(index, state, penalize, current_slot): validator.exit_slot = current_slot validator.exit_seq = state.current_exit_seq state.current_exit_seq += 1 + for committee in state.persistent_committees: + for i, vindex in committee: + if vindex == index: + committee.pop(i) + break if penalize: validator.status = PENALIZED state.deposits_penalized_in_period[current_slot // COLLECTIVE_PENALTY_CALCULATION_PERIOD] += validator.balance @@ -906,18 +909,15 @@ Verify that `deposit_data.msg_value == DEPOSIT_SIZE` and `block.slot - (deposit_ Run `add_validator(validators, deposit_data.deposit_params.pubkey, deposit_data.deposit_params.proof_of_possession, deposit_data.deposit_params.withdrawal_shard, data.deposit_params.withdrawal_address, deposit_data.deposit_params.randao_commitment, PENDING_ACTIVATION, block.slot)`. -## State recalculations (every `CYCLE_LENGTH` slots) +## Cycle boundary processing (every `CYCLE_LENGTH` slots) -Repeat while `slot - last_state_recalculation_slot >= CYCLE_LENGTH`: +Repeat the steps in this section while `block.slot - last_state_recalculation_slot >= CYCLE_LENGTH`. For simplicity, we'll use `s` as `last_state_recalculation_slot`. #### Adjust justified slots and crosslink status -For every slot `s` in the range `last_state_recalculation_slot - CYCLE_LENGTH ... last_state_recalculation_slot - 1`: - * Let `total_balance` be the total balance of active validators. * Let `total_balance_attesting_at_s` be the total balance of validators that attested to the beacon block at slot `s`. -* If `3 * total_balance_attesting_at_s >= 2 * total_balance` set `last_justified_slot = max(last_justified_slot, s)` and `justified_streak += 1`. Otherwise set `justified_streak = 0`. -* If `justified_streak >= CYCLE_LENGTH + 1` set `last_finalized_slot = max(last_finalized_slot, s - CYCLE_LENGTH - 1)`. +* If `3 * total_balance_attesting_at_s >= 2 * total_balance` then first (i) if `last_justified_slot == s - CYCLE_LENGTH`, set `last_finalized_slot = last_justified_slot`, then (ii) set `last_justified_slot = s`. For every `(shard, shard_block_hash)` tuple: @@ -932,12 +932,11 @@ Note: When applying penalties in the following balance recalculations implemente * Let `total_balance` be the total balance of active validators. * Let `total_balance_in_eth = total_balance // GWEI_PER_ETH`. * Let `reward_quotient = BASE_REWARD_QUOTIENT * int_sqrt(total_balance_in_eth)`. (The per-slot maximum interest rate is `1/reward_quotient`.) -* Let `quadratic_penalty_quotient = SQRT_E_DROP_TIME**2`. (The portion lost by offline validators after `D` slots is about `D*D/2/quadratic_penalty_quotient`.) -* Let `time_since_finality = block.slot - last_finalized_slot`. +* Let `quadratic_penalty_quotient = SQRT_E_DROP_TIME**2`. (The portion lost by offline validators after `D` cycles is about `D*D/2/quadratic_penalty_quotient`.) +* Let `time_since_finality = slot - last_finalized_slot`. -For every slot `s` in the range `last_state_recalculation_slot - CYCLE_LENGTH ... last_state_recalculation_slot - 1`: +* Let `total_balance_participating` be the total balance of validators that voted for the canonical beacon block at slot `s` (note: every attestation for a block in the slot span `s ... s + CYCLE_LENGTH - 1` counts as this) -* Let `total_balance_participating` be the total balance of validators that voted for the canonical beacon block at slot `s`. In the normal case every validator will be in one of the `CYCLE_LENGTH` slots following slot `s` and so can vote for a block at slot `s`. * Let `B` be the balance of any given validator whose balance we are adjusting, not including any balance changes from this round of state recalculation. * If `time_since_finality <= 3 * CYCLE_LENGTH` adjust the balance of participating and non-participating validators as follows: * Participating validators gain `B // reward_quotient * (2 * total_balance_participating - total_balance) // total_balance`. (Note that this value may be negative.) @@ -950,13 +949,13 @@ In addition, validators with `status == PENALIZED` lose `B // reward_quotient + #### Balance recalculations related to crosslink rewards -For every shard number `shard` for which a crosslink committee exists in the cycle prior to the most recent cycle (`last_state_recalculation_slot - CYCLE_LENGTH ... last_state_recalculation_slot - 1`), let `V` be the corresponding validator set. Let `B` be the balance of any given validator whose balance we are adjusting, not including any balance changes from this round of state recalculation. For each `shard`, `V`: +For every shard number `shard` for which a crosslink committee exists in the cycle prior to the most recent cycle (`s - CYCLE_LENGTH ... s - 1`), let `V` be the corresponding validator set. Let `B` be the balance of any given validator whose balance we are adjusting, not including any balance changes from this round of state recalculation. For each `shard`, `V`: * Let `total_balance_of_v` be the total balance of `V`. * Let `winning_shard_hash` be the hash that the largest total deposits signed for the `shard` during the cycle. * Define a "participating validator" as a member of `V` that signed a crosslink of `winning_shard_hash`. * Let `total_balance_of_v_participating` be the total balance of the subset of `V` that participated. -* Let `time_since_last_confirmation = block.slot - crosslinks[shard].slot`. +* Let `time_since_last_confirmation = s - crosslinks[shard].slot`. * Adjust balances as follows: * Participating validators gain `B // reward_quotient * (2 * total_balance_of_v_participating - total_balance_of_v) // total_balance_of_v`. * Non-participating validators lose `B // reward_quotient`. @@ -969,11 +968,39 @@ If `last_state_recalculation_slot % POW_HASH_VOTING_PERIOD == 0`, then: * Set `state.candidate_hash_chain_tip = block.candidate_pow_hash_chain_tip` * Set `state.candidate_hash_chain_tip_votes = 0` -### Validator set change +#### Proposer reshuffling -A validator set change can happen after a state recalculation if all of the following criteria are satisfied: +Run the following code: + +```python +active_validator_indices = get_active_validator_indices(validators) +num_validators_to_reshuffle = len(active_validator_indices) // SHARD_PERSISTENT_COMMITTEE_CHANGE_PERIOD +for i in range(num_validators_to_reshuffle): + # Multiplying i to 2 to ensure we have different input to all the required hashes in the shuffling + # and none of the hashes used for entropy in this loop will be the same + vid = active_validator_indices[hash(state.randao_mix + bytes8(i * 2)) % len(active_validator_indices)] + new_shard = hash(state.randao_mix + bytes8(i * 2 + 1)) % SHARD_COUNT + shard_reassignment_record = ShardReassignmentRecord( + validator_index=vid, + shard=new_shard, + slot=s + SHARD_PERSISTENT_COMMITTEE_CHANGE_PERIOD + ) + state.persistent_committee_reassignments.append(shard_reassignment_record) + +while len(state.persistent_committee_reassignments) > 0 and state.persistent_committee_reassignments[0].slot <= s: + rec = state.persistent_committee_reassignments.pop(0) + for committee in state.persistent_committees: + if rec.validator_index in committee: + committee.pop( + committee.index(rec.validator_index) + ) + state.persistent_committees[rec.shard].append(rec.validator_index) +``` + +#### Validator set change + +A validator set change can happen if all of the following criteria are satisfied: -* `block.slot - state.validator_set_change_slot >= MIN_VALIDATOR_SET_CHANGE_INTERVAL` * `last_finalized_slot > state.validator_set_change_slot` * For every shard number `shard` in `shard_and_committee_for_slots`, `crosslinks[shard].slot > state.validator_set_change_slot` @@ -1040,13 +1067,13 @@ def change_validators(validators: List[ValidatorRecord], current_slot: int) -> N # STUB: withdraw to shard chain ``` -* Set `state.validator_set_change_slot = state.last_state_recalculation_slot` +* Set `state.validator_set_change_slot = s` * Set `shard_and_committee_for_slots[:CYCLE_LENGTH] = shard_and_committee_for_slots[CYCLE_LENGTH:]` * Let `next_start_shard = (shard_and_committee_for_slots[-1][-1].shard + 1) % SHARD_COUNT` * Set `shard_and_committee_for_slots[CYCLE_LENGTH:] = get_new_shuffling(state.next_shuffling_seed, validators, next_start_shard)` * Set `state.next_shuffling_seed = state.randao_mix` -### If a validator set change does NOT happen +#### If a validator set change does NOT happen * Set `shard_and_committee_for_slots[:CYCLE_LENGTH] = shard_and_committee_for_slots[CYCLE_LENGTH:]` * Let `time_since_finality = block.slot - state.validator_set_change_slot` @@ -1055,47 +1082,11 @@ def change_validators(validators: List[ValidatorRecord], current_slot: int) -> N #### Finally... -* Remove all attestation records older than slot `state.last_state_recalculation_slot` -* Empty the `state.pending_specials` list +* Remove all attestation records older than slot `s` * For any validator with index `v` with balance less than `MIN_ONLINE_DEPOSIT_SIZE` and status `ACTIVE`, run `exit_validator(v, state, penalize=False, current_slot=block.slot)` * Set `state.recent_block_hashes = state.recent_block_hashes[CYCLE_LENGTH:]` * Set `state.last_state_recalculation_slot += CYCLE_LENGTH` -For any validator that was added or removed from the active validator list during this state recalculation: - -* If the validator was removed, remove their index from the `persistent_committees` and remove any `ShardReassignmentRecord`s containing their index from `persistent_committee_reassignments`. -* If the validator was added with index `validator_index`: - * let `assigned_shard = hash(state.randao_mix + bytes8(validator_index)) % SHARD_COUNT` - * let `reassignment_record = ShardReassignmentRecord(validator_index=validator_index, shard=assigned_shard, slot=block.slot + SHARD_PERSISTENT_COMMITTEE_CHANGE_PERIOD)` - * Append `reassignment_record` to the end of `persistent_committee_reassignments` - -Now run the following code to reshuffle a few proposers: - -```python -active_validator_indices = get_active_validator_indices(validators) -num_validators_to_reshuffle = len(active_validator_indices) // SHARD_PERSISTENT_COMMITTEE_CHANGE_PERIOD -for i in range(num_validators_to_reshuffle): - # Multiplying i to 2 to ensure we have different input to all the required hashes in the shuffling - # and none of the hashes used for entropy in this loop will be the same - vid = active_validator_indices[hash(state.randao_mix + bytes8(i * 2)) % len(active_validator_indices)] - new_shard = hash(state.randao_mix + bytes8(i * 2 + 1)) % SHARD_COUNT - shard_reassignment_record = ShardReassignmentRecord( - validator_index=vid, - shard=new_shard, - slot=block.slot + SHARD_PERSISTENT_COMMITTEE_CHANGE_PERIOD - ) - state.persistent_committee_reassignments.append(shard_reassignment_record) - -while len(state.persistent_committee_reassignments) > 0 and state.persistent_committee_reassignments[0].slot <= block.slot: - rec = state.persistent_committee_reassignments.pop(0) - for committee in state.persistent_committees: - if rec.validator_index in committee: - committee.pop( - committee.index(rec.validator_index) - ) - state.persistent_committees[rec.shard].append(rec.validator_index) -``` - ### TODO Note: This spec is ~65% complete. From 00f7554ec4356e48fae51f61db51735c280aeab1 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 20 Nov 2018 11:18:58 +0900 Subject: [PATCH 02/36] Remove justified_streak --- specs/core/0_beacon-chain.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 6ee2f1fbb..a0d689644 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -618,7 +618,6 @@ def on_startup(initial_validator_entries: List[Any], genesis_time: uint64, pow_h last_state_recalculation_slot=0, last_finalized_slot=0, last_justified_slot=0, - justified_streak=0, shard_and_committee_for_slots=x + x, persistent_committees=split(shuffle(validators, bytes([0] * 32)), SHARD_COUNT), persistent_committee_reassignments=[], @@ -756,7 +755,6 @@ def on_startup(initial_validator_entries: List[Any]) -> BeaconState: last_state_recalculation_slot=0, last_finalized_slot=0, last_justified_slot=0, - justified_streak=0, shard_and_committee_for_slots=x + x, persistent_committees=split(shuffle(validators, bytes([0] * 32)), SHARD_COUNT), persistent_committee_reassignments=[], From b4dfdb5ed4b386511107e51ddf654491b044dd2b Mon Sep 17 00:00:00 2001 From: vbuterin Date: Tue, 20 Nov 2018 16:49:43 -0500 Subject: [PATCH 03/36] Partial edits.... --- specs/core/0_beacon-chain.md | 47 ++++++++++++++++++++++++++++-------- 1 file changed, 37 insertions(+), 10 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 28c07f9ce..383a69942 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -133,7 +133,7 @@ An `AttestationRecord` has the following fields: # Shard number 'shard': 'uint16', # Beacon block hashes not part of the current chain, oldest to newest - 'oblique_parent_hashes': ['hash32'], + 'parent_hashes': ['hash32'], # Shard block hash being attested to 'shard_block_hash': 'hash32', # Last crosslink hash @@ -517,6 +517,24 @@ def get_beacon_proposer(state:BeaconState, slot: int) -> ValidatorRecord: return state.validators[index] ``` +The following is a function that determines the validators that participated in an attestation: + +```python +def get_attestation_participants(state: State, attestation: AttestationRecord): + sncs_for_slot = get_shards_and_committees_for_slot(state, attestation.slot) + snc = [x for x in sncs_for_slot if x.shard == attestation.shard][0] + assert len(attestation.attester_bitfield) == ceil_div8(len(snc.committee) * 2) + bit0_participants, bit1_participants = [], [] + for i, vindex in snc.committee: + bits = (attestation.attester_bitfield[i//4] >> (3 - (i % 4)) * 2) % 4 + assert bits in (0, 2, 3) + if bits == 2: + bit0_participants.append(vindex) + elif bits == 3: + bit1_participants.append(vindex) + return bit0_participants, bit1_participants +``` + We define another set of helpers to be used throughout: `bytes1(x): return x.to_bytes(1, 'big')`, `bytes2(x): return x.to_bytes(2, 'big')`, and so on for all integers, particularly 1, 2, 3, 4, 8, 32. We define a function to "add a link" to the validator hash chain, used when a validator is added or removed: @@ -834,20 +852,19 @@ def update_ancestor_hashes(parent_ancestor_hashes: List[Hash32], ### Verify attestations -For each `AttestationRecord` object: +For each `AttestationRecord` object `obj`: * Verify that `slot <= block.slot - MIN_ATTESTATION_INCLUSION_DELAY` and `slot >= max(parent.slot - CYCLE_LENGTH + 1, 0)`. * Verify that `justified_slot` is equal to or earlier than `last_justified_slot`. * Verify that `justified_block_hash` is the hash of the block in the current chain at the slot -- `justified_slot`. * Verify that either `last_crosslink_hash` or `shard_block_hash` equals `state.crosslinks[shard].shard_block_hash`. -* Compute `parent_hashes` = `[get_block_hash(state, block, slot - CYCLE_LENGTH + i) for i in range(1, CYCLE_LENGTH - len(oblique_parent_hashes) + 1)] + oblique_parent_hashes` (eg, if `CYCLE_LENGTH = 4`, `slot = 5`, the actual block hashes starting from slot 0 are `Z A B C D E F G H I J`, and `oblique_parent_hashes = [D', E']` then `parent_hashes = [B, C, D' E']`). Note that when *creating* an attestation for a block, the hash of that block itself won't yet be in the `state`, so you would need to add it explicitly. -* Let `attestation_indices` be `get_shards_and_committees_for_slot(state, slot)[x]`, choosing `x` so that `attestation_indices.shard` equals the `shard` value provided to find the set of validators that is creating this attestation record. -* Verify that `len(attester_bitfield) == ceil_div8(len(attestation_indices))`, where `ceil_div8 = (x + 7) // 8`. Verify that bits `len(attestation_indices)....` and higher, if present (i.e. `len(attestation_indices)` is not a multiple of 8), are all zero. -* Derive a group public key by adding the public keys of all of the attesters in `attestation_indices` for whom the corresponding bit in `attester_bitfield` (the ith bit is `(attester_bitfield[i // 8] >> (7 - (i %8))) % 2`) equals 1. +* Compute `full_parent_hashes` = `[get_block_hash(state, block, slot - CYCLE_LENGTH + i) for i in range(1, CYCLE_LENGTH - len(parent_hashes) + 1)] + parent_hashes` (eg, if `CYCLE_LENGTH = 4`, `slot = 5`, the actual block hashes starting from slot 0 are `Z A B C D E F G H I J`, and `parent_hashes = [D', E']` then `parent_hashes = [B, C, D' E']`). Note that when *creating* an attestation for a block, the hash of that block itself won't yet be in the `state`, so you would need to add it explicitly. +* Let `bit0_attestation_indices, bit1_attestation_indices = get_attestation_participants(state, obj)` (and verify that the method returns successfully) +* Let `bit0_group_public_key = BLSAddPubkeys(bit0_attestation_indices)` and `bit1_group_public_key = BLSAddPubkeys(bit1_attestation_indices)` * Let `fork_version = pre_fork_version if slot < fork_slot_number else post_fork_version`. * Verify that `aggregate_sig` verifies using the group pubkey generated and the serialized form of `AttestationSignedData(fork_version, slot, shard, parent_hashes, shard_block_hash, last_crosslinked_hash, shard_block_combined_data_root, justified_slot)` as the message. -Extend the list of `AttestationRecord` objects in the `state` with those included in the block, ordering the new additions in the same order as they came in the block. +Extend the list of `AttestationRecord` objects in the `state` with those included in the block, ordering the new additions in the same order as they came in the block, and replacing `obj.parent_hashes` with the calculated value of `full_parent_hashes`. ### Verify proposer signature @@ -953,14 +970,24 @@ Verify that `deposit_data.msg_value == DEPOSIT_SIZE` and `block.slot - (deposit_ Run `add_validator(validators, deposit_data.deposit_params.pubkey, deposit_data.deposit_params.proof_of_possession, deposit_data.deposit_params.withdrawal_shard, data.deposit_params.withdrawal_address, deposit_data.deposit_params.randao_commitment, PENDING_ACTIVATION, block.slot)`. -## Cycle boundary processing (every `CYCLE_LENGTH` slots) +## Cycle boundary processing Repeat the steps in this section while `block.slot - last_state_recalculation_slot >= CYCLE_LENGTH`. For simplicity, we'll use `s` as `last_state_recalculation_slot`. +_Note: `last_state_recalculation_slot` will always be a multiple of `CYCLE_LENGTH`. In the "happy case", this process will trigger, and loop once, every time `block.slot` passes a new exact multiple of `CYCLE_LENGTH`, but if a chain skips more than an entire cycle then the loop may run multiple times, incrementing `last_state_recalculation_slot` by `CYCLE_LENGTH` with each iteration._ + +#### Precomputation + +* Let `active_validators = [state.validators[i] for i in get_active_validator_indices(state.validators)]` +* Let `total_balance = sum([v.balance for v in active_validators])` +* Let `this_cycle_attestations = [a for a in state.pending_attestations if s <= a.slot < s + CYCLE_LENGTH]` (note: this is the set of attestations _of slots in the cycle `s...s+CYCLE_LENGTH-1`_, not attestations _that got included in the chain during the cycle `s...s+CYCLE_LENGTH-1`) +* Let `prev_cycle_attestations = [a for a in state.pending_attestations if s - CYCLE_LENGTH <= a.slot < s] +* Let `this_cycle_s_attestations = [a for a in this_cycle_attestations if get_block_hash(state, block, s) in a.parent_hashes] +* Let `s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_s_attestations]` +* Let `total_balance_attesting_at_s = sum([v.balance for v in s_attesters])` + #### Adjust justified slots and crosslink status -* Let `total_balance` be the total balance of active validators. -* Let `total_balance_attesting_at_s` be the total balance of validators that attested to the beacon block at slot `s`. * If `3 * total_balance_attesting_at_s >= 2 * total_balance` then first (i) if `last_justified_slot == s - CYCLE_LENGTH`, set `last_finalized_slot = last_justified_slot`, then (ii) set `last_justified_slot = s`. For every `(shard, shard_block_hash)` tuple: From 252a24cff0557933758fe234225692458cbb644c Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sat, 24 Nov 2018 01:38:57 +0800 Subject: [PATCH 04/36] Minor fix: markdown grammar and typing hints --- specs/core/0_beacon-chain.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index fdd51733f..7e3a9955d 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -521,7 +521,8 @@ def get_beacon_proposer(state:BeaconState, slot: int) -> ValidatorRecord: The following is a function that determines the validators that participated in an attestation: ```python -def get_attestation_participants(state: State, attestation: AttestationRecord): +def get_attestation_participants(state: State, + attestation: AttestationRecord) -> Tuple[List[int], List[int]]: sncs_for_slot = get_shards_and_committees_for_slot(state, attestation.slot) snc = [x for x in sncs_for_slot if x.shard == attestation.shard][0] assert len(attestation.attester_bitfield) == ceil_div8(len(snc.committee) * 2) @@ -984,7 +985,7 @@ _Note: `last_state_recalculation_slot` will always be a multiple of `CYCLE_LENGT * Let `active_validators = [state.validators[i] for i in get_active_validator_indices(state.validators)]` * Let `total_balance = sum([v.balance for v in active_validators])` -* Let `this_cycle_attestations = [a for a in state.pending_attestations if s <= a.slot < s + CYCLE_LENGTH]` (note: this is the set of attestations _of slots in the cycle `s...s+CYCLE_LENGTH-1`_, not attestations _that got included in the chain during the cycle `s...s+CYCLE_LENGTH-1`) +* Let `this_cycle_attestations = [a for a in state.pending_attestations if s <= a.slot < s + CYCLE_LENGTH]` (note: this is the set of attestations _of slots in the cycle `s...s+CYCLE_LENGTH-1`_, not attestations _that got included in the chain during the cycle `s...s+CYCLE_LENGTH-1`_) * Let `prev_cycle_attestations = [a for a in state.pending_attestations if s - CYCLE_LENGTH <= a.slot < s] * Let `this_cycle_s_attestations = [a for a in this_cycle_attestations if get_block_hash(state, block, s) in a.parent_hashes] * Let `s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_s_attestations]` From 403253c8799b765bdd152bbd87a217fe3a73ee4c Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sat, 24 Nov 2018 09:41:25 -0500 Subject: [PATCH 05/36] Added 2/3/4 cycle finalization options Suppose B1, B2, B3, B4 are consecutive blocks and we are now processing the end of the cycle containing B4. * If B4 is justified using source B3, then B3 is finalized. * If B4 is justified using source B2, and B3 has been justified, then B2 is finalized. * If B3 is justified using source B1, and B1 has been justified, then B1 is finalized. --- specs/core/0_beacon-chain.md | 79 +++++++++--------------------------- 1 file changed, 20 insertions(+), 59 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 7e3a9955d..052323add 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -217,8 +217,11 @@ The `BeaconState` has the following fields: 'last_state_recalculation_slot': 'uint64', # Last finalized slot 'last_finalized_slot': 'uint64', - # Last justified slot - 'last_justified_slot': 'uint64', + # Justification source + 'justification_source': 'uint64', + 'prev_cycle_justification_source': 'uint64', + # Recent justified slot bitmask + 'justified_slot_bitfield': 'uint64', # Committee members and their assigned shard, per slot 'shard_and_committee_for_slots': [[ShardAndCommittee]], # Persistent shard committees @@ -666,7 +669,9 @@ def on_startup(initial_validator_entries: List[Any], genesis_time: uint64, pow_h crosslinks=crosslinks, last_state_recalculation_slot=0, last_finalized_slot=0, - last_justified_slot=0, + justification_source=0, + prev_cycle_justification_source=0, + justified_slot_bitfield=0, shard_and_committee_for_slots=x + x, persistent_committees=split(shuffle(validators, bytes([0] * 32)), SHARD_COUNT), persistent_committee_reassignments=[], @@ -771,59 +776,6 @@ def exit_validator(index, state, block, penalize, current_slot): add_validator_set_change_record(state, index, validator.pubkey, EXIT) ``` -## On startup - -Run the following code: - -```python -def on_startup(initial_validator_entries: List[Any]) -> BeaconState: - # Induct validators - validators = [] - for pubkey, proof_of_possession, withdrawal_shard, withdrawal_address, \ - randao_commitment in initial_validator_entries: - add_validator( - validators=validators, - pubkey=pubkey, - proof_of_possession=proof_of_possession, - withdrawal_shard=withdrawal_shard, - withdrawal_address=withdrawal_address, - randao_commitment=randao_commitment, - current_slot=0, - status=ACTIVE, - ) - # Setup state - x = get_new_shuffling(bytes([0] * 32), validators, 0) - crosslinks = [ - CrosslinkRecord( - slot=0, - hash=bytes([0] * 32) - ) - for i in range(SHARD_COUNT) - ] - state = BeaconState( - validator_set_change_slot=0, - validators=validators, - crosslinks=crosslinks, - last_state_recalculation_slot=0, - last_finalized_slot=0, - last_justified_slot=0, - shard_and_committee_for_slots=x + x, - persistent_committees=split(shuffle(validators, bytes([0] * 32)), SHARD_COUNT), - persistent_committee_reassignments=[], - deposits_penalized_in_period=[], - next_shuffling_seed=b'\x00'*32, - validator_set_delta_hash_chain=bytes([0] * 32), # stub - pre_fork_version=INITIAL_FORK_VERSION, - post_fork_version=INITIAL_FORK_VERSION, - fork_slot_number=0, - pending_attestations=[], - recent_block_hashes=[bytes([0] * 32) for _ in range(CYCLE_LENGTH * 2)], - randao_mix=bytes([0] * 32) # stub - ) - - return state -``` - ## Per-block processing This procedure should be carried out every beacon block. @@ -860,7 +812,7 @@ def update_ancestor_hashes(parent_ancestor_hashes: List[Hash32], For each `AttestationRecord` object `obj`: * Verify that `slot <= block.slot - MIN_ATTESTATION_INCLUSION_DELAY` and `slot >= max(parent.slot - CYCLE_LENGTH + 1, 0)`. -* Verify that `justified_slot` is equal to or earlier than `last_justified_slot`. +* Verify that `justified_slot` is equal to `justification_source if slot >= block.slot - (block.slot % CYCLE_LENGTH) else prev_cycle_justification_source` * Verify that `justified_block_hash` is the hash of the block in the current chain at the slot -- `justified_slot`. * Verify that either `last_crosslink_hash` or `shard_block_hash` equals `state.crosslinks[shard].shard_block_hash`. * Compute `full_parent_hashes` = `[get_block_hash(state, block, slot - CYCLE_LENGTH + i) for i in range(1, CYCLE_LENGTH - len(parent_hashes) + 1)] + parent_hashes` (eg, if `CYCLE_LENGTH = 4`, `slot = 5`, the actual block hashes starting from slot 0 are `Z A B C D E F G H I J`, and `parent_hashes = [D', E']` then `parent_hashes = [B, C, D' E']`). Note that when *creating* an attestation for a block, the hash of that block itself won't yet be in the `state`, so you would need to add it explicitly. @@ -987,13 +939,22 @@ _Note: `last_state_recalculation_slot` will always be a multiple of `CYCLE_LENGT * Let `total_balance = sum([v.balance for v in active_validators])` * Let `this_cycle_attestations = [a for a in state.pending_attestations if s <= a.slot < s + CYCLE_LENGTH]` (note: this is the set of attestations _of slots in the cycle `s...s+CYCLE_LENGTH-1`_, not attestations _that got included in the chain during the cycle `s...s+CYCLE_LENGTH-1`_) * Let `prev_cycle_attestations = [a for a in state.pending_attestations if s - CYCLE_LENGTH <= a.slot < s] -* Let `this_cycle_s_attestations = [a for a in this_cycle_attestations if get_block_hash(state, block, s) in a.parent_hashes] +* Let `this_cycle_s_attestations = [a for a in this_cycle_attestations if get_block_hash(state, block, s) in a.parent_hashes and a.justified_slot == state.justification_source] +* Let `prev_s_attestations = [a for a in this_cycle_attestations + prev_cycle_attestations if get_block_hash(state, block, s - CYCLE_LENGTH) in a.parent_hashes and a.justified_slot == state.prev_cycle_justification_source] * Let `s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_s_attestations]` +* Let `prev_s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_s_attestations]` * Let `total_balance_attesting_at_s = sum([v.balance for v in s_attesters])` +* Let `total_balance_attesting_at_prev_s = sum([v.balance for v in prev_s_attesters])` #### Adjust justified slots and crosslink status -* If `3 * total_balance_attesting_at_s >= 2 * total_balance` then first (i) if `last_justified_slot == s - CYCLE_LENGTH`, set `last_finalized_slot = last_justified_slot`, then (ii) set `last_justified_slot = s`. +* Set `state.justified_slot_bitfield = (state.justified_slot_bitfield * 2) % 2**64`. +* If `3 * total_balance_attesting_at_prev_s >= 2 * total_balance` then set `state.justified_slot_bitfield &= 2` (ie. flip the second lowest bit to 1) and `new_justification_source = prev_s`. +* If `3 * total_balance_attesting_at_s >= 2 * total_balance` then set `state.justified_slot_bitfield &= 1` (ie. flip the lowest bit to 1) and `new_justification_source = s`. +* If `justification_source == prev_s and state.justified_slot_bitfield % 4 == 3`, set `last_finalized_slot = justification_source`. +* If `justification_source == prev_s - 2 * CYCLE_LENGTH and state.justified_slot_bitfield % 16 in (15, 14)`, set `last_finalized_slot = justification_source`. +* If `justification_source == prev_s - CYCLE_LENGTH and state.justified_slot_bitfield % 8 == 7`, set `last_finalized_slot = justification_source`. +* Set `prev_cycle_justification_source = justification_source` and `justification_source = new_justification_source`. For every `(shard, shard_block_hash)` tuple: From a818575ca8214c5bbf545fa11c982f506b2ed414 Mon Sep 17 00:00:00 2001 From: Justin Date: Sat, 24 Nov 2018 15:15:19 +0000 Subject: [PATCH 06/36] Minor cosmetic fixes --- specs/core/0_beacon-chain.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 052323add..e3b7e51d5 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -938,9 +938,9 @@ _Note: `last_state_recalculation_slot` will always be a multiple of `CYCLE_LENGT * Let `active_validators = [state.validators[i] for i in get_active_validator_indices(state.validators)]` * Let `total_balance = sum([v.balance for v in active_validators])` * Let `this_cycle_attestations = [a for a in state.pending_attestations if s <= a.slot < s + CYCLE_LENGTH]` (note: this is the set of attestations _of slots in the cycle `s...s+CYCLE_LENGTH-1`_, not attestations _that got included in the chain during the cycle `s...s+CYCLE_LENGTH-1`_) -* Let `prev_cycle_attestations = [a for a in state.pending_attestations if s - CYCLE_LENGTH <= a.slot < s] -* Let `this_cycle_s_attestations = [a for a in this_cycle_attestations if get_block_hash(state, block, s) in a.parent_hashes and a.justified_slot == state.justification_source] -* Let `prev_s_attestations = [a for a in this_cycle_attestations + prev_cycle_attestations if get_block_hash(state, block, s - CYCLE_LENGTH) in a.parent_hashes and a.justified_slot == state.prev_cycle_justification_source] +* Let `prev_cycle_attestations = [a for a in state.pending_attestations if s - CYCLE_LENGTH <= a.slot < s]` +* Let `this_cycle_s_attestations = [a for a in this_cycle_attestations if get_block_hash(state, block, s) in a.parent_hashes and a.justified_slot == state.justification_source]` +* Let `prev_s_attestations = [a for a in this_cycle_attestations + prev_cycle_attestations if get_block_hash(state, block, s - CYCLE_LENGTH) in a.parent_hashes and a.justified_slot == state.prev_cycle_justification_source]` * Let `s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_s_attestations]` * Let `prev_s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_s_attestations]` * Let `total_balance_attesting_at_s = sum([v.balance for v in s_attesters])` @@ -971,9 +971,7 @@ Note: When applying penalties in the following balance recalculations implemente * Let `reward_quotient = BASE_REWARD_QUOTIENT * int_sqrt(total_balance_in_eth)`. (The per-slot maximum interest rate is `2/reward_quotient`.) * Let `quadratic_penalty_quotient = SQRT_E_DROP_TIME**2`. (The portion lost by offline validators after `D` cycles is about `D*D/2/quadratic_penalty_quotient`.) * Let `time_since_finality = slot - last_finalized_slot`. - * Let `total_balance_participating` be the total balance of validators that voted for the canonical beacon block at slot `s` (note: every attestation for a block in the slot span `s ... s + CYCLE_LENGTH - 1` counts as this) - * Let `B` be the balance of any given validator whose balance we are adjusting, not including any balance changes from this round of state recalculation. * If `time_since_finality <= 3 * CYCLE_LENGTH` adjust the balance of participating and non-participating validators as follows: * Participating validators gain `B // reward_quotient * (2 * total_balance_participating - total_balance) // total_balance`. (Note that this value may be negative.) From b61f7d57c9670933f41602cfd274380c5165a8fe Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sat, 24 Nov 2018 12:34:26 -0500 Subject: [PATCH 07/36] Further incremental steps on balances --- specs/core/0_beacon-chain.md | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index e3b7e51d5..f17e00222 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -942,9 +942,10 @@ _Note: `last_state_recalculation_slot` will always be a multiple of `CYCLE_LENGT * Let `this_cycle_s_attestations = [a for a in this_cycle_attestations if get_block_hash(state, block, s) in a.parent_hashes and a.justified_slot == state.justification_source]` * Let `prev_s_attestations = [a for a in this_cycle_attestations + prev_cycle_attestations if get_block_hash(state, block, s - CYCLE_LENGTH) in a.parent_hashes and a.justified_slot == state.prev_cycle_justification_source]` * Let `s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_s_attestations]` -* Let `prev_s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_s_attestations]` +* Let `prev_s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in prev_s_attestations]` * Let `total_balance_attesting_at_s = sum([v.balance for v in s_attesters])` * Let `total_balance_attesting_at_prev_s = sum([v.balance for v in prev_s_attesters])` +* For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots`, let `validators(obj, shard_block_hash)` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_attestations + prev_cycle_attestations if a.shard == obj.shard and a.shard_block_hash == shard_block_hash]`. Let `validators(obj)` be equal to `validators(obj, shard_block_hash)` for the value of `shard_block_hash` such that `sum([v.balance for v in validators(obj, shard_block_hash)])` is maximized (ties broken by favoring lower `shard_block_hash` values). Let `total_attesting_balance(obj)` be this maximal sum balance, and `winning_hash(obj)` be the winning `shard_block_hash` value. #### Adjust justified slots and crosslink status @@ -956,34 +957,34 @@ _Note: `last_state_recalculation_slot` will always be a multiple of `CYCLE_LENGT * If `justification_source == prev_s - CYCLE_LENGTH and state.justified_slot_bitfield % 8 == 7`, set `last_finalized_slot = justification_source`. * Set `prev_cycle_justification_source = justification_source` and `justification_source = new_justification_source`. -For every `(shard, shard_block_hash)` tuple: +For every `ShardAndCommittee` object `obj`: -* Let `total_balance_attesting_to_h` be the total balance of validators that attested to the shard block with hash `shard_block_hash`. -* Let `total_committee_balance` be the total balance in the committee of validators that could have attested to the shard block with hash `shard_block_hash`. -* If `3 * total_balance_attesting_to_h >= 2 * total_committee_balance`, set `crosslinks[shard] = CrosslinkRecord(slot=last_state_recalculation_slot + CYCLE_LENGTH, hash=shard_block_hash)`. +* Let `total_committee_balance = sum([state.validators[v].balance for v in obj.committee])` +* If `3 * total_attesting_balance(obj) >= 2 * total_committee_balance`, set `crosslinks[shard] = CrosslinkRecord(slot=last_state_recalculation_slot + CYCLE_LENGTH, hash=winning_hash(obj))`. #### Balance recalculations related to FFG rewards Note: When applying penalties in the following balance recalculations implementers should make sure the `uint64` does not underflow. -* Let `total_balance` be the total balance of active validators. * Let `total_balance_in_eth = total_balance // GWEI_PER_ETH`. * Let `reward_quotient = BASE_REWARD_QUOTIENT * int_sqrt(total_balance_in_eth)`. (The per-slot maximum interest rate is `2/reward_quotient`.) * Let `quadratic_penalty_quotient = SQRT_E_DROP_TIME**2`. (The portion lost by offline validators after `D` cycles is about `D*D/2/quadratic_penalty_quotient`.) * Let `time_since_finality = slot - last_finalized_slot`. -* Let `total_balance_participating` be the total balance of validators that voted for the canonical beacon block at slot `s` (note: every attestation for a block in the slot span `s ... s + CYCLE_LENGTH - 1` counts as this) -* Let `B` be the balance of any given validator whose balance we are adjusting, not including any balance changes from this round of state recalculation. -* If `time_since_finality <= 3 * CYCLE_LENGTH` adjust the balance of participating and non-participating validators as follows: - * Participating validators gain `B // reward_quotient * (2 * total_balance_participating - total_balance) // total_balance`. (Note that this value may be negative.) - * Non-participating validators lose `B // reward_quotient`. -* Otherwise: - * Participating validators gain nothing. - * Non-participating validators lose `B // reward_quotient + B * time_since_finality // quadratic_penalty_quotient`. -In addition, validators with `status == PENALIZED` lose `B // reward_quotient + B * time_since_finality // quadratic_penalty_quotient`. +Case 1: `time_since_finality <= 4 * CYCLE_LENGTH`: + +* Any validator in `prev_s_attesters` with balance `B` gains `B // reward_quotient * (prev_s_attesters - total_balance) // total_balance`. +* All other active validators lose `B // reward_quotient`. + +Case 2: `time_since_finality > 4 * CYCLE_LENGTH`: + +* Any validator in `prev_s_attesters` with balance `B` sees their balance unchanged. +* All other active validators, and validators with `status == PENALIZED`, lose `B // reward_quotient + B * time_since_finality // quadratic_penalty_quotient`. #### Balance recalculations related to crosslink rewards +For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots[:CYCLE_LENGTH]` (ie. the objects corresponding to the slot before the current one): + For every shard number `shard` for which a crosslink committee exists in the cycle prior to the most recent cycle (`s - CYCLE_LENGTH ... s - 1`), let `V` be the corresponding validator set. Let `B` be the balance of any given validator whose balance we are adjusting, not including any balance changes from this round of state recalculation. For each `shard`, `V`: * Let `total_balance_of_v` be the total balance of `V`. From 0d26da4a7d0f7a6adcda9122b19c126e0c21dde5 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sat, 24 Nov 2018 12:44:43 -0500 Subject: [PATCH 08/36] Cleaned up committee balance changes --- specs/core/0_beacon-chain.md | 43 +++++++++++++++++------------------- 1 file changed, 20 insertions(+), 23 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index f17e00222..a1535f174 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -945,7 +945,12 @@ _Note: `last_state_recalculation_slot` will always be a multiple of `CYCLE_LENGT * Let `prev_s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in prev_s_attestations]` * Let `total_balance_attesting_at_s = sum([v.balance for v in s_attesters])` * Let `total_balance_attesting_at_prev_s = sum([v.balance for v in prev_s_attesters])` -* For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots`, let `validators(obj, shard_block_hash)` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_attestations + prev_cycle_attestations if a.shard == obj.shard and a.shard_block_hash == shard_block_hash]`. Let `validators(obj)` be equal to `validators(obj, shard_block_hash)` for the value of `shard_block_hash` such that `sum([v.balance for v in validators(obj, shard_block_hash)])` is maximized (ties broken by favoring lower `shard_block_hash` values). Let `total_attesting_balance(obj)` be this maximal sum balance, and `winning_hash(obj)` be the winning `shard_block_hash` value. +* For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots`, let: + * `attesting_validators(obj, shard_block_hash)` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_attestations + prev_cycle_attestations if a.shard == obj.shard and a.shard_block_hash == shard_block_hash]` + * `attesting_validators(obj)` be equal to `attesting_validators(obj, shard_block_hash)` for the value of `shard_block_hash` such that `sum([v.balance for v in attesting_validators(obj, shard_block_hash)])` is maximized (ties broken by favoring lower `shard_block_hash` values) + * `total_attesting_balance(obj)` be the maximal sum balance + * `winning_hash(obj)` be the winning `shard_block_hash` value + * `total_balance(obj) = sum([v.balance for v in obj.committee])` #### Adjust justified slots and crosslink status @@ -959,8 +964,7 @@ _Note: `last_state_recalculation_slot` will always be a multiple of `CYCLE_LENGT For every `ShardAndCommittee` object `obj`: -* Let `total_committee_balance = sum([state.validators[v].balance for v in obj.committee])` -* If `3 * total_attesting_balance(obj) >= 2 * total_committee_balance`, set `crosslinks[shard] = CrosslinkRecord(slot=last_state_recalculation_slot + CYCLE_LENGTH, hash=winning_hash(obj))`. +* If `3 * total_attesting_balance(obj) >= 2 * total_balance(obj)`, set `crosslinks[shard] = CrosslinkRecord(slot=last_state_recalculation_slot + CYCLE_LENGTH, hash=winning_hash(obj))`. #### Balance recalculations related to FFG rewards @@ -983,18 +987,10 @@ Case 2: `time_since_finality > 4 * CYCLE_LENGTH`: #### Balance recalculations related to crosslink rewards -For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots[:CYCLE_LENGTH]` (ie. the objects corresponding to the slot before the current one): +For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots[:CYCLE_LENGTH]` (ie. the objects corresponding to the slot before the current one), for each `v in obj.committee`, where `V = state.validators[b]` adjust balances as follows: -For every shard number `shard` for which a crosslink committee exists in the cycle prior to the most recent cycle (`s - CYCLE_LENGTH ... s - 1`), let `V` be the corresponding validator set. Let `B` be the balance of any given validator whose balance we are adjusting, not including any balance changes from this round of state recalculation. For each `shard`, `V`: - -* Let `total_balance_of_v` be the total balance of `V`. -* Let `winning_shard_hash` be the hash that the largest total deposits signed for the `shard` during the cycle. -* Define a "participating validator" as a member of `V` that signed a crosslink of `winning_shard_hash`. -* Let `total_balance_of_v_participating` be the total balance of the subset of `V` that participated. -* Let `time_since_last_confirmation = s - crosslinks[shard].slot`. -* Adjust balances as follows: - * Participating validators gain `B // reward_quotient * (2 * total_balance_of_v_participating - total_balance_of_v) // total_balance_of_v`. - * Non-participating validators lose `B // reward_quotient`. +* If `v in attesting_validators(obj)`, `V.balance += V.balance // reward_quotient * total_attesting_balance(obj) // total_balance(obj))`. +* If `v not in attesting_validators(obj)`, `V.balance -= V.balance // reward_quotient`. #### PoW chain related rules @@ -1006,7 +1002,7 @@ If `last_state_recalculation_slot % POW_HASH_VOTING_PERIOD == 0`, then: #### Proposer reshuffling -Run the following code: +Run the following code to update the shard proposer set: ```python active_validator_indices = get_active_validator_indices(validators) @@ -1099,22 +1095,23 @@ def change_validators(validators: List[ValidatorRecord], current_slot: int) -> N v.exit_slot = current_slot withdraw_amount = v.balance - ... - # STUB: withdraw to shard chain + # STUB: withdraw to shard chain ``` +And perform the following updates to the `state`: + * Set `state.validator_set_change_slot = s` -* Set `shard_and_committee_for_slots[:CYCLE_LENGTH] = shard_and_committee_for_slots[CYCLE_LENGTH:]` -* Let `next_start_shard = (shard_and_committee_for_slots[-1][-1].shard + 1) % SHARD_COUNT` -* Set `shard_and_committee_for_slots[CYCLE_LENGTH:] = get_new_shuffling(state.next_shuffling_seed, validators, next_start_shard)` +* Set `state.shard_and_committee_for_slots[:CYCLE_LENGTH] = state.shard_and_committee_for_slots[CYCLE_LENGTH:]` +* Let `state.next_start_shard = (shard_and_committee_for_slots[-1][-1].shard + 1) % SHARD_COUNT` +* Set `state.shard_and_committee_for_slots[CYCLE_LENGTH:] = get_new_shuffling(state.next_shuffling_seed, validators, next_start_shard)` * Set `state.next_shuffling_seed = state.randao_mix` #### If a validator set change does NOT happen -* Set `shard_and_committee_for_slots[:CYCLE_LENGTH] = shard_and_committee_for_slots[CYCLE_LENGTH:]` +* Set `state.shard_and_committee_for_slots[:CYCLE_LENGTH] = state.shard_and_committee_for_slots[CYCLE_LENGTH:]` * Let `time_since_finality = block.slot - state.validator_set_change_slot` -* Let `start_shard = shard_and_committee_for_slots[0][0].shard` -* If `time_since_finality * CYCLE_LENGTH <= MIN_VALIDATOR_SET_CHANGE_INTERVAL` or `time_since_finality` is an exact power of 2, set `shard_and_committee_for_slots[CYCLE_LENGTH:] = get_new_shuffling(state.next_shuffling_seed, validators, start_shard)` and set `state.next_shuffling_seed = state.randao_mix`. Note that `start_shard` is not changed from last cycle. +* Let `start_shard = state.shard_and_committee_for_slots[0][0].shard` +* If `time_since_finality * CYCLE_LENGTH <= MIN_VALIDATOR_SET_CHANGE_INTERVAL` or `time_since_finality` is an exact power of 2, set `state.shard_and_committee_for_slots[CYCLE_LENGTH:] = get_new_shuffling(state.next_shuffling_seed, validators, start_shard)` and set `state.next_shuffling_seed = state.randao_mix`. Note that `start_shard` is not changed from last cycle. #### Finally... From 117b7b695368e9b4c265925ef536e41699e7675a Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sat, 24 Nov 2018 13:30:18 -0500 Subject: [PATCH 09/36] More reward changes * Added rewards for proposers * Added declining rewards for greater inclusion distance --- specs/core/0_beacon-chain.md | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index a1535f174..05a767e54 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -52,6 +52,7 @@ The primary source of load on the beacon chain are "attestations". Attestations | `COLLECTIVE_PENALTY_CALCULATION_PERIOD` | 2**20 (= 1,048,576) | slots | ~2.4 months | | `SLASHING_WHISTLEBLOWER_REWARD_DENOMINATOR` | 2**9 (= 512) | | `BASE_REWARD_QUOTIENT` | 2**11 (= 2,048) | — | +| `INCLUDER_REWARD_QUOTIENT` | 2**14 (= 16,384) | — | | `MAX_VALIDATOR_CHURN_QUOTIENT` | 2**5 (= 32) | — | | `POW_HASH_VOTING_PERIOD` | 2**10 (=1024) | - | | `POW_CONTRACT_MERKLE_TREE_DEPTH` | 2**5 (=32) | - | @@ -951,6 +952,13 @@ _Note: `last_state_recalculation_slot` will always be a multiple of `CYCLE_LENGT * `total_attesting_balance(obj)` be the maximal sum balance * `winning_hash(obj)` be the winning `shard_block_hash` value * `total_balance(obj) = sum([v.balance for v in obj.committee])` + +Let `inclusion_slot(a)` equal the slot in which attestation `a` was included, and `inclusion_distance(a) = inclusion_slot(a) - a.slot`. Let `inclusion_slot(v)` equal `inclusion_distance(a)` for the attestation `a` where `v` is in `get_attestation_participants(state, a)`, and define `inclusion_distance(v)` similarly. We define a function `adjust_for_inclusion_distance(magnitude, dist)` which adjusts the reward of an attestation based on how long it took to get included (the longer, the lower the reward). Returns a value between 0 and `magnitude` + +```python +def adjust_for_inclusion_distance(magnitude, dist): + return magnitude // 2 + (magnitude // 2) * MIN_ATTESTATION_INCLUSION_DELAY // dist +``` #### Adjust justified slots and crosslink status @@ -977,19 +985,21 @@ Note: When applying penalties in the following balance recalculations implemente Case 1: `time_since_finality <= 4 * CYCLE_LENGTH`: -* Any validator in `prev_s_attesters` with balance `B` gains `B // reward_quotient * (prev_s_attesters - total_balance) // total_balance`. -* All other active validators lose `B // reward_quotient`. +* Any validator `V` in `prev_s_attesters` gains `adjust_for_inclusion_distance(V.balance // reward_quotient * (prev_s_attesters - total_balance) // total_balance, inclusion_distance(V))``. +* All other active validators lose `V.balance // reward_quotient`. Case 2: `time_since_finality > 4 * CYCLE_LENGTH`: -* Any validator in `prev_s_attesters` with balance `B` sees their balance unchanged. -* All other active validators, and validators with `status == PENALIZED`, lose `B // reward_quotient + B * time_since_finality // quadratic_penalty_quotient`. +* Any validator in `prev_s_attesters` sees their balance unchanged. +* All other active validators, and validators with `status == PENALIZED`, lose `V.balance // reward_quotient + V.balance * time_since_finality // quadratic_penalty_quotient`. + +For each `V` in `prev_s_attesters`, the validator `proposer = get_beacon_proposer(state, inclusion_slot(V))` gains `proposer.balance // INCLUDER_REWARD_QUOTIENT`. #### Balance recalculations related to crosslink rewards For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots[:CYCLE_LENGTH]` (ie. the objects corresponding to the slot before the current one), for each `v in obj.committee`, where `V = state.validators[b]` adjust balances as follows: -* If `v in attesting_validators(obj)`, `V.balance += V.balance // reward_quotient * total_attesting_balance(obj) // total_balance(obj))`. +* If `v in attesting_validators(obj)`, `V.balance += adjust_for_inclusion_distance(V.balance // reward_quotient * total_attesting_balance(obj) // total_balance(obj)), inclusion_distance(V))`. * If `v not in attesting_validators(obj)`, `V.balance -= V.balance // reward_quotient`. #### PoW chain related rules From 99d7f5c1b89c66983d7be7de4b4e143f0782118d Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sun, 25 Nov 2018 06:33:05 -0500 Subject: [PATCH 10/36] Address Justin's #1 --- specs/core/0_beacon-chain.md | 37 +++++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 05a767e54..2dc4748f2 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -543,6 +543,13 @@ def get_attestation_participants(state: State, We define another set of helpers to be used throughout: `bytes1(x): return x.to_bytes(1, 'big')`, `bytes2(x): return x.to_bytes(2, 'big')`, and so on for all integers, particularly 1, 2, 3, 4, 8, 32. +We define a function to determine the balance of a validator used for determining punishments and calculating stake: + + ```python +def balance_at_stake(validator): + return min(validator.balance, DEPOSIT_SIZE) +``` + We define a function to "add a link" to the validator hash chain, used when a validator is added or removed: ```python @@ -771,7 +778,7 @@ def exit_validator(index, state, block, penalize, current_slot): whistleblower_xfer_amount = validator.deposit // SLASHING_WHISTLEBLOWER_REWARD_DENOMINATOR validator.deposit -= whistleblower_xfer_amount get_beacon_proposer(state, block.slot).deposit += whistleblower_xfer_amount - state.deposits_penalized_in_period[current_slot // COLLECTIVE_PENALTY_CALCULATION_PERIOD] += validator.balance + state.deposits_penalized_in_period[current_slot // COLLECTIVE_PENALTY_CALCULATION_PERIOD] += balance_at_stake(validator) else: validator.status = PENDING_EXIT add_validator_set_change_record(state, index, validator.pubkey, EXIT) @@ -937,21 +944,21 @@ _Note: `last_state_recalculation_slot` will always be a multiple of `CYCLE_LENGT #### Precomputation * Let `active_validators = [state.validators[i] for i in get_active_validator_indices(state.validators)]` -* Let `total_balance = sum([v.balance for v in active_validators])` +* Let `total_balance = sum([balance_at_stake(v) for v in active_validators])` * Let `this_cycle_attestations = [a for a in state.pending_attestations if s <= a.slot < s + CYCLE_LENGTH]` (note: this is the set of attestations _of slots in the cycle `s...s+CYCLE_LENGTH-1`_, not attestations _that got included in the chain during the cycle `s...s+CYCLE_LENGTH-1`_) * Let `prev_cycle_attestations = [a for a in state.pending_attestations if s - CYCLE_LENGTH <= a.slot < s]` * Let `this_cycle_s_attestations = [a for a in this_cycle_attestations if get_block_hash(state, block, s) in a.parent_hashes and a.justified_slot == state.justification_source]` * Let `prev_s_attestations = [a for a in this_cycle_attestations + prev_cycle_attestations if get_block_hash(state, block, s - CYCLE_LENGTH) in a.parent_hashes and a.justified_slot == state.prev_cycle_justification_source]` * Let `s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_s_attestations]` * Let `prev_s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in prev_s_attestations]` -* Let `total_balance_attesting_at_s = sum([v.balance for v in s_attesters])` -* Let `total_balance_attesting_at_prev_s = sum([v.balance for v in prev_s_attesters])` +* Let `total_balance_attesting_at_s = sum([balance_at_stake(v) for v in s_attesters])` +* Let `total_balance_attesting_at_prev_s = sum([balance_at_stake(v) for v in prev_s_attesters])` * For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots`, let: * `attesting_validators(obj, shard_block_hash)` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_attestations + prev_cycle_attestations if a.shard == obj.shard and a.shard_block_hash == shard_block_hash]` - * `attesting_validators(obj)` be equal to `attesting_validators(obj, shard_block_hash)` for the value of `shard_block_hash` such that `sum([v.balance for v in attesting_validators(obj, shard_block_hash)])` is maximized (ties broken by favoring lower `shard_block_hash` values) + * `attesting_validators(obj)` be equal to `attesting_validators(obj, shard_block_hash)` for the value of `shard_block_hash` such that `sum([balance_at_stake(v) for v in attesting_validators(obj, shard_block_hash)])` is maximized (ties broken by favoring lower `shard_block_hash` values) * `total_attesting_balance(obj)` be the maximal sum balance * `winning_hash(obj)` be the winning `shard_block_hash` value - * `total_balance(obj) = sum([v.balance for v in obj.committee])` + * `total_balance(obj) = sum([balance_at_stake(v) for v in obj.committee])` Let `inclusion_slot(a)` equal the slot in which attestation `a` was included, and `inclusion_distance(a) = inclusion_slot(a) - a.slot`. Let `inclusion_slot(v)` equal `inclusion_distance(a)` for the attestation `a` where `v` is in `get_attestation_participants(state, a)`, and define `inclusion_distance(v)` similarly. We define a function `adjust_for_inclusion_distance(magnitude, dist)` which adjusts the reward of an attestation based on how long it took to get included (the longer, the lower the reward). Returns a value between 0 and `magnitude` @@ -985,22 +992,22 @@ Note: When applying penalties in the following balance recalculations implemente Case 1: `time_since_finality <= 4 * CYCLE_LENGTH`: -* Any validator `V` in `prev_s_attesters` gains `adjust_for_inclusion_distance(V.balance // reward_quotient * (prev_s_attesters - total_balance) // total_balance, inclusion_distance(V))``. -* All other active validators lose `V.balance // reward_quotient`. +* Any validator `V` in `prev_s_attesters` gains `adjust_for_inclusion_distance(balance_at_stake(V) // reward_quotient * (prev_s_attesters - total_balance) // total_balance, inclusion_distance(V))``. +* Any active validator `V` not in `prev_s_attesters` loses `balance_at_stake(V) // reward_quotient`. Case 2: `time_since_finality > 4 * CYCLE_LENGTH`: * Any validator in `prev_s_attesters` sees their balance unchanged. -* All other active validators, and validators with `status == PENALIZED`, lose `V.balance // reward_quotient + V.balance * time_since_finality // quadratic_penalty_quotient`. +* Any active validator `V` not in `prev_s_attesters`, and any validator with `status == PENALIZED`, loses `balance_at_stake(V) // reward_quotient + balance_at_stake(V) * time_since_finality // quadratic_penalty_quotient`. -For each `V` in `prev_s_attesters`, the validator `proposer = get_beacon_proposer(state, inclusion_slot(V))` gains `proposer.balance // INCLUDER_REWARD_QUOTIENT`. +For each `V` in `prev_s_attesters`, the validator `proposer = get_beacon_proposer(state, inclusion_slot(V))` gains `balance_at_stake(proposer) // INCLUDER_REWARD_QUOTIENT`. #### Balance recalculations related to crosslink rewards For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots[:CYCLE_LENGTH]` (ie. the objects corresponding to the slot before the current one), for each `v in obj.committee`, where `V = state.validators[b]` adjust balances as follows: -* If `v in attesting_validators(obj)`, `V.balance += adjust_for_inclusion_distance(V.balance // reward_quotient * total_attesting_balance(obj) // total_balance(obj)), inclusion_distance(V))`. -* If `v not in attesting_validators(obj)`, `V.balance -= V.balance // reward_quotient`. +* If `v in attesting_validators(obj)`, `V.balance += adjust_for_inclusion_distance(balance_at_stake(V) // reward_quotient * total_attesting_balance(obj) // total_balance(obj)), inclusion_distance(V))`. +* If `v not in attesting_validators(obj)`, `V.balance -= balance_at_stake(V) // reward_quotient`. #### PoW chain related rules @@ -1053,7 +1060,7 @@ def change_validators(validators: List[ValidatorRecord], current_slot: int) -> N # The active validator set active_validators = get_active_validator_indices(validators) # The total balance of active validators - total_balance = sum([v.balance for i, v in enumerate(validators) if i in active_validators]) + total_balance = sum([balance_at_stake(v) for i, v in enumerate(validators) if i in active_validators]) # The maximum total wei that can deposit+withdraw max_allowable_change = max( 2 * DEPOSIT_SIZE * GWEI_PER_ETH, @@ -1074,7 +1081,7 @@ def change_validators(validators: List[ValidatorRecord], current_slot: int) -> N if validators[i].status == PENDING_EXIT: validators[i].status = PENDING_WITHDRAW validators[i].exit_slot = current_slot - total_changed += validators[i].balance + total_changed += balance_at_stake(validators[i]) add_validator_set_change_record( state=state, index=i, @@ -1100,7 +1107,7 @@ def change_validators(validators: List[ValidatorRecord], current_slot: int) -> N withdrawable_validators = sorted(filter(withdrawable, validators), key=lambda v: v.exit_seq) for v in withdrawable_validators[:WITHDRAWALS_PER_CYCLE]: if v.status == PENALIZED: - v.balance -= v.balance * min(total_penalties * 3, total_balance) // total_balance + v.balance -= balance_at_stake(v) * min(total_penalties * 3, total_balance) // total_balance v.status = WITHDRAWN v.exit_slot = current_slot From 2dad2d4c42f0f17a26b399dd86b547bee91b6dc9 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sun, 25 Nov 2018 21:14:13 -0500 Subject: [PATCH 11/36] V -> v --- specs/core/0_beacon-chain.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 2dc4748f2..4051e1959 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -840,9 +840,9 @@ Verify that `BLSVerify(pubkey=get_beacon_proposer(state, block.slot).pubkey, dat ### Verify and process RANDAO reveal * Let `repeat_hash(x, n) = x if n == 0 else repeat_hash(hash(x), n-1)`. -* Let `V = get_beacon_proposer(state, block.slot). -* Verify that `repeat_hash(block.randao_reveal, (block.slot - V.randao_last_change) // RANDAO_SLOTS_PER_LAYER + 1) == V.randao_commitment` -* Set `state.randao_mix = xor(state.randao_mix, block.randao_reveal)`, `V.randao_commitment = block.randao_reveal`, `V.randao_last_change = block.slot` +* Let `proposer = get_beacon_proposer(state, block.slot). +* Verify that `repeat_hash(block.randao_reveal, (block.slot - proposer.randao_last_change) // RANDAO_SLOTS_PER_LAYER + 1) == proposer.randao_commitment` +* Set `state.randao_mix = xor(state.randao_mix, block.randao_reveal)`, `proposer.randao_commitment = block.randao_reveal`, `proposer.randao_last_change = block.slot` Finally, if `block.candidate_pow_hash_chain_tip = state.candidate_pow_hash_chain_tip`, set `state.candidate_hash_chain_tip_votes += 1`. @@ -992,22 +992,22 @@ Note: When applying penalties in the following balance recalculations implemente Case 1: `time_since_finality <= 4 * CYCLE_LENGTH`: -* Any validator `V` in `prev_s_attesters` gains `adjust_for_inclusion_distance(balance_at_stake(V) // reward_quotient * (prev_s_attesters - total_balance) // total_balance, inclusion_distance(V))``. -* Any active validator `V` not in `prev_s_attesters` loses `balance_at_stake(V) // reward_quotient`. +* Any validator `v` in `prev_s_attesters` gains `adjust_for_inclusion_distance(balance_at_stake(v) // reward_quotient * (prev_s_attesters - total_balance) // total_balance, inclusion_distance(v))``. +* Any active validator `v` not in `prev_s_attesters` loses `balance_at_stake(v) // reward_quotient`. Case 2: `time_since_finality > 4 * CYCLE_LENGTH`: * Any validator in `prev_s_attesters` sees their balance unchanged. -* Any active validator `V` not in `prev_s_attesters`, and any validator with `status == PENALIZED`, loses `balance_at_stake(V) // reward_quotient + balance_at_stake(V) * time_since_finality // quadratic_penalty_quotient`. +* Any active validator `v` not in `prev_s_attesters`, and any validator with `status == PENALIZED`, loses `balance_at_stake(v) // reward_quotient + balance_at_stake(v) * time_since_finality // quadratic_penalty_quotient`. -For each `V` in `prev_s_attesters`, the validator `proposer = get_beacon_proposer(state, inclusion_slot(V))` gains `balance_at_stake(proposer) // INCLUDER_REWARD_QUOTIENT`. +For each `v` in `prev_s_attesters`, the validator `proposer = get_beacon_proposer(state, inclusion_slot(v))` gains `balance_at_stake(proposer) // INCLUDER_REWARD_QUOTIENT`. #### Balance recalculations related to crosslink rewards -For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots[:CYCLE_LENGTH]` (ie. the objects corresponding to the slot before the current one), for each `v in obj.committee`, where `V = state.validators[b]` adjust balances as follows: +For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots[:CYCLE_LENGTH]` (ie. the objects corresponding to the slot before the current one), for each `v in obj.committee`, where `v = state.validators[b]` adjust balances as follows: -* If `v in attesting_validators(obj)`, `V.balance += adjust_for_inclusion_distance(balance_at_stake(V) // reward_quotient * total_attesting_balance(obj) // total_balance(obj)), inclusion_distance(V))`. -* If `v not in attesting_validators(obj)`, `V.balance -= balance_at_stake(V) // reward_quotient`. +* If `v in attesting_validators(obj)`, `v.balance += adjust_for_inclusion_distance(balance_at_stake(v) // reward_quotient * total_attesting_balance(obj) // total_balance(obj)), inclusion_distance(v))`. +* If `v not in attesting_validators(obj)`, `v.balance -= balance_at_stake(v) // reward_quotient`. #### PoW chain related rules From 04ac4ea54d4b24b9fce5d0a023ab8011cc0d0c47 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Sun, 25 Nov 2018 21:15:46 -0500 Subject: [PATCH 12/36] Fixed more nitpicks --- specs/core/0_beacon-chain.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 4051e1959..ea93b5ccb 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -823,7 +823,7 @@ For each `AttestationRecord` object `obj`: * Verify that `justified_slot` is equal to `justification_source if slot >= block.slot - (block.slot % CYCLE_LENGTH) else prev_cycle_justification_source` * Verify that `justified_block_hash` is the hash of the block in the current chain at the slot -- `justified_slot`. * Verify that either `last_crosslink_hash` or `shard_block_hash` equals `state.crosslinks[shard].shard_block_hash`. -* Compute `full_parent_hashes` = `[get_block_hash(state, block, slot - CYCLE_LENGTH + i) for i in range(1, CYCLE_LENGTH - len(parent_hashes) + 1)] + parent_hashes` (eg, if `CYCLE_LENGTH = 4`, `slot = 5`, the actual block hashes starting from slot 0 are `Z A B C D E F G H I J`, and `parent_hashes = [D', E']` then `parent_hashes = [B, C, D' E']`). Note that when *creating* an attestation for a block, the hash of that block itself won't yet be in the `state`, so you would need to add it explicitly. +* Compute `full_parent_hashes` = `[get_block_hash(state, block, slot - CYCLE_LENGTH + i) for i in range(1, CYCLE_LENGTH - len(parent_hashes) + 1)] + parent_hashes` (eg, if `CYCLE_LENGTH = 4`, `slot = 5`, the actual block hashes starting from slot 0 are `Z A B C D E F G H I J`, and `parent_hashes = [D', E']` then `full_parent_hashes = [B, C, D' E']`). Note that when *creating* an attestation for a block, the hash of that block itself won't yet be in the `state`, so you would need to add it explicitly. * Let `bit0_attestation_indices, bit1_attestation_indices = get_attestation_participants(state, obj)` (and verify that the method returns successfully) * Let `bit0_group_public_key = BLSAddPubkeys(bit0_attestation_indices)` and `bit1_group_public_key = BLSAddPubkeys(bit1_attestation_indices)` * Let `fork_version = pre_fork_version if slot < fork_slot_number else post_fork_version`. @@ -992,7 +992,7 @@ Note: When applying penalties in the following balance recalculations implemente Case 1: `time_since_finality <= 4 * CYCLE_LENGTH`: -* Any validator `v` in `prev_s_attesters` gains `adjust_for_inclusion_distance(balance_at_stake(v) // reward_quotient * (prev_s_attesters - total_balance) // total_balance, inclusion_distance(v))``. +* Any validator `v` in `prev_s_attesters` gains `adjust_for_inclusion_distance(balance_at_stake(v) // reward_quotient * (prev_s_attesters - total_balance) // total_balance, inclusion_distance(v))`. * Any active validator `v` not in `prev_s_attesters` loses `balance_at_stake(v) // reward_quotient`. Case 2: `time_since_finality > 4 * CYCLE_LENGTH`: From b1740ba0c5b68bef28933838130e4c6a3a777ae9 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Mon, 26 Nov 2018 17:14:42 +0800 Subject: [PATCH 13/36] Proofread --- specs/core/0_beacon-chain.md | 38 ++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 01d72ad7c..4a1611588 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -150,7 +150,7 @@ An `AttestationRecord` has the following fields: 'last_crosslink_hash': 'hash32', # Root of data between last hash and this one 'shard_block_combined_data_root': 'hash32', - # Attester participation bitfield (1 bit per attester) + # Attester participation bitfield (2 bits per attester) 'attester_bitfield': 'bytes', # Slot of last justified beacon block 'justified_slot': 'uint64', @@ -577,7 +577,7 @@ We define another set of helpers to be used throughout: `bytes1(x): return x.to_ We define a function to determine the balance of a validator used for determining punishments and calculating stake: ```python -def balance_at_stake(validator): +def balance_at_stake(validator: ValidatorRecord) -> int: return min(validator.balance, DEPOSIT_SIZE) ``` @@ -825,7 +825,7 @@ def exit_validator(index, state, block, penalize, current_slot): This procedure should be carried out every beacon block. * Let `parent_hash` be the hash of the immediate previous beacon block (ie. equal to `ancestor_hashes[0]`). -* Let `parent` be the beacon block with the hash `parent_hash` +* Let `parent` be the beacon block with the hash `parent_hash`. First, set `recent_block_hashes` to the output of the following: @@ -860,10 +860,11 @@ For each `AttestationRecord` object `obj`: * Verify that `justified_block_hash` is the hash of the block in the current chain at the slot -- `justified_slot`. * Verify that either `last_crosslink_hash` or `shard_block_hash` equals `state.crosslinks[shard].shard_block_hash`. * Compute `full_parent_hashes` = `[get_block_hash(state, block, slot - CYCLE_LENGTH + i) for i in range(1, CYCLE_LENGTH - len(parent_hashes) + 1)] + parent_hashes` (eg, if `CYCLE_LENGTH = 4`, `slot = 5`, the actual block hashes starting from slot 0 are `Z A B C D E F G H I J`, and `parent_hashes = [D', E']` then `full_parent_hashes = [B, C, D' E']`). Note that when *creating* an attestation for a block, the hash of that block itself won't yet be in the `state`, so you would need to add it explicitly. -* Let `bit0_attestation_indices, bit1_attestation_indices = get_attestation_participants(state, obj)` (and verify that the method returns successfully) -* Let `bit0_group_public_key = BLSAddPubkeys(bit0_attestation_indices)` and `bit1_group_public_key = BLSAddPubkeys(bit1_attestation_indices)` -* Let `data = AttestationSignedData(slot, shard, parent_hashes, shard_block_hash, last_crosslinked_hash, shard_block_combined_data_root, justified_slot)`. -* Check `BLSVerify(pubkey=group_public_key, msg=data, sig=aggregate_sig, domain=get_domain(state, slot, DOMAIN_ATTESTATION))`. +* `aggregate_sig` verification: + * Let `bit0_attestation_indices, bit1_attestation_indices = get_attestation_participants(state, obj)` (and verify that the method returns successfully) + * Let `bit0_group_public_key = BLSAddPubkeys(bit0_attestation_indices)` and `bit1_group_public_key = BLSAddPubkeys(bit1_attestation_indices)`. + * Let `data = AttestationSignedData(slot, shard, parent_hashes, shard_block_hash, last_crosslinked_hash, shard_block_combined_data_root, justified_slot)`. + * Check `BLSVerify(pubkey=group_public_key, msg=data, sig=aggregate_sig, domain=get_domain(state, slot, DOMAIN_ATTESTATION))`. * [TO BE REMOVED IN PHASE 1] Verify that `shard_block_hash == bytes([0] * 32)`. Extend the list of `AttestationRecord` objects in the `state` with those included in the block, ordering the new additions in the same order as they came in the block, and replacing `obj.parent_hashes` with the calculated value of `full_parent_hashes`. @@ -983,16 +984,16 @@ _Note: `last_state_recalculation_slot` will always be a multiple of `CYCLE_LENGT #### Precomputation -* Let `active_validators = [state.validators[i] for i in get_active_validator_indices(state.validators)]` -* Let `total_balance = sum([balance_at_stake(v) for v in active_validators])` -* Let `this_cycle_attestations = [a for a in state.pending_attestations if s <= a.slot < s + CYCLE_LENGTH]` (note: this is the set of attestations _of slots in the cycle `s...s+CYCLE_LENGTH-1`_, not attestations _that got included in the chain during the cycle `s...s+CYCLE_LENGTH-1`_) -* Let `prev_cycle_attestations = [a for a in state.pending_attestations if s - CYCLE_LENGTH <= a.slot < s]` -* Let `this_cycle_s_attestations = [a for a in this_cycle_attestations if get_block_hash(state, block, s) in a.parent_hashes and a.justified_slot == state.justification_source]` -* Let `prev_s_attestations = [a for a in this_cycle_attestations + prev_cycle_attestations if get_block_hash(state, block, s - CYCLE_LENGTH) in a.parent_hashes and a.justified_slot == state.prev_cycle_justification_source]` -* Let `s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_s_attestations]` -* Let `prev_s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in prev_s_attestations]` -* Let `total_balance_attesting_at_s = sum([balance_at_stake(v) for v in s_attesters])` -* Let `total_balance_attesting_at_prev_s = sum([balance_at_stake(v) for v in prev_s_attesters])` +* Let `active_validators = [state.validators[i] for i in get_active_validator_indices(state.validators)]`. +* Let `total_balance = sum([balance_at_stake(v) for v in active_validators])`. +* Let `this_cycle_attestations = [a for a in state.pending_attestations if s <= a.slot < s + CYCLE_LENGTH]`. (note: this is the set of attestations _of slots in the cycle `s...s+CYCLE_LENGTH-1`_, not attestations _that got included in the chain during the cycle `s...s+CYCLE_LENGTH-1`_) +* Let `prev_cycle_attestations = [a for a in state.pending_attestations if s - CYCLE_LENGTH <= a.slot < s]`. +* Let `this_cycle_s_attestations = [a for a in this_cycle_attestations if get_block_hash(state, block, s) in a.parent_hashes and a.justified_slot == state.justification_source]`. +* Let `prev_s_attestations = [a for a in this_cycle_attestations + prev_cycle_attestations if get_block_hash(state, block, s - CYCLE_LENGTH) in a.parent_hashes and a.justified_slot == state.prev_cycle_justification_source]`. +* Let `s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_s_attestations]`. +* Let `prev_s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in prev_s_attestations]`. +* Let `total_balance_attesting_at_s = sum([balance_at_stake(v) for v in s_attesters])`. +* Let `total_balance_attesting_at_prev_s = sum([balance_at_stake(v) for v in prev_s_attesters])`. * For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots`, let: * `attesting_validators(obj, shard_block_hash)` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_attestations + prev_cycle_attestations if a.shard == obj.shard and a.shard_block_hash == shard_block_hash]` * `attesting_validators(obj)` be equal to `attesting_validators(obj, shard_block_hash)` for the value of `shard_block_hash` such that `sum([balance_at_stake(v) for v in attesting_validators(obj, shard_block_hash)])` is maximized (ties broken by favoring lower `shard_block_hash` values) @@ -1003,7 +1004,7 @@ _Note: `last_state_recalculation_slot` will always be a multiple of `CYCLE_LENGT Let `inclusion_slot(a)` equal the slot in which attestation `a` was included, and `inclusion_distance(a) = inclusion_slot(a) - a.slot`. Let `inclusion_slot(v)` equal `inclusion_distance(a)` for the attestation `a` where `v` is in `get_attestation_participants(state, a)`, and define `inclusion_distance(v)` similarly. We define a function `adjust_for_inclusion_distance(magnitude, dist)` which adjusts the reward of an attestation based on how long it took to get included (the longer, the lower the reward). Returns a value between 0 and `magnitude` ```python -def adjust_for_inclusion_distance(magnitude, dist): +def adjust_for_inclusion_distance(magnitude: int, dist: int) -> int: return magnitude // 2 + (magnitude // 2) * MIN_ATTESTATION_INCLUSION_DELAY // dist ``` @@ -1214,7 +1215,6 @@ Note: This spec is ~65% complete. * [ ] Add penalties for deposits below 32 ETH (or some other threshold) * [ ] Add a `SpecialRecord` to (re)register - # Appendix ## Appendix A - Hash function From 25e5408ed80a2a5659ce4250ecf945d2a4a952bc Mon Sep 17 00:00:00 2001 From: terence tsao Date: Mon, 26 Nov 2018 06:36:19 -1000 Subject: [PATCH 14/36] shard block sigs to uint384 --- specs/core/1_shard-data-chains.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/core/1_shard-data-chains.md b/specs/core/1_shard-data-chains.md index 59a6f6c1a..27b9d9ef1 100644 --- a/specs/core/1_shard-data-chains.md +++ b/specs/core/1_shard-data-chains.md @@ -51,10 +51,10 @@ A `ShardBlock` object has the following fields: # State root (placeholder for now) 'state_root': 'hash32', # Block signature - 'signature': ['uint256'], + 'signature': ['uint384'], # Attestation 'attester_bitfield': 'bytes', - 'aggregate_sig': ['uint256'], + 'aggregate_sig': ['uint384'], } ``` From 386dab1140b87f367e0e5089a50cc9e8add36611 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Mon, 26 Nov 2018 13:40:02 -0500 Subject: [PATCH 15/36] Update 0_beacon-chain.md --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 5117ea664..8d9010c4a 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -815,11 +815,11 @@ def exit_validator(index, state, block, penalize, current_slot): committee.pop(i) break if penalize: + state.deposits_penalized_in_period[current_slot // COLLECTIVE_PENALTY_CALCULATION_PERIOD] += balance_at_stake(validator) validator.status = PENALIZED whistleblower_xfer_amount = validator.deposit // SLASHING_WHISTLEBLOWER_REWARD_DENOMINATOR validator.deposit -= whistleblower_xfer_amount get_beacon_proposer(state, block.slot).deposit += whistleblower_xfer_amount - state.deposits_penalized_in_period[current_slot // COLLECTIVE_PENALTY_CALCULATION_PERIOD] += balance_at_stake(validator) else: validator.status = PENDING_EXIT add_validator_set_change_record(state, index, validator.pubkey, EXIT) From 170962c80f7919d0e985fb05fc4545c2d037772a Mon Sep 17 00:00:00 2001 From: vbuterin Date: Mon, 26 Nov 2018 17:12:39 -0500 Subject: [PATCH 16/36] Big refactor + simplification --- specs/core/0_beacon-chain.md | 198 +++++++++++++++++++---------------- 1 file changed, 105 insertions(+), 93 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 8d9010c4a..d18ccbcfc 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -51,7 +51,7 @@ The primary source of load on the beacon chain are "attestations". Attestations | `POW_RECEIPT_ROOT_VOTING_PERIOD` | 2**10 (= 1,024) | slots | ~1.7 hours | | `SLASHING_WHISTLEBLOWER_REWARD_DENOMINATOR` | 2**9 (= 512) | | `BASE_REWARD_QUOTIENT` | 2**11 (= 2,048) | — | -| `INCLUDER_REWARD_QUOTIENT` | 2**14 (= 16,384) | — | +| `INCLUDER_REWARD_SHARE_QUOTIENT` | 2**3 (= 8) | — | | `MAX_VALIDATOR_CHURN_QUOTIENT` | 2**5 (= 32) | — | | `POW_CONTRACT_MERKLE_TREE_DEPTH` | 2**5 (= 32) | - | | `MAX_ATTESTATION_COUNT` | 2**7 (= 128) | - | @@ -138,29 +138,39 @@ An `AttestationRecord` has the following fields: ```python { - # Slot number - 'slot': 'uint64', - # Shard number - 'shard': 'uint64', - # Beacon block hashes not part of the current chain, oldest to newest - 'parent_hashes': ['hash32'], - # Shard block hash being attested to - 'shard_block_hash': 'hash32', - # Last crosslink hash - 'last_crosslink_hash': 'hash32', - # Root of data between last hash and this one - 'shard_block_combined_data_root': 'hash32', + data: AttestationSignedData # Attester participation bitfield (2 bits per attester) 'attester_bitfield': 'bytes', - # Slot of last justified beacon block - 'justified_slot': 'uint64', - # Hash of last justified beacon block - 'justified_block_hash': 'hash32', + # Proof of custody bitfield + 'poc_bitfield': 'bytes', # BLS aggregate signature 'aggregate_sig': ['uint384'] } ``` +`AttestationSignedData`: + +```python +{ + # Slot number + 'slot': 'uint64', + # Shard number + 'shard': 'uint64', + # Hash of the block we're signing + 'block_hash': 'hash32', + # Hash of the ancestor at the cycle boundary + 'cycle_boundary_hash': 'hash32', + # Shard block hash being attested to + 'shard_block_hash': 'hash32', + # Last crosslink hash + 'last_crosslink_hash': 'hash32', + # Slot of last justified beacon block + 'justified_slot': 'uint64', + # Hash of last justified beacon block + 'justified_block_hash': 'hash32', +} +``` + A `ProposalSignedData` has the following fields: ```python @@ -174,27 +184,6 @@ A `ProposalSignedData` has the following fields: } ``` -An `AttestationSignedData` has the following fields: - -```python -{ - # Slot number - 'slot': 'uint64', - # Shard number - 'shard': 'uint64', - # CYCLE_LENGTH parent hashes - 'parent_hashes': ['hash32'], - # Shard block hash - 'shard_block_hash': 'hash32', - # Last crosslink hash - 'last_crosslink_hash': 'hash32', - # Root of data between last hash and this one - 'shard_block_combined_data_root': 'hash32', - # Slot of last justified beacon block referenced in the attestation - 'justified_slot': 'uint64' -} -``` - A `SpecialRecord` has the following fields: ```python @@ -251,7 +240,7 @@ The `BeaconState` has the following fields: 'post_fork_version': 'uint64', 'fork_slot_number': 'uint64', # Attestations not yet processed - 'pending_attestations': [AttestationRecord], + 'pending_attestations': [ProcessedAttestations], # recent beacon block hashes needed to process attestations, older to newer 'recent_block_hashes': ['hash32'], # RANDAO state @@ -328,6 +317,21 @@ A `CandidatePoWReceiptRootRecord` object contains the following fields: } ``` +A `ProcessedAttestation` object has the following fields: + +```python +{ + # Signed data + 'data': AttestationSignedData, + # Attester participation bitfield (2 bits per attester) + 'attester_bitfield': 'bytes', + # Proof of custody bitfield + 'poc_bitfield': 'bytes', + # Slot in which it was included + 'slot_included': 'uint64' +} +``` + ## Beacon chain processing The beacon chain is the "main chain" of the PoS system. The beacon chain's main responsibilities are: @@ -557,19 +561,17 @@ The following is a function that determines the validators that participated in ```python def get_attestation_participants(state: State, - attestation: AttestationRecord) -> Tuple[List[int], List[int]]: - sncs_for_slot = get_shards_and_committees_for_slot(state, attestation.slot) - snc = [x for x in sncs_for_slot if x.shard == attestation.shard][0] - assert len(attestation.attester_bitfield) == ceil_div8(len(snc.committee) * 2) - bit0_participants, bit1_participants = [], [] + attestation_data: AttestationSignedData, + attester_bitfield: bytes) -> List[int]: + sncs_for_slot = get_shards_and_committees_for_slot(state, attestation_data.slot) + snc = [x for x in sncs_for_slot if x.shard == attestation_data.shard][0] + assert len(attester_bitfield) == ceil_div8(len(snc.committee)) + participants = [] for i, vindex in snc.committee: - bits = (attestation.attester_bitfield[i//4] >> (3 - (i % 4)) * 2) % 4 - assert bits in (0, 2, 3) - if bits == 2: - bit0_participants.append(vindex) - elif bits == 3: - bit1_participants.append(vindex) - return bit0_participants, bit1_participants + bit = (attester_bitfield[i//8] >> (7 - (i % 8))) % 2 + if bit == 1: + participants.append(vindex) + return participants ``` We define another set of helpers to be used throughout: `bytes1(x): return x.to_bytes(1, 'big')`, `bytes2(x): return x.to_bytes(2, 'big')`, and so on for all integers, particularly 1, 2, 3, 4, 8, 32. @@ -860,21 +862,18 @@ def update_ancestor_hashes(parent_ancestor_hashes: List[Hash32], Verify that there are at most `MAX_ATTESTATION_COUNT` `AttestationRecord` objects. -For each `AttestationRecord` object: +For each `AttestationRecord` object `obj`: -* Verify that `slot <= block.slot - MIN_ATTESTATION_INCLUSION_DELAY` and `slot >= max(parent.slot - CYCLE_LENGTH + 1, 0)`. -* Verify that `justified_slot` is equal to `justification_source if slot >= block.slot - (block.slot % CYCLE_LENGTH) else prev_cycle_justification_source` -* Verify that `justified_block_hash` is the hash of the block in the current chain at the slot -- `justified_slot`. -* Verify that either `last_crosslink_hash` or `shard_block_hash` equals `state.crosslinks[shard].shard_block_hash`. -* Compute `full_parent_hashes` = `[get_block_hash(state, block, slot - CYCLE_LENGTH + i) for i in range(1, CYCLE_LENGTH - len(parent_hashes) + 1)] + parent_hashes` (eg, if `CYCLE_LENGTH = 4`, `slot = 5`, the actual block hashes starting from slot 0 are `Z A B C D E F G H I J`, and `parent_hashes = [D', E']` then `full_parent_hashes = [B, C, D' E']`). Note that when *creating* an attestation for a block, the hash of that block itself won't yet be in the `state`, so you would need to add it explicitly. +* Verify that `obj.data.slot <= block.slot - MIN_ATTESTATION_INCLUSION_DELAY` and `obj.data.slot >= max(parent.slot - CYCLE_LENGTH + 1, 0)`. +* Verify that `obj.data.justified_slot` is equal to `justification_source if state.last_state_recalculation_slot else prev_cycle_justification_source` +* Verify that `obj.data.justified_block_hash` is the hash of the block in the current chain at the slot -- `obj.data.justified_slot`. +* Verify that either `obj.data.last_crosslink_hash` or `obj.data.shard_block_hash` equals `state.crosslinks[shard].shard_block_hash`. * `aggregate_sig` verification: - * Let `bit0_attestation_indices, bit1_attestation_indices = get_attestation_participants(state, obj)` (and verify that the method returns successfully) - * Let `bit0_group_public_key = BLSAddPubkeys(bit0_attestation_indices)` and `bit1_group_public_key = BLSAddPubkeys(bit1_attestation_indices)`. - * Let `data = AttestationSignedData(slot, shard, parent_hashes, shard_block_hash, last_crosslinked_hash, shard_block_combined_data_root, justified_slot)`. - * Check `BLSVerify(pubkey=group_public_key, msg=data, sig=aggregate_sig, domain=get_domain(state, slot, DOMAIN_ATTESTATION))`. + * Let `participants = get_attestation_participants(state, obj.data, obj.attester_bitfield)` + * Let `group_public_key = BLSAddPubkeys([state.validators[v].pubkey for v in participants])` + * Check `BLSVerify(pubkey=group_public_key, msg=obj.data, sig=aggregate_sig, domain=get_domain(state, slot, DOMAIN_ATTESTATION))`. * [TO BE REMOVED IN PHASE 1] Verify that `shard_block_hash == bytes([0] * 32)`. - -Extend the list of `AttestationRecord` objects in the `state` with those included in the block, ordering the new additions in the same order as they came in the block, and replacing `obj.parent_hashes` with the calculated value of `full_parent_hashes`. +* Append `ProcessedAttestation(data=obj.data, attester_bitfield=obj.attester_bitfield, poc_bitfield=obj.poc_bitfield, slot_included=block.slot)` to `state.pending_attestations`. ### Verify proposer signature @@ -1001,39 +1000,52 @@ _Note: `last_state_recalculation_slot` will always be a multiple of `CYCLE_LENGT #### Precomputation +All validators: + * Let `active_validators = [state.validators[i] for i in get_active_validator_indices(state.validators)]`. * Let `total_balance = sum([balance_at_stake(v) for v in active_validators])`. -* Let `this_cycle_attestations = [a for a in state.pending_attestations if s <= a.slot < s + CYCLE_LENGTH]`. (note: this is the set of attestations _of slots in the cycle `s...s+CYCLE_LENGTH-1`_, not attestations _that got included in the chain during the cycle `s...s+CYCLE_LENGTH-1`_) + +Validators justifying the cycle boundary block at the start of the current cycle: + +* Let `this_cycle_attestations = [a for a in state.pending_attestations if s <= a.data.slot < s + CYCLE_LENGTH]`. (note: this is the set of attestations _of slots in the cycle `s...s+CYCLE_LENGTH-1`_, not attestations _that got included in the chain during the cycle `s...s+CYCLE_LENGTH-1`_) +* Let `this_cycle_boundary_attestations = [a for a in this_cycle_attestations if a.data.cycle_boundary_hash == get_block_hash(state, block, s) a.justified_slot == state.justification_source]`. +* Let `this_cycle_boundary_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a.data, a.attester_bitfield) for a in this_cycle_boundary_attestations]`. +* Let `this_cycle_boundary_attesting_balance = sum([balance_at_stake(v) for v in this_cycle_boundary_attesters])`. + +Validators justifying the cycle boundary block at the start of the previous cycle: + * Let `prev_cycle_attestations = [a for a in state.pending_attestations if s - CYCLE_LENGTH <= a.slot < s]`. -* Let `this_cycle_s_attestations = [a for a in this_cycle_attestations if get_block_hash(state, block, s) in a.parent_hashes and a.justified_slot == state.justification_source]`. -* Let `prev_s_attestations = [a for a in this_cycle_attestations + prev_cycle_attestations if get_block_hash(state, block, s - CYCLE_LENGTH) in a.parent_hashes and a.justified_slot == state.prev_cycle_justification_source]`. -* Let `s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_s_attestations]`. -* Let `prev_s_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in prev_s_attestations]`. -* Let `total_balance_attesting_at_s = sum([balance_at_stake(v) for v in s_attesters])`. -* Let `total_balance_attesting_at_prev_s = sum([balance_at_stake(v) for v in prev_s_attesters])`. -* For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots`, let: - * `attesting_validators(obj, shard_block_hash)` be the union of the validator index sets given by `[get_attestation_participants(state, a) for a in this_cycle_attestations + prev_cycle_attestations if a.shard == obj.shard and a.shard_block_hash == shard_block_hash]` - * `attesting_validators(obj)` be equal to `attesting_validators(obj, shard_block_hash)` for the value of `shard_block_hash` such that `sum([balance_at_stake(v) for v in attesting_validators(obj, shard_block_hash)])` is maximized (ties broken by favoring lower `shard_block_hash` values) - * `total_attesting_balance(obj)` be the maximal sum balance - * `winning_hash(obj)` be the winning `shard_block_hash` value - * `total_balance(obj) = sum([balance_at_stake(v) for v in obj.committee])` +* Let `prev_cycle_boundary_attestations = [a for a in this_cycle_attestations + prev_cycle_attestations if a.cycle_boundary_hash == get_block_hash(state, block, s - CYCLE_LENGTH) and a.justified_slot == state.prev_cycle_justification_source]`. +* Let `prev_cycle_boundary_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a.data, a.attester_bitfield) for a in prev_cycle_boundary_attestations]`. +* Let `prev_cycle_boundary_attesting_balance = sum([balance_at_stake(v) for v in prev_cycle_boundary_attesters])`. + +For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots`, let: + + +* `attesting_validators(obj, shard_block_hash)` be the union of the validator index sets given by `[get_attestation_participants(state, a.data, a.attester_bitfield) for a in this_cycle_attestations + prev_cycle_attestations if a.shard == obj.shard and a.shard_block_hash == shard_block_hash]` +* `attesting_validators(obj)` be equal to `attesting_validators(obj, shard_block_hash)` for the value of `shard_block_hash` such that `sum([balance_at_stake(v) for v in attesting_validators(obj, shard_block_hash)])` is maximized (ties broken by favoring lower `shard_block_hash` values) +* `total_attesting_balance(obj)` be the sum of the balances-at-stake of `attesting_validators(obj)` +* `winning_hash(obj)` be the winning `shard_block_hash` value +* `total_balance(obj) = sum([balance_at_stake(v) for v in obj.committee])` -Let `inclusion_slot(a)` equal the slot in which attestation `a` was included, and `inclusion_distance(a) = inclusion_slot(a) - a.slot`. Let `inclusion_slot(v)` equal `inclusion_distance(a)` for the attestation `a` where `v` is in `get_attestation_participants(state, a)`, and define `inclusion_distance(v)` similarly. We define a function `adjust_for_inclusion_distance(magnitude, dist)` which adjusts the reward of an attestation based on how long it took to get included (the longer, the lower the reward). Returns a value between 0 and `magnitude` +Let `inclusion_slot(v)` equal `a.slot_included` for the attestation `a` where `v` is in `get_attestation_participants(state, a.data, a.attester_bitfield)`, and `inclusion_distance(v) = a.slot_included - a.data.slot` for the same attestation. We define a function `adjust_for_inclusion_distance(magnitude, dist)` which adjusts the reward of an attestation based on how long it took to get included (the longer, the lower the reward). Returns a value between 0 and `magnitude` ```python def adjust_for_inclusion_distance(magnitude: int, dist: int) -> int: return magnitude // 2 + (magnitude // 2) * MIN_ATTESTATION_INCLUSION_DELAY // dist ``` +For any validator `v`, `base_reward(v) = balance_at_stake(v) // BASE_REWARD_QUOTIENT` + #### Adjust justified slots and crosslink status * Set `state.justified_slot_bitfield = (state.justified_slot_bitfield * 2) % 2**64`. -* If `3 * total_balance_attesting_at_prev_s >= 2 * total_balance` then set `state.justified_slot_bitfield &= 2` (ie. flip the second lowest bit to 1) and `new_justification_source = prev_s`. -* If `3 * total_balance_attesting_at_s >= 2 * total_balance` then set `state.justified_slot_bitfield &= 1` (ie. flip the lowest bit to 1) and `new_justification_source = s`. -* If `justification_source == prev_s and state.justified_slot_bitfield % 4 == 3`, set `last_finalized_slot = justification_source`. -* If `justification_source == prev_s - 2 * CYCLE_LENGTH and state.justified_slot_bitfield % 16 in (15, 14)`, set `last_finalized_slot = justification_source`. -* If `justification_source == prev_s - CYCLE_LENGTH and state.justified_slot_bitfield % 8 == 7`, set `last_finalized_slot = justification_source`. -* Set `prev_cycle_justification_source = justification_source` and `justification_source = new_justification_source`. +* If `3 * prev_cycle_boundary_attesting_balance >= 2 * total_balance` then set `state.justified_slot_bitfield &= 2` (ie. flip the second lowest bit to 1) and `new_justification_source = s - CYCLE_LENGTH`. +* If `3 * this_cycle_boundary_attesting_balance >= 2 * total_balance` then set `state.justified_slot_bitfield &= 1` (ie. flip the lowest bit to 1) and `new_justification_source = s`. +* If `state.justification_source == s - CYCLE_LENGTH and state.justified_slot_bitfield % 4 == 3`, set `last_finalized_slot = justification_source`. +* If `state.justification_source == s - CYCLE_LENGTH - 2 * CYCLE_LENGTH and state.justified_slot_bitfield % 16 in (15, 14)`, set `last_finalized_slot = justification_source`. +* If `state.justification_source == s - CYCLE_LENGTH - CYCLE_LENGTH and state.justified_slot_bitfield % 8 == 7`, set `state.last_finalized_slot = state.justification_source`. +* Set `state.prev_cycle_justification_source = state.justification_source` and if `new_justification_source` has been set, set `state.justification_source = new_justification_source`. For every `ShardAndCommittee` object `obj`: @@ -1046,26 +1058,26 @@ Note: When applying penalties in the following balance recalculations implemente * Let `total_balance_in_eth = total_balance // GWEI_PER_ETH`. * Let `reward_quotient = BASE_REWARD_QUOTIENT * int_sqrt(total_balance_in_eth)`. (The per-slot maximum interest rate is `2/reward_quotient`.) * Let `quadratic_penalty_quotient = SQRT_E_DROP_TIME**2`. (The portion lost by offline validators after `D` cycles is about `D*D/2/quadratic_penalty_quotient`.) -* Let `time_since_finality = slot - last_finalized_slot`. +* Let `time_since_finality = slot - state.last_finalized_slot`. Case 1: `time_since_finality <= 4 * CYCLE_LENGTH`: -* Any validator `v` in `prev_s_attesters` gains `adjust_for_inclusion_distance(balance_at_stake(v) // reward_quotient * (prev_s_attesters - total_balance) // total_balance, inclusion_distance(v))`. -* Any active validator `v` not in `prev_s_attesters` loses `balance_at_stake(v) // reward_quotient`. +* Any validator `v` in `prev_cycle_boundary_attesters` gains `adjust_for_inclusion_distance(base_reward(v) * prev_cycle_boundary_attesting_balance // total_balance, inclusion_distance(v))`. +* Any active validator `v` not in `prev_cycle_boundary_attesters` loses `base_reward(v)`. Case 2: `time_since_finality > 4 * CYCLE_LENGTH`: -* Any validator in `prev_s_attesters` sees their balance unchanged. -* Any active validator `v` not in `prev_s_attesters`, and any validator with `status == PENALIZED`, loses `balance_at_stake(v) // reward_quotient + balance_at_stake(v) * time_since_finality // quadratic_penalty_quotient`. +* Any validator in `prev_cycle_boundary_attesters` sees their balance unchanged. +* Any active validator `v` not in `prev_cycle_boundary_attesters`, and any validator with `status == PENALIZED`, loses `base_reward(v) + balance_at_stake(v) * time_since_finality // quadratic_penalty_quotient`. -For each `v` in `prev_s_attesters`, the validator `proposer = get_beacon_proposer(state, inclusion_slot(v))` gains `balance_at_stake(proposer) // INCLUDER_REWARD_QUOTIENT`. +For each `v` in `prev_cycle_boundary_attesters`, the validator `proposer = get_beacon_proposer(state, inclusion_slot(v))` gains `base_reward(v) // INCLUDER_REWARD_SHARE_QUOTIENT`. #### Balance recalculations related to crosslink rewards -For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots[:CYCLE_LENGTH]` (ie. the objects corresponding to the slot before the current one), for each `v in obj.committee`, where `v = state.validators[b]` adjust balances as follows: +For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots[:CYCLE_LENGTH]` (ie. the objects corresponding to the cycle before the current one), for each `v` in `[validators[index] for index in obj.committee]`, adjust balances as follows: -* If `v in attesting_validators(obj)`, `v.balance += adjust_for_inclusion_distance(balance_at_stake(v) // reward_quotient * total_attesting_balance(obj) // total_balance(obj)), inclusion_distance(v))`. -* If `v not in attesting_validators(obj)`, `v.balance -= balance_at_stake(v) // reward_quotient`. +* If `v in attesting_validators(obj)`, `v.balance += adjust_for_inclusion_distance(base_reward(v) * total_attesting_balance(obj) // total_balance(obj)), inclusion_distance(v))`. +* If `v not in attesting_validators(obj)`, `v.balance -= base_reward(v)`. #### PoW chain related rules @@ -1174,7 +1186,7 @@ def change_validators(validators: List[ValidatorRecord], current_slot: int) -> N And perform the following updates to the `state`: -* Set `state.validator_set_change_slot = s` +* Set `state.validator_set_change_slot = s + CYCLE_LENGTH` * Set `state.shard_and_committee_for_slots[:CYCLE_LENGTH] = state.shard_and_committee_for_slots[CYCLE_LENGTH:]` * Let `state.next_start_shard = (shard_and_committee_for_slots[-1][-1].shard + 1) % SHARD_COUNT` * Set `state.shard_and_committee_for_slots[CYCLE_LENGTH:] = get_new_shuffling(state.next_shuffling_seed, validators, next_start_shard)` From 9f1869cad91e8a77f2f39c172d190907abdde8bc Mon Sep 17 00:00:00 2001 From: vbuterin Date: Mon, 26 Nov 2018 17:14:38 -0500 Subject: [PATCH 17/36] Remove todos --- specs/core/0_beacon-chain.md | 38 ------------------------------------ 1 file changed, 38 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index d18ccbcfc..db919c668 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1206,44 +1206,6 @@ And perform the following updates to the `state`: * Set `state.recent_block_hashes = state.recent_block_hashes[CYCLE_LENGTH:]` * Set `state.last_state_recalculation_slot += CYCLE_LENGTH` -### TODO - -Note: This spec is ~65% complete. - -**Missing** - -* [ ] Specify the rules around acceptable values for `pow_chain_reference` ([issue 58](https://github.com/ethereum/eth2.0-specs/issues/58)) -* [ ] Specify the shard chain blocks, blobs, proposers, etc. -* [ ] Specify the deposit contract on the PoW chain in Vyper -* [ ] Specify the beacon chain genesis rules ([issue 58](https://github.com/ethereum/eth2.0-specs/issues/58)) -* [ ] Specify the logic for proofs of custody, including slashing conditions -* [ ] Specify BLSVerify and rework the spec for BLS12-381 throughout -* [ ] Specify the constraints for `SpecialRecord`s ([issue 43](https://github.com/ethereum/eth2.0-specs/issues/43)) -* [ ] Specify the calculation and validation of `BeaconBlock.state_root` -* [ ] Undergo peer review, security audits and formal verification - -**Documentation** - -* [ ] Specify the various assumptions (global clock, networking latency, validator honesty, validator liveness, etc.) -* [ ] Add an appendix on gossip networks and the offchain signature aggregation logic -* [ ] Add a glossary (in a separate `glossary.md`) to comprehensively and precisely define all the terms -* [ ] Clearly document the various edge cases, e.g. with committee sizing -* [ ] Rework the document for readability - -**Possible modifications and additions** - -* [ ] Replace the IMD fork choice rule with LMD -* [ ] Homogenise types to `uint64` ([PR 36](https://github.com/ethereum/eth2.0-specs/pull/36)) -* [ ] Reduce the slot duration to 8 seconds -* [ ] Allow for the delayed inclusion of aggregated signatures -* [ ] Introduce a RANDAO slashing condition for early reveals -* [ ] Use a separate hash function for the proof of possession -* [ ] Rework the `ShardAndCommittee` data structures -* [ ] Add a double-batched Merkle accumulator for historical beacon chain blocks -* [ ] Allow for deposits larger than 32 ETH, as well as deposit top-ups -* [ ] Add penalties for deposits below 32 ETH (or some other threshold) -* [ ] Add a `SpecialRecord` to (re)register - # Appendix ## Appendix A - Hash function From 6015242a567a4fc04124b1806d5c05311492eaa3 Mon Sep 17 00:00:00 2001 From: ncsolar Date: Mon, 26 Nov 2018 17:57:01 -0500 Subject: [PATCH 18/36] Update README.md (#176) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2c4b76f38..fd8748d5e 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ Core specifications for eth2.0 client validation can be found in [specs/core](sp ## Design goals The following are the broad design goals for Ethereum 2.0: * to minimize complexity, even at the cost of some losses in efficiency -* to remain live through major network partitions and when very large portions of nodes going offline +* to remain live through major network partitions and when very large portions of nodes go offline * to select all components such that they are either quantum secure or can be easily swapped out for quantum secure counterparts when available * to utilize crypto and design techniques that allow for a large participation of validators in total and per unit time * to allow for a typical consumer laptop with `O(C)` resources to process/validate `O(1)` shards (including any system level validation such as the beacon chain) From be56e58c1a995a208c622a847325db8521c6536d Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 26 Nov 2018 18:32:53 -0500 Subject: [PATCH 19/36] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index db919c668..cd80499a0 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -139,7 +139,7 @@ An `AttestationRecord` has the following fields: ```python { data: AttestationSignedData - # Attester participation bitfield (2 bits per attester) + # Attester participation bitfield 'attester_bitfield': 'bytes', # Proof of custody bitfield 'poc_bitfield': 'bytes', From fe4496dd7456abca2f8acdd299505babcab11ffd Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 26 Nov 2018 18:33:06 -0500 Subject: [PATCH 20/36] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index cd80499a0..4c785ab16 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -865,7 +865,7 @@ Verify that there are at most `MAX_ATTESTATION_COUNT` `AttestationRecord` object For each `AttestationRecord` object `obj`: * Verify that `obj.data.slot <= block.slot - MIN_ATTESTATION_INCLUSION_DELAY` and `obj.data.slot >= max(parent.slot - CYCLE_LENGTH + 1, 0)`. -* Verify that `obj.data.justified_slot` is equal to `justification_source if state.last_state_recalculation_slot else prev_cycle_justification_source` +* Verify that `obj.data.justified_slot` is equal to `justification_source if obj.data.slot >= state.last_state_recalculation_slot else prev_cycle_justification_source` * Verify that `obj.data.justified_block_hash` is the hash of the block in the current chain at the slot -- `obj.data.justified_slot`. * Verify that either `obj.data.last_crosslink_hash` or `obj.data.shard_block_hash` equals `state.crosslinks[shard].shard_block_hash`. * `aggregate_sig` verification: From bcd9eb53fbcf4eb42edbdf40c93ba7045ac7cd39 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 26 Nov 2018 18:33:18 -0500 Subject: [PATCH 21/36] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 4c785ab16..0ea8bc829 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1058,7 +1058,7 @@ Note: When applying penalties in the following balance recalculations implemente * Let `total_balance_in_eth = total_balance // GWEI_PER_ETH`. * Let `reward_quotient = BASE_REWARD_QUOTIENT * int_sqrt(total_balance_in_eth)`. (The per-slot maximum interest rate is `2/reward_quotient`.) * Let `quadratic_penalty_quotient = SQRT_E_DROP_TIME**2`. (The portion lost by offline validators after `D` cycles is about `D*D/2/quadratic_penalty_quotient`.) -* Let `time_since_finality = slot - state.last_finalized_slot`. +* Let `time_since_finality = block.slot - state.last_finalized_slot`. Case 1: `time_since_finality <= 4 * CYCLE_LENGTH`: From 973d0ab5913346e7e0182d2657721a33cd30d810 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 26 Nov 2018 18:33:31 -0500 Subject: [PATCH 22/36] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 0ea8bc829..6064c3a36 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1074,7 +1074,7 @@ For each `v` in `prev_cycle_boundary_attesters`, the validator `proposer = get_b #### Balance recalculations related to crosslink rewards -For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots[:CYCLE_LENGTH]` (ie. the objects corresponding to the cycle before the current one), for each `v` in `[validators[index] for index in obj.committee]`, adjust balances as follows: +For every `ShardAndCommittee` object `obj` in `shard_and_committee_for_slots[:CYCLE_LENGTH]` (ie. the objects corresponding to the cycle before the current one), for each `v` in `[state.validators[index] for index in obj.committee]`, adjust balances as follows: * If `v in attesting_validators(obj)`, `v.balance += adjust_for_inclusion_distance(base_reward(v) * total_attesting_balance(obj) // total_balance(obj)), inclusion_distance(v))`. * If `v not in attesting_validators(obj)`, `v.balance -= base_reward(v)`. From e651e992108859b3558a94debeab10f2a17e3d0f Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 26 Nov 2018 18:35:02 -0500 Subject: [PATCH 23/36] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 6064c3a36..14874b562 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1008,7 +1008,7 @@ All validators: Validators justifying the cycle boundary block at the start of the current cycle: * Let `this_cycle_attestations = [a for a in state.pending_attestations if s <= a.data.slot < s + CYCLE_LENGTH]`. (note: this is the set of attestations _of slots in the cycle `s...s+CYCLE_LENGTH-1`_, not attestations _that got included in the chain during the cycle `s...s+CYCLE_LENGTH-1`_) -* Let `this_cycle_boundary_attestations = [a for a in this_cycle_attestations if a.data.cycle_boundary_hash == get_block_hash(state, block, s) a.justified_slot == state.justification_source]`. +* Let `this_cycle_boundary_attestations = [a for a in this_cycle_attestations if a.data.cycle_boundary_hash == get_block_hash(state, block, s) and a.justified_slot == state.justification_source]`. * Let `this_cycle_boundary_attesters` be the union of the validator index sets given by `[get_attestation_participants(state, a.data, a.attester_bitfield) for a in this_cycle_boundary_attestations]`. * Let `this_cycle_boundary_attesting_balance = sum([balance_at_stake(v) for v in this_cycle_boundary_attesters])`. From 7fcd9930df3f433407bf174f9ae61c615824149c Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 26 Nov 2018 18:35:25 -0500 Subject: [PATCH 24/36] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 14874b562..ad821605b 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -866,7 +866,7 @@ For each `AttestationRecord` object `obj`: * Verify that `obj.data.slot <= block.slot - MIN_ATTESTATION_INCLUSION_DELAY` and `obj.data.slot >= max(parent.slot - CYCLE_LENGTH + 1, 0)`. * Verify that `obj.data.justified_slot` is equal to `justification_source if obj.data.slot >= state.last_state_recalculation_slot else prev_cycle_justification_source` -* Verify that `obj.data.justified_block_hash` is the hash of the block in the current chain at the slot -- `obj.data.justified_slot`. +* Verify that `obj.data.justified_block_hash` is equal to `get_block_hash(state, block, obj.data.justified_slot)`. * Verify that either `obj.data.last_crosslink_hash` or `obj.data.shard_block_hash` equals `state.crosslinks[shard].shard_block_hash`. * `aggregate_sig` verification: * Let `participants = get_attestation_participants(state, obj.data, obj.attester_bitfield)` From b8689e17274ed4571f82de98d8f73b7d3def04b5 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Mon, 26 Nov 2018 18:35:32 -0500 Subject: [PATCH 25/36] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index ad821605b..50d69cf36 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -567,7 +567,7 @@ def get_attestation_participants(state: State, snc = [x for x in sncs_for_slot if x.shard == attestation_data.shard][0] assert len(attester_bitfield) == ceil_div8(len(snc.committee)) participants = [] - for i, vindex in snc.committee: + for i, vindex in enumerate(snc.committee): bit = (attester_bitfield[i//8] >> (7 - (i % 8))) % 2 if bit == 1: participants.append(vindex) From 3b7164c2d6fbfdd2ce6555e5b83aa25580a38f88 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Mon, 26 Nov 2018 18:39:35 -0500 Subject: [PATCH 26/36] Resolved some of @djrtwo's nitpicks --- specs/core/0_beacon-chain.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 50d69cf36..db6dea88a 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1003,7 +1003,8 @@ _Note: `last_state_recalculation_slot` will always be a multiple of `CYCLE_LENGT All validators: * Let `active_validators = [state.validators[i] for i in get_active_validator_indices(state.validators)]`. -* Let `total_balance = sum([balance_at_stake(v) for v in active_validators])`. +* Let `total_balance = sum([balance_at_stake(v) for v in active_validators])`. Let `total_balance_in_eth = total_balance // GWEI_PER_ETH`. +* Let `reward_quotient = BASE_REWARD_QUOTIENT * int_sqrt(total_balance_in_eth)`. (The per-slot maximum interest rate is `2/reward_quotient`.) Validators justifying the cycle boundary block at the start of the current cycle: @@ -1035,7 +1036,12 @@ def adjust_for_inclusion_distance(magnitude: int, dist: int) -> int: return magnitude // 2 + (magnitude // 2) * MIN_ATTESTATION_INCLUSION_DELAY // dist ``` -For any validator `v`, `base_reward(v) = balance_at_stake(v) // BASE_REWARD_QUOTIENT` +For any validator `v`, `base_reward(v) = balance_at_stake(v) // reward_quotient` + +Miscellaneous: + +* Let `quadratic_penalty_quotient = SQRT_E_DROP_TIME**2`. (The portion lost by offline validators after `D` cycles is about `D*D/2/quadratic_penalty_quotient`.) +* Let `time_since_finality = block.slot - state.last_finalized_slot`. #### Adjust justified slots and crosslink status @@ -1043,8 +1049,8 @@ For any validator `v`, `base_reward(v) = balance_at_stake(v) // BASE_REWARD_QUOT * If `3 * prev_cycle_boundary_attesting_balance >= 2 * total_balance` then set `state.justified_slot_bitfield &= 2` (ie. flip the second lowest bit to 1) and `new_justification_source = s - CYCLE_LENGTH`. * If `3 * this_cycle_boundary_attesting_balance >= 2 * total_balance` then set `state.justified_slot_bitfield &= 1` (ie. flip the lowest bit to 1) and `new_justification_source = s`. * If `state.justification_source == s - CYCLE_LENGTH and state.justified_slot_bitfield % 4 == 3`, set `last_finalized_slot = justification_source`. -* If `state.justification_source == s - CYCLE_LENGTH - 2 * CYCLE_LENGTH and state.justified_slot_bitfield % 16 in (15, 14)`, set `last_finalized_slot = justification_source`. * If `state.justification_source == s - CYCLE_LENGTH - CYCLE_LENGTH and state.justified_slot_bitfield % 8 == 7`, set `state.last_finalized_slot = state.justification_source`. +* If `state.justification_source == s - CYCLE_LENGTH - 2 * CYCLE_LENGTH and state.justified_slot_bitfield % 16 in (15, 14)`, set `last_finalized_slot = justification_source`. * Set `state.prev_cycle_justification_source = state.justification_source` and if `new_justification_source` has been set, set `state.justification_source = new_justification_source`. For every `ShardAndCommittee` object `obj`: @@ -1055,11 +1061,6 @@ For every `ShardAndCommittee` object `obj`: Note: When applying penalties in the following balance recalculations implementers should make sure the `uint64` does not underflow. -* Let `total_balance_in_eth = total_balance // GWEI_PER_ETH`. -* Let `reward_quotient = BASE_REWARD_QUOTIENT * int_sqrt(total_balance_in_eth)`. (The per-slot maximum interest rate is `2/reward_quotient`.) -* Let `quadratic_penalty_quotient = SQRT_E_DROP_TIME**2`. (The portion lost by offline validators after `D` cycles is about `D*D/2/quadratic_penalty_quotient`.) -* Let `time_since_finality = block.slot - state.last_finalized_slot`. - Case 1: `time_since_finality <= 4 * CYCLE_LENGTH`: * Any validator `v` in `prev_cycle_boundary_attesters` gains `adjust_for_inclusion_distance(base_reward(v) * prev_cycle_boundary_attesting_balance // total_balance, inclusion_distance(v))`. From f2af5fd9df62f2eaac82b14ebd57118c323d0577 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Mon, 26 Nov 2018 18:44:29 -0500 Subject: [PATCH 27/36] Moved two calculations back to the FFG reward section --- specs/core/0_beacon-chain.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index db6dea88a..84b7773ce 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1038,11 +1038,6 @@ def adjust_for_inclusion_distance(magnitude: int, dist: int) -> int: For any validator `v`, `base_reward(v) = balance_at_stake(v) // reward_quotient` -Miscellaneous: - -* Let `quadratic_penalty_quotient = SQRT_E_DROP_TIME**2`. (The portion lost by offline validators after `D` cycles is about `D*D/2/quadratic_penalty_quotient`.) -* Let `time_since_finality = block.slot - state.last_finalized_slot`. - #### Adjust justified slots and crosslink status * Set `state.justified_slot_bitfield = (state.justified_slot_bitfield * 2) % 2**64`. @@ -1061,6 +1056,9 @@ For every `ShardAndCommittee` object `obj`: Note: When applying penalties in the following balance recalculations implementers should make sure the `uint64` does not underflow. +* Let `quadratic_penalty_quotient = SQRT_E_DROP_TIME**2`. (The portion lost by offline validators after `D` cycles is about `D*D/2/quadratic_penalty_quotient`.) +* Let `time_since_finality = block.slot - state.last_finalized_slot`. + Case 1: `time_since_finality <= 4 * CYCLE_LENGTH`: * Any validator `v` in `prev_cycle_boundary_attesters` gains `adjust_for_inclusion_distance(base_reward(v) * prev_cycle_boundary_attesting_balance // total_balance, inclusion_distance(v))`. From cd9c47af5aea8a9d23b79aaa492d6f11cc4b4e29 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Mon, 26 Nov 2018 18:51:34 -0500 Subject: [PATCH 28/36] Made pure functions return indices of validators instead of validator objects Makes the code cleaner this way; returning an object that then gets mutated is confusing. --- specs/core/0_beacon-chain.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 84b7773ce..261593b47 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -403,7 +403,7 @@ def get_active_validator_indices(validators) return [i for i, v in enumerate(validators) if v.status == ACTIVE] ``` -The following is a function that shuffles the validator list: +The following is a function that shuffles any list; we primarily use it for the validator list: ```python def shuffle(values: List[Any], @@ -551,10 +551,10 @@ def get_block_hash(state: BeaconState, The following is a function that determines the proposer of a beacon block: ```python -def get_beacon_proposer(state:BeaconState, slot: int) -> ValidatorRecord: +def get_beacon_proposer_index(state:BeaconState, slot: int) -> ValidatorRecord: first_committee = get_shards_and_committees_for_slot(state, slot)[0].committee index = first_committee[slot % len(first_committee)] - return state.validators[index] + return index ``` The following is a function that determines the validators that participated in an attestation: @@ -747,7 +747,7 @@ This routine should be run for every validator that is inducted as part of a log First, some helper functions: ```python -def min_empty_validator(validators: List[ValidatorRecord], current_slot: int): +def min_empty_validator_index(validators: List[ValidatorRecord], current_slot: int): for i, v in enumerate(validators): if v.status == WITHDRAWN and v.last_status_change_slot + DELETION_PERIOD <= current_slot: return i @@ -792,7 +792,7 @@ def add_validator(state: State, exit_seq=0 ) # Add the validator - index = min_empty_validator(state.validators) + index = min_empty_validator_index(state.validators) if index is None: state.validators.append(rec) index = len(state.validators) - 1 @@ -821,7 +821,7 @@ def exit_validator(index, state, block, penalize, current_slot): validator.status = PENALIZED whistleblower_xfer_amount = validator.deposit // SLASHING_WHISTLEBLOWER_REWARD_DENOMINATOR validator.deposit -= whistleblower_xfer_amount - get_beacon_proposer(state, block.slot).deposit += whistleblower_xfer_amount + state.validators[get_beacon_proposer_index(state, block.slot)].deposit += whistleblower_xfer_amount else: validator.status = PENDING_EXIT add_validator_set_change_record(state, index, validator.pubkey, EXIT) @@ -879,7 +879,7 @@ For each `AttestationRecord` object `obj`: Let `proposal_hash = hash(ProposalSignedData(block.slot, 2**64 - 1, block_hash_without_sig))` where `block_hash_without_sig` is the hash of the block except setting `proposer_signature` to `[0, 0]`. -Verify that `BLSVerify(pubkey=get_beacon_proposer(state, block.slot).pubkey, data=proposal_hash, sig=block.proposer_signature, domain=get_domain(state, block.slot, DOMAIN_PROPOSAL))` passes. +Verify that `BLSVerify(pubkey=state.validators[get_beacon_proposer_index(state, block.slot)].pubkey, data=proposal_hash, sig=block.proposer_signature, domain=get_domain(state, block.slot, DOMAIN_PROPOSAL))` passes. ### Verify and process RANDAO reveal @@ -887,16 +887,16 @@ First run the following state transition to update `randao_skips` variables for ```python for slot in range(parent.slot + 1, block.slot): - proposer = get_beacon_proposer(state, slot) - proposer.randao_skips += 1 + proposer_index = get_beacon_proposer_index(state, slot) + state.validators[proposer_index].randao_skips += 1 ``` Then: * Let `repeat_hash(x, n) = x if n == 0 else repeat_hash(hash(x), n-1)`. -* Let `proposer = get_beacon_proposer(state, block.slot)`. -* Verify that `repeat_hash(block.randao_reveal, proposer.randao_skips + 1) == V.randao_commitment` -* Set `state.randao_mix = xor(state.randao_mix, block.randao_reveal)`, `proposer.randao_commitment = block.randao_reveal`, `V.randao_skips = 0` +* Let `proposer = state.validators[get_beacon_proposer_index(state, block.slot)]`. +* Verify that `repeat_hash(block.randao_reveal, proposer.randao_skips + 1) == proposer.randao_commitment` +* Set `state.randao_mix = xor(state.randao_mix, block.randao_reveal)`, `proposer.randao_commitment = block.randao_reveal`, `proposer.randao_skips = 0` ### Process PoW receipt root @@ -1069,7 +1069,7 @@ Case 2: `time_since_finality > 4 * CYCLE_LENGTH`: * Any validator in `prev_cycle_boundary_attesters` sees their balance unchanged. * Any active validator `v` not in `prev_cycle_boundary_attesters`, and any validator with `status == PENALIZED`, loses `base_reward(v) + balance_at_stake(v) * time_since_finality // quadratic_penalty_quotient`. -For each `v` in `prev_cycle_boundary_attesters`, the validator `proposer = get_beacon_proposer(state, inclusion_slot(v))` gains `base_reward(v) // INCLUDER_REWARD_SHARE_QUOTIENT`. +For each `v` in `prev_cycle_boundary_attesters`, we determine the proposer `proposer_index = get_beacon_proposer_index(state, inclusion_slot(v))` and set `state.validators[proposer_index].balance += base_reward(v) // INCLUDER_REWARD_SHARE_QUOTIENT`. #### Balance recalculations related to crosslink rewards From 472a117162c3dd756375aa50a8d0be199dfd7944 Mon Sep 17 00:00:00 2001 From: ncsolar Date: Tue, 27 Nov 2018 00:12:10 -0600 Subject: [PATCH 29/36] Update 0_beacon-chain.md Assumed it should read "A block `B` is justified if there is a descendant of `B`..." ? --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 3ee0a1646..564e100c0 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -353,7 +353,7 @@ The beacon chain fork choice rule is a hybrid that combines justification and fi * Let `store` be the set of attestations and blocks that the validator `v` has observed and verified (in particular, block ancestors must be recursively verified). Attestations not part of any chain are still included in `store`. * Let `finalized_head` be the finalized block with the highest slot number. (A block `B` is finalized if there is a descendant of `B` in `store` the processing of which sets `B` as finalized.) -* Let `justified_head` be the descendant of `finalized_head` with the highest slot number that has been justified for at least `CYCLE_LENGTH` slots. (A block `B` is justified is there is a descendant of `B` in `store` the processing of which sets `B` as justified.) If no such descendant exists set `justified_head` to `finalized_head`. +* Let `justified_head` be the descendant of `finalized_head` with the highest slot number that has been justified for at least `CYCLE_LENGTH` slots. (A block `B` is justified if there is a descendant of `B` in `store` the processing of which sets `B` as justified.) If no such descendant exists set `justified_head` to `finalized_head`. * Let `get_ancestor(store, block, slot)` be the ancestor of `block` with slot number `slot`. The `get_ancestor` function can be defined recursively as `def get_ancestor(store, block, slot): return block if block.slot == slot else get_ancestor(store, store.get_parent(block), slot)`. * Let `get_latest_attestation(store, validator)` be the attestation with the highest slot number in `store` from `validator`. If several such attestations exist use the one the validator `v` observed first. * Let `get_latest_attestation_target(store, validator)` be the target block in the attestation `get_latest_attestation(store, validator)`. From fc059f2ec12dfbd2f0b35f3ec02c4e920013a2b9 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Tue, 27 Nov 2018 06:30:19 -0500 Subject: [PATCH 30/36] Moved proposer reshuffling section --- specs/core/0_beacon-chain.md | 58 ++++++++++++++++++------------------ 1 file changed, 29 insertions(+), 29 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 261593b47..8991961c7 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -1085,35 +1085,6 @@ If `last_state_recalculation_slot % POW_RECEIPT_ROOT_VOTING_PERIOD == 0`, then: * If for any `x` in `state.candidate_pow_receipt_root`, `x.votes * 2 >= POW_RECEIPT_ROOT_VOTING_PERIOD` set `state.processed_pow_receipt_root = x.receipt_root`. * Set `state.candidate_pow_receipt_roots = []`. -#### Proposer reshuffling - -Run the following code to update the shard proposer set: - -```python -active_validator_indices = get_active_validator_indices(validators) -num_validators_to_reshuffle = len(active_validator_indices) // SHARD_PERSISTENT_COMMITTEE_CHANGE_PERIOD -for i in range(num_validators_to_reshuffle): - # Multiplying i to 2 to ensure we have different input to all the required hashes in the shuffling - # and none of the hashes used for entropy in this loop will be the same - vid = active_validator_indices[hash(state.randao_mix + bytes8(i * 2)) % len(active_validator_indices)] - new_shard = hash(state.randao_mix + bytes8(i * 2 + 1)) % SHARD_COUNT - shard_reassignment_record = ShardReassignmentRecord( - validator_index=vid, - shard=new_shard, - slot=s + SHARD_PERSISTENT_COMMITTEE_CHANGE_PERIOD - ) - state.persistent_committee_reassignments.append(shard_reassignment_record) - -while len(state.persistent_committee_reassignments) > 0 and state.persistent_committee_reassignments[0].slot <= s: - rec = state.persistent_committee_reassignments.pop(0) - for committee in state.persistent_committees: - if rec.validator_index in committee: - committee.pop( - committee.index(rec.validator_index) - ) - state.persistent_committees[rec.shard].append(rec.validator_index) -``` - #### Validator set change A validator set change can happen if all of the following criteria are satisfied: @@ -1198,6 +1169,35 @@ And perform the following updates to the `state`: * Let `start_shard = state.shard_and_committee_for_slots[0][0].shard` * If `time_since_finality * CYCLE_LENGTH <= MIN_VALIDATOR_SET_CHANGE_INTERVAL` or `time_since_finality` is an exact power of 2, set `state.shard_and_committee_for_slots[CYCLE_LENGTH:] = get_new_shuffling(state.next_shuffling_seed, validators, start_shard)` and set `state.next_shuffling_seed = state.randao_mix`. Note that `start_shard` is not changed from last cycle. +#### Proposer reshuffling + +Run the following code to update the shard proposer set: + +```python +active_validator_indices = get_active_validator_indices(validators) +num_validators_to_reshuffle = len(active_validator_indices) // SHARD_PERSISTENT_COMMITTEE_CHANGE_PERIOD +for i in range(num_validators_to_reshuffle): + # Multiplying i to 2 to ensure we have different input to all the required hashes in the shuffling + # and none of the hashes used for entropy in this loop will be the same + vid = active_validator_indices[hash(state.randao_mix + bytes8(i * 2)) % len(active_validator_indices)] + new_shard = hash(state.randao_mix + bytes8(i * 2 + 1)) % SHARD_COUNT + shard_reassignment_record = ShardReassignmentRecord( + validator_index=vid, + shard=new_shard, + slot=s + SHARD_PERSISTENT_COMMITTEE_CHANGE_PERIOD + ) + state.persistent_committee_reassignments.append(shard_reassignment_record) + +while len(state.persistent_committee_reassignments) > 0 and state.persistent_committee_reassignments[0].slot <= s: + rec = state.persistent_committee_reassignments.pop(0) + for committee in state.persistent_committees: + if rec.validator_index in committee: + committee.pop( + committee.index(rec.validator_index) + ) + state.persistent_committees[rec.shard].append(rec.validator_index) +``` + #### Finally... * Remove all attestation records older than slot `s` From 2d766e4aaeeae239d985f8c74d63346d43bef9a4 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 27 Nov 2018 06:35:55 -0500 Subject: [PATCH 31/36] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 8991961c7..08f89de2a 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -138,7 +138,7 @@ An `AttestationRecord` has the following fields: ```python { - data: AttestationSignedData + data: AttestationSignedData, # Attester participation bitfield 'attester_bitfield': 'bytes', # Proof of custody bitfield From 034e2d45469e76201bcb4046a515e81a82c6897c Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 27 Nov 2018 06:36:10 -0500 Subject: [PATCH 32/36] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 08f89de2a..af6a33c6c 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -551,7 +551,7 @@ def get_block_hash(state: BeaconState, The following is a function that determines the proposer of a beacon block: ```python -def get_beacon_proposer_index(state:BeaconState, slot: int) -> ValidatorRecord: +def get_beacon_proposer_index(state:BeaconState, slot: int) -> int: first_committee = get_shards_and_committees_for_slot(state, slot)[0].committee index = first_committee[slot % len(first_committee)] return index From 2795af7bae0744fb017a39ead03c0b7b49d047b1 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 27 Nov 2018 06:36:47 -0500 Subject: [PATCH 33/36] Update specs/core/0_beacon-chain.md Co-Authored-By: vbuterin --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index af6a33c6c..78c2e7df3 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -747,7 +747,7 @@ This routine should be run for every validator that is inducted as part of a log First, some helper functions: ```python -def min_empty_validator_index(validators: List[ValidatorRecord], current_slot: int): +def min_empty_validator_index(validators: List[ValidatorRecord], current_slot: int) -> int: for i, v in enumerate(validators): if v.status == WITHDRAWN and v.last_status_change_slot + DELETION_PERIOD <= current_slot: return i From 1df0f9f1b7dc7c633c5bcdff00607045fbb439b0 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Tue, 27 Nov 2018 07:25:31 -0500 Subject: [PATCH 34/36] Fixed indent. --- specs/core/0_beacon-chain.md | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 78c2e7df3..1403f404e 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -152,22 +152,22 @@ An `AttestationRecord` has the following fields: ```python { - # Slot number - 'slot': 'uint64', - # Shard number - 'shard': 'uint64', - # Hash of the block we're signing - 'block_hash': 'hash32', - # Hash of the ancestor at the cycle boundary - 'cycle_boundary_hash': 'hash32', - # Shard block hash being attested to - 'shard_block_hash': 'hash32', - # Last crosslink hash - 'last_crosslink_hash': 'hash32', - # Slot of last justified beacon block - 'justified_slot': 'uint64', - # Hash of last justified beacon block - 'justified_block_hash': 'hash32', + # Slot number + 'slot': 'uint64', + # Shard number + 'shard': 'uint64', + # Hash of the block we're signing + 'block_hash': 'hash32', + # Hash of the ancestor at the cycle boundary + 'cycle_boundary_hash': 'hash32', + # Shard block hash being attested to + 'shard_block_hash': 'hash32', + # Last crosslink hash + 'last_crosslink_hash': 'hash32', + # Slot of last justified beacon block + 'justified_slot': 'uint64', + # Hash of last justified beacon block + 'justified_block_hash': 'hash32', } ``` From cae363f20990ddbc3848c15988be6d334d6df6fc Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Tue, 27 Nov 2018 08:21:42 -0600 Subject: [PATCH 35/36] add missing quotes for field --- specs/core/0_beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/core/0_beacon-chain.md b/specs/core/0_beacon-chain.md index 1403f404e..a9e9731ce 100644 --- a/specs/core/0_beacon-chain.md +++ b/specs/core/0_beacon-chain.md @@ -138,7 +138,7 @@ An `AttestationRecord` has the following fields: ```python { - data: AttestationSignedData, + 'data': AttestationSignedData, # Attester participation bitfield 'attester_bitfield': 'bytes', # Proof of custody bitfield From 822fbed83fef77ffd1dd889c417855f145abcd16 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Tue, 27 Nov 2018 11:08:43 -0500 Subject: [PATCH 36/36] BLS-12-381 specification (#141) * Partial draft of BLS-12-381 specification * Some updates * Updated point serialization to standard * Added BLSMultiVerify * Added domain separation logic * Added point to hash algorithm * Represented field_modulus as decimal along with the cofactor * Added what e(x,y) means. * Pass domain into hash_to_G2 * Updated code to make it more easily runnable in py_ecc * Add pending review warning --- specs/bls_verify.md | 64 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 specs/bls_verify.md diff --git a/specs/bls_verify.md b/specs/bls_verify.md new file mode 100644 index 000000000..9736c0151 --- /dev/null +++ b/specs/bls_verify.md @@ -0,0 +1,64 @@ +### BLS Verification + +**Warning: This document is pending academic review and should not yet be considered secure.** + +See https://z.cash/blog/new-snark-curve/ for BLS-12-381 parameters. + +We represent coordinates as defined in https://github.com/zkcrypto/pairing/tree/master/src/bls12_381/. + +Specifically, a point in G1 as a 384-bit integer `z`, which we decompose into: + +* `x = z % 2**381` +* `highflag = z // 2**382` +* `lowflag = (z % 2**382) // 2**381` + +If `highflag == 3`, the point is the point at infinity and we require `lowflag = x = 0`. Otherwise, we require `highflag == 2`, in which case the point is `(x, y)` where `y` is the valid coordinate such that `(y * 2) // q == lowflag`. + +We represent a point in G2 as a pair of 384-bit integers `(z1, z2)` that are each decomposed into `x1`, `highflag1`, `lowflag1`, `x2`, `highflag2`, `lowflag2` as above. We require `lowflag2 == highflag2 == 0`. If `highflag1 == 3`, the point is the point at infinity and we require `lowflag1 == x1 == x2 == 0`. Otherwise, we require `highflag == 2`, in which case the point is `(x1 * i + x2, y)` where `y` is the valid coordinate such that the imaginary part of `y` satisfies `(y_im * 2) // q == lowflag1`. + +`BLSVerify(pubkey: uint384, msg: bytes32, sig: [uint384], domain: uint64)` is done as follows: + +* Verify that `pubkey` is a valid G1 point and `sig` is a valid G2 point. +* Convert `msg` to a G2 point using `hash_to_G2` defined below. +* Do the pairing check: verify `e(pubkey, hash_to_G2(msg, domain)) == e(G1, sig)` (where `e` is the BLS pairing function) + +Here is the `hash_to_G2` definition: + +```python +G2_cofactor = 305502333931268344200999753193121504214466019254188142667664032982267604182971884026507427359259977847832272839041616661285803823378372096355777062779109 +field_modulus = 4002409555221667393417789825735904156556882819939007885332058136124031650490837864442687629129015664037894272559787 + +def hash_to_G2(m, domain): + x1 = hash(bytes8(domain) + b'\x01' + m) + x2 = hash(bytes8(domain) + b'\x02' + m) + x_coord = FQ2([x1, x2]) # x1 + x2 * i + while 1: + x_cubed_plus_b2 = x_coord ** 3 + FQ2([4,4]) + y_coord = mod_sqrt(x_cubed_plus_b2) + if y_coord is not None: + break + x_coord += FQ2([1, 0]) # Add one until we get a quadratic residue + assert is_on_curve((x_coord, y_coord)) + return multiply((x_coord, y_coord), G2_cofactor) +``` + +Here is a sample implementation of `mod_sqrt`: + +```python +qmod = field_modulus ** 2 - 1 +eighth_roots_of_unity = [FQ2([1,1]) ** ((qmod * k) // 8) for k in range(8)] + +def mod_sqrt(val): + candidate_sqrt = val ** ((qmod + 8) // 16) + check = candidate_sqrt ** 2 / val + if check in eighth_roots_of_unity[::2]: + return candidate_sqrt / eighth_roots_of_unity[eighth_roots_of_unity.index(check) // 2] + return None +``` + +`BLSMultiVerify(pubkeys: [uint384], msgs: [bytes32], sig: [uint384], domain: uint64)` is done as follows: + +* Verify that each element of `pubkeys` is a valid G1 point and `sig` is a valid G2 point. +* Convert each element of `msg` to a G2 point using `hash_to_G2` defined above, using the specified `domain`. +* Check that the length of `pubkeys` and `msgs` is the same, call the length `L` +* Do the pairing check: verify `e(pubkeys[0], hash_to_G2(msgs[0], domain)) * ... * e(pubkeys[L-1], hash_to_G2(msgs[L-1], domain)) == e(G1, sig)`