status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth2/beacon_chain/spec/state_transition_epoch.nim

931 lines
41 KiB
Nim
Raw Normal View History

Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# beacon_chain
per HF1, split process_final_updates() (#2319)
2021-02-14 19:31:01 +00:00
# Copyright (c) 2018-2021 Status Research & Development GmbH
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Licensed and distributed under either of
replace links to Apache and MIT licenses with HTTPS versions (#592)
2019-11-25 15:30:02 +00:00
# * MIT license (license terms in the root directory or at https://opensource.org/licenses/MIT).
# * Apache v2 license (license terms in the root directory or at https://www.apache.org/licenses/LICENSE-2.0).
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# at your option. This file may not be copied, modified, or distributed except according to those terms.
# State transition - epoch processing, as described in
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.2/specs/phase0/beacon-chain.md#epoch-processing and
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.2/specs/altair/beacon-chain.md#epoch-processing
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
#
complete re-implmentation of reasonable asymptotic-complexity algorithms from 0.11.3
2020-06-09 09:42:53 +00:00
# The entry point is `process_epoch`, which is at the bottom of this file.
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
#
Datatype cleanup (#1953) * clear up spec todo * test fix * remove unnecessary toSszType * type * one more
2020-11-04 21:52:47 +00:00
# General notes about the code:
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# * Weird styling - the sections taken from the spec use python styling while
# the others use NEP-1 - helps grepping identifiers in spec
Datatype cleanup (#1953) * clear up spec todo * test fix * remove unnecessary toSszType * type * one more
2020-11-04 21:52:47 +00:00
# * When updating the code, add TODO sections to mark where there are clear
# improvements to be made - other than that, keep things similar to spec unless
# motivated by security or performance considerations
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
spec: regulate exceptions (#913) * spec: regulate exceptions * a few more simple raises
2020-04-22 05:53:02 +00:00
{.push raises: [Defect].}
fix network sim finalization; remove get_attestation_data_slot(...); remove 2 more get_crosslink_committee(...) calls
2019-11-12 05:35:52 +00:00
import
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
std/[math, sequtils, sets, tables, algorithm],
fix broken metrics during replay (#2090) * move metrics out of state transition * add validator count metric * remove expensive beacon_current_validators, beacon_previous_validators metrics (they should be reimplemented with cache), add cheap beacon_active_validators to approximate * remove unused validator count metrics * tidy imports/defects
2020-11-27 22:16:13 +00:00
stew/[bitops2], chronicles,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
../extras,
rework spec imports (#2779) The spec imports are a mess to work with, so this branch cleans them up a bit to ensure that we avoid generic sandwitches and that importing stuff generally becomes easier. * reexport crypto/digest/presets because these are part of the public symbol set of the rest of the spec types * don't export `merge` types from `base` - this causes circular deps * fix circular deps in `ssz/spec_types` - this is the first step in disentangling ssz from spec * be explicit about phase0 vs altair - longer term, `altair` will become the "natural" type set, then merge and so on, so no point in giving `phase0` special preferential treatment
2021-08-12 13:08:20 +00:00
./datatypes/[phase0, altair],
disentangle eth2 types from the ssz library (#2785) * reorganize ssz dependencies This PR continues the work in https://github.com/status-im/nimbus-eth2/pull/2646, https://github.com/status-im/nimbus-eth2/pull/2779 as well as past issues with serialization and type, to disentangle SSZ from eth2 and at the same time simplify imports and exports with a structured approach. The principal idea here is that when a library wants to introduce SSZ support, they do so via 3 files: * `ssz_codecs` which imports and reexports `codecs` - this covers the basic byte conversions and ensures no overloads get lost * `xxx_merkleization` imports and exports `merkleization` to specialize and get access to `hash_tree_root` and friends * `xxx_ssz_serialization` imports and exports `ssz_serialization` to specialize ssz for a specific library Those that need to interact with SSZ always import the `xxx_` versions of the modules and never `ssz` itself so as to keep imports simple and safe. This is similar to how the REST / JSON-RPC serializers are structured in that someone wanting to serialize spec types to REST-JSON will import `eth2_rest_serialization` and nothing else. * split up ssz into a core library that is independendent of eth2 types * rename `bytes_reader` to `codec` to highlight that it contains coding and decoding of bytes and native ssz types * remove tricky List init overload that causes compile issues * get rid of top-level ssz import * reenable merkleization tests * move some "standard" json serializers to spec * remove `ValidatorIndex` serialization for now * remove test_ssz_merkleization * add tests for over/underlong byte sequences * fix broken seq[byte] test - seq[byte] is not an SSZ type There are a few things this PR doesn't solve: * like #2646 this PR is weak on how to handle root and other dontSerialize fields that "sometimes" should be computed - the same problem appears in REST / JSON-RPC etc * Fix a build problem on macOS * Another way to fix the macOS builds Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2021-08-18 18:57:58 +00:00
"."/[beaconstate, eth2_merkleization, helpers, validator],
Nbench - Flexible benchmarking of Nimbus internals (#641) * nbench PoC * Remove the yaml files from the example scenarios * update README with current status * Add an alternative implementation that uses defer * Forgot to add the old proc body * slots-processing * allow benching state_transition failures * Add Attestations processing (workaround confutils bug: - https://github.com/status-im/nim-confutils/issues/10 - https://github.com/status-im/nim-confutils/issues/11 - https://github.com/status-im/nim-confutils/issues/12 * Add CLI command in the readme * Filter report and add notes about CPU cycles * Report averages * Add debugecho style time/cycle print * Report when we skip BLS and state root verification * Update to 0.9.3 * Generalize scenario parsing * Support all block processing scenarios * parallel bench runner PoC * gitBetter load issues reporting (the load issues were invalid signature and expected to fail)
2019-12-20 16:14:43 +00:00
../../nbench/bench_lab
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
rework spec imports (#2779) The spec imports are a mess to work with, so this branch cleans them up a bit to ensure that we avoid generic sandwitches and that importing stuff generally becomes easier. * reexport crypto/digest/presets because these are part of the public symbol set of the rest of the spec types * don't export `merge` types from `base` - this causes circular deps * fix circular deps in `ssz/spec_types` - this is the first step in disentangling ssz from spec * be explicit about phase0 vs altair - longer term, `altair` will become the "natural" type set, then merge and so on, so no point in giving `phase0` special preferential treatment
2021-08-12 13:08:20 +00:00
export extras, phase0, altair
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
# Logging utilities
# --------------------------------------------------------
logScope: topics = "consens"
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
# Accessors that implement the max condition in `get_total_balance`:
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#get_total_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
template current_epoch*(v: TotalBalances): Gwei =
max(EFFECTIVE_BALANCE_INCREMENT, v.current_epoch_raw)
template previous_epoch*(v: TotalBalances): Gwei =
max(EFFECTIVE_BALANCE_INCREMENT, v.previous_epoch_raw)
template current_epoch_attesters*(v: TotalBalances): Gwei =
max(EFFECTIVE_BALANCE_INCREMENT, v.current_epoch_attesters_raw)
template current_epoch_target_attesters*(v: TotalBalances): Gwei =
max(EFFECTIVE_BALANCE_INCREMENT, v.current_epoch_target_attesters_raw)
template previous_epoch_attesters*(v: TotalBalances): Gwei =
max(EFFECTIVE_BALANCE_INCREMENT, v.previous_epoch_attesters_raw)
template previous_epoch_target_attesters*(v: TotalBalances): Gwei =
max(EFFECTIVE_BALANCE_INCREMENT, v.previous_epoch_target_attesters_raw)
template previous_epoch_head_attesters*(v: TotalBalances): Gwei =
max(EFFECTIVE_BALANCE_INCREMENT, v.previous_epoch_head_attesters_raw)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
func init*(rewards: var RewardInfo, state: SomeBeaconState) =
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
rewards.total_balances = TotalBalances()
rewards.statuses.setLen(state.validators.len)
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
for i in 0..<state.validators.len:
let v = unsafeAddr state.validators[i]
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
var flags: set[RewardFlags]
if v[].slashed:
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl(RewardFlags.isSlashed)
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
if state.get_current_epoch() >= v[].withdrawable_epoch:
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.canWithdrawInCurrentEpoch
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
if v[].is_active_validator(state.get_current_epoch()):
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
rewards.total_balances.current_epoch_raw += v[].effective_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
if v[].is_active_validator(state.get_previous_epoch()):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.isActiveInPreviousEpoch
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
rewards.total_balances.previous_epoch_raw += v[].effective_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
rewards.statuses[i] = RewardStatus(
current_epoch_effective_balance: v[].effective_balance,
flags: flags,
)
func add(a: var RewardDelta, b: RewardDelta) =
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
a.rewards += b.rewards
a.penalties += b.penalties
func process_attestation(
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
self: var RewardInfo, state: phase0.BeaconState, a: PendingAttestation,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
cache: var StateCache) =
# Collect information about the attestation
var
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
flags: set[RewardFlags]
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
is_previous_epoch_attester: Option[InclusionInfo]
if a.data.target.epoch == state.get_current_epoch():
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.isCurrentEpochAttester
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if a.data.target.root == get_block_root(state, state.get_current_epoch()):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.isCurrentEpochTargetAttester
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
elif a.data.target.epoch == state.get_previous_epoch():
is_previous_epoch_attester = some(InclusionInfo(
delay: a.inclusion_delay,
proposer_index: a.proposer_index,
))
if a.data.target.root == get_block_root(state, state.get_previous_epoch()):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.isPreviousEpochTargetAttester
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if a.data.beacon_block_root == get_block_root_at_slot(state, a.data.slot):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.isPreviousEpochHeadAttester
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
# Update the cache for all participants
for validator_index in get_attesting_indices(
state, a.data, a.aggregation_bits, cache):
template v(): untyped = self.statuses[validator_index]
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
v.flags = v.flags + flags
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if is_previous_epoch_attester.isSome:
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
if v.isPreviousEpochAttester.isSome:
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if is_previous_epoch_attester.get().delay <
v.is_previous_epoch_attester.get().delay:
v.is_previous_epoch_attester = is_previous_epoch_attester
else:
v.is_previous_epoch_attester = is_previous_epoch_attester
func process_attestations*(
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
self: var RewardInfo, state: phase0.BeaconState, cache: var StateCache) =
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
# Walk state attestations and update the status information
for a in state.previous_epoch_attestations:
process_attestation(self, state, a, cache)
for a in state.current_epoch_attestations:
process_attestation(self, state, a, cache)
for idx, v in self.statuses:
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if v.flags.contains RewardFlags.isSlashed:
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
continue
let validator_balance = state.validators[idx].effective_balance
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if v.flags.contains RewardFlags.isCurrentEpochAttester:
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
self.total_balances.current_epoch_attesters_raw += validator_balance
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if v.flags.contains RewardFlags.isCurrentEpochTargetAttester:
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
self.total_balances.current_epoch_target_attesters_raw += validator_balance
if v.is_previous_epoch_attester.isSome():
self.total_balances.previous_epoch_attesters_raw += validator_balance
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if v.flags.contains RewardFlags.isPreviousEpochTargetAttester:
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
self.total_balances.previous_epoch_target_attesters_raw += validator_balance
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if v.flags.contains RewardFlags.isPreviousEpochHeadAttester:
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
self.total_balances.previous_epoch_head_attesters_raw += validator_balance
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
func is_eligible_validator*(validator: RewardStatus): bool =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
validator.flags.contains(RewardFlags.isActiveInPreviousEpoch) or
(validator.flags.contains(RewardFlags.isSlashed) and not
(validator.flags.contains RewardFlags.canWithdrawInCurrentEpoch))
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
# Spec
# --------------------------------------------------------
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.2/specs/altair/beacon-chain.md#get_unslashed_participating_indices
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
func get_unslashed_participating_indices(
state: altair.BeaconState, flag_index: int, epoch: Epoch):
HashSet[ValidatorIndex] =
## Return the set of validator indices that are both active and unslashed for
## the given ``flag_index`` and ``epoch``.
doAssert epoch in [get_previous_epoch(state), get_current_epoch(state)]
let
epoch_participation =
if epoch == get_current_epoch(state):
state.current_epoch_participation
else:
state.previous_epoch_participation
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
# TODO use cached version, or similar
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
active_validator_indices = get_active_validator_indices(state, epoch)
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
var res: HashSet[ValidatorIndex]
for validator_index in active_validator_indices:
if has_flag(epoch_participation[validator_index], flag_index) and
not state.validators[validator_index].slashed:
res.incl validator_index
res
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#justification-and-finalization
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
proc process_justification_and_finalization*(state: var phase0.BeaconState,
avoid extraneous hash root calculation (#2537) When applying a block, we'll currently compute a state root for the state after slot processing but before block processing - this is unnecessary when a block is being applied because the intermediate state root is never observed.
2021-05-05 06:54:21 +00:00
total_balances: TotalBalances, flags: UpdateFlags = {}) {.nbench.} =
update 120+ beacon_chain and validator spec refs from v0.12.2 to v0.12.3 (#1740)
2020-09-24 17:04:10 +00:00
# Initial FFG checkpoint values have a `0x00` stub for `root`.
# Skip FFG updates in the first two epochs to avoid corner cases that might
# result in modifying this stub.
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
if get_current_epoch(state) <= GENESIS_EPOCH + 1:
return
let
previous_epoch = get_previous_epoch(state)
current_epoch = get_current_epoch(state)
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
old_previous_justified_checkpoint = state.previous_justified_checkpoint
old_current_justified_checkpoint = state.current_justified_checkpoint
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Process justifications
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
state.previous_justified_checkpoint = state.current_justified_checkpoint
## Spec:
## state.justification_bits[1:] = state.justification_bits[:-1]
## state.justification_bits[0] = 0b0
address some TODOs (#2005)
2020-11-12 19:24:07 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#constants
Fix an overflow in the justification bits processing
2019-09-27 16:40:33 +00:00
const JUSTIFICATION_BITS_LENGTH = 4
address some TODOs (#2005)
2020-11-12 19:24:07 +00:00
Fix an overflow in the justification bits processing
2019-09-27 16:40:33 +00:00
state.justification_bits = (state.justification_bits shl 1) and
cast[uint8]((2^JUSTIFICATION_BITS_LENGTH) - 1)
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
let total_active_balance = total_balances.current_epoch
if total_balances.previous_epoch_target_attesters * 3 >=
total_active_balance * 2:
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
state.current_justified_checkpoint =
Checkpoint(epoch: previous_epoch,
root: get_block_root(state, previous_epoch))
bump stew, fix warnings (#655)
2019-12-20 13:25:33 +00:00
state.justification_bits.setBit 1
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
small perf fixes * don't sort shuffled_validator_indices, just get them directly with iteration * grab full epoch of proposer indices while we have the data available - they'll get cached and reused * avoid computing active validator set when not used for logging
2020-08-19 08:03:50 +00:00
trace "Justified with previous epoch",
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
current_epoch = current_epoch,
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
checkpoint = shortLog(state.current_justified_checkpoint)
avoid extraneous hash root calculation (#2537) When applying a block, we'll currently compute a state root for the state after slot processing but before block processing - this is unnecessary when a block is being applied because the intermediate state root is never observed.
2021-05-05 06:54:21 +00:00
elif verifyFinalization in flags:
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
warn "Low attestation participation in previous epoch",
total_balances, epoch = get_current_epoch(state)
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if total_balances.current_epoch_target_attesters * 3 >=
total_active_balance * 2:
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
state.current_justified_checkpoint =
Checkpoint(epoch: current_epoch,
root: get_block_root(state, current_epoch))
bump stew, fix warnings (#655)
2019-12-20 13:25:33 +00:00
state.justification_bits.setBit 0
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
small perf fixes * don't sort shuffled_validator_indices, just get them directly with iteration * grab full epoch of proposer indices while we have the data available - they'll get cached and reused * avoid computing active validator set when not used for logging
2020-08-19 08:03:50 +00:00
trace "Justified with current epoch",
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
current_epoch = current_epoch,
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
checkpoint = shortLog(state.current_justified_checkpoint)
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Process finalizations
do most of 0.8.0 BeaconState updating, in particular field renamings, re-orderings, etc; rm now-unnecessary genesis block creation aspects, towards the near-stub version in current spec as mostfields are 0/default/etc values
2019-07-01 09:42:37 +00:00
let bitfield = state.justification_bits
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
## The 2nd/3rd/4th most recent epochs are justified, the 2nd using the 4th
## as source
Implement the latest SSZ specification and integrate the official SSZ test suite
2019-07-03 07:35:05 +00:00
if (bitfield and 0b1110) == 0b1110 and
old_previous_justified_checkpoint.epoch + 3 == current_epoch:
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
state.finalized_checkpoint = old_previous_justified_checkpoint
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
small perf fixes * don't sort shuffled_validator_indices, just get them directly with iteration * grab full epoch of proposer indices while we have the data available - they'll get cached and reused * avoid computing active validator set when not used for logging
2020-08-19 08:03:50 +00:00
trace "Finalized with rule 234",
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
current_epoch = current_epoch,
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
checkpoint = shortLog(state.finalized_checkpoint)
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
## The 2nd/3rd most recent epochs are justified, the 2nd using the 3rd as
## source
Implement the latest SSZ specification and integrate the official SSZ test suite
2019-07-03 07:35:05 +00:00
if (bitfield and 0b110) == 0b110 and
old_previous_justified_checkpoint.epoch + 2 == current_epoch:
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
state.finalized_checkpoint = old_previous_justified_checkpoint
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
small perf fixes * don't sort shuffled_validator_indices, just get them directly with iteration * grab full epoch of proposer indices while we have the data available - they'll get cached and reused * avoid computing active validator set when not used for logging
2020-08-19 08:03:50 +00:00
trace "Finalized with rule 23",
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
current_epoch = current_epoch,
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
checkpoint = shortLog(state.finalized_checkpoint)
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
## The 1st/2nd/3rd most recent epochs are justified, the 1st using the 3rd as
## source
Implement the latest SSZ specification and integrate the official SSZ test suite
2019-07-03 07:35:05 +00:00
if (bitfield and 0b111) == 0b111 and
old_current_justified_checkpoint.epoch + 2 == current_epoch:
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
state.finalized_checkpoint = old_current_justified_checkpoint
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
small perf fixes * don't sort shuffled_validator_indices, just get them directly with iteration * grab full epoch of proposer indices while we have the data available - they'll get cached and reused * avoid computing active validator set when not used for logging
2020-08-19 08:03:50 +00:00
trace "Finalized with rule 123",
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
current_epoch = current_epoch,
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
checkpoint = shortLog(state.finalized_checkpoint)
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
## The 1st/2nd most recent epochs are justified, the 1st using the 2nd as
## source
Implement the latest SSZ specification and integrate the official SSZ test suite
2019-07-03 07:35:05 +00:00
if (bitfield and 0b11) == 0b11 and
old_current_justified_checkpoint.epoch + 1 == current_epoch:
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
state.finalized_checkpoint = old_current_justified_checkpoint
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
small perf fixes * don't sort shuffled_validator_indices, just get them directly with iteration * grab full epoch of proposer indices while we have the data available - they'll get cached and reused * avoid computing active validator set when not used for logging
2020-08-19 08:03:50 +00:00
trace "Finalized with rule 12",
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
current_epoch = current_epoch,
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
checkpoint = shortLog(state.finalized_checkpoint)
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-alpha.7/specs/altair/beacon-chain.md#justification-and-finalization
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-alpha.7/specs/phase0/beacon-chain.md#justification-and-finalization
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
# TODO merge these things -- effectively, the phase0 process_justification_and_finalization is mostly a stub in this world
proc weigh_justification_and_finalization(state: var altair.BeaconState,
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
total_active_balance: Gwei,
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
previous_epoch_target_balance: Gwei,
current_epoch_target_balance: Gwei,
flags: UpdateFlags = {}) =
let
previous_epoch = get_previous_epoch(state)
current_epoch = get_current_epoch(state)
old_previous_justified_checkpoint = state.previous_justified_checkpoint
old_current_justified_checkpoint = state.current_justified_checkpoint
# Process justifications
state.previous_justified_checkpoint = state.current_justified_checkpoint
## Spec:
## state.justification_bits[1:] = state.justification_bits[:-1]
## state.justification_bits[0] = 0b0
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#constants
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
const JUSTIFICATION_BITS_LENGTH = 4
state.justification_bits = (state.justification_bits shl 1) and
cast[uint8]((2^JUSTIFICATION_BITS_LENGTH) - 1)
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
if previous_epoch_target_balance * 3 >= total_active_balance * 2:
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
state.current_justified_checkpoint =
Checkpoint(epoch: previous_epoch,
root: get_block_root(state, previous_epoch))
state.justification_bits.setBit 1
trace "Justified with previous epoch",
current_epoch = current_epoch,
checkpoint = shortLog(state.current_justified_checkpoint)
elif verifyFinalization in flags:
warn "Low attestation participation in previous epoch",
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
total_active_balance,
previous_epoch_target_balance,
current_epoch_target_balance,
epoch = get_current_epoch(state)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
if current_epoch_target_balance * 3 >= total_active_balance * 2:
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
state.current_justified_checkpoint =
Checkpoint(epoch: current_epoch,
root: get_block_root(state, current_epoch))
state.justification_bits.setBit 0
trace "Justified with current epoch",
current_epoch = current_epoch,
checkpoint = shortLog(state.current_justified_checkpoint)
# Process finalizations
let bitfield = state.justification_bits
## The 2nd/3rd/4th most recent epochs are justified, the 2nd using the 4th
## as source
if (bitfield and 0b1110) == 0b1110 and
old_previous_justified_checkpoint.epoch + 3 == current_epoch:
state.finalized_checkpoint = old_previous_justified_checkpoint
trace "Finalized with rule 234",
current_epoch = current_epoch,
checkpoint = shortLog(state.finalized_checkpoint)
## The 2nd/3rd most recent epochs are justified, the 2nd using the 3rd as
## source
if (bitfield and 0b110) == 0b110 and
old_previous_justified_checkpoint.epoch + 2 == current_epoch:
state.finalized_checkpoint = old_previous_justified_checkpoint
trace "Finalized with rule 23",
current_epoch = current_epoch,
checkpoint = shortLog(state.finalized_checkpoint)
## The 1st/2nd/3rd most recent epochs are justified, the 1st using the 3rd as
## source
if (bitfield and 0b111) == 0b111 and
old_current_justified_checkpoint.epoch + 2 == current_epoch:
state.finalized_checkpoint = old_current_justified_checkpoint
trace "Finalized with rule 123",
current_epoch = current_epoch,
checkpoint = shortLog(state.finalized_checkpoint)
## The 1st/2nd most recent epochs are justified, the 1st using the 2nd as
## source
if (bitfield and 0b11) == 0b11 and
old_current_justified_checkpoint.epoch + 1 == current_epoch:
state.finalized_checkpoint = old_current_justified_checkpoint
trace "Finalized with rule 12",
current_epoch = current_epoch,
checkpoint = shortLog(state.finalized_checkpoint)
proc process_justification_and_finalization*(state: var altair.BeaconState,
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
total_active_balance: Gwei, flags: UpdateFlags = {}) {.nbench.} =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
# Initial FFG checkpoint values have a `0x00` stub for `root`.
# Skip FFG updates in the first two epochs to avoid corner cases that might
# result in modifying this stub.
if get_current_epoch(state) <= GENESIS_EPOCH + 1:
return
let
# these ultimately differ from phase0 only in these lines
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.2/specs/phase0/beacon-chain.md#justification-and-finalization
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
previous_indices = get_unslashed_participating_indices(
state, TIMELY_TARGET_FLAG_INDEX, get_previous_epoch(state))
current_indices = get_unslashed_participating_indices(
state, TIMELY_TARGET_FLAG_INDEX, get_current_epoch(state))
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
previous_target_balance = get_total_balance(state, previous_indices)
current_target_balance = get_total_balance(state, current_indices)
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
weigh_justification_and_finalization(
state, total_active_balance, previous_target_balance,
current_target_balance, flags)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#helpers
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
func get_base_reward_sqrt*(state: phase0.BeaconState, index: ValidatorIndex,
improve slot processing speeds (#1670) about 40% better slot processing times (with LTO enabled) - these don't do BLS but are used heavily during replay (state transition = slot + block transition) tests using a recent medalla state and advancing it 1000 slots: ``` ./ncli slots --preState2:state-302271-3c1dbf19-c1f944bf.ssz --slot:1000 --postState2:xx.ssz ``` pre: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 39.236, 0.000, 39.236, 39.236, 1, Load state from file 0.049, 0.002, 0.046, 0.063, 968, Apply slot 256.504, 81.008, 213.471, 591.902, 32, Apply epoch slot 28.597, 0.000, 28.597, 28.597, 1, Save state to file ``` cast: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 37.079, 0.000, 37.079, 37.079, 1, Load state from file 0.042, 0.002, 0.040, 0.090, 968, Apply slot 215.552, 68.763, 180.155, 500.103, 32, Apply epoch slot 25.106, 0.000, 25.106, 25.106, 1, Save state to file ``` cast+rewards: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 40.049, 0.000, 40.049, 40.049, 1, Load state from file 0.048, 0.001, 0.045, 0.060, 968, Apply slot 164.981, 76.273, 142.099, 477.868, 32, Apply epoch slot 28.498, 0.000, 28.498, 28.498, 1, Save state to file ``` cast+rewards+shr ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 12.898, 0.000, 12.898, 12.898, 1, Load state from file 0.039, 0.002, 0.038, 0.054, 968, Apply slot 139.971, 68.797, 120.088, 428.844, 32, Apply epoch slot 24.761, 0.000, 24.761, 24.761, 1, Save state to file ```
2020-09-16 20:59:33 +00:00
total_balance_sqrt: auto): Gwei =
verify that state_sim is justifying and finalizing; fix 3 more warnings; rename crosslink_committee_cache to beacon_committee_cache; fix O(n^2) usage of get_base_reward(...)
2019-11-15 22:37:39 +00:00
# Spec function recalculates total_balance every time, which creates an
# O(n^2) situation.
let effective_balance = state.validators[index].effective_balance
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
effective_balance * BASE_REWARD_FACTOR div
improve slot processing speeds (#1670) about 40% better slot processing times (with LTO enabled) - these don't do BLS but are used heavily during replay (state transition = slot + block transition) tests using a recent medalla state and advancing it 1000 slots: ``` ./ncli slots --preState2:state-302271-3c1dbf19-c1f944bf.ssz --slot:1000 --postState2:xx.ssz ``` pre: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 39.236, 0.000, 39.236, 39.236, 1, Load state from file 0.049, 0.002, 0.046, 0.063, 968, Apply slot 256.504, 81.008, 213.471, 591.902, 32, Apply epoch slot 28.597, 0.000, 28.597, 28.597, 1, Save state to file ``` cast: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 37.079, 0.000, 37.079, 37.079, 1, Load state from file 0.042, 0.002, 0.040, 0.090, 968, Apply slot 215.552, 68.763, 180.155, 500.103, 32, Apply epoch slot 25.106, 0.000, 25.106, 25.106, 1, Save state to file ``` cast+rewards: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 40.049, 0.000, 40.049, 40.049, 1, Load state from file 0.048, 0.001, 0.045, 0.060, 968, Apply slot 164.981, 76.273, 142.099, 477.868, 32, Apply epoch slot 28.498, 0.000, 28.498, 28.498, 1, Save state to file ``` cast+rewards+shr ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 12.898, 0.000, 12.898, 12.898, 1, Load state from file 0.039, 0.002, 0.038, 0.054, 968, Apply slot 139.971, 68.797, 120.088, 428.844, 32, Apply epoch slot 24.761, 0.000, 24.761, 24.761, 1, Save state to file ```
2020-09-16 20:59:33 +00:00
total_balance_sqrt div BASE_REWARDS_PER_EPOCH
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
func get_proposer_reward(base_reward: Gwei): Gwei =
allow 50k validators on block_sim by avoiding repeated get_total_active_balance() in get_proposer_reward()
2020-07-07 16:52:12 +00:00
# Spec version recalculates get_total_active_balance(state) quadratically
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward div PROPOSER_REWARD_QUOTIENT
func is_in_inactivity_leak(finality_delay: uint64): bool =
finality_delay > MIN_EPOCHS_TO_INACTIVITY_PENALTY
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
func get_finality_delay(state: SomeBeaconState): uint64 =
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
get_previous_epoch(state) - state.finalized_checkpoint.epoch
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-alpha.8/specs/phase0/beacon-chain.md#rewards-and-penalties-1
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
func is_in_inactivity_leak(state: altair.BeaconState): bool =
# TODO remove this, see above
get_finality_delay(state) > MIN_EPOCHS_TO_INACTIVITY_PENALTY
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
func get_attestation_component_delta(is_unslashed_attester: bool,
attesting_balance: Gwei,
total_balance: Gwei,
base_reward: uint64,
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
finality_delay: uint64): RewardDelta =
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
# Helper with shared logic for use by get source, target, and head deltas
# functions
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if is_unslashed_attester:
if is_in_inactivity_leak(finality_delay):
# Since full base reward will be canceled out by inactivity penalty deltas,
# optimal participation receives full base reward compensation here.
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
RewardDelta(rewards: base_reward)
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
else:
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
let reward_numerator =
base_reward * (attesting_balance div EFFECTIVE_BALANCE_INCREMENT)
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
RewardDelta(rewards:
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
reward_numerator div (total_balance div EFFECTIVE_BALANCE_INCREMENT))
else:
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
RewardDelta(penalties: base_reward)
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#components-of-attestation-deltas
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
func get_source_delta*(validator: RewardStatus,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward: uint64,
total_balances: TotalBalances,
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
finality_delay: uint64): RewardDelta =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Return attester micro-rewards/penalties for source-vote for each validator.
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
get_attestation_component_delta(
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
validator.is_previous_epoch_attester.isSome() and
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
not (validator.flags.contains RewardFlags.isSlashed),
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
total_balances.previous_epoch_attesters,
total_balances.current_epoch,
base_reward,
finality_delay)
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
func get_target_delta*(validator: RewardStatus,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward: uint64,
total_balances: TotalBalances,
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
finality_delay: uint64): RewardDelta =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Return attester micro-rewards/penalties for target-vote for each validator.
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
get_attestation_component_delta(
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
validator.flags.contains(RewardFlags.isPreviousEpochTargetAttester) and
not (validator.flags.contains(RewardFlags.isSlashed)),
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
total_balances.previous_epoch_target_attesters,
total_balances.current_epoch,
base_reward,
finality_delay)
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
func get_head_delta*(validator: RewardStatus,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward: uint64,
total_balances: TotalBalances,
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
finality_delay: uint64): RewardDelta =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Return attester micro-rewards/penalties for head-vote for each validator.
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
get_attestation_component_delta(
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
validator.flags.contains(RewardFlags.isPreviousEpochHeadAttester) and
((not validator.flags.contains(RewardFlags.isSlashed))),
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
total_balances.previous_epoch_head_attesters,
total_balances.current_epoch,
base_reward,
finality_delay)
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
func get_inclusion_delay_delta*(validator: RewardStatus,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward: uint64):
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
(RewardDelta, Option[(uint64, RewardDelta)]) =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Return proposer and inclusion delay micro-rewards/penalties for each validator.
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if validator.is_previous_epoch_attester.isSome() and ((not validator.flags.contains(RewardFlags.isSlashed))):
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
let
inclusion_info = validator.is_previous_epoch_attester.get()
proposer_reward = get_proposer_reward(base_reward)
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
proposer_delta = RewardDelta(rewards: proposer_reward)
fix attestation handling bottlenecks in slot processing and quadratic-time inclusion delay rewards/palties in epoch processing; create infrastructure for dynamically (un)subscribing to/from attestation topics (#1294)
2020-07-10 09:24:04 +00:00
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
let
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
max_attester_reward = base_reward - proposer_reward
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
delta = RewardDelta(rewards: max_attester_reward div inclusion_info.delay)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
proposer_index = inclusion_info.proposer_index;
return (delta, some((proposer_index, proposer_delta)))
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
func get_inactivity_penalty_delta*(validator: RewardStatus,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward: Gwei,
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
finality_delay: uint64): RewardDelta =
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
## Return inactivity reward/penalty deltas for each validator.
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
var delta: RewardDelta
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if is_in_inactivity_leak(finality_delay):
# If validator is performing optimally this cancels all rewards for a neutral balance
delta.penalties +=
BASE_REWARDS_PER_EPOCH * base_reward - get_proposer_reward(base_reward)
# Additionally, all validators whose FFG target didn't match are penalized extra
# This condition is equivalent to this condition from the spec:
# `index not in get_unslashed_attesting_indices(state, matching_target_attestations)`
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if (validator.flags.contains(RewardFlags.isSlashed)) or
((not validator.flags.contains(RewardFlags.isPreviousEpochTargetAttester))):
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
delta.penalties +=
validator.current_epoch_effective_balance * finality_delay div
INACTIVITY_PENALTY_QUOTIENT
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
delta
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#get_attestation_deltas
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
func get_attestation_deltas(state: phase0.BeaconState, rewards: var RewardInfo) =
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
## Update rewards with attestation reward/penalty deltas for each validator.
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
remove v0.11.3 support; add block_sim to CI (#1253) * remove v0.11.3 support; add block_sim to CI * rm stray PERSISTENT_COMMITTEE_PERIOD * remove TopicFilter.InteropAttestations * bump two comment-spec-refs to v0.12.1
2020-06-29 18:08:58 +00:00
let
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
finality_delay = get_finality_delay(state)
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
total_balance = rewards.total_balances.current_epoch
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
total_balance_sqrt = integer_squareroot(total_balance)
# Filter out ineligible validators. All sub-functions of the spec do this
# except for `get_inclusion_delay_deltas`. It's safe to do so here because
# any validator that is in the unslashed indices of the matching source
# attestations is active, and therefore eligible.
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
for index, validator in rewards.statuses.mpairs():
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if not is_eligible_validator(validator):
continue
let
base_reward = get_base_reward_sqrt(
state, index.ValidatorIndex, total_balance_sqrt)
improve slot processing speeds (#1670) about 40% better slot processing times (with LTO enabled) - these don't do BLS but are used heavily during replay (state transition = slot + block transition) tests using a recent medalla state and advancing it 1000 slots: ``` ./ncli slots --preState2:state-302271-3c1dbf19-c1f944bf.ssz --slot:1000 --postState2:xx.ssz ``` pre: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 39.236, 0.000, 39.236, 39.236, 1, Load state from file 0.049, 0.002, 0.046, 0.063, 968, Apply slot 256.504, 81.008, 213.471, 591.902, 32, Apply epoch slot 28.597, 0.000, 28.597, 28.597, 1, Save state to file ``` cast: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 37.079, 0.000, 37.079, 37.079, 1, Load state from file 0.042, 0.002, 0.040, 0.090, 968, Apply slot 215.552, 68.763, 180.155, 500.103, 32, Apply epoch slot 25.106, 0.000, 25.106, 25.106, 1, Save state to file ``` cast+rewards: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 40.049, 0.000, 40.049, 40.049, 1, Load state from file 0.048, 0.001, 0.045, 0.060, 968, Apply slot 164.981, 76.273, 142.099, 477.868, 32, Apply epoch slot 28.498, 0.000, 28.498, 28.498, 1, Save state to file ``` cast+rewards+shr ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 12.898, 0.000, 12.898, 12.898, 1, Load state from file 0.039, 0.002, 0.038, 0.054, 968, Apply slot 139.971, 68.797, 120.088, 428.844, 32, Apply epoch slot 24.761, 0.000, 24.761, 24.761, 1, Save state to file ```
2020-09-16 20:59:33 +00:00
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
let
source_delta = get_source_delta(
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
validator, base_reward, rewards.total_balances, finality_delay)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
target_delta = get_target_delta(
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
validator, base_reward, rewards.total_balances, finality_delay)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
head_delta = get_head_delta(
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
validator, base_reward, rewards.total_balances, finality_delay)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
(inclusion_delay_delta, proposer_delta) =
get_inclusion_delay_delta(validator, base_reward)
inactivity_delta = get_inactivity_penalty_delta(
validator, base_reward, finality_delay)
validator.delta.add source_delta
validator.delta.add target_delta
validator.delta.add head_delta
validator.delta.add inclusion_delay_delta
validator.delta.add inactivity_delta
if proposer_delta.isSome:
let proposer_index = proposer_delta.get()[0]
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
if proposer_index < rewards.statuses.lenu64:
rewards.statuses[proposer_index].delta.add(
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
proposer_delta.get()[1])
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.2/specs/altair/beacon-chain.md#get_base_reward_per_increment
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
func get_base_reward_per_increment(
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
state: altair.BeaconState, total_active_balance_sqrt: uint64): Gwei =
EFFECTIVE_BALANCE_INCREMENT * BASE_REWARD_FACTOR div total_active_balance_sqrt
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.2/specs/altair/beacon-chain.md#get_base_reward
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
func get_base_reward(
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
state: altair.BeaconState, index: ValidatorIndex,
total_active_balance_sqrt: uint64): Gwei =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
## Return the base reward for the validator defined by ``index`` with respect
## to the current ``state``.
let increments =
state.validators[index].effective_balance div EFFECTIVE_BALANCE_INCREMENT
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
increments * get_base_reward_per_increment(state, total_active_balance_sqrt)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.2/specs/altair/beacon-chain.md#get_flag_index_deltas
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
iterator get_flag_index_deltas(
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
state: altair.BeaconState, flag_index: int, total_active_balance: Gwei,
total_active_balance_sqrt: uint64): (ValidatorIndex, Gwei, Gwei) =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
## Return the deltas for a given ``flag_index`` by scanning through the
## participation flags.
let
previous_epoch = get_previous_epoch(state)
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
unslashed_participating_indices =
get_unslashed_participating_indices(state, flag_index, previous_epoch)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
weight = PARTICIPATION_FLAG_WEIGHTS[flag_index].uint64 # safe
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
unslashed_participating_balance =
get_total_balance(state, unslashed_participating_indices)
unslashed_participating_increments =
unslashed_participating_balance div EFFECTIVE_BALANCE_INCREMENT
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
active_increments = total_active_balance div EFFECTIVE_BALANCE_INCREMENT
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
for index in 0 ..< state.validators.len:
# TODO Obviously not great
let v = state.validators[index]
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
if not (is_active_validator(v, previous_epoch) or
(v.slashed and previous_epoch + 1 < v.withdrawable_epoch)):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
continue
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
template vidx: ValidatorIndex = index.ValidatorIndex
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
let base_reward = get_base_reward(state, vidx, total_active_balance_sqrt)
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
yield
if vidx in unslashed_participating_indices:
if not is_in_inactivity_leak(state):
let reward_numerator =
base_reward * weight * unslashed_participating_increments
(vidx, reward_numerator div (active_increments * WEIGHT_DENOMINATOR), 0.Gwei)
else:
(vidx, 0.Gwei, 0.Gwei)
elif flag_index != TIMELY_HEAD_FLAG_INDEX:
(vidx, 0.Gwei, base_reward * weight div WEIGHT_DENOMINATOR)
else:
(vidx, 0.Gwei, 0.Gwei)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.2/specs/altair/beacon-chain.md#modified-get_inactivity_penalty_deltas
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
iterator get_inactivity_penalty_deltas(cfg: RuntimeConfig, state: altair.BeaconState):
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
(ValidatorIndex, Gwei) =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
## Return the inactivity penalty deltas by considering timely target
## participation flags and inactivity scores.
let
previous_epoch = get_previous_epoch(state)
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
matching_target_indices =
get_unslashed_participating_indices(state, TIMELY_TARGET_FLAG_INDEX, previous_epoch)
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
penalty_denominator =
cfg.INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
for index in 0 ..< state.validators.len:
# get_eligible_validator_indices()
let v = state.validators[index]
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
if not (is_active_validator(v, previous_epoch) or
(v.slashed and previous_epoch + 1 < v.withdrawable_epoch)):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
continue
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
template vidx: untyped = index.ValidatorIndex
if not (vidx in matching_target_indices):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
let
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
penalty_numerator = state.validators[index].effective_balance *
state.inactivity_scores[index]
yield (vidx, Gwei(penalty_numerator div penalty_denominator))
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.2/specs/altair/beacon-chain.md#rewards-and-penalties
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
func process_rewards_and_penalties(
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
state: var phase0.BeaconState, rewards: var RewardInfo) {.nbench.} =
update 120+ beacon_chain and validator spec refs from v0.12.2 to v0.12.3 (#1740)
2020-09-24 17:04:10 +00:00
# No rewards are applied at the end of `GENESIS_EPOCH` because rewards are
# for work done in the previous epoch
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
doAssert rewards.statuses.len == state.validators.len
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
if get_current_epoch(state) == GENESIS_EPOCH:
return
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
get_attestation_deltas(state, rewards)
fix attestation selection when calculating rewards and penalties Previously, the we'd use the first source attestation regardless if it matches the filter or not leading to the wrong minimum being used
2019-10-01 14:40:41 +00:00
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
# Here almost all balances are updated (assuming most validators are active) -
# clearing the cache becomes a bottleneck if done item by item because of the
# recursive nature of cache clearing - instead, we clear the whole cache then
# update the raw list directly
state.balances.clearCache()
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
for idx, v in rewards.statuses:
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
increase_balance(state.balances.asSeq()[idx], v.delta.rewards)
decrease_balance(state.balances.asSeq()[idx], v.delta.penalties)
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.2/specs/altair/beacon-chain.md#rewards-and-penalties
split state_transition() into slots/block parts and use only block where appropriate (#2630)
2021-06-03 09:42:25 +00:00
func process_rewards_and_penalties(
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
cfg: RuntimeConfig, state: var altair.BeaconState, total_active_balance: Gwei) {.nbench.} =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if get_current_epoch(state) == GENESIS_EPOCH:
return
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
# TODO assess relevance of missing phase0 optimizations
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
# TODO probably both of these aren't necessary, but need to verify
# commutativity & associativity. Probably, since active validators
# get ejected at 16 Gwei, either it avoids over or underflow there
# or doesn't receive rewards or penalties so both are 0. But start
# with this.
var
rewards = newSeq[Gwei](state.validators.len)
penalties = newSeq[Gwei](state.validators.len)
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
let total_active_balance_sqrt = integer_squareroot(total_active_balance)
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
for flag_index in 0 ..< PARTICIPATION_FLAG_WEIGHTS.len:
for validator_index, reward, penalty in get_flag_index_deltas(
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
state, flag_index, total_active_balance, total_active_balance_sqrt):
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
rewards[validator_index] += reward
penalties[validator_index] += penalty
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
for validator_index, penalty in get_inactivity_penalty_deltas(cfg, state):
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
penalties[validator_index] += penalty
for index in 0 ..< len(state.validators):
increase_balance(state, ValidatorIndex(index), rewards[index])
decrease_balance(state, ValidatorIndex(index), penalties[index])
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#slashings
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.2/specs/altair/beacon-chain.md#slashings
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
func process_slashings*(state: var SomeBeaconState, total_balance: Gwei) {.nbench.}=
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
let
More 0.8.0 updates (#311) * replace BeaconState.finalized_{epoch,root} with BeaconState.finalized_checkpoint; rename get_delayed_activation_exit_epoch(...) to compute_activation_exit_epoch(...) and mark as 0.8.0; update get_churn_limit(...)/get_validator_churn_limit(...) to 0.8.0; update process_registry_updates(...) to 0.8.0 * update process_crosslinks(...) to 0.8.0; mark compute_start_slot_of_epoch(...) and get_committee_count(...) as 0.8.0 * mark Fork, is_slashable_validator(...), and get_beacon_proposer_index(...) as 0.8.0 * rename LATEST_SLASHED_EXIT_LENGTH to EPOCHS_PER_SLASHINGS_VECTOR; update process_slashings(...) to 0.8.0; remove pointless type conversion warning in get_previous_epoch(...) * convert remaining references to finalized_epoch to finalized_checkpoint.epoch * update slash_validator(...) to 0.8.0; mark inital value, Gwei, and time constants as 0.8.0; mark hash(...) and processBlockHeader(...) as 0.8.0 * rename WHISTLEBLOWING_REWARD_QUOTIENT to WHISTLEBLOWER_REWARD_QUOTIENT; rename LATEST_ACTIVE_INDEX_ROOTS_LENGTH to EPOCHS_PER_HISTORICAL_VECTOR (randao will also get merged into this); remove get_active_index_root(...); mark time parameter, signature domain types, and max operations per block constants as 0.8.0; update rewards and penalties constants to 0.8.0 * update is_valid_indexed_attestation(...) to 0.8.0; mark process_slot(...) as 0.8.0 * replace BeaconState.{current,previous}_justified_{epoch,root} with BeaconState.{current,previous}_justified_checkpoint
2019-07-05 08:30:05 +00:00
epoch = get_current_epoch(state)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
multiplier =
# tradeoff here about interleaving phase0/altair, but for these
# single-constant changes...
uint64(when state is phase0.BeaconState:
PROPORTIONAL_SLASHING_MULTIPLIER
elif state is altair.BeaconState:
PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR
else:
raiseAssert "process_slashings: incorrect BeaconState type")
update process_slashings() to use PROPORTIONAL_SLASHING_MULTIPLIER (#1841)
2020-10-08 18:00:07 +00:00
adjusted_total_slashing_balance =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
min(sum(state.slashings) * multiplier, total_balance)
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
for index in 0..<state.validators.len:
let validator = unsafeAddr state.validators.asSeq()[index]
if validator[].slashed and epoch + EPOCHS_PER_SLASHINGS_VECTOR div 2 ==
validator[].withdrawable_epoch:
V0.8.3 tests part2 - BLS + Shuffling + State transition epoch (#372) * Update BLS fixtures to 0.8.3 * Bump fixtures with shuffling with minimal preset * Update shuffling tests * parseTest generic over file format (Json or SSZ) * Initial crosslink parsing commit for debugging Nim crash * Workaround https://github.com/status-im/nim-beacon-chain/issues/369 * Crosslink test works for minimal - https://github.com/status-im/nim-beacon-chain/issues/369 is back on mainnet * Use ref objects to workaround https://github.com/status-im/nim-beacon-chain/issues/369 * Generalize state transition epoch test to all epoch tests * Fix slashing (potential uint64 overflow in previous spec) * Add a state debugging macro to deeply inspect the wrong fields * make reportDiff visible * Improve the debug state macro for containers
2019-09-03 03:12:09 +00:00
let increment = EFFECTIVE_BALANCE_INCREMENT # Factored out from penalty
# numerator to avoid uint64 overflow
let penalty_numerator =
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
validator[].effective_balance div increment *
update process_slashings() to use PROPORTIONAL_SLASHING_MULTIPLIER (#1841)
2020-10-08 18:00:07 +00:00
adjusted_total_slashing_balance
V0.8.3 tests part2 - BLS + Shuffling + State transition epoch (#372) * Update BLS fixtures to 0.8.3 * Bump fixtures with shuffling with minimal preset * Update shuffling tests * parseTest generic over file format (Json or SSZ) * Initial crosslink parsing commit for debugging Nim crash * Workaround https://github.com/status-im/nim-beacon-chain/issues/369 * Crosslink test works for minimal - https://github.com/status-im/nim-beacon-chain/issues/369 is back on mainnet * Use ref objects to workaround https://github.com/status-im/nim-beacon-chain/issues/369 * Generalize state transition epoch test to all epoch tests * Fix slashing (potential uint64 overflow in previous spec) * Add a state debugging macro to deeply inspect the wrong fields * make reportDiff visible * Improve the debug state macro for containers
2019-09-03 03:12:09 +00:00
let penalty = penalty_numerator div total_balance * increment
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
decrease_balance(state, index.ValidatorIndex, penalty)
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/34cea67b91/specs/phase0/beacon-chain.md#eth1-data-votes-updates
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
func process_eth1_data_reset*(state: var SomeBeaconState) {.nbench.} =
per HF1, split process_final_updates() (#2319)
2021-02-14 19:31:01 +00:00
let next_epoch = get_current_epoch(state) + 1
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Reset eth1 data votes
initial 0.11.0 spec version update
2020-03-14 21:54:45 +00:00
if next_epoch mod EPOCHS_PER_ETH1_VOTING_PERIOD == 0:
SSZ cleanup * be stricter about SSZ length prefix * compute zeroHash list at compile time * remove SSZ schema stuff * move SSZ navigation to ncli * cleanup a few leftover openArray uses
2020-05-27 11:36:02 +00:00
state.eth1_data_votes = default(type state.eth1_data_votes)
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/34cea67b91/specs/phase0/beacon-chain.md#effective-balances-updates
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
func process_effective_balance_updates*(state: var SomeBeaconState) {.nbench.} =
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Update effective balances with hysteresis
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
for index in 0..<state.validators.len:
let balance = state.balances.asSeq()[index]
initial 0.11.0 spec version update
2020-03-14 21:54:45 +00:00
const
HYSTERESIS_INCREMENT =
EFFECTIVE_BALANCE_INCREMENT div HYSTERESIS_QUOTIENT
DOWNWARD_THRESHOLD =
HYSTERESIS_INCREMENT * HYSTERESIS_DOWNWARD_MULTIPLIER
UPWARD_THRESHOLD = HYSTERESIS_INCREMENT * HYSTERESIS_UPWARD_MULTIPLIER
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
let effective_balance = state.validators.asSeq()[index].effective_balance
if balance + DOWNWARD_THRESHOLD < effective_balance or
effective_balance + UPWARD_THRESHOLD < balance:
rename BeaconState.validator_registry to BeaconState.validators
2019-07-01 09:13:14 +00:00
state.validators[index].effective_balance =
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
min(
balance - balance mod EFFECTIVE_BALANCE_INCREMENT,
MAX_EFFECTIVE_BALANCE)
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/34cea67b91/specs/phase0/beacon-chain.md#slashings-balances-updates
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
func process_slashings_reset*(state: var SomeBeaconState) {.nbench.} =
per HF1, split process_final_updates() (#2319)
2021-02-14 19:31:01 +00:00
let next_epoch = get_current_epoch(state) + 1
0.8.0, continued (#315) * rename get_genesis_beacon_state(...) to initialize_beacon_state_from_eth1(...); remove unused vestiges get_temporary_block_header(...) and get_empty_block(...); partly align initialize_beacon_state_from_eth1(...) with 0.8.0 version; implement CompactCommittee object type; update process_final_updates(...) to 0.8.0 * mark get_shard_delta(...) as 0.8.0 * mark get_total_active_balance(...) and epoch transition helper functions as 0.8.0 * move FORK_CHOICE_BALANCE_INCREMENT, not in spec anymore, to sole user in fork_choice
2019-07-10 12:23:02 +00:00
# Reset slashings
Small fixes (#1165) * random fixes * create dump dir on startup * don't crash on failure to write dump * fix a few `uint64` instances being used when indexing arrays - this should be a compile error but isn't due to compiler bugs * fix standalone test_block_pool compilation * add signed block processing in ncli * reuse cache entry instead of allocating a new one * allow for small clock disparities when validating blocks
2020-06-12 16:43:20 +00:00
state.slashings[int(next_epoch mod EPOCHS_PER_SLASHINGS_VECTOR)] = 0.Gwei
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/34cea67b91/specs/phase0/beacon-chain.md#randao-mixes-updates
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
func process_randao_mixes_reset*(state: var SomeBeaconState) {.nbench.} =
per HF1, split process_final_updates() (#2319)
2021-02-14 19:31:01 +00:00
let
current_epoch = get_current_epoch(state)
next_epoch = current_epoch + 1
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Set randao mix
Fix #378, int to bytes32, LATEST_RANDAO_MIXES, get_seed, get_crosslink_committee, get_compact_committee_root (#380)
2019-09-04 13:57:18 +00:00
state.randao_mixes[next_epoch mod EPOCHS_PER_HISTORICAL_VECTOR] =
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
get_randao_mix(state, current_epoch)
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/34cea67b91/specs/phase0/beacon-chain.md#historical-roots-updates
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
func process_historical_roots_update*(state: var SomeBeaconState) {.nbench.} =
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Set historical root accumulator
per HF1, split process_final_updates() (#2319)
2021-02-14 19:31:01 +00:00
let next_epoch = get_current_epoch(state) + 1
Restore compilation with custom presets (#1309) This also assigns precise types to the constants in the minimal and mainnet presets in order to reduce the chance of compilation errors when custom presets are used (previously, only the custom presets have precisely assigned types for the constants).
2020-07-13 14:44:58 +00:00
if next_epoch mod (SLOTS_PER_HISTORICAL_ROOT div SLOTS_PER_EPOCH) == 0:
reduce stack space usage in process_final_updates(...) (#925) * reduce stack space usage in process_final_updates(...) to avoid fuzzed segfault in https://github.com/status-im/nim-beacon-chain/issues/921 * document motivation behind manually constructing hash_tree_root of a HistoricalBatch
2020-04-23 11:39:38 +00:00
# Equivalent to hash_tree_root(foo: HistoricalBatch), but without using
# significant additional stack or heap.
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#historicalbatch
update nim-beacon-chain to nimbus-eth2 in beacon_chain/, ncli/, tests/, and README.md (#1843)
2020-10-08 19:02:05 +00:00
# In response to https://github.com/status-im/nimbus-eth2/issues/921
Reset cached indices when resetting cache on SSZ read (#2480) * Reset cached indices when resetting cache on SSZ read When deserializing into an existing structure, the cache should be cleared - goes for json also. Also improve error messages.
2021-04-08 10:11:04 +00:00
if not state.historical_roots.add hash_tree_root(
[hash_tree_root(state.block_roots), hash_tree_root(state.state_roots)]):
raiseAssert "no more room for historical roots, so long and thanks for the fish!"
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/34cea67b91/specs/phase0/beacon-chain.md#participation-records-rotation
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
func process_participation_record_updates*(state: var phase0.BeaconState) {.nbench.} =
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
# Rotate current/previous epoch attestations - using swap avoids copying all
# elements using a slow genericSeqAssign
state.previous_epoch_attestations.clear()
swap(state.previous_epoch_attestations, state.current_epoch_attestations)
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.2/specs/altair/beacon-chain.md#participation-flags-updates
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
func process_participation_flag_updates*(state: var altair.BeaconState) =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
state.previous_epoch_participation = state.current_epoch_participation
block_clearance, ncli, and ncli_db Altair state saving (#2672) * block_clearance, ncli, and ncli_db Altair state saving * avoid invalidating SSZ hash caches with every assignment
2021-06-24 18:34:08 +00:00
const zero = 0.ParticipationFlags
for i in 0 ..< state.current_epoch_participation.len:
state.current_epoch_participation.data[i] = zero
# Shouldn't be wasted zeroing, because state.current_epoch_participation only
# grows. New elements are automatically initialized to 0, as required.
doAssert state.current_epoch_participation.data.setLen(state.validators.len)
state.current_epoch_participation.resetCache()
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.2/specs/altair/beacon-chain.md#sync-committee-updates
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
proc process_sync_committee_updates*(state: var altair.BeaconState) =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
let next_epoch = get_current_epoch(state) + 1
if next_epoch mod EPOCHS_PER_SYNC_COMMITTEE_PERIOD == 0:
state.current_sync_committee = state.next_sync_committee
state.next_sync_committee = get_next_sync_committee(state)
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.2/specs/altair/beacon-chain.md#inactivity-scores
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
func process_inactivity_updates*(cfg: RuntimeConfig, state: var altair.BeaconState) =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
# Score updates based on previous epoch participation, skip genesis epoch
if get_current_epoch(state) == GENESIS_EPOCH:
return
# TODO actually implement get_eligible_validator_indices() as an iterator
remove false OnBlockAdded dependency on phase0 HashedBeaconState (#2661) * remove false OnBlockAdded dependency on phase.HashedBeaconState * introduce altair data types into block_clearance; update some alpha.6 spec refs to alpha.7; add get_active_validator_indices_len ForkedHashedBeaconState wrapper * switch many modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 and remove rm'd G2_POINT_AT_INFINITY * switch more modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 * remove unnecessary phase0-only wrapper of get_attesting_indices(); allow signatures_batch to process either fork; remove O(n^2) nested loop in process_inactivity_updates(); add altair support to getAttestationsforTestBlock() * add Altair versions of asSigVerified(), asTrusted(), and makeBeaconBlock() * fix spec URL to be Altair for Altair makeBeaconBlock()
2021-06-21 08:35:24 +00:00
let
previous_epoch = get_previous_epoch(state) # get_eligible_validator_indices()
unslashed_participating_indices =
get_unslashed_participating_indices(
state, TIMELY_TARGET_FLAG_INDEX, get_previous_epoch(state))
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
for index in 0'u64 ..< state.validators.lenu64:
# get_eligible_validator_indices()
let v = state.validators[index]
if not (is_active_validator(v, previous_epoch) or (v.slashed and previous_epoch + 1 < v.withdrawable_epoch)):
continue
remove false OnBlockAdded dependency on phase0 HashedBeaconState (#2661) * remove false OnBlockAdded dependency on phase.HashedBeaconState * introduce altair data types into block_clearance; update some alpha.6 spec refs to alpha.7; add get_active_validator_indices_len ForkedHashedBeaconState wrapper * switch many modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 and remove rm'd G2_POINT_AT_INFINITY * switch more modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 * remove unnecessary phase0-only wrapper of get_attesting_indices(); allow signatures_batch to process either fork; remove O(n^2) nested loop in process_inactivity_updates(); add altair support to getAttestationsforTestBlock() * add Altair versions of asSigVerified(), asTrusted(), and makeBeaconBlock() * fix spec URL to be Altair for Altair makeBeaconBlock()
2021-06-21 08:35:24 +00:00
# Increase the inactivity score of inactive validators
if index.ValidatorIndex in unslashed_participating_indices:
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
state.inactivity_scores[index] -= min(1'u64, state.inactivity_scores[index])
else:
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
state.inactivity_scores[index] += cfg.INACTIVITY_SCORE_BIAS
remove false OnBlockAdded dependency on phase0 HashedBeaconState (#2661) * remove false OnBlockAdded dependency on phase.HashedBeaconState * introduce altair data types into block_clearance; update some alpha.6 spec refs to alpha.7; add get_active_validator_indices_len ForkedHashedBeaconState wrapper * switch many modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 and remove rm'd G2_POINT_AT_INFINITY * switch more modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 * remove unnecessary phase0-only wrapper of get_attesting_indices(); allow signatures_batch to process either fork; remove O(n^2) nested loop in process_inactivity_updates(); add altair support to getAttestationsforTestBlock() * add Altair versions of asSigVerified(), asTrusted(), and makeBeaconBlock() * fix spec URL to be Altair for Altair makeBeaconBlock()
2021-06-21 08:35:24 +00:00
# Decrease the inactivity score of all eligible validators during a
# leak-free epoch
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if not is_in_inactivity_leak(state):
state.inactivity_scores[index] -= min(INACTIVITY_SCORE_RECOVERY_RATE.uint64, state.inactivity_scores[index])
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#epoch-processing
avoid extraneous hash root calculation (#2537) When applying a block, we'll currently compute a state root for the state after slot processing but before block processing - this is unnecessary when a block is being applied because the intermediate state root is never observed.
2021-05-05 06:54:21 +00:00
proc process_epoch*(
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
cfg: RuntimeConfig, state: var phase0.BeaconState, flags: UpdateFlags,
cache: var StateCache, rewards: var RewardInfo) {.nbench.} =
more metrics (#1004)
2020-05-11 06:25:49 +00:00
let currentEpoch = get_current_epoch(state)
restore network sim finalization * relax attestation validation when attestation is incoming but make it more strict when adding to block * share attestation validation logic between attestation pool and state transition * remove a bunch of redundant logging * fix potential underflow in attestation delay checking * fix committee used for attestation shard selection when attesting * fix attestation data construction
2019-08-19 16:41:13 +00:00
trace "process_epoch",
more metrics (#1004)
2020-05-11 06:25:49 +00:00
current_epoch = currentEpoch
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
init(rewards, state)
rewards.process_attestations(state, cache)
Justification/finalization amelioration using testnet1 shard count and epoch length (#331) * working justification, 1 node, 64 validators * closer to tn1 params: 256 validators * more debugging output; switch to minimum test case * working justification and finalization in local network simulation * fix currently incorrect state transition/attestation test assumption
2019-08-14 08:56:32 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#justification-and-finalization
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
process_justification_and_finalization(
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
state, rewards.total_balances, flags)
Increase finalization and finalization checking robustness (#990) * fix some warnings related to beacon_node splitting; reimplement finalization verification more robustly; improve attestation pool block selection logic * re-add missing import * whitelist allowed state transition flags and make rollback/restore naming more consistent * restore usage of update flags passed into skipAndUpdateState(...) in addition to the potential verifyFinalization flag * switch rest of rollback -> restore
2020-05-09 12:43:15 +00:00
# state.slot hasn't been incremented yet.
avoid extraneous hash root calculation (#2537) When applying a block, we'll currently compute a state root for the state after slot processing but before block processing - this is unnecessary when a block is being applied because the intermediate state root is never observed.
2021-05-05 06:54:21 +00:00
if verifyFinalization in flags and currentEpoch >= 2:
verifyFinalization checks for justification being consistent with later finalization (#1386)
2020-07-28 17:35:32 +00:00
doAssert state.current_justified_checkpoint.epoch + 2 >= currentEpoch
avoid extraneous hash root calculation (#2537) When applying a block, we'll currently compute a state root for the state after slot processing but before block processing - this is unnecessary when a block is being applied because the intermediate state root is never observed.
2021-05-05 06:54:21 +00:00
if verifyFinalization in flags and currentEpoch >= 3:
Increase finalization and finalization checking robustness (#990) * fix some warnings related to beacon_node splitting; reimplement finalization verification more robustly; improve attestation pool block selection logic * re-add missing import * whitelist allowed state transition flags and make rollback/restore naming more consistent * restore usage of update flags passed into skipAndUpdateState(...) in addition to the potential verifyFinalization flag * switch rest of rollback -> restore
2020-05-09 12:43:15 +00:00
# Rule 2/3/4 finalization results in the most pessimal case. The other
# three finalization rules finalize more quickly as long as the any of
# the finalization rules triggered.
more metrics (#1004)
2020-05-11 06:25:49 +00:00
doAssert state.finalized_checkpoint.epoch + 3 >= currentEpoch
Justification/finalization amelioration using testnet1 shard count and epoch length (#331) * working justification, 1 node, 64 validators * closer to tn1 params: 256 validators * more debugging output; switch to minimum test case * working justification and finalization in local network simulation * fix currently incorrect state transition/attestation test assumption
2019-08-14 08:56:32 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#rewards-and-penalties-1
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
process_rewards_and_penalties(state, rewards)
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#registry-updates
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
process_registry_updates(cfg, state, cache)
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#slashings
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
process_slashings(state, rewards.total_balances.current_epoch)
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#final-updates
split state_transition() into slots/block parts and use only block where appropriate (#2630)
2021-06-03 09:42:25 +00:00
process_eth1_data_reset(state)
process_effective_balance_updates(state)
process_slashings_reset(state)
process_randao_mixes_reset(state)
process_historical_roots_update(state)
process_participation_record_updates(state)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-alpha.7/specs/altair/beacon-chain.md#epoch-processing
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
proc process_epoch*(
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
cfg: RuntimeConfig, state: var altair.BeaconState, flags: UpdateFlags,
cache: var StateCache, rewards: var RewardInfo) {.nbench.} =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
let currentEpoch = get_current_epoch(state)
trace "process_epoch",
current_epoch = currentEpoch
init(rewards, state)
when false:
rewards.process_attestations(state, cache)
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
let total_active_balance = state.get_total_active_balance(cache)
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#justification-and-finalization
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
process_justification_and_finalization(state, total_active_balance, flags)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
# state.slot hasn't been incremented yet.
if verifyFinalization in flags and currentEpoch >= 2:
doAssert state.current_justified_checkpoint.epoch + 2 >= currentEpoch
if verifyFinalization in flags and currentEpoch >= 3:
# Rule 2/3/4 finalization results in the most pessimal case. The other
# three finalization rules finalize more quickly as long as the any of
# the finalization rules triggered.
doAssert state.finalized_checkpoint.epoch + 3 >= currentEpoch
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
process_inactivity_updates(cfg, state) # [New in Altair]
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#rewards-and-penalties-1
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
process_rewards_and_penalties(cfg, state, total_active_balance)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#registry-updates
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
process_registry_updates(cfg, state, cache)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#slashings
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
process_slashings(state, total_active_balance)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
process_eth1_data_reset(state)
process_effective_balance_updates(state)
process_slashings_reset(state)
process_randao_mixes_reset(state)
process_historical_roots_update(state)
process_participation_flag_updates(state) # [New in Altair]
process_sync_committee_updates(state) # [New in Altair]