status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth2/beacon_chain/spec/state_transition_epoch.nim

1130 lines
49 KiB
Nim
Raw Normal View History

Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# beacon_chain
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
# Copyright (c) 2018-2022 Status Research & Development GmbH
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Licensed and distributed under either of
replace links to Apache and MIT licenses with HTTPS versions (#592)
2019-11-25 15:30:02 +00:00
# * MIT license (license terms in the root directory or at https://opensource.org/licenses/MIT).
# * Apache v2 license (license terms in the root directory or at https://www.apache.org/licenses/LICENSE-2.0).
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# at your option. This file may not be copied, modified, or distributed except according to those terms.
# State transition - epoch processing, as described in
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#epoch-processing and
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#epoch-processing
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
#
complete re-implmentation of reasonable asymptotic-complexity algorithms from 0.11.3
2020-06-09 09:42:53 +00:00
# The entry point is `process_epoch`, which is at the bottom of this file.
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
#
Datatype cleanup (#1953) * clear up spec todo * test fix * remove unnecessary toSszType * type * one more
2020-11-04 21:52:47 +00:00
# General notes about the code:
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# * Weird styling - the sections taken from the spec use python styling while
# the others use NEP-1 - helps grepping identifiers in spec
Datatype cleanup (#1953) * clear up spec todo * test fix * remove unnecessary toSszType * type * one more
2020-11-04 21:52:47 +00:00
# * When updating the code, add TODO sections to mark where there are clear
# improvements to be made - other than that, keep things similar to spec unless
# motivated by security or performance considerations
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
spec: regulate exceptions (#913) * spec: regulate exceptions * a few more simple raises
2020-04-22 05:53:02 +00:00
{.push raises: [Defect].}
fix network sim finalization; remove get_attestation_data_slot(...); remove 2 more get_crosslink_committee(...) calls
2019-11-12 05:35:52 +00:00
import
test fixture cleanups (#2976) * "official" -> "scenarios", like the submodule * fewer test binaries - various compile hacks have been improved over time, test suite should follow * remove obsolete bls tests - there are better test vectors in nim-blscurve * remove obsolete mentions of `ssz_testing` * remove obsolete comments about proc vs globals, unittest2 already uses proc's
2021-10-12 11:36:52 +00:00
std/[algorithm, math],
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
stew/bitops2, chronicles,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
../extras,
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
./datatypes/[phase0, altair, bellatrix],
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
"."/[beaconstate, eth2_merkleization, helpers, validator]
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
rework spec imports (#2779) The spec imports are a mess to work with, so this branch cleans them up a bit to ensure that we avoid generic sandwitches and that importing stuff generally becomes easier. * reexport crypto/digest/presets because these are part of the public symbol set of the rest of the spec types * don't export `merge` types from `base` - this causes circular deps * fix circular deps in `ssz/spec_types` - this is the first step in disentangling ssz from spec * be explicit about phase0 vs altair - longer term, `altair` will become the "natural" type set, then merge and so on, so no point in giving `phase0` special preferential treatment
2021-08-12 13:08:20 +00:00
export extras, phase0, altair
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
# Logging utilities
# --------------------------------------------------------
logScope: topics = "consens"
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
# Accessors that implement the max condition in `get_total_balance`:
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#get_total_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
template current_epoch*(v: TotalBalances): Gwei =
max(EFFECTIVE_BALANCE_INCREMENT, v.current_epoch_raw)
template previous_epoch*(v: TotalBalances): Gwei =
max(EFFECTIVE_BALANCE_INCREMENT, v.previous_epoch_raw)
template current_epoch_attesters*(v: TotalBalances): Gwei =
max(EFFECTIVE_BALANCE_INCREMENT, v.current_epoch_attesters_raw)
template current_epoch_target_attesters*(v: TotalBalances): Gwei =
max(EFFECTIVE_BALANCE_INCREMENT, v.current_epoch_target_attesters_raw)
template previous_epoch_attesters*(v: TotalBalances): Gwei =
max(EFFECTIVE_BALANCE_INCREMENT, v.previous_epoch_attesters_raw)
template previous_epoch_target_attesters*(v: TotalBalances): Gwei =
max(EFFECTIVE_BALANCE_INCREMENT, v.previous_epoch_target_attesters_raw)
template previous_epoch_head_attesters*(v: TotalBalances): Gwei =
max(EFFECTIVE_BALANCE_INCREMENT, v.previous_epoch_head_attesters_raw)
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
func init*(info: var phase0.EpochInfo, state: phase0.BeaconState) =
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.balances = TotalBalances()
info.validators.setLen(state.validators.len)
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
for i in 0..<state.validators.len:
let v = unsafeAddr state.validators[i]
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
var flags: set[RewardFlags]
if v[].slashed:
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl(RewardFlags.isSlashed)
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
if state.get_current_epoch() >= v[].withdrawable_epoch:
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.canWithdrawInCurrentEpoch
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
if v[].is_active_validator(state.get_current_epoch()):
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.balances.current_epoch_raw += v[].effective_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
if v[].is_active_validator(state.get_previous_epoch()):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.isActiveInPreviousEpoch
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.balances.previous_epoch_raw += v[].effective_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.validators[i] = RewardStatus(
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
current_epoch_effective_balance: v[].effective_balance,
flags: flags,
)
func add(a: var RewardDelta, b: RewardDelta) =
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
a.rewards += b.rewards
a.penalties += b.penalties
func process_attestation(
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
info: var phase0.EpochInfo, state: phase0.BeaconState, a: PendingAttestation,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
cache: var StateCache) =
# Collect information about the attestation
var
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
flags: set[RewardFlags]
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
is_previous_epoch_attester: Option[InclusionInfo]
if a.data.target.epoch == state.get_current_epoch():
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.isCurrentEpochAttester
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if a.data.target.root == get_block_root(state, state.get_current_epoch()):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.isCurrentEpochTargetAttester
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
elif a.data.target.epoch == state.get_previous_epoch():
is_previous_epoch_attester = some(InclusionInfo(
delay: a.inclusion_delay,
proposer_index: a.proposer_index,
))
if a.data.target.root == get_block_root(state, state.get_previous_epoch()):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.isPreviousEpochTargetAttester
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if a.data.beacon_block_root == get_block_root_at_slot(state, a.data.slot):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.isPreviousEpochHeadAttester
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
# Update the cache for all participants
for validator_index in get_attesting_indices(
state, a.data, a.aggregation_bits, cache):
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
template v(): untyped = info.validators[validator_index]
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
v.flags = v.flags + flags
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if is_previous_epoch_attester.isSome:
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
if v.is_previous_epoch_attester.isSome:
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if is_previous_epoch_attester.get().delay <
v.is_previous_epoch_attester.get().delay:
v.is_previous_epoch_attester = is_previous_epoch_attester
else:
v.is_previous_epoch_attester = is_previous_epoch_attester
func process_attestations*(
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
info: var phase0.EpochInfo, state: phase0.BeaconState, cache: var StateCache) =
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
# Walk state attestations and update the status information
for a in state.previous_epoch_attestations:
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
process_attestation(info, state, a, cache)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
for a in state.current_epoch_attestations:
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
process_attestation(info, state, a, cache)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
for idx, v in info.validators:
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if v.flags.contains RewardFlags.isSlashed:
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
continue
let validator_balance = state.validators[idx].effective_balance
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if v.flags.contains RewardFlags.isCurrentEpochAttester:
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.balances.current_epoch_attesters_raw += validator_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if v.flags.contains RewardFlags.isCurrentEpochTargetAttester:
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.balances.current_epoch_target_attesters_raw += validator_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if v.is_previous_epoch_attester.isSome():
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.balances.previous_epoch_attesters_raw += validator_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if v.flags.contains RewardFlags.isPreviousEpochTargetAttester:
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.balances.previous_epoch_target_attesters_raw += validator_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if v.flags.contains RewardFlags.isPreviousEpochHeadAttester:
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.balances.previous_epoch_head_attesters_raw += validator_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
spec URL updates (#3254)
2022-01-06 18:35:38 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.1/specs/phase0/beacon-chain.md#helpers
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
# get_eligible_validator_indices
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
func is_eligible_validator*(validator: RewardStatus): bool =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
validator.flags.contains(RewardFlags.isActiveInPreviousEpoch) or
(validator.flags.contains(RewardFlags.isSlashed) and not
(validator.flags.contains RewardFlags.canWithdrawInCurrentEpoch))
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
func is_eligible_validator*(validator: Validator, previous_epoch: Epoch): bool =
is_active_validator(validator, previous_epoch) or
(validator.slashed and previous_epoch + 1 < validator.withdrawable_epoch)
func is_eligible_validator*(validator: ParticipationInfo): bool =
validator.flags.contains(ParticipationFlag.eligible)
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
# Spec
# --------------------------------------------------------
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#get_unslashed_participating_indices
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#get_total_balance
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
func get_unslashed_participating_balances*(state: altair.BeaconState | bellatrix.BeaconState):
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
UnslashedParticipatingBalances =
let
previous_epoch = get_previous_epoch(state)
current_epoch = get_current_epoch(state)
var res: UnslashedParticipatingBalances
for validator_index in 0'u64 ..< state.validators.lenu64:
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
let
is_active_current_epoch = is_active_validator(
state.validators[validator_index], current_epoch)
validator_effective_balance =
state.validators[validator_index].effective_balance
if is_active_current_epoch:
# Active balance counted also for slashed validators
res.current_epoch += validator_effective_balance
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
if state.validators[validator_index].slashed:
continue
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
let
is_active_previous_epoch = is_active_validator(
state.validators[validator_index], previous_epoch)
previous_epoch_participation =
state.previous_epoch_participation[validator_index]
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
if is_active_previous_epoch:
for flag_index in 0 ..< PARTICIPATION_FLAG_WEIGHTS.len:
if has_flag(previous_epoch_participation, flag_index):
res.previous_epoch[flag_index] += validator_effective_balance
# Only TIMELY_TARGET_FLAG_INDEX is used with the current epoch in Altair
cleanups, partly from kintsugi branch (#3161) * cleanups, partly from kintsugi branch * re-export shortLog(EthBlock) and preserve exception messages in batchVerify and processBatch
2021-12-05 17:32:41 +00:00
# and merge
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
if is_active_current_epoch and has_flag(
state.current_epoch_participation[validator_index],
TIMELY_TARGET_FLAG_INDEX):
res.current_epoch_TIMELY_TARGET += validator_effective_balance
for flag_index in 0 ..< PARTICIPATION_FLAG_WEIGHTS.len:
res.previous_epoch[flag_index] =
max(EFFECTIVE_BALANCE_INCREMENT, res.previous_epoch[flag_index])
res.current_epoch_TIMELY_TARGET =
max(EFFECTIVE_BALANCE_INCREMENT, res.current_epoch_TIMELY_TARGET)
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
res.current_epoch = max(EFFECTIVE_BALANCE_INCREMENT, res.current_epoch)
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
res
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
func is_unslashed_participating_index(
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
state: altair.BeaconState | bellatrix.BeaconState, flag_index: int, epoch: Epoch,
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
validator_index: ValidatorIndex): bool =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
doAssert epoch in [get_previous_epoch(state), get_current_epoch(state)]
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
# TODO hoist this conditional
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
let epoch_participation =
if epoch == get_current_epoch(state):
unsafeAddr state.current_epoch_participation
else:
unsafeAddr state.previous_epoch_participation
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
is_active_validator(state.validators[validator_index], epoch) and
has_flag(epoch_participation[].asSeq()[validator_index], flag_index) and
not state.validators[validator_index].slashed
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#justification-and-finalization
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
proc process_justification_and_finalization*(state: var phase0.BeaconState,
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
balances: TotalBalances, flags: UpdateFlags = {}) =
update 120+ beacon_chain and validator spec refs from v0.12.2 to v0.12.3 (#1740)
2020-09-24 17:04:10 +00:00
# Initial FFG checkpoint values have a `0x00` stub for `root`.
# Skip FFG updates in the first two epochs to avoid corner cases that might
# result in modifying this stub.
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
if get_current_epoch(state) <= GENESIS_EPOCH + 1:
return
let
previous_epoch = get_previous_epoch(state)
current_epoch = get_current_epoch(state)
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
old_previous_justified_checkpoint = state.previous_justified_checkpoint
old_current_justified_checkpoint = state.current_justified_checkpoint
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Process justifications
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
state.previous_justified_checkpoint = state.current_justified_checkpoint
## Spec:
## state.justification_bits[1:] = state.justification_bits[:-1]
## state.justification_bits[0] = 0b0
address some TODOs (#2005)
2020-11-12 19:24:07 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#misc
Fix an overflow in the justification bits processing
2019-09-27 16:40:33 +00:00
const JUSTIFICATION_BITS_LENGTH = 4
address some TODOs (#2005)
2020-11-12 19:24:07 +00:00
REST JSON support improvements (#3232) * support downloading blocks / states via JSON in addition to SSZ - slow, but needed for infura support - SSZ is still used when server supports it * use common forked block/state reader in REST API * fix stack overflows in REST JSON decoder * fix invalid serialization of `justification_bits` in `/eth/v1/debug/beacon/states` and `/eth/v2/debug/beacon/states` * fix REST client to use `/eth/...` instead of `/api/eth/...`, update "default" urls to expose REST api via `/eth` as well as this is what the standard says - `/api` was added early on based on an example "base url" in the spec that has been removed since * expose Nimbus REST extensions via `/nimbus` in addition to `/api/nimbus` to stay consistent with `/eth` * fix invalid state root when reading states via REST * fix recursive imports in `spec/ssz_codec` * remove usages of `serialization.useCustomSerialization` - fickle
2022-01-06 07:38:40 +00:00
state.justification_bits = JustificationBits(
(uint8(state.justification_bits) shl 1) and
uint8((2^JUSTIFICATION_BITS_LENGTH) - 1))
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
let total_active_balance = balances.current_epoch
if balances.previous_epoch_target_attesters * 3 >=
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
total_active_balance * 2:
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
state.current_justified_checkpoint =
Checkpoint(epoch: previous_epoch,
root: get_block_root(state, previous_epoch))
REST JSON support improvements (#3232) * support downloading blocks / states via JSON in addition to SSZ - slow, but needed for infura support - SSZ is still used when server supports it * use common forked block/state reader in REST API * fix stack overflows in REST JSON decoder * fix invalid serialization of `justification_bits` in `/eth/v1/debug/beacon/states` and `/eth/v2/debug/beacon/states` * fix REST client to use `/eth/...` instead of `/api/eth/...`, update "default" urls to expose REST api via `/eth` as well as this is what the standard says - `/api` was added early on based on an example "base url" in the spec that has been removed since * expose Nimbus REST extensions via `/nimbus` in addition to `/api/nimbus` to stay consistent with `/eth` * fix invalid state root when reading states via REST * fix recursive imports in `spec/ssz_codec` * remove usages of `serialization.useCustomSerialization` - fickle
2022-01-06 07:38:40 +00:00
uint8(state.justification_bits).setBit 1
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
small perf fixes * don't sort shuffled_validator_indices, just get them directly with iteration * grab full epoch of proposer indices while we have the data available - they'll get cached and reused * avoid computing active validator set when not used for logging
2020-08-19 08:03:50 +00:00
trace "Justified with previous epoch",
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
current_epoch = current_epoch,
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
checkpoint = shortLog(state.current_justified_checkpoint)
avoid extraneous hash root calculation (#2537) When applying a block, we'll currently compute a state root for the state after slot processing but before block processing - this is unnecessary when a block is being applied because the intermediate state root is never observed.
2021-05-05 06:54:21 +00:00
elif verifyFinalization in flags:
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
warn "Low attestation participation in previous epoch",
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
balances, epoch = get_current_epoch(state)
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
if balances.current_epoch_target_attesters * 3 >=
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
total_active_balance * 2:
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
state.current_justified_checkpoint =
Checkpoint(epoch: current_epoch,
root: get_block_root(state, current_epoch))
REST JSON support improvements (#3232) * support downloading blocks / states via JSON in addition to SSZ - slow, but needed for infura support - SSZ is still used when server supports it * use common forked block/state reader in REST API * fix stack overflows in REST JSON decoder * fix invalid serialization of `justification_bits` in `/eth/v1/debug/beacon/states` and `/eth/v2/debug/beacon/states` * fix REST client to use `/eth/...` instead of `/api/eth/...`, update "default" urls to expose REST api via `/eth` as well as this is what the standard says - `/api` was added early on based on an example "base url" in the spec that has been removed since * expose Nimbus REST extensions via `/nimbus` in addition to `/api/nimbus` to stay consistent with `/eth` * fix invalid state root when reading states via REST * fix recursive imports in `spec/ssz_codec` * remove usages of `serialization.useCustomSerialization` - fickle
2022-01-06 07:38:40 +00:00
uint8(state.justification_bits).setBit 0
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
small perf fixes * don't sort shuffled_validator_indices, just get them directly with iteration * grab full epoch of proposer indices while we have the data available - they'll get cached and reused * avoid computing active validator set when not used for logging
2020-08-19 08:03:50 +00:00
trace "Justified with current epoch",
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
current_epoch = current_epoch,
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
checkpoint = shortLog(state.current_justified_checkpoint)
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Process finalizations
REST JSON support improvements (#3232) * support downloading blocks / states via JSON in addition to SSZ - slow, but needed for infura support - SSZ is still used when server supports it * use common forked block/state reader in REST API * fix stack overflows in REST JSON decoder * fix invalid serialization of `justification_bits` in `/eth/v1/debug/beacon/states` and `/eth/v2/debug/beacon/states` * fix REST client to use `/eth/...` instead of `/api/eth/...`, update "default" urls to expose REST api via `/eth` as well as this is what the standard says - `/api` was added early on based on an example "base url" in the spec that has been removed since * expose Nimbus REST extensions via `/nimbus` in addition to `/api/nimbus` to stay consistent with `/eth` * fix invalid state root when reading states via REST * fix recursive imports in `spec/ssz_codec` * remove usages of `serialization.useCustomSerialization` - fickle
2022-01-06 07:38:40 +00:00
let bitfield = uint8(state.justification_bits)
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
## The 2nd/3rd/4th most recent epochs are justified, the 2nd using the 4th
## as source
Implement the latest SSZ specification and integrate the official SSZ test suite
2019-07-03 07:35:05 +00:00
if (bitfield and 0b1110) == 0b1110 and
old_previous_justified_checkpoint.epoch + 3 == current_epoch:
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
state.finalized_checkpoint = old_previous_justified_checkpoint
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
small perf fixes * don't sort shuffled_validator_indices, just get them directly with iteration * grab full epoch of proposer indices while we have the data available - they'll get cached and reused * avoid computing active validator set when not used for logging
2020-08-19 08:03:50 +00:00
trace "Finalized with rule 234",
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
current_epoch = current_epoch,
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
checkpoint = shortLog(state.finalized_checkpoint)
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
## The 2nd/3rd most recent epochs are justified, the 2nd using the 3rd as
## source
Implement the latest SSZ specification and integrate the official SSZ test suite
2019-07-03 07:35:05 +00:00
if (bitfield and 0b110) == 0b110 and
old_previous_justified_checkpoint.epoch + 2 == current_epoch:
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
state.finalized_checkpoint = old_previous_justified_checkpoint
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
small perf fixes * don't sort shuffled_validator_indices, just get them directly with iteration * grab full epoch of proposer indices while we have the data available - they'll get cached and reused * avoid computing active validator set when not used for logging
2020-08-19 08:03:50 +00:00
trace "Finalized with rule 23",
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
current_epoch = current_epoch,
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
checkpoint = shortLog(state.finalized_checkpoint)
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
## The 1st/2nd/3rd most recent epochs are justified, the 1st using the 3rd as
## source
Implement the latest SSZ specification and integrate the official SSZ test suite
2019-07-03 07:35:05 +00:00
if (bitfield and 0b111) == 0b111 and
old_current_justified_checkpoint.epoch + 2 == current_epoch:
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
state.finalized_checkpoint = old_current_justified_checkpoint
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
small perf fixes * don't sort shuffled_validator_indices, just get them directly with iteration * grab full epoch of proposer indices while we have the data available - they'll get cached and reused * avoid computing active validator set when not used for logging
2020-08-19 08:03:50 +00:00
trace "Finalized with rule 123",
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
current_epoch = current_epoch,
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
checkpoint = shortLog(state.finalized_checkpoint)
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
## The 1st/2nd most recent epochs are justified, the 1st using the 2nd as
## source
Implement the latest SSZ specification and integrate the official SSZ test suite
2019-07-03 07:35:05 +00:00
if (bitfield and 0b11) == 0b11 and
old_current_justified_checkpoint.epoch + 1 == current_epoch:
Add non-attesting-validators instrumentation in process_justification_and_finalization(...) (#323) * update process_justifications_and_finalizations(...) to 0.8.1
2019-07-16 07:34:11 +00:00
state.finalized_checkpoint = old_current_justified_checkpoint
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
small perf fixes * don't sort shuffled_validator_indices, just get them directly with iteration * grab full epoch of proposer indices while we have the data available - they'll get cached and reused * avoid computing active validator set when not used for logging
2020-08-19 08:03:50 +00:00
trace "Finalized with rule 12",
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
current_epoch = current_epoch,
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
checkpoint = shortLog(state.finalized_checkpoint)
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#justification-and-finalization
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#justification-and-finalization
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
# TODO merge these things -- effectively, the phase0 process_justification_and_finalization is mostly a stub in this world
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
proc weigh_justification_and_finalization(state: var (altair.BeaconState | bellatrix.BeaconState),
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
total_active_balance: Gwei,
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
previous_epoch_target_balance: Gwei,
current_epoch_target_balance: Gwei,
flags: UpdateFlags = {}) =
let
previous_epoch = get_previous_epoch(state)
current_epoch = get_current_epoch(state)
old_previous_justified_checkpoint = state.previous_justified_checkpoint
old_current_justified_checkpoint = state.current_justified_checkpoint
# Process justifications
state.previous_justified_checkpoint = state.current_justified_checkpoint
## Spec:
## state.justification_bits[1:] = state.justification_bits[:-1]
## state.justification_bits[0] = 0b0
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#misc
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
const JUSTIFICATION_BITS_LENGTH = 4
REST JSON support improvements (#3232) * support downloading blocks / states via JSON in addition to SSZ - slow, but needed for infura support - SSZ is still used when server supports it * use common forked block/state reader in REST API * fix stack overflows in REST JSON decoder * fix invalid serialization of `justification_bits` in `/eth/v1/debug/beacon/states` and `/eth/v2/debug/beacon/states` * fix REST client to use `/eth/...` instead of `/api/eth/...`, update "default" urls to expose REST api via `/eth` as well as this is what the standard says - `/api` was added early on based on an example "base url" in the spec that has been removed since * expose Nimbus REST extensions via `/nimbus` in addition to `/api/nimbus` to stay consistent with `/eth` * fix invalid state root when reading states via REST * fix recursive imports in `spec/ssz_codec` * remove usages of `serialization.useCustomSerialization` - fickle
2022-01-06 07:38:40 +00:00
state.justification_bits = JustificationBits(
(uint8(state.justification_bits) shl 1) and
uint8((2^JUSTIFICATION_BITS_LENGTH) - 1))
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
if previous_epoch_target_balance * 3 >= total_active_balance * 2:
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
state.current_justified_checkpoint =
Checkpoint(epoch: previous_epoch,
root: get_block_root(state, previous_epoch))
REST JSON support improvements (#3232) * support downloading blocks / states via JSON in addition to SSZ - slow, but needed for infura support - SSZ is still used when server supports it * use common forked block/state reader in REST API * fix stack overflows in REST JSON decoder * fix invalid serialization of `justification_bits` in `/eth/v1/debug/beacon/states` and `/eth/v2/debug/beacon/states` * fix REST client to use `/eth/...` instead of `/api/eth/...`, update "default" urls to expose REST api via `/eth` as well as this is what the standard says - `/api` was added early on based on an example "base url" in the spec that has been removed since * expose Nimbus REST extensions via `/nimbus` in addition to `/api/nimbus` to stay consistent with `/eth` * fix invalid state root when reading states via REST * fix recursive imports in `spec/ssz_codec` * remove usages of `serialization.useCustomSerialization` - fickle
2022-01-06 07:38:40 +00:00
uint8(state.justification_bits).setBit 1
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
trace "Justified with previous epoch",
current_epoch = current_epoch,
checkpoint = shortLog(state.current_justified_checkpoint)
elif verifyFinalization in flags:
warn "Low attestation participation in previous epoch",
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
total_active_balance,
previous_epoch_target_balance,
current_epoch_target_balance,
epoch = get_current_epoch(state)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
if current_epoch_target_balance * 3 >= total_active_balance * 2:
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
state.current_justified_checkpoint =
Checkpoint(epoch: current_epoch,
root: get_block_root(state, current_epoch))
REST JSON support improvements (#3232) * support downloading blocks / states via JSON in addition to SSZ - slow, but needed for infura support - SSZ is still used when server supports it * use common forked block/state reader in REST API * fix stack overflows in REST JSON decoder * fix invalid serialization of `justification_bits` in `/eth/v1/debug/beacon/states` and `/eth/v2/debug/beacon/states` * fix REST client to use `/eth/...` instead of `/api/eth/...`, update "default" urls to expose REST api via `/eth` as well as this is what the standard says - `/api` was added early on based on an example "base url" in the spec that has been removed since * expose Nimbus REST extensions via `/nimbus` in addition to `/api/nimbus` to stay consistent with `/eth` * fix invalid state root when reading states via REST * fix recursive imports in `spec/ssz_codec` * remove usages of `serialization.useCustomSerialization` - fickle
2022-01-06 07:38:40 +00:00
uint8(state.justification_bits).setBit 0
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
trace "Justified with current epoch",
current_epoch = current_epoch,
checkpoint = shortLog(state.current_justified_checkpoint)
# Process finalizations
REST JSON support improvements (#3232) * support downloading blocks / states via JSON in addition to SSZ - slow, but needed for infura support - SSZ is still used when server supports it * use common forked block/state reader in REST API * fix stack overflows in REST JSON decoder * fix invalid serialization of `justification_bits` in `/eth/v1/debug/beacon/states` and `/eth/v2/debug/beacon/states` * fix REST client to use `/eth/...` instead of `/api/eth/...`, update "default" urls to expose REST api via `/eth` as well as this is what the standard says - `/api` was added early on based on an example "base url" in the spec that has been removed since * expose Nimbus REST extensions via `/nimbus` in addition to `/api/nimbus` to stay consistent with `/eth` * fix invalid state root when reading states via REST * fix recursive imports in `spec/ssz_codec` * remove usages of `serialization.useCustomSerialization` - fickle
2022-01-06 07:38:40 +00:00
let bitfield = uint8(state.justification_bits)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
## The 2nd/3rd/4th most recent epochs are justified, the 2nd using the 4th
## as source
if (bitfield and 0b1110) == 0b1110 and
old_previous_justified_checkpoint.epoch + 3 == current_epoch:
state.finalized_checkpoint = old_previous_justified_checkpoint
trace "Finalized with rule 234",
current_epoch = current_epoch,
checkpoint = shortLog(state.finalized_checkpoint)
## The 2nd/3rd most recent epochs are justified, the 2nd using the 3rd as
## source
if (bitfield and 0b110) == 0b110 and
old_previous_justified_checkpoint.epoch + 2 == current_epoch:
state.finalized_checkpoint = old_previous_justified_checkpoint
trace "Finalized with rule 23",
current_epoch = current_epoch,
checkpoint = shortLog(state.finalized_checkpoint)
## The 1st/2nd/3rd most recent epochs are justified, the 1st using the 3rd as
## source
if (bitfield and 0b111) == 0b111 and
old_current_justified_checkpoint.epoch + 2 == current_epoch:
state.finalized_checkpoint = old_current_justified_checkpoint
trace "Finalized with rule 123",
current_epoch = current_epoch,
checkpoint = shortLog(state.finalized_checkpoint)
## The 1st/2nd most recent epochs are justified, the 1st using the 2nd as
## source
if (bitfield and 0b11) == 0b11 and
old_current_justified_checkpoint.epoch + 1 == current_epoch:
state.finalized_checkpoint = old_current_justified_checkpoint
trace "Finalized with rule 12",
current_epoch = current_epoch,
checkpoint = shortLog(state.finalized_checkpoint)
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
proc process_justification_and_finalization*(state: var (altair.BeaconState | bellatrix.BeaconState),
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
balances: UnslashedParticipatingBalances,
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
flags: UpdateFlags = {}) =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
# Initial FFG checkpoint values have a `0x00` stub for `root`.
# Skip FFG updates in the first two epochs to avoid corner cases that might
# result in modifying this stub.
if get_current_epoch(state) <= GENESIS_EPOCH + 1:
return
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
# These ultimately differ from phase0 only in these lines, with the phase 0
# version effectively embedding weigh_justification_and_finalization(), for
# historical reasons.
# https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.2/specs/phase0/beacon-chain.md#justification-and-finalization
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
weigh_justification_and_finalization(
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
state, balances.current_epoch,
balances.previous_epoch[TIMELY_TARGET_FLAG_INDEX],
balances.current_epoch_TIMELY_TARGET, flags)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#helpers
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
func get_base_reward_sqrt*(state: phase0.BeaconState, index: ValidatorIndex,
improve slot processing speeds (#1670) about 40% better slot processing times (with LTO enabled) - these don't do BLS but are used heavily during replay (state transition = slot + block transition) tests using a recent medalla state and advancing it 1000 slots: ``` ./ncli slots --preState2:state-302271-3c1dbf19-c1f944bf.ssz --slot:1000 --postState2:xx.ssz ``` pre: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 39.236, 0.000, 39.236, 39.236, 1, Load state from file 0.049, 0.002, 0.046, 0.063, 968, Apply slot 256.504, 81.008, 213.471, 591.902, 32, Apply epoch slot 28.597, 0.000, 28.597, 28.597, 1, Save state to file ``` cast: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 37.079, 0.000, 37.079, 37.079, 1, Load state from file 0.042, 0.002, 0.040, 0.090, 968, Apply slot 215.552, 68.763, 180.155, 500.103, 32, Apply epoch slot 25.106, 0.000, 25.106, 25.106, 1, Save state to file ``` cast+rewards: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 40.049, 0.000, 40.049, 40.049, 1, Load state from file 0.048, 0.001, 0.045, 0.060, 968, Apply slot 164.981, 76.273, 142.099, 477.868, 32, Apply epoch slot 28.498, 0.000, 28.498, 28.498, 1, Save state to file ``` cast+rewards+shr ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 12.898, 0.000, 12.898, 12.898, 1, Load state from file 0.039, 0.002, 0.038, 0.054, 968, Apply slot 139.971, 68.797, 120.088, 428.844, 32, Apply epoch slot 24.761, 0.000, 24.761, 24.761, 1, Save state to file ```
2020-09-16 20:59:33 +00:00
total_balance_sqrt: auto): Gwei =
verify that state_sim is justifying and finalizing; fix 3 more warnings; rename crosslink_committee_cache to beacon_committee_cache; fix O(n^2) usage of get_base_reward(...)
2019-11-15 22:37:39 +00:00
# Spec function recalculates total_balance every time, which creates an
# O(n^2) situation.
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
let effective_balance = state.validators.asSeq()[index].effective_balance
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
effective_balance * BASE_REWARD_FACTOR div
improve slot processing speeds (#1670) about 40% better slot processing times (with LTO enabled) - these don't do BLS but are used heavily during replay (state transition = slot + block transition) tests using a recent medalla state and advancing it 1000 slots: ``` ./ncli slots --preState2:state-302271-3c1dbf19-c1f944bf.ssz --slot:1000 --postState2:xx.ssz ``` pre: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 39.236, 0.000, 39.236, 39.236, 1, Load state from file 0.049, 0.002, 0.046, 0.063, 968, Apply slot 256.504, 81.008, 213.471, 591.902, 32, Apply epoch slot 28.597, 0.000, 28.597, 28.597, 1, Save state to file ``` cast: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 37.079, 0.000, 37.079, 37.079, 1, Load state from file 0.042, 0.002, 0.040, 0.090, 968, Apply slot 215.552, 68.763, 180.155, 500.103, 32, Apply epoch slot 25.106, 0.000, 25.106, 25.106, 1, Save state to file ``` cast+rewards: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 40.049, 0.000, 40.049, 40.049, 1, Load state from file 0.048, 0.001, 0.045, 0.060, 968, Apply slot 164.981, 76.273, 142.099, 477.868, 32, Apply epoch slot 28.498, 0.000, 28.498, 28.498, 1, Save state to file ``` cast+rewards+shr ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 12.898, 0.000, 12.898, 12.898, 1, Load state from file 0.039, 0.002, 0.038, 0.054, 968, Apply slot 139.971, 68.797, 120.088, 428.844, 32, Apply epoch slot 24.761, 0.000, 24.761, 24.761, 1, Save state to file ```
2020-09-16 20:59:33 +00:00
total_balance_sqrt div BASE_REWARDS_PER_EPOCH
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func get_proposer_reward*(base_reward: Gwei): Gwei =
allow 50k validators on block_sim by avoiding repeated get_total_active_balance() in get_proposer_reward()
2020-07-07 16:52:12 +00:00
# Spec version recalculates get_total_active_balance(state) quadratically
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward div PROPOSER_REWARD_QUOTIENT
func is_in_inactivity_leak(finality_delay: uint64): bool =
finality_delay > MIN_EPOCHS_TO_INACTIVITY_PENALTY
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func get_finality_delay*(state: ForkyBeaconState): uint64 =
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
get_previous_epoch(state) - state.finalized_checkpoint.epoch
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func get_attestation_component_reward*(attesting_balance: Gwei,
total_balance: Gwei,
base_reward: uint64,
finality_delay: uint64): Gwei =
if is_in_inactivity_leak(finality_delay):
# Since full base reward will be canceled out by inactivity penalty deltas,
# optimal participation receives full base reward compensation here.
base_reward
else:
let reward_numerator =
base_reward * (attesting_balance div EFFECTIVE_BALANCE_INCREMENT)
reward_numerator div (total_balance div EFFECTIVE_BALANCE_INCREMENT)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
func get_attestation_component_delta(is_unslashed_attester: bool,
attesting_balance: Gwei,
total_balance: Gwei,
base_reward: uint64,
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
finality_delay: uint64): RewardDelta =
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
# Helper with shared logic for use by get source, target, and head deltas
# functions
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if is_unslashed_attester:
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
RewardDelta(rewards: get_attestation_component_reward(
attesting_balance,
total_balance,
base_reward,
finality_delay))
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
else:
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
RewardDelta(penalties: base_reward)
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
spec URL updates (#3254)
2022-01-06 18:35:38 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.1/specs/phase0/beacon-chain.md#components-of-attestation-deltas
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
func get_source_delta*(validator: RewardStatus,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward: uint64,
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
balances: TotalBalances,
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
finality_delay: uint64): RewardDelta =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Return attester micro-rewards/penalties for source-vote for each validator.
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
get_attestation_component_delta(
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
validator.is_previous_epoch_attester.isSome() and
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
not (validator.flags.contains RewardFlags.isSlashed),
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
balances.previous_epoch_attesters,
balances.current_epoch,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward,
finality_delay)
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
func get_target_delta*(validator: RewardStatus,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward: uint64,
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
balances: TotalBalances,
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
finality_delay: uint64): RewardDelta =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Return attester micro-rewards/penalties for target-vote for each validator.
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
get_attestation_component_delta(
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
validator.flags.contains(RewardFlags.isPreviousEpochTargetAttester) and
not (validator.flags.contains(RewardFlags.isSlashed)),
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
balances.previous_epoch_target_attesters,
balances.current_epoch,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward,
finality_delay)
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
func get_head_delta*(validator: RewardStatus,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward: uint64,
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
balances: TotalBalances,
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
finality_delay: uint64): RewardDelta =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Return attester micro-rewards/penalties for head-vote for each validator.
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
get_attestation_component_delta(
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
validator.flags.contains(RewardFlags.isPreviousEpochHeadAttester) and
((not validator.flags.contains(RewardFlags.isSlashed))),
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
balances.previous_epoch_head_attesters,
balances.current_epoch,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward,
finality_delay)
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
func get_inclusion_delay_delta*(validator: RewardStatus,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward: uint64):
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
(RewardDelta, Option[(uint64, RewardDelta)]) =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Return proposer and inclusion delay micro-rewards/penalties for each validator.
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if validator.is_previous_epoch_attester.isSome() and ((not validator.flags.contains(RewardFlags.isSlashed))):
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
let
inclusion_info = validator.is_previous_epoch_attester.get()
proposer_reward = get_proposer_reward(base_reward)
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
proposer_delta = RewardDelta(rewards: proposer_reward)
fix attestation handling bottlenecks in slot processing and quadratic-time inclusion delay rewards/palties in epoch processing; create infrastructure for dynamically (un)subscribing to/from attestation topics (#1294)
2020-07-10 09:24:04 +00:00
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
let
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
max_attester_reward = base_reward - proposer_reward
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
delta = RewardDelta(rewards: max_attester_reward div inclusion_info.delay)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
proposer_index = inclusion_info.proposer_index;
return (delta, some((proposer_index, proposer_delta)))
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
func get_inactivity_penalty_delta*(validator: RewardStatus,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward: Gwei,
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
finality_delay: uint64): RewardDelta =
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
## Return inactivity reward/penalty deltas for each validator.
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
var delta: RewardDelta
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if is_in_inactivity_leak(finality_delay):
# If validator is performing optimally this cancels all rewards for a neutral balance
delta.penalties +=
BASE_REWARDS_PER_EPOCH * base_reward - get_proposer_reward(base_reward)
# Additionally, all validators whose FFG target didn't match are penalized extra
# This condition is equivalent to this condition from the spec:
# `index not in get_unslashed_attesting_indices(state, matching_target_attestations)`
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if (validator.flags.contains(RewardFlags.isSlashed)) or
((not validator.flags.contains(RewardFlags.isPreviousEpochTargetAttester))):
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
delta.penalties +=
validator.current_epoch_effective_balance * finality_delay div
INACTIVITY_PENALTY_QUOTIENT
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
delta
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#get_attestation_deltas
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
func get_attestation_deltas(state: phase0.BeaconState, info: var phase0.EpochInfo) =
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
## Update rewards with attestation reward/penalty deltas for each validator.
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
remove v0.11.3 support; add block_sim to CI (#1253) * remove v0.11.3 support; add block_sim to CI * rm stray PERSISTENT_COMMITTEE_PERIOD * remove TopicFilter.InteropAttestations * bump two comment-spec-refs to v0.12.1
2020-06-29 18:08:58 +00:00
let
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
finality_delay = get_finality_delay(state)
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
total_balance = info.balances.current_epoch
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
total_balance_sqrt = integer_squareroot(total_balance)
# Filter out ineligible validators. All sub-functions of the spec do this
# except for `get_inclusion_delay_deltas`. It's safe to do so here because
# any validator that is in the unslashed indices of the matching source
# attestations is active, and therefore eligible.
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
for index, validator in info.validators.mpairs():
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if not is_eligible_validator(validator):
continue
let
base_reward = get_base_reward_sqrt(
state, index.ValidatorIndex, total_balance_sqrt)
improve slot processing speeds (#1670) about 40% better slot processing times (with LTO enabled) - these don't do BLS but are used heavily during replay (state transition = slot + block transition) tests using a recent medalla state and advancing it 1000 slots: ``` ./ncli slots --preState2:state-302271-3c1dbf19-c1f944bf.ssz --slot:1000 --postState2:xx.ssz ``` pre: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 39.236, 0.000, 39.236, 39.236, 1, Load state from file 0.049, 0.002, 0.046, 0.063, 968, Apply slot 256.504, 81.008, 213.471, 591.902, 32, Apply epoch slot 28.597, 0.000, 28.597, 28.597, 1, Save state to file ``` cast: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 37.079, 0.000, 37.079, 37.079, 1, Load state from file 0.042, 0.002, 0.040, 0.090, 968, Apply slot 215.552, 68.763, 180.155, 500.103, 32, Apply epoch slot 25.106, 0.000, 25.106, 25.106, 1, Save state to file ``` cast+rewards: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 40.049, 0.000, 40.049, 40.049, 1, Load state from file 0.048, 0.001, 0.045, 0.060, 968, Apply slot 164.981, 76.273, 142.099, 477.868, 32, Apply epoch slot 28.498, 0.000, 28.498, 28.498, 1, Save state to file ``` cast+rewards+shr ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 12.898, 0.000, 12.898, 12.898, 1, Load state from file 0.039, 0.002, 0.038, 0.054, 968, Apply slot 139.971, 68.797, 120.088, 428.844, 32, Apply epoch slot 24.761, 0.000, 24.761, 24.761, 1, Save state to file ```
2020-09-16 20:59:33 +00:00
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
let
source_delta = get_source_delta(
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
validator, base_reward, info.balances, finality_delay)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
target_delta = get_target_delta(
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
validator, base_reward, info.balances, finality_delay)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
head_delta = get_head_delta(
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
validator, base_reward, info.balances, finality_delay)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
(inclusion_delay_delta, proposer_delta) =
get_inclusion_delay_delta(validator, base_reward)
inactivity_delta = get_inactivity_penalty_delta(
validator, base_reward, finality_delay)
validator.delta.add source_delta
validator.delta.add target_delta
validator.delta.add head_delta
validator.delta.add inclusion_delay_delta
validator.delta.add inactivity_delta
if proposer_delta.isSome:
let proposer_index = proposer_delta.get()[0]
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
if proposer_index < info.validators.lenu64:
info.validators[proposer_index].delta.add(
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
proposer_delta.get()[1])
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#get_base_reward
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func get_base_reward_increment*(
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
state: altair.BeaconState | bellatrix.BeaconState, index: ValidatorIndex,
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
base_reward_per_increment: Gwei): Gwei =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
## Return the base reward for the validator defined by ``index`` with respect
## to the current ``state``.
let increments =
state.validators[index].effective_balance div EFFECTIVE_BALANCE_INCREMENT
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
increments * base_reward_per_increment
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#get_flag_index_deltas
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func get_flag_index_reward*(state: altair.BeaconState | bellatrix.BeaconState,
base_reward: Gwei, active_increments: Gwei,
unslashed_participating_increments: Gwei,
[ncli_db] bugfix: take into account finalization delay in reward calc post Altair This fixes a problem affecting Prater's epoch 64444.
2022-01-27 16:37:30 +00:00
weight, finality_delay: uint64): Gwei =
if not is_in_inactivity_leak(finality_delay):
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
let reward_numerator =
base_reward * weight * unslashed_participating_increments
reward_numerator div (active_increments * WEIGHT_DENOMINATOR)
else:
0.Gwei
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#get_flag_index_deltas
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func get_unslashed_participating_increment*(
info: altair.EpochInfo, flag_index: int): Gwei =
info.balances.previous_epoch[flag_index] div EFFECTIVE_BALANCE_INCREMENT
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#get_flag_index_deltas
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func get_active_increments*(info: altair.EpochInfo): Gwei =
info.balances.current_epoch div EFFECTIVE_BALANCE_INCREMENT
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#get_flag_index_deltas
add Altair and Merge rewards tests (#2995)
2021-10-18 14:54:17 +00:00
iterator get_flag_index_deltas*(
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
state: altair.BeaconState | bellatrix.BeaconState, flag_index: int,
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
base_reward_per_increment: Gwei,
[ncli_db] bugfix: take into account finalization delay in reward calc post Altair This fixes a problem affecting Prater's epoch 64444.
2022-01-27 16:37:30 +00:00
info: var altair.EpochInfo,
finality_delay: uint64):
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
(ValidatorIndex, RewardDelta) =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
## Return the deltas for a given ``flag_index`` by scanning through the
## participation flags.
let
previous_epoch = get_previous_epoch(state)
weight = PARTICIPATION_FLAG_WEIGHTS[flag_index].uint64 # safe
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
unslashed_participating_increments = get_unslashed_participating_increment(
info, flag_index)
active_increments = get_active_increments(info)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
for index in 0 ..< state.validators.len:
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
if not is_eligible_validator(info.validators[index]):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
continue
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
template vidx: ValidatorIndex = index.ValidatorIndex
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
let base_reward = get_base_reward_increment(state, vidx, base_reward_per_increment)
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
yield
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
if is_unslashed_participating_index(
state, flag_index, previous_epoch, vidx):
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
let pflag = case flag_index
of TIMELY_SOURCE_FLAG_INDEX: ParticipationFlag.timelySourceAttester
of TIMELY_TARGET_FLAG_INDEX: ParticipationFlag.timelyTargetAttester
of TIMELY_HEAD_FLAG_INDEX: ParticipationFlag.timelyHeadAttester
else: raiseAssert "Unknown flag index " & $flag_index
info.validators[vidx].flags.incl pflag
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
(vidx, RewardDelta(
rewards: get_flag_index_reward(
state, base_reward, active_increments,
[ncli_db] bugfix: take into account finalization delay in reward calc post Altair This fixes a problem affecting Prater's epoch 64444.
2022-01-27 16:37:30 +00:00
unslashed_participating_increments, weight, finality_delay),
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
penalties: 0.Gwei))
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
elif flag_index != TIMELY_HEAD_FLAG_INDEX:
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
(vidx, RewardDelta(
rewards: 0.Gwei,
penalties: base_reward * weight div WEIGHT_DENOMINATOR))
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
else:
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
(vidx, RewardDelta(rewards: 0.Gwei, penalties: 0.Gwei))
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#modified-get_inactivity_penalty_deltas
add Altair and Merge rewards tests (#2995)
2021-10-18 14:54:17 +00:00
iterator get_inactivity_penalty_deltas*(
use v1.1.4 test vectors
2021-11-05 16:49:03 +00:00
cfg: RuntimeConfig, state: altair.BeaconState, info: altair.EpochInfo):
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
(ValidatorIndex, Gwei) =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
## Return the inactivity penalty deltas by considering timely target
## participation flags and inactivity scores.
let
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
penalty_denominator =
cfg.INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_ALTAIR
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
previous_epoch = get_previous_epoch(state)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
for index in 0 ..< state.validators.len:
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
if not is_eligible_validator(info.validators[index]):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
continue
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
template vidx: untyped = index.ValidatorIndex
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
if not is_unslashed_participating_index(
state, TIMELY_TARGET_FLAG_INDEX, previous_epoch, vidx):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
let
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
penalty_numerator = state.validators[index].effective_balance *
state.inactivity_scores[index]
yield (vidx, Gwei(penalty_numerator div penalty_denominator))
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
spec URL updates (#3342)
2022-01-31 09:56:59 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/bellatrix/beacon-chain.md#modified-get_inactivity_penalty_deltas
use v1.1.4 test vectors
2021-11-05 16:49:03 +00:00
iterator get_inactivity_penalty_deltas*(
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
cfg: RuntimeConfig, state: bellatrix.BeaconState, info: altair.EpochInfo):
use v1.1.4 test vectors
2021-11-05 16:49:03 +00:00
(ValidatorIndex, Gwei) =
## Return the inactivity penalty deltas by considering timely target
## participation flags and inactivity scores.
let
rename *_{MERGE => BELLATRIX} constant names (#3296)
2022-01-18 16:31:05 +00:00
# [Modified in Bellatrix]
use v1.1.4 test vectors
2021-11-05 16:49:03 +00:00
penalty_denominator =
rename *_{MERGE => BELLATRIX} constant names (#3296)
2022-01-18 16:31:05 +00:00
cfg.INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT_BELLATRIX
use v1.1.4 test vectors
2021-11-05 16:49:03 +00:00
previous_epoch = get_previous_epoch(state)
for index in 0 ..< state.validators.len:
if not is_eligible_validator(info.validators[index]):
continue
template vidx: untyped = index.ValidatorIndex
if not is_unslashed_participating_index(
state, TIMELY_TARGET_FLAG_INDEX, previous_epoch, vidx):
let
penalty_numerator = state.validators[index].effective_balance *
state.inactivity_scores[index]
yield (vidx, Gwei(penalty_numerator div penalty_denominator))
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#rewards-and-penalties-1
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
func process_rewards_and_penalties(
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
state: var phase0.BeaconState, info: var phase0.EpochInfo) =
update 120+ beacon_chain and validator spec refs from v0.12.2 to v0.12.3 (#1740)
2020-09-24 17:04:10 +00:00
# No rewards are applied at the end of `GENESIS_EPOCH` because rewards are
# for work done in the previous epoch
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
doAssert info.validators.len == state.validators.len
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
if get_current_epoch(state) == GENESIS_EPOCH:
return
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
get_attestation_deltas(state, info)
fix attestation selection when calculating rewards and penalties Previously, the we'd use the first source attestation regardless if it matches the filter or not leading to the wrong minimum being used
2019-10-01 14:40:41 +00:00
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
# Here almost all balances are updated (assuming most validators are active) -
# clearing the cache becomes a bottleneck if done item by item because of the
# recursive nature of cache clearing - instead, we clear the whole cache then
# update the raw list directly
state.balances.clearCache()
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
for idx, v in info.validators:
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
var balance = state.balances.asSeq()[idx]
increase_balance(balance, v.delta.rewards)
decrease_balance(balance, v.delta.penalties)
state.balances.asSeq()[idx] = balance
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#rewards-and-penalties
split state_transition() into slots/block parts and use only block where appropriate (#2630)
2021-06-03 09:42:25 +00:00
func process_rewards_and_penalties(
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
cfg: RuntimeConfig, state: var (altair.BeaconState | bellatrix.BeaconState),
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
info: var altair.EpochInfo)
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
=
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if get_current_epoch(state) == GENESIS_EPOCH:
return
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
let
total_active_balance = info.balances.current_epoch
base_reward_per_increment = get_base_reward_per_increment(
total_active_balance)
[ncli_db] bugfix: take into account finalization delay in reward calc post Altair This fixes a problem affecting Prater's epoch 64444.
2022-01-27 16:37:30 +00:00
finality_delay = get_finality_delay(state)
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
doAssert state.validators.len() == info.validators.len()
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
for flag_index in 0 ..< PARTICIPATION_FLAG_WEIGHTS.len:
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
for validator_index, delta in get_flag_index_deltas(
[ncli_db] bugfix: take into account finalization delay in reward calc post Altair This fixes a problem affecting Prater's epoch 64444.
2022-01-27 16:37:30 +00:00
state, flag_index, base_reward_per_increment, info, finality_delay):
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
info.validators[validator_index].delta.add(delta)
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
for validator_index, penalty in get_inactivity_penalty_deltas(
cfg, state, info):
info.validators[validator_index].delta.penalties += penalty
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
# Here almost all balances are updated (assuming most validators are active) -
# clearing the cache becomes a bottleneck if done item by item because of the
# recursive nature of cache clearing - instead, we clear the whole cache then
# update the raw list directly
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
state.balances.clearCache()
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
for index in 0 ..< len(state.validators):
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
var balance = state.balances.asSeq()[index]
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
increase_balance(balance, info.validators[index].delta.rewards)
decrease_balance(balance, info.validators[index].delta.penalties)
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
state.balances.asSeq()[index] = balance
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#registry-updates
test fixture cleanups (#2976) * "official" -> "scenarios", like the submodule * fewer test binaries - various compile hacks have been improved over time, test suite should follow * remove obsolete bls tests - there are better test vectors in nim-blscurve * remove obsolete mentions of `ssz_testing` * remove obsolete comments about proc vs globals, unittest2 already uses proc's
2021-10-12 11:36:52 +00:00
func process_registry_updates*(
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
cfg: RuntimeConfig, state: var ForkyBeaconState, cache: var StateCache) =
test fixture cleanups (#2976) * "official" -> "scenarios", like the submodule * fewer test binaries - various compile hacks have been improved over time, test suite should follow * remove obsolete bls tests - there are better test vectors in nim-blscurve * remove obsolete mentions of `ssz_testing` * remove obsolete comments about proc vs globals, unittest2 already uses proc's
2021-10-12 11:36:52 +00:00
## Process activation eligibility and ejections
# Make visible, e.g.,
# https://github.com/status-im/nimbus-eth2/pull/608
# https://github.com/sigp/lighthouse/pull/657
let epoch {.used.} = get_current_epoch(state)
trace "process_registry_updates validator balances",
balances=state.balances,
active_validator_indices=get_active_validator_indices(state, epoch),
epoch=epoch
# is_active_validator(...) is activation_epoch <= epoch < exit_epoch,
# and changes here to either activation_epoch or exit_epoch only take
# effect with a compute_activation_exit_epoch(...) delay of, based on
# the current epoch, 1 + MAX_SEED_LOOKAHEAD epochs ahead. Thus caches
# remain valid for this epoch through though this function along with
# the rest of the epoch transition.
for index in 0..<state.validators.len():
if is_eligible_for_activation_queue(state.validators.asSeq()[index]):
state.validators[index].activation_eligibility_epoch =
get_current_epoch(state) + 1
if is_active_validator(state.validators.asSeq()[index], get_current_epoch(state)) and
state.validators.asSeq()[index].effective_balance <= cfg.EJECTION_BALANCE:
initiate_validator_exit(cfg, state, index.ValidatorIndex, cache)
## Queue validators eligible for activation and not dequeued for activation
var activation_queue : seq[tuple[a: Epoch, b: int]] = @[]
for index in 0..<state.validators.len():
let validator = unsafeAddr state.validators.asSeq()[index]
if is_eligible_for_activation(state, validator[]):
activation_queue.add (
validator[].activation_eligibility_epoch, index)
activation_queue.sort(system.cmp)
## Dequeued validators for activation up to churn limit (without resetting
## activation epoch)
let churn_limit = get_validator_churn_limit(cfg, state, cache)
for i, epoch_and_index in activation_queue:
if i.uint64 >= churn_limit:
break
let
(_, index) = epoch_and_index
state.validators[index].activation_epoch =
compute_activation_exit_epoch(get_current_epoch(state))
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#slashings
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#slashings
spec URL updates (#3342)
2022-01-31 09:56:59 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/bellatrix/beacon-chain.md#slashings
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func get_adjusted_total_slashing_balance*(
state: ForkyBeaconState, total_balance: Gwei): Gwei =
let multiplier =
# tradeoff here about interleaving phase0/altair, but for these
# single-constant changes...
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
when state is phase0.BeaconState:
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
PROPORTIONAL_SLASHING_MULTIPLIER
elif state is altair.BeaconState:
PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR
elif state is bellatrix.BeaconState:
rename *_{MERGE => BELLATRIX} constant names (#3296)
2022-01-18 16:31:05 +00:00
PROPORTIONAL_SLASHING_MULTIPLIER_BELLATRIX
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
else:
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
{.fatal: "process_slashings: incorrect BeaconState type".}
min(sum(state.slashings.data) * multiplier, total_balance)
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#slashings
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#slashings
spec URL updates (#3342)
2022-01-31 09:56:59 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/bellatrix/beacon-chain.md#slashings
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func slashing_penalty_applies*(validator: Validator, epoch: Epoch): bool =
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
validator.slashed and
epoch + EPOCHS_PER_SLASHINGS_VECTOR div 2 == validator.withdrawable_epoch
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#slashings
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#slashings
spec URL updates (#3342)
2022-01-31 09:56:59 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/bellatrix/beacon-chain.md#slashings
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func get_slashing_penalty*(validator: Validator,
adjusted_total_slashing_balance,
total_balance: Gwei): Gwei =
const increment = EFFECTIVE_BALANCE_INCREMENT # Factored out from penalty
# numerator to avoid uint64 overflow
let penalty_numerator = validator.effective_balance div increment *
adjusted_total_slashing_balance
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
penalty_numerator div total_balance * increment
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#slashings
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#slashings
spec URL updates (#3342)
2022-01-31 09:56:59 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/bellatrix/beacon-chain.md#slashings
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
func process_slashings*(state: var ForkyBeaconState, total_balance: Gwei) =
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
let
More 0.8.0 updates (#311) * replace BeaconState.finalized_{epoch,root} with BeaconState.finalized_checkpoint; rename get_delayed_activation_exit_epoch(...) to compute_activation_exit_epoch(...) and mark as 0.8.0; update get_churn_limit(...)/get_validator_churn_limit(...) to 0.8.0; update process_registry_updates(...) to 0.8.0 * update process_crosslinks(...) to 0.8.0; mark compute_start_slot_of_epoch(...) and get_committee_count(...) as 0.8.0 * mark Fork, is_slashable_validator(...), and get_beacon_proposer_index(...) as 0.8.0 * rename LATEST_SLASHED_EXIT_LENGTH to EPOCHS_PER_SLASHINGS_VECTOR; update process_slashings(...) to 0.8.0; remove pointless type conversion warning in get_previous_epoch(...) * convert remaining references to finalized_epoch to finalized_checkpoint.epoch * update slash_validator(...) to 0.8.0; mark inital value, Gwei, and time constants as 0.8.0; mark hash(...) and processBlockHeader(...) as 0.8.0 * rename WHISTLEBLOWING_REWARD_QUOTIENT to WHISTLEBLOWER_REWARD_QUOTIENT; rename LATEST_ACTIVE_INDEX_ROOTS_LENGTH to EPOCHS_PER_HISTORICAL_VECTOR (randao will also get merged into this); remove get_active_index_root(...); mark time parameter, signature domain types, and max operations per block constants as 0.8.0; update rewards and penalties constants to 0.8.0 * update is_valid_indexed_attestation(...) to 0.8.0; mark process_slot(...) as 0.8.0 * replace BeaconState.{current,previous}_justified_{epoch,root} with BeaconState.{current,previous}_justified_checkpoint
2019-07-05 08:30:05 +00:00
epoch = get_current_epoch(state)
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
adjusted_total_slashing_balance = get_adjusted_total_slashing_balance(
state, total_balance)
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
for index in 0..<state.validators.len:
let validator = unsafeAddr state.validators.asSeq()[index]
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
if slashing_penalty_applies(validator[], epoch):
let penalty = validator[].get_slashing_penalty(
adjusted_total_slashing_balance, total_balance)
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
decrease_balance(state, index.ValidatorIndex, penalty)
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#eth1-data-votes-updates
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
func process_eth1_data_reset*(state: var ForkyBeaconState) =
per HF1, split process_final_updates() (#2319)
2021-02-14 19:31:01 +00:00
let next_epoch = get_current_epoch(state) + 1
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Reset eth1 data votes
initial 0.11.0 spec version update
2020-03-14 21:54:45 +00:00
if next_epoch mod EPOCHS_PER_ETH1_VOTING_PERIOD == 0:
SSZ cleanup * be stricter about SSZ length prefix * compute zeroHash list at compile time * remove SSZ schema stuff * move SSZ navigation to ncli * cleanup a few leftover openArray uses
2020-05-27 11:36:02 +00:00
state.eth1_data_votes = default(type state.eth1_data_votes)
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#effective-balances-updates
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
func process_effective_balance_updates*(state: var ForkyBeaconState) =
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Update effective balances with hysteresis
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
for index in 0..<state.validators.len:
let balance = state.balances.asSeq()[index]
initial 0.11.0 spec version update
2020-03-14 21:54:45 +00:00
const
HYSTERESIS_INCREMENT =
EFFECTIVE_BALANCE_INCREMENT div HYSTERESIS_QUOTIENT
DOWNWARD_THRESHOLD =
HYSTERESIS_INCREMENT * HYSTERESIS_DOWNWARD_MULTIPLIER
UPWARD_THRESHOLD = HYSTERESIS_INCREMENT * HYSTERESIS_UPWARD_MULTIPLIER
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
let effective_balance = state.validators.asSeq()[index].effective_balance
if balance + DOWNWARD_THRESHOLD < effective_balance or
effective_balance + UPWARD_THRESHOLD < balance:
speed up epoch processing 6x+ (#3089) * speed up epoch processing 6x+ This change above all helps contain long replay times on epoch change, reorg and deep history inspection via REST/RPC * most effective balances don't actually change due to MAX_EFFECTIVE_BALANCE * ditto for inactivity scores * avoid signature check for trusted sync aggregates pre: ``` ./ncli_db --db:mainnet_0/db bench --start-slot=-3200 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3468.621, 0.000, 3468.621, 3468.621, 1, Initialize DB 0.357, 0.938, 0.171, 52.752, 3155, Load block from database 15691.471, 0.000, 15691.471, 15691.471, 1, Load state from database 6.100, 9.469, 0.033, 526.816, 3101, Advance slot, non-epoch 579.131, 9.523, 566.936, 610.328, 100, Advance slot, epoch 18.551, 16.317, 12.664, 136.668, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` post: ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3488.541, 0.000, 3488.541, 3488.541, 1, Initialize DB 0.369, 1.123, 0.183, 63.208, 3155, Load block from database 13430.642, 0.000, 13430.642, 13430.642, 1, Load state from database 6.522, 1.721, 0.034, 36.708, 3101, Advance slot, non-epoch 89.074, 3.162, 83.573, 101.436, 100, Advance slot, epoch 18.325, 18.346, 13.005, 145.040, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * Update beacon_chain/spec/state_transition_block.nim Co-authored-by: zah <zahary@gmail.com> * avoid copying validator data in accessor ``` 5291.227, 0.000, 5291.227, 5291.227, 1, Initialize DB 0.436, 0.928, 0.138, 51.438, 3155, Load block from database 11962.826, 0.000, 11962.826, 11962.826, 1, Load state from database 6.477, 1.675, 0.037, 34.174, 3101, Advance slot, non-epoch 76.633, 3.705, 71.106, 98.085, 100, Advance slot, epoch 18.301, 18.593, 13.208, 149.153, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * work around compiler bug Co-authored-by: zah <zahary@gmail.com>
2021-11-11 19:24:29 +00:00
let new_effective_balance =
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
min(
balance - balance mod EFFECTIVE_BALANCE_INCREMENT,
MAX_EFFECTIVE_BALANCE)
speed up epoch processing 6x+ (#3089) * speed up epoch processing 6x+ This change above all helps contain long replay times on epoch change, reorg and deep history inspection via REST/RPC * most effective balances don't actually change due to MAX_EFFECTIVE_BALANCE * ditto for inactivity scores * avoid signature check for trusted sync aggregates pre: ``` ./ncli_db --db:mainnet_0/db bench --start-slot=-3200 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3468.621, 0.000, 3468.621, 3468.621, 1, Initialize DB 0.357, 0.938, 0.171, 52.752, 3155, Load block from database 15691.471, 0.000, 15691.471, 15691.471, 1, Load state from database 6.100, 9.469, 0.033, 526.816, 3101, Advance slot, non-epoch 579.131, 9.523, 566.936, 610.328, 100, Advance slot, epoch 18.551, 16.317, 12.664, 136.668, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` post: ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3488.541, 0.000, 3488.541, 3488.541, 1, Initialize DB 0.369, 1.123, 0.183, 63.208, 3155, Load block from database 13430.642, 0.000, 13430.642, 13430.642, 1, Load state from database 6.522, 1.721, 0.034, 36.708, 3101, Advance slot, non-epoch 89.074, 3.162, 83.573, 101.436, 100, Advance slot, epoch 18.325, 18.346, 13.005, 145.040, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * Update beacon_chain/spec/state_transition_block.nim Co-authored-by: zah <zahary@gmail.com> * avoid copying validator data in accessor ``` 5291.227, 0.000, 5291.227, 5291.227, 1, Initialize DB 0.436, 0.928, 0.138, 51.438, 3155, Load block from database 11962.826, 0.000, 11962.826, 11962.826, 1, Load state from database 6.477, 1.675, 0.037, 34.174, 3101, Advance slot, non-epoch 76.633, 3.705, 71.106, 98.085, 100, Advance slot, epoch 18.301, 18.593, 13.208, 149.153, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * work around compiler bug Co-authored-by: zah <zahary@gmail.com>
2021-11-11 19:24:29 +00:00
# Protect against unnecessary cache invalidation
if new_effective_balance != effective_balance:
state.validators[index].effective_balance = new_effective_balance
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#slashings-balances-updates
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
func process_slashings_reset*(state: var ForkyBeaconState) =
per HF1, split process_final_updates() (#2319)
2021-02-14 19:31:01 +00:00
let next_epoch = get_current_epoch(state) + 1
0.8.0, continued (#315) * rename get_genesis_beacon_state(...) to initialize_beacon_state_from_eth1(...); remove unused vestiges get_temporary_block_header(...) and get_empty_block(...); partly align initialize_beacon_state_from_eth1(...) with 0.8.0 version; implement CompactCommittee object type; update process_final_updates(...) to 0.8.0 * mark get_shard_delta(...) as 0.8.0 * mark get_total_active_balance(...) and epoch transition helper functions as 0.8.0 * move FORK_CHOICE_BALANCE_INCREMENT, not in spec anymore, to sole user in fork_choice
2019-07-10 12:23:02 +00:00
# Reset slashings
Small fixes (#1165) * random fixes * create dump dir on startup * don't crash on failure to write dump * fix a few `uint64` instances being used when indexing arrays - this should be a compile error but isn't due to compiler bugs * fix standalone test_block_pool compilation * add signed block processing in ncli * reuse cache entry instead of allocating a new one * allow for small clock disparities when validating blocks
2020-06-12 16:43:20 +00:00
state.slashings[int(next_epoch mod EPOCHS_PER_SLASHINGS_VECTOR)] = 0.Gwei
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#randao-mixes-updates
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
func process_randao_mixes_reset*(state: var ForkyBeaconState) =
per HF1, split process_final_updates() (#2319)
2021-02-14 19:31:01 +00:00
let
current_epoch = get_current_epoch(state)
next_epoch = current_epoch + 1
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Set randao mix
Fix #378, int to bytes32, LATEST_RANDAO_MIXES, get_seed, get_crosslink_committee, get_compact_committee_root (#380)
2019-09-04 13:57:18 +00:00
state.randao_mixes[next_epoch mod EPOCHS_PER_HISTORICAL_VECTOR] =
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
get_randao_mix(state, current_epoch)
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#historical-roots-updates
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
func process_historical_roots_update*(state: var ForkyBeaconState) =
update 62 spec URLs to v1.1.2 (#2979)
2021-10-12 10:17:37 +00:00
## Set historical root accumulator
per HF1, split process_final_updates() (#2319)
2021-02-14 19:31:01 +00:00
let next_epoch = get_current_epoch(state) + 1
Restore compilation with custom presets (#1309) This also assigns precise types to the constants in the minimal and mainnet presets in order to reduce the chance of compilation errors when custom presets are used (previously, only the custom presets have precisely assigned types for the constants).
2020-07-13 14:44:58 +00:00
if next_epoch mod (SLOTS_PER_HISTORICAL_ROOT div SLOTS_PER_EPOCH) == 0:
reduce stack space usage in process_final_updates(...) (#925) * reduce stack space usage in process_final_updates(...) to avoid fuzzed segfault in https://github.com/status-im/nim-beacon-chain/issues/921 * document motivation behind manually constructing hash_tree_root of a HistoricalBatch
2020-04-23 11:39:38 +00:00
# Equivalent to hash_tree_root(foo: HistoricalBatch), but without using
# significant additional stack or heap.
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#historicalbatch
update nim-beacon-chain to nimbus-eth2 in beacon_chain/, ncli/, tests/, and README.md (#1843)
2020-10-08 19:02:05 +00:00
# In response to https://github.com/status-im/nimbus-eth2/issues/921
Reset cached indices when resetting cache on SSZ read (#2480) * Reset cached indices when resetting cache on SSZ read When deserializing into an existing structure, the cache should be cleared - goes for json also. Also improve error messages.
2021-04-08 10:11:04 +00:00
if not state.historical_roots.add hash_tree_root(
[hash_tree_root(state.block_roots), hash_tree_root(state.state_roots)]):
raiseAssert "no more room for historical roots, so long and thanks for the fish!"
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#participation-records-rotation
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
func process_participation_record_updates*(state: var phase0.BeaconState) =
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
# Rotate current/previous epoch attestations - using swap avoids copying all
# elements using a slow genericSeqAssign
state.previous_epoch_attestations.clear()
swap(state.previous_epoch_attestations, state.current_epoch_attestations)
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#participation-flags-updates
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
func process_participation_flag_updates*(state: var (altair.BeaconState | bellatrix.BeaconState)) =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
state.previous_epoch_participation = state.current_epoch_participation
block_clearance, ncli, and ncli_db Altair state saving (#2672) * block_clearance, ncli, and ncli_db Altair state saving * avoid invalidating SSZ hash caches with every assignment
2021-06-24 18:34:08 +00:00
const zero = 0.ParticipationFlags
for i in 0 ..< state.current_epoch_participation.len:
state.current_epoch_participation.data[i] = zero
# Shouldn't be wasted zeroing, because state.current_epoch_participation only
# grows. New elements are automatically initialized to 0, as required.
doAssert state.current_epoch_participation.data.setLen(state.validators.len)
state.current_epoch_participation.resetCache()
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#sync-committee-updates
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
proc process_sync_committee_updates*(state: var (altair.BeaconState | bellatrix.BeaconState)) =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
let next_epoch = get_current_epoch(state) + 1
time: spring cleaning (#3262) Time in the beacon chain is expressed relative to the genesis time - this PR creates a `beacon_time` module that collects helpers and utilities for dealing the time units - the new module does not deal with actual wall time (that's remains in `beacon_clock`). Collecting the time related stuff in one place makes it easier to find, avoids some circular imports and allows more easily identifying the code actually needs wall time to operate. * move genesis-time-related functionality into `spec/beacon_time` * avoid using `chronos.Duration` for time differences - it does not support negative values (such as when something happens earlier than it should) * saturate conversions between `FAR_FUTURE_XXX`, so as to avoid overflows * fix delay reporting in validator client so it uses the expected deadline of the slot, not "closest wall slot" * simplify looping over the slots of an epoch * `compute_start_slot_at_epoch` -> `start_slot` * `compute_epoch_at_slot` -> `epoch` A follow-up PR will (likely) introduce saturating arithmetic for the time units - this is merely code moves, renames and fixing of small bugs.
2022-01-11 10:01:54 +00:00
if next_epoch.is_sync_committee_period():
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
state.current_sync_committee = state.next_sync_committee
state.next_sync_committee = get_next_sync_committee(state)
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#inactivity-scores
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
func process_inactivity_updates*(
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
cfg: RuntimeConfig, state: var (altair.BeaconState | bellatrix.BeaconState),
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
info: altair.EpochInfo) =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
# Score updates based on previous epoch participation, skip genesis epoch
if get_current_epoch(state) == GENESIS_EPOCH:
return
remove false OnBlockAdded dependency on phase0 HashedBeaconState (#2661) * remove false OnBlockAdded dependency on phase.HashedBeaconState * introduce altair data types into block_clearance; update some alpha.6 spec refs to alpha.7; add get_active_validator_indices_len ForkedHashedBeaconState wrapper * switch many modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 and remove rm'd G2_POINT_AT_INFINITY * switch more modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 * remove unnecessary phase0-only wrapper of get_attesting_indices(); allow signatures_batch to process either fork; remove O(n^2) nested loop in process_inactivity_updates(); add altair support to getAttestationsforTestBlock() * add Altair versions of asSigVerified(), asTrusted(), and makeBeaconBlock() * fix spec URL to be Altair for Altair makeBeaconBlock()
2021-06-21 08:35:24 +00:00
let
previous_epoch = get_previous_epoch(state) # get_eligible_validator_indices()
[ncli_db] bugfix: take into account finalization delay in reward calc post Altair This fixes a problem affecting Prater's epoch 64444.
2022-01-27 16:37:30 +00:00
finality_delay = get_finality_delay(state)
not_in_inactivity_leak = not is_in_inactivity_leak(finality_delay)
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
for index in 0'u64 ..< state.validators.lenu64:
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
if not is_eligible_validator(info.validators[index]):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
continue
remove false OnBlockAdded dependency on phase0 HashedBeaconState (#2661) * remove false OnBlockAdded dependency on phase.HashedBeaconState * introduce altair data types into block_clearance; update some alpha.6 spec refs to alpha.7; add get_active_validator_indices_len ForkedHashedBeaconState wrapper * switch many modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 and remove rm'd G2_POINT_AT_INFINITY * switch more modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 * remove unnecessary phase0-only wrapper of get_attesting_indices(); allow signatures_batch to process either fork; remove O(n^2) nested loop in process_inactivity_updates(); add altair support to getAttestationsforTestBlock() * add Altair versions of asSigVerified(), asTrusted(), and makeBeaconBlock() * fix spec URL to be Altair for Altair makeBeaconBlock()
2021-06-21 08:35:24 +00:00
# Increase the inactivity score of inactive validators
speed up epoch processing 6x+ (#3089) * speed up epoch processing 6x+ This change above all helps contain long replay times on epoch change, reorg and deep history inspection via REST/RPC * most effective balances don't actually change due to MAX_EFFECTIVE_BALANCE * ditto for inactivity scores * avoid signature check for trusted sync aggregates pre: ``` ./ncli_db --db:mainnet_0/db bench --start-slot=-3200 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3468.621, 0.000, 3468.621, 3468.621, 1, Initialize DB 0.357, 0.938, 0.171, 52.752, 3155, Load block from database 15691.471, 0.000, 15691.471, 15691.471, 1, Load state from database 6.100, 9.469, 0.033, 526.816, 3101, Advance slot, non-epoch 579.131, 9.523, 566.936, 610.328, 100, Advance slot, epoch 18.551, 16.317, 12.664, 136.668, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` post: ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3488.541, 0.000, 3488.541, 3488.541, 1, Initialize DB 0.369, 1.123, 0.183, 63.208, 3155, Load block from database 13430.642, 0.000, 13430.642, 13430.642, 1, Load state from database 6.522, 1.721, 0.034, 36.708, 3101, Advance slot, non-epoch 89.074, 3.162, 83.573, 101.436, 100, Advance slot, epoch 18.325, 18.346, 13.005, 145.040, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * Update beacon_chain/spec/state_transition_block.nim Co-authored-by: zah <zahary@gmail.com> * avoid copying validator data in accessor ``` 5291.227, 0.000, 5291.227, 5291.227, 1, Initialize DB 0.436, 0.928, 0.138, 51.438, 3155, Load block from database 11962.826, 0.000, 11962.826, 11962.826, 1, Load state from database 6.477, 1.675, 0.037, 34.174, 3101, Advance slot, non-epoch 76.633, 3.705, 71.106, 98.085, 100, Advance slot, epoch 18.301, 18.593, 13.208, 149.153, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * work around compiler bug Co-authored-by: zah <zahary@gmail.com>
2021-11-11 19:24:29 +00:00
let pre_inactivity_score = state.inactivity_scores.asSeq()[index]
var inactivity_score = pre_inactivity_score
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
# TODO activeness already checked; remove redundant checks between
# is_active_validator and is_unslashed_participating_index
if is_unslashed_participating_index(
state, TIMELY_TARGET_FLAG_INDEX, previous_epoch, index.ValidatorIndex):
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
inactivity_score -= min(1'u64, inactivity_score)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
else:
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
inactivity_score += cfg.INACTIVITY_SCORE_BIAS
remove false OnBlockAdded dependency on phase0 HashedBeaconState (#2661) * remove false OnBlockAdded dependency on phase.HashedBeaconState * introduce altair data types into block_clearance; update some alpha.6 spec refs to alpha.7; add get_active_validator_indices_len ForkedHashedBeaconState wrapper * switch many modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 and remove rm'd G2_POINT_AT_INFINITY * switch more modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 * remove unnecessary phase0-only wrapper of get_attesting_indices(); allow signatures_batch to process either fork; remove O(n^2) nested loop in process_inactivity_updates(); add altair support to getAttestationsforTestBlock() * add Altair versions of asSigVerified(), asTrusted(), and makeBeaconBlock() * fix spec URL to be Altair for Altair makeBeaconBlock()
2021-06-21 08:35:24 +00:00
# Decrease the inactivity score of all eligible validators during a
# leak-free epoch
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
if not_in_inactivity_leak:
inactivity_score -= min(INACTIVITY_SCORE_RECOVERY_RATE.uint64, inactivity_score)
speed up epoch processing 6x+ (#3089) * speed up epoch processing 6x+ This change above all helps contain long replay times on epoch change, reorg and deep history inspection via REST/RPC * most effective balances don't actually change due to MAX_EFFECTIVE_BALANCE * ditto for inactivity scores * avoid signature check for trusted sync aggregates pre: ``` ./ncli_db --db:mainnet_0/db bench --start-slot=-3200 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3468.621, 0.000, 3468.621, 3468.621, 1, Initialize DB 0.357, 0.938, 0.171, 52.752, 3155, Load block from database 15691.471, 0.000, 15691.471, 15691.471, 1, Load state from database 6.100, 9.469, 0.033, 526.816, 3101, Advance slot, non-epoch 579.131, 9.523, 566.936, 610.328, 100, Advance slot, epoch 18.551, 16.317, 12.664, 136.668, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` post: ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3488.541, 0.000, 3488.541, 3488.541, 1, Initialize DB 0.369, 1.123, 0.183, 63.208, 3155, Load block from database 13430.642, 0.000, 13430.642, 13430.642, 1, Load state from database 6.522, 1.721, 0.034, 36.708, 3101, Advance slot, non-epoch 89.074, 3.162, 83.573, 101.436, 100, Advance slot, epoch 18.325, 18.346, 13.005, 145.040, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * Update beacon_chain/spec/state_transition_block.nim Co-authored-by: zah <zahary@gmail.com> * avoid copying validator data in accessor ``` 5291.227, 0.000, 5291.227, 5291.227, 1, Initialize DB 0.436, 0.928, 0.138, 51.438, 3155, Load block from database 11962.826, 0.000, 11962.826, 11962.826, 1, Load state from database 6.477, 1.675, 0.037, 34.174, 3101, Advance slot, non-epoch 76.633, 3.705, 71.106, 98.085, 100, Advance slot, epoch 18.301, 18.593, 13.208, 149.153, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * work around compiler bug Co-authored-by: zah <zahary@gmail.com>
2021-11-11 19:24:29 +00:00
# Most inactivity scores remain at 0 - avoid invalidating cache
if pre_inactivity_score != inactivity_score:
state.inactivity_scores[index] = inactivity_score
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#epoch-processing
avoid extraneous hash root calculation (#2537) When applying a block, we'll currently compute a state root for the state after slot processing but before block processing - this is unnecessary when a block is being applied because the intermediate state root is never observed.
2021-05-05 06:54:21 +00:00
proc process_epoch*(
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
cfg: RuntimeConfig, state: var phase0.BeaconState, flags: UpdateFlags,
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
cache: var StateCache, info: var phase0.EpochInfo) =
more metrics (#1004)
2020-05-11 06:25:49 +00:00
let currentEpoch = get_current_epoch(state)
restore network sim finalization * relax attestation validation when attestation is incoming but make it more strict when adding to block * share attestation validation logic between attestation pool and state transition * remove a bunch of redundant logging * fix potential underflow in attestation delay checking * fix committee used for attestation shard selection when attesting * fix attestation data construction
2019-08-19 16:41:13 +00:00
trace "process_epoch",
more metrics (#1004)
2020-05-11 06:25:49 +00:00
current_epoch = currentEpoch
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
init(info, state)
info.process_attestations(state, cache)
Justification/finalization amelioration using testnet1 shard count and epoch length (#331) * working justification, 1 node, 64 validators * closer to tn1 params: 256 validators * more debugging output; switch to minimum test case * working justification and finalization in local network simulation * fix currently incorrect state transition/attestation test assumption
2019-08-14 08:56:32 +00:00
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
process_justification_and_finalization(state, info.balances, flags)
Increase finalization and finalization checking robustness (#990) * fix some warnings related to beacon_node splitting; reimplement finalization verification more robustly; improve attestation pool block selection logic * re-add missing import * whitelist allowed state transition flags and make rollback/restore naming more consistent * restore usage of update flags passed into skipAndUpdateState(...) in addition to the potential verifyFinalization flag * switch rest of rollback -> restore
2020-05-09 12:43:15 +00:00
# state.slot hasn't been incremented yet.
avoid extraneous hash root calculation (#2537) When applying a block, we'll currently compute a state root for the state after slot processing but before block processing - this is unnecessary when a block is being applied because the intermediate state root is never observed.
2021-05-05 06:54:21 +00:00
if verifyFinalization in flags and currentEpoch >= 2:
verifyFinalization checks for justification being consistent with later finalization (#1386)
2020-07-28 17:35:32 +00:00
doAssert state.current_justified_checkpoint.epoch + 2 >= currentEpoch
avoid extraneous hash root calculation (#2537) When applying a block, we'll currently compute a state root for the state after slot processing but before block processing - this is unnecessary when a block is being applied because the intermediate state root is never observed.
2021-05-05 06:54:21 +00:00
if verifyFinalization in flags and currentEpoch >= 3:
Increase finalization and finalization checking robustness (#990) * fix some warnings related to beacon_node splitting; reimplement finalization verification more robustly; improve attestation pool block selection logic * re-add missing import * whitelist allowed state transition flags and make rollback/restore naming more consistent * restore usage of update flags passed into skipAndUpdateState(...) in addition to the potential verifyFinalization flag * switch rest of rollback -> restore
2020-05-09 12:43:15 +00:00
# Rule 2/3/4 finalization results in the most pessimal case. The other
# three finalization rules finalize more quickly as long as the any of
# the finalization rules triggered.
more metrics (#1004)
2020-05-11 06:25:49 +00:00
doAssert state.finalized_checkpoint.epoch + 3 >= currentEpoch
Justification/finalization amelioration using testnet1 shard count and epoch length (#331) * working justification, 1 node, 64 validators * closer to tn1 params: 256 validators * more debugging output; switch to minimum test case * working justification and finalization in local network simulation * fix currently incorrect state transition/attestation test assumption
2019-08-14 08:56:32 +00:00
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
process_rewards_and_penalties(state, info)
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
process_registry_updates(cfg, state, cache)
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
process_slashings(state, info.balances.current_epoch)
split state_transition() into slots/block parts and use only block where appropriate (#2630)
2021-06-03 09:42:25 +00:00
process_eth1_data_reset(state)
process_effective_balance_updates(state)
process_slashings_reset(state)
process_randao_mixes_reset(state)
process_historical_roots_update(state)
process_participation_record_updates(state)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
func init*(
info: var altair.EpochInfo,
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
state: altair.BeaconState | bellatrix.BeaconState) =
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
# init participation, overwriting the full structure
info.balances = get_unslashed_participating_balances(state)
info.validators.setLen(state.validators.len())
let previous_epoch = get_previous_epoch(state)
for index in 0..<state.validators.len():
var flags: set[ParticipationFlag]
if is_eligible_validator(state.validators[index], previous_epoch):
flags.incl ParticipationFlag.eligible
info.validators[index] = ParticipationInfo(
flags: flags
)
func init*(
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
T: type altair.EpochInfo, state: altair.BeaconState | bellatrix.BeaconState): T =
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
init(result, state)
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/altair/beacon-chain.md#epoch-processing
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
proc process_epoch*(
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
cfg: RuntimeConfig, state: var (altair.BeaconState | bellatrix.BeaconState),
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
flags: UpdateFlags, cache: var StateCache, info: var altair.EpochInfo)
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
=
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
let currentEpoch = get_current_epoch(state)
trace "process_epoch",
current_epoch = currentEpoch
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
info.init(state)
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#justification-and-finalization
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
process_justification_and_finalization(state, info.balances, flags)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
# state.slot hasn't been incremented yet.
if verifyFinalization in flags and currentEpoch >= 2:
doAssert state.current_justified_checkpoint.epoch + 2 >= currentEpoch
if verifyFinalization in flags and currentEpoch >= 3:
# Rule 2/3/4 finalization results in the most pessimal case. The other
# three finalization rules finalize more quickly as long as the any of
# the finalization rules triggered.
doAssert state.finalized_checkpoint.epoch + 3 >= currentEpoch
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
process_inactivity_updates(cfg, state, info) # [New in Altair]
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
spec URL updates (#3254)
2022-01-06 18:35:38 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.1/specs/phase0/beacon-chain.md#rewards-and-penalties-1
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
process_rewards_and_penalties(cfg, state, info)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
phase 0 & altair beacon chain and altair validator spec URL updates (#3339)
2022-01-29 13:53:31 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.9/specs/phase0/beacon-chain.md#registry-updates
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
process_registry_updates(cfg, state, cache)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.0.1/specs/phase0/beacon-chain.md#slashings
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
process_slashings(state, info.balances.current_epoch)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
process_eth1_data_reset(state)
process_effective_balance_updates(state)
process_slashings_reset(state)
process_randao_mixes_reset(state)
process_historical_roots_update(state)
process_participation_flag_updates(state) # [New in Altair]
process_sync_committee_updates(state) # [New in Altair]