status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth2/beacon_chain/spec/state_transition_epoch.nim

1510 lines
64 KiB
Nim
Raw Normal View History

Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# beacon_chain
verify `genesis_time` more strictly (fixes #1667) (#5694) Bogus values lead to crashes down the line when timers overflow
2024-01-06 14:26:56 +00:00
# Copyright (c) 2018-2024 Status Research & Development GmbH
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Licensed and distributed under either of
replace links to Apache and MIT licenses with HTTPS versions (#592)
2019-11-25 15:30:02 +00:00
# * MIT license (license terms in the root directory or at https://opensource.org/licenses/MIT).
# * Apache v2 license (license terms in the root directory or at https://www.apache.org/licenses/LICENSE-2.0).
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# at your option. This file may not be copied, modified, or distributed except according to those terms.
add `{.push raises: [].}` to recently modified files (#5908) Status Nim style mandates `{.push raises: []}.` at start of modules. Ensure that's the case so that exceptions are properly tracked. - https://status-im.github.io/nim-style-guide/errors.exceptions.html - https://github.com/status-im/nim-eth/pull/614#discussion_r1220906149
2024-02-18 01:16:49 +00:00
{.push raises: [].}
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# State transition - epoch processing, as described in
automated consensus spec URL updating to v1.4.0-beta.6 (#5804)
2024-01-20 11:19:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.6/specs/phase0/beacon-chain.md#epoch-processing
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#epoch-processing
automated consensus spec URL updating to v1.5.0-alpha.3 (#6364)
2024-06-16 00:59:25 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.3/specs/bellatrix/beacon-chain.md#epoch-processing
automatic update of v1.3.0-rc.5 consensus spec URLs to v1.3.0 (#4848)
2023-04-21 18:52:43 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.3.0/specs/capella/beacon-chain.md#epoch-processing
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
#
complete re-implmentation of reasonable asymptotic-complexity algorithms from 0.11.3
2020-06-09 09:42:53 +00:00
# The entry point is `process_epoch`, which is at the bottom of this file.
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
#
Datatype cleanup (#1953) * clear up spec todo * test fix * remove unnecessary toSszType * type * one more
2020-11-04 21:52:47 +00:00
# General notes about the code:
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# * Weird styling - the sections taken from the spec use python styling while
# the others use NEP-1 - helps grepping identifiers in spec
Datatype cleanup (#1953) * clear up spec todo * test fix * remove unnecessary toSszType * type * one more
2020-11-04 21:52:47 +00:00
# * When updating the code, add TODO sections to mark where there are clear
# improvements to be made - other than that, keep things similar to spec unless
# motivated by security or performance considerations
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
fix network sim finalization; remove get_attestation_data_slot(...); remove 2 more get_crosslink_committee(...) calls
2019-11-12 05:35:52 +00:00
import
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
stew/assign2, chronicles,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
../extras,
optimize epoch transition via get_flag_index_deltas() and get_inactivity_penalty_deltas() (#5404) * optimize epoch transition via get_flag_index_deltas() and get_inactivity_penalty_deltas() * directly mutate Altair info in {get,update}_flag_and_inactivity_deltas
2023-09-07 11:14:52 +00:00
"."/[beaconstate, eth2_merkleization, validator]
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
eip4844 light client tests; avoid case object out-of-bound array reads (#4404)
2022-12-08 16:21:53 +00:00
from std/math import sum, `^`
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
from stew/bitops2 import setBit
use v1.3.0-rc.0 consensus spec test vectors (#4472)
2023-01-09 14:15:43 +00:00
from ./datatypes/capella import
BeaconState, HistoricalSummary, Withdrawal, WithdrawalIndex
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
rework spec imports (#2779) The spec imports are a mess to work with, so this branch cleans them up a bit to ensure that we avoid generic sandwitches and that importing stuff generally becomes easier. * reexport crypto/digest/presets because these are part of the public symbol set of the rest of the spec types * don't export `merge` types from `base` - this causes circular deps * fix circular deps in `ssz/spec_types` - this is the first step in disentangling ssz from spec * be explicit about phase0 vs altair - longer term, `altair` will become the "natural" type set, then merge and so on, so no point in giving `phase0` special preferential treatment
2021-08-12 13:08:20 +00:00
export extras, phase0, altair
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
# Logging utilities
# --------------------------------------------------------
logScope: topics = "consens"
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
# Accessors that implement the max condition in `get_total_balance`:
some consensus spec v1.4.0 spec URL updates (#6208)
2024-04-17 03:51:16 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/phase0/beacon-chain.md#get_total_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
template current_epoch*(v: TotalBalances): Gwei =
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
max(EFFECTIVE_BALANCE_INCREMENT.Gwei, v.current_epoch_raw)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
template previous_epoch*(v: TotalBalances): Gwei =
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
max(EFFECTIVE_BALANCE_INCREMENT.Gwei, v.previous_epoch_raw)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
template current_epoch_attesters*(v: TotalBalances): Gwei =
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
max(EFFECTIVE_BALANCE_INCREMENT.Gwei, v.current_epoch_attesters_raw)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
template current_epoch_target_attesters*(v: TotalBalances): Gwei =
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
max(EFFECTIVE_BALANCE_INCREMENT.Gwei, v.current_epoch_target_attesters_raw)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
template previous_epoch_attesters*(v: TotalBalances): Gwei =
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
max(EFFECTIVE_BALANCE_INCREMENT.Gwei, v.previous_epoch_attesters_raw)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
template previous_epoch_target_attesters*(v: TotalBalances): Gwei =
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
max(EFFECTIVE_BALANCE_INCREMENT.Gwei, v.previous_epoch_target_attesters_raw)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
template previous_epoch_head_attesters*(v: TotalBalances): Gwei =
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
max(EFFECTIVE_BALANCE_INCREMENT.Gwei, v.previous_epoch_head_attesters_raw)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
func init*(info: var phase0.EpochInfo, state: phase0.BeaconState) =
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.balances = TotalBalances()
info.validators.setLen(state.validators.len)
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
for i in 0..<state.validators.len:
let v = unsafeAddr state.validators[i]
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
var flags: set[RewardFlags]
if v[].slashed:
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl(RewardFlags.isSlashed)
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
if state.get_current_epoch() >= v[].withdrawable_epoch:
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.canWithdrawInCurrentEpoch
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
if v[].is_active_validator(state.get_current_epoch()):
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.balances.current_epoch_raw += v[].effective_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
if v[].is_active_validator(state.get_previous_epoch()):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.isActiveInPreviousEpoch
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.balances.previous_epoch_raw += v[].effective_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.validators[i] = RewardStatus(
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
current_epoch_effective_balance: v[].effective_balance,
flags: flags,
)
func add(a: var RewardDelta, b: RewardDelta) =
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
a.rewards += b.rewards
a.penalties += b.penalties
func process_attestation(
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
info: var phase0.EpochInfo, state: phase0.BeaconState, a: PendingAttestation,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
cache: var StateCache) =
# Collect information about the attestation
var
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
flags: set[RewardFlags]
spec: Option -> Opt (#4488)
2023-01-11 12:29:21 +00:00
is_previous_epoch_attester: Opt[InclusionInfo]
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if a.data.target.epoch == state.get_current_epoch():
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.isCurrentEpochAttester
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if a.data.target.root == get_block_root(state, state.get_current_epoch()):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.isCurrentEpochTargetAttester
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
elif a.data.target.epoch == state.get_previous_epoch():
spec: Option -> Opt (#4488)
2023-01-11 12:29:21 +00:00
is_previous_epoch_attester = Opt.some(InclusionInfo(
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
delay: a.inclusion_delay,
proposer_index: a.proposer_index,
))
if a.data.target.root == get_block_root(state, state.get_previous_epoch()):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.isPreviousEpochTargetAttester
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if a.data.beacon_block_root == get_block_root_at_slot(state, a.data.slot):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags.incl RewardFlags.isPreviousEpochHeadAttester
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
# Update the cache for all participants
reduce memory allocations during state transition (#5235) This PR removes a few hundred thousand temporary seq allocations during state transition - in particular, the flag seq was allocated per validator while committees are computed per attestation.
2023-08-02 23:03:40 +00:00
for validator_index in get_attesting_indices_iter(
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
state, a.data, a.aggregation_bits, cache):
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
template v(): untyped = info.validators[validator_index]
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
v.flags = v.flags + flags
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if is_previous_epoch_attester.isSome:
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
if v.is_previous_epoch_attester.isSome:
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if is_previous_epoch_attester.get().delay <
v.is_previous_epoch_attester.get().delay:
v.is_previous_epoch_attester = is_previous_epoch_attester
else:
v.is_previous_epoch_attester = is_previous_epoch_attester
func process_attestations*(
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
info: var phase0.EpochInfo, state: phase0.BeaconState, cache: var StateCache) =
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
# Walk state attestations and update the status information
for a in state.previous_epoch_attestations:
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
process_attestation(info, state, a, cache)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
for a in state.current_epoch_attestations:
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
process_attestation(info, state, a, cache)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
for idx, v in info.validators:
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if v.flags.contains RewardFlags.isSlashed:
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
continue
let validator_balance = state.validators[idx].effective_balance
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if v.flags.contains RewardFlags.isCurrentEpochAttester:
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.balances.current_epoch_attesters_raw += validator_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if v.flags.contains RewardFlags.isCurrentEpochTargetAttester:
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.balances.current_epoch_target_attesters_raw += validator_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if v.is_previous_epoch_attester.isSome():
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.balances.previous_epoch_attesters_raw += validator_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if v.flags.contains RewardFlags.isPreviousEpochTargetAttester:
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.balances.previous_epoch_target_attesters_raw += validator_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if v.flags.contains RewardFlags.isPreviousEpochHeadAttester:
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
info.balances.previous_epoch_head_attesters_raw += validator_balance
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
automated consensus spec URL updating to v1.4.0-beta.6 (#5804)
2024-01-20 11:19:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.6/specs/phase0/beacon-chain.md#helpers
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
# get_eligible_validator_indices
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
func is_eligible_validator*(validator: RewardStatus): bool =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
validator.flags.contains(RewardFlags.isActiveInPreviousEpoch) or
(validator.flags.contains(RewardFlags.isSlashed) and not
(validator.flags.contains RewardFlags.canWithdrawInCurrentEpoch))
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
func is_eligible_validator*(validator: Validator, previous_epoch: Epoch): bool =
is_active_validator(validator, previous_epoch) or
(validator.slashed and previous_epoch + 1 < validator.withdrawable_epoch)
func is_eligible_validator*(validator: ParticipationInfo): bool =
validator.flags.contains(ParticipationFlag.eligible)
Add detailed finalization and justification debug logs (#449) * Add detailed finalization and justification debug logs * log was wrongfully reporting rule 123 instead of rule 12
2019-09-23 13:48:25 +00:00
# Spec
# --------------------------------------------------------
use 'deneb' in 4 more modules (#4658)
2023-02-23 18:06:57 +00:00
from ./datatypes/deneb import BeaconState
eip4844 fork and epoch transition tests; some eip4844 gossip (#4393)
2022-12-06 16:43:11 +00:00
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#get_unslashed_participating_indices
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
func get_unslashed_participating_balances*(
eip4844 fork and epoch transition tests; some eip4844 gossip (#4393)
2022-12-06 16:43:11 +00:00
state: altair.BeaconState | bellatrix.BeaconState | capella.BeaconState |
add Electra overloads for spec functions; add Electra block processing (#5963)
2024-02-26 02:38:21 +00:00
deneb.BeaconState | electra.BeaconState):
UnslashedParticipatingBalances =
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
let
previous_epoch = get_previous_epoch(state)
current_epoch = get_current_epoch(state)
var res: UnslashedParticipatingBalances
for validator_index in 0'u64 ..< state.validators.lenu64:
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
let
is_active_current_epoch = is_active_validator(
state.validators[validator_index], current_epoch)
validator_effective_balance =
state.validators[validator_index].effective_balance
if is_active_current_epoch:
# Active balance counted also for slashed validators
res.current_epoch += validator_effective_balance
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
if state.validators[validator_index].slashed:
continue
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
let
is_active_previous_epoch = is_active_validator(
state.validators[validator_index], previous_epoch)
previous_epoch_participation =
state.previous_epoch_participation[validator_index]
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
if is_active_previous_epoch:
reduce memory allocations during state transition (#5235) This PR removes a few hundred thousand temporary seq allocations during state transition - in particular, the flag seq was allocated per validator while committees are computed per attestation.
2023-08-02 23:03:40 +00:00
for flag_index in TimelyFlag:
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
if has_flag(previous_epoch_participation, flag_index):
res.previous_epoch[flag_index] += validator_effective_balance
# Only TIMELY_TARGET_FLAG_INDEX is used with the current epoch in Altair
cleanups, partly from kintsugi branch (#3161) * cleanups, partly from kintsugi branch * re-export shortLog(EthBlock) and preserve exception messages in batchVerify and processBatch
2021-12-05 17:32:41 +00:00
# and merge
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
if is_active_current_epoch and has_flag(
state.current_epoch_participation[validator_index],
TIMELY_TARGET_FLAG_INDEX):
res.current_epoch_TIMELY_TARGET += validator_effective_balance
reduce memory allocations during state transition (#5235) This PR removes a few hundred thousand temporary seq allocations during state transition - in particular, the flag seq was allocated per validator while committees are computed per attestation.
2023-08-02 23:03:40 +00:00
for flag_index in TimelyFlag:
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
res.previous_epoch[flag_index] =
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
max(EFFECTIVE_BALANCE_INCREMENT.Gwei, res.previous_epoch[flag_index])
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
res.current_epoch_TIMELY_TARGET =
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
max(EFFECTIVE_BALANCE_INCREMENT.Gwei, res.current_epoch_TIMELY_TARGET)
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
res.current_epoch = max(EFFECTIVE_BALANCE_INCREMENT.Gwei, res.current_epoch)
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
res
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
func is_unslashed_participating_index(
eip4844 fork and epoch transition tests; some eip4844 gossip (#4393)
2022-12-06 16:43:11 +00:00
state: altair.BeaconState | bellatrix.BeaconState | capella.BeaconState |
add Electra overloads for spec functions; add Electra block processing (#5963)
2024-02-26 02:38:21 +00:00
deneb.BeaconState | electra.BeaconState,
reduce memory allocations during state transition (#5235) This PR removes a few hundred thousand temporary seq allocations during state transition - in particular, the flag seq was allocated per validator while committees are computed per attestation.
2023-08-02 23:03:40 +00:00
flag_index: TimelyFlag, epoch: Epoch, validator_index: ValidatorIndex): bool =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
doAssert epoch in [get_previous_epoch(state), get_current_epoch(state)]
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
# TODO hoist this conditional
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
let epoch_participation =
if epoch == get_current_epoch(state):
unsafeAddr state.current_epoch_participation
else:
unsafeAddr state.previous_epoch_participation
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
is_active_validator(state.validators[validator_index], epoch) and
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
has_flag(epoch_participation[].item(validator_index), flag_index) and
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
not state.validators[validator_index].slashed
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
automated consensus spec URL updating to v1.4.0-beta.6 (#5804)
2024-01-20 11:19:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.6/specs/phase0/beacon-chain.md#justification-and-finalization
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
type FinalityState = object
slot: Slot
current_epoch_ancestor_root: Eth2Digest
previous_epoch_ancestor_root: Eth2Digest
justification_bits: JustificationBits
previous_justified_checkpoint: Checkpoint
current_justified_checkpoint: Checkpoint
finalized_checkpoint: Checkpoint
func toFinalityState(state: ForkyBeaconState): FinalityState =
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
let
current_epoch = get_current_epoch(state)
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
previous_epoch = get_previous_epoch(state)
FinalityState(
slot: state.slot,
current_epoch_ancestor_root:
if state.slot > current_epoch.start_slot:
get_block_root(state, current_epoch)
else:
ZERO_HASH,
previous_epoch_ancestor_root:
if state.slot > previous_epoch.start_slot:
get_block_root(state, previous_epoch)
else:
ZERO_HASH,
justification_bits:
state.justification_bits,
previous_justified_checkpoint:
state.previous_justified_checkpoint,
current_justified_checkpoint:
state.current_justified_checkpoint,
finalized_checkpoint:
state.finalized_checkpoint)
func get_current_epoch(state: FinalityState): Epoch =
state.slot.epoch
func get_previous_epoch(state: FinalityState): Epoch =
get_previous_epoch(get_current_epoch(state))
func get_block_root(state: FinalityState, epoch: Epoch): Eth2Digest =
doAssert state.slot > epoch.start_slot
if epoch == get_current_epoch(state):
state.current_epoch_ancestor_root
else:
doAssert epoch == get_previous_epoch(state)
state.previous_epoch_ancestor_root
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
type
JustificationAndFinalizationInfo = object
previous_justified_checkpoint: Checkpoint
current_justified_checkpoint: Checkpoint
finalized_checkpoint: Checkpoint
justification_bits: JustificationBits
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
proc weigh_justification_and_finalization(
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
state: ForkyBeaconState | FinalityState,
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
total_active_balance: Gwei,
previous_epoch_target_balance: Gwei,
current_epoch_target_balance: Gwei,
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
flags: UpdateFlags = {}): JustificationAndFinalizationInfo =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
let
previous_epoch = get_previous_epoch(state)
current_epoch = get_current_epoch(state)
old_previous_justified_checkpoint = state.previous_justified_checkpoint
old_current_justified_checkpoint = state.current_justified_checkpoint
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
var res = JustificationAndFinalizationInfo(
previous_justified_checkpoint: state.previous_justified_checkpoint,
current_justified_checkpoint: state.current_justified_checkpoint,
finalized_checkpoint: state.finalized_checkpoint,
justification_bits: state.justification_bits)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
# Process justifications
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
res.previous_justified_checkpoint = res.current_justified_checkpoint
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
## Spec:
## state.justification_bits[1:] = state.justification_bits[:-1]
## state.justification_bits[0] = 0b0
automated consensus spec URL updating to v1.4.0-beta.6 (#5804)
2024-01-20 11:19:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.6/specs/phase0/beacon-chain.md#misc
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
const JUSTIFICATION_BITS_LENGTH = 4
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
res.justification_bits = JustificationBits(
(uint8(res.justification_bits) shl 1) and
REST JSON support improvements (#3232) * support downloading blocks / states via JSON in addition to SSZ - slow, but needed for infura support - SSZ is still used when server supports it * use common forked block/state reader in REST API * fix stack overflows in REST JSON decoder * fix invalid serialization of `justification_bits` in `/eth/v1/debug/beacon/states` and `/eth/v2/debug/beacon/states` * fix REST client to use `/eth/...` instead of `/api/eth/...`, update "default" urls to expose REST api via `/eth` as well as this is what the standard says - `/api` was added early on based on an example "base url" in the spec that has been removed since * expose Nimbus REST extensions via `/nimbus` in addition to `/api/nimbus` to stay consistent with `/eth` * fix invalid state root when reading states via REST * fix recursive imports in `spec/ssz_codec` * remove usages of `serialization.useCustomSerialization` - fickle
2022-01-06 07:38:40 +00:00
uint8((2^JUSTIFICATION_BITS_LENGTH) - 1))
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
if previous_epoch_target_balance * 3 >= total_active_balance * 2:
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
res.current_justified_checkpoint = Checkpoint(
epoch: previous_epoch, root: get_block_root(state, previous_epoch))
uint8(res.justification_bits).setBit 1
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
trace "Justified with previous epoch",
current_epoch = current_epoch,
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
checkpoint = shortLog(res.current_justified_checkpoint)
rename verifyFinalization internal flag to strictVerification (#3866) * rename verifyFinalization internal flag to strictVerification * Update beacon_chain/extras.nim Co-authored-by: Etan Kissling <etan@status.im> Co-authored-by: Etan Kissling <etan@status.im>
2022-07-13 13:48:09 +00:00
elif strictVerification in flags:
low attestations during epoch should instafail in CI; dbg -> warn level in `newPayload` log (#4830) * low attestations during epoch should instafail in CI; dbg -> warn level on newPayload log * improve newPayload warning message when no valid EL connected * reduce potential spam; make log spelling more consistent; use fatal/quit
2023-04-19 19:42:30 +00:00
fatal "Low attestation participation in previous epoch",
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
total_active_balance,
previous_epoch_target_balance,
current_epoch_target_balance,
epoch = get_current_epoch(state)
low attestations during epoch should instafail in CI; dbg -> warn level in `newPayload` log (#4830) * low attestations during epoch should instafail in CI; dbg -> warn level on newPayload log * improve newPayload warning message when no valid EL connected * reduce potential spam; make log spelling more consistent; use fatal/quit
2023-04-19 19:42:30 +00:00
quit 1
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
if current_epoch_target_balance * 3 >= total_active_balance * 2:
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
res.current_justified_checkpoint = Checkpoint(
epoch: current_epoch, root: get_block_root(state, current_epoch))
uint8(res.justification_bits).setBit 0
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
trace "Justified with current epoch",
current_epoch = current_epoch,
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
checkpoint = shortLog(res.current_justified_checkpoint)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
# Process finalizations
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
let bitfield = uint8(res.justification_bits)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
## The 2nd/3rd/4th most recent epochs are justified, the 2nd using the 4th
## as source
if (bitfield and 0b1110) == 0b1110 and
old_previous_justified_checkpoint.epoch + 3 == current_epoch:
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
res.finalized_checkpoint = old_previous_justified_checkpoint
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
trace "Finalized with rule 234",
current_epoch = current_epoch,
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
checkpoint = shortLog(res.finalized_checkpoint)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
## The 2nd/3rd most recent epochs are justified, the 2nd using the 3rd as
## source
if (bitfield and 0b110) == 0b110 and
old_previous_justified_checkpoint.epoch + 2 == current_epoch:
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
res.finalized_checkpoint = old_previous_justified_checkpoint
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
trace "Finalized with rule 23",
current_epoch = current_epoch,
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
checkpoint = shortLog(res.finalized_checkpoint)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
## The 1st/2nd/3rd most recent epochs are justified, the 1st using the 3rd as
## source
if (bitfield and 0b111) == 0b111 and
old_current_justified_checkpoint.epoch + 2 == current_epoch:
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
res.finalized_checkpoint = old_current_justified_checkpoint
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
trace "Finalized with rule 123",
current_epoch = current_epoch,
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
checkpoint = shortLog(res.finalized_checkpoint)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
## The 1st/2nd most recent epochs are justified, the 1st using the 2nd as
## source
if (bitfield and 0b11) == 0b11 and
old_current_justified_checkpoint.epoch + 1 == current_epoch:
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
res.finalized_checkpoint = old_current_justified_checkpoint
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
trace "Finalized with rule 12",
current_epoch = current_epoch,
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
checkpoint = shortLog(res.finalized_checkpoint)
res
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
automated consensus spec URL updating to v1.4.0-beta.6 (#5804)
2024-01-20 11:19:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.6/specs/phase0/beacon-chain.md#justification-and-finalization
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
proc process_justification_and_finalization*(
state: var phase0.BeaconState,
balances: TotalBalances, flags: UpdateFlags = {}) =
# Initial FFG checkpoint values have a `0x00` stub for `root`.
# Skip FFG updates in the first two epochs to avoid corner cases that might
# result in modifying this stub.
if get_current_epoch(state) <= GENESIS_EPOCH + 1:
return
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
let jfRes = weigh_justification_and_finalization(
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
state, balances.current_epoch,
balances.previous_epoch_target_attesters,
balances.current_epoch_target_attesters, flags)
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
assign(
state.previous_justified_checkpoint, jfRes.previous_justified_checkpoint)
assign(
state.current_justified_checkpoint, jfRes.current_justified_checkpoint)
assign(state.finalized_checkpoint, jfRes.finalized_checkpoint)
assign(state.justification_bits, jfRes.justification_bits)
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
proc compute_unrealized_finality*(
state: phase0.BeaconState, cache: var StateCache): FinalityCheckpoints =
if get_current_epoch(state) <= GENESIS_EPOCH + 1:
return FinalityCheckpoints(
justified: state.current_justified_checkpoint,
finalized: state.finalized_checkpoint)
var info: phase0.EpochInfo
info.init(state)
info.process_attestations(state, cache)
template balances(): auto = info.balances
var finalityState = state.toFinalityState()
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
let jfRes = weigh_justification_and_finalization(
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
finalityState, balances.current_epoch,
balances.previous_epoch_target_attesters,
balances.current_epoch_target_attesters)
FinalityCheckpoints(
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
justified: jfRes.current_justified_checkpoint,
finalized: jfRes.finalized_checkpoint)
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#justification-and-finalization
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
proc process_justification_and_finalization*(
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
state: var (altair.BeaconState | bellatrix.BeaconState |
add Electra overloads for spec functions; add Electra block processing (#5963)
2024-02-26 02:38:21 +00:00
capella.BeaconState | deneb.BeaconState | electra.BeaconState),
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
balances: UnslashedParticipatingBalances,
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
flags: UpdateFlags = {}) =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
# Initial FFG checkpoint values have a `0x00` stub for `root`.
# Skip FFG updates in the first two epochs to avoid corner cases that might
# result in modifying this stub.
if get_current_epoch(state) <= GENESIS_EPOCH + 1:
return
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
let jfRes = weigh_justification_and_finalization(
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
state, balances.current_epoch,
balances.previous_epoch[TIMELY_TARGET_FLAG_INDEX],
balances.current_epoch_TIMELY_TARGET, flags)
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
assign(
state.previous_justified_checkpoint, jfRes.previous_justified_checkpoint)
assign(
state.current_justified_checkpoint, jfRes.current_justified_checkpoint)
assign(state.finalized_checkpoint, jfRes.finalized_checkpoint)
assign(state.justification_bits, jfRes.justification_bits)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
proc compute_unrealized_finality*(
eip4844 light client tests; avoid case object out-of-bound array reads (#4404)
2022-12-08 16:21:53 +00:00
state: altair.BeaconState | bellatrix.BeaconState | capella.BeaconState |
add Electra overloads for spec functions; add Electra block processing (#5963)
2024-02-26 02:38:21 +00:00
deneb.BeaconState | electra.BeaconState): FinalityCheckpoints =
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
if get_current_epoch(state) <= GENESIS_EPOCH + 1:
return FinalityCheckpoints(
justified: state.current_justified_checkpoint,
finalized: state.finalized_checkpoint)
let balances = get_unslashed_participating_balances(state)
var finalityState = state.toFinalityState()
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
let jfRes = weigh_justification_and_finalization(
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
finalityState, balances.current_epoch,
balances.previous_epoch[TIMELY_TARGET_FLAG_INDEX],
balances.current_epoch_TIMELY_TARGET)
FinalityCheckpoints(
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
justified: jfRes.current_justified_checkpoint,
finalized: jfRes.finalized_checkpoint)
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
automated consensus spec URL updating to v1.4.0-beta.6 (#5804)
2024-01-20 11:19:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.6/specs/phase0/beacon-chain.md#helpers
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
func get_base_reward_sqrt*(state: phase0.BeaconState, index: ValidatorIndex,
improve slot processing speeds (#1670) about 40% better slot processing times (with LTO enabled) - these don't do BLS but are used heavily during replay (state transition = slot + block transition) tests using a recent medalla state and advancing it 1000 slots: ``` ./ncli slots --preState2:state-302271-3c1dbf19-c1f944bf.ssz --slot:1000 --postState2:xx.ssz ``` pre: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 39.236, 0.000, 39.236, 39.236, 1, Load state from file 0.049, 0.002, 0.046, 0.063, 968, Apply slot 256.504, 81.008, 213.471, 591.902, 32, Apply epoch slot 28.597, 0.000, 28.597, 28.597, 1, Save state to file ``` cast: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 37.079, 0.000, 37.079, 37.079, 1, Load state from file 0.042, 0.002, 0.040, 0.090, 968, Apply slot 215.552, 68.763, 180.155, 500.103, 32, Apply epoch slot 25.106, 0.000, 25.106, 25.106, 1, Save state to file ``` cast+rewards: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 40.049, 0.000, 40.049, 40.049, 1, Load state from file 0.048, 0.001, 0.045, 0.060, 968, Apply slot 164.981, 76.273, 142.099, 477.868, 32, Apply epoch slot 28.498, 0.000, 28.498, 28.498, 1, Save state to file ``` cast+rewards+shr ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 12.898, 0.000, 12.898, 12.898, 1, Load state from file 0.039, 0.002, 0.038, 0.054, 968, Apply slot 139.971, 68.797, 120.088, 428.844, 32, Apply epoch slot 24.761, 0.000, 24.761, 24.761, 1, Save state to file ```
2020-09-16 20:59:33 +00:00
total_balance_sqrt: auto): Gwei =
verify that state_sim is justifying and finalizing; fix 3 more warnings; rename crosslink_committee_cache to beacon_committee_cache; fix O(n^2) usage of get_base_reward(...)
2019-11-15 22:37:39 +00:00
# Spec function recalculates total_balance every time, which creates an
# O(n^2) situation.
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
let effective_balance = state.validators[index].effective_balance
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
effective_balance * BASE_REWARD_FACTOR div
improve slot processing speeds (#1670) about 40% better slot processing times (with LTO enabled) - these don't do BLS but are used heavily during replay (state transition = slot + block transition) tests using a recent medalla state and advancing it 1000 slots: ``` ./ncli slots --preState2:state-302271-3c1dbf19-c1f944bf.ssz --slot:1000 --postState2:xx.ssz ``` pre: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 39.236, 0.000, 39.236, 39.236, 1, Load state from file 0.049, 0.002, 0.046, 0.063, 968, Apply slot 256.504, 81.008, 213.471, 591.902, 32, Apply epoch slot 28.597, 0.000, 28.597, 28.597, 1, Save state to file ``` cast: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 37.079, 0.000, 37.079, 37.079, 1, Load state from file 0.042, 0.002, 0.040, 0.090, 968, Apply slot 215.552, 68.763, 180.155, 500.103, 32, Apply epoch slot 25.106, 0.000, 25.106, 25.106, 1, Save state to file ``` cast+rewards: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 40.049, 0.000, 40.049, 40.049, 1, Load state from file 0.048, 0.001, 0.045, 0.060, 968, Apply slot 164.981, 76.273, 142.099, 477.868, 32, Apply epoch slot 28.498, 0.000, 28.498, 28.498, 1, Save state to file ``` cast+rewards+shr ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 12.898, 0.000, 12.898, 12.898, 1, Load state from file 0.039, 0.002, 0.038, 0.054, 968, Apply slot 139.971, 68.797, 120.088, 428.844, 32, Apply epoch slot 24.761, 0.000, 24.761, 24.761, 1, Save state to file ```
2020-09-16 20:59:33 +00:00
total_balance_sqrt div BASE_REWARDS_PER_EPOCH
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func get_proposer_reward*(base_reward: Gwei): Gwei =
allow 50k validators on block_sim by avoiding repeated get_total_active_balance() in get_proposer_reward()
2020-07-07 16:52:12 +00:00
# Spec version recalculates get_total_active_balance(state) quadratically
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward div PROPOSER_REWARD_QUOTIENT
func is_in_inactivity_leak(finality_delay: uint64): bool =
finality_delay > MIN_EPOCHS_TO_INACTIVITY_PENALTY
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func get_finality_delay*(state: ForkyBeaconState): uint64 =
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
get_previous_epoch(state) - state.finalized_checkpoint.epoch
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
func get_attestation_component_reward*(
attesting_balance: Gwei,
total_balance: Gwei,
base_reward: Gwei,
finality_delay: uint64): Gwei =
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
if is_in_inactivity_leak(finality_delay):
# Since full base reward will be canceled out by inactivity penalty deltas,
# optimal participation receives full base reward compensation here.
base_reward
else:
let reward_numerator =
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
base_reward * (attesting_balance div EFFECTIVE_BALANCE_INCREMENT.Gwei)
reward_numerator div (total_balance div EFFECTIVE_BALANCE_INCREMENT.Gwei)
func get_attestation_component_delta(
is_unslashed_attester: bool,
attesting_balance: Gwei,
total_balance: Gwei,
base_reward: Gwei,
finality_delay: uint64): RewardDelta =
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
# Helper with shared logic for use by get source, target, and head deltas
# functions
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if is_unslashed_attester:
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
RewardDelta(rewards: get_attestation_component_reward(
attesting_balance,
total_balance,
base_reward,
finality_delay))
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
else:
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
RewardDelta(penalties: base_reward)
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
some consensus spec URL updates to v1.5.0-alpha.3 (#6382)
2024-06-22 05:28:19 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.3/specs/phase0/beacon-chain.md#components-of-attestation-deltas
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
func get_source_delta*(
validator: RewardStatus,
base_reward: Gwei,
balances: TotalBalances,
finality_delay: uint64): RewardDelta =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Return attester micro-rewards/penalties for source-vote for each validator.
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
get_attestation_component_delta(
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
validator.is_previous_epoch_attester.isSome() and
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
not (validator.flags.contains RewardFlags.isSlashed),
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
balances.previous_epoch_attesters,
balances.current_epoch,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward,
finality_delay)
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
func get_target_delta*(
validator: RewardStatus,
base_reward: Gwei,
balances: TotalBalances,
finality_delay: uint64): RewardDelta =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Return attester micro-rewards/penalties for target-vote for each validator.
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
get_attestation_component_delta(
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
validator.flags.contains(RewardFlags.isPreviousEpochTargetAttester) and
not (validator.flags.contains(RewardFlags.isSlashed)),
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
balances.previous_epoch_target_attesters,
balances.current_epoch,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward,
finality_delay)
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
func get_head_delta*(
validator: RewardStatus,
base_reward: Gwei,
balances: TotalBalances,
finality_delay: uint64): RewardDelta =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Return attester micro-rewards/penalties for head-vote for each validator.
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
get_attestation_component_delta(
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
validator.flags.contains(RewardFlags.isPreviousEpochHeadAttester) and
((not validator.flags.contains(RewardFlags.isSlashed))),
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
balances.previous_epoch_head_attesters,
balances.current_epoch,
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
base_reward,
finality_delay)
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
func get_inclusion_delay_delta*(
validator: RewardStatus,
base_reward: Gwei): (RewardDelta, Opt[(uint64, RewardDelta)]) =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Return proposer and inclusion delay micro-rewards/penalties for each validator.
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if validator.is_previous_epoch_attester.isSome() and ((not validator.flags.contains(RewardFlags.isSlashed))):
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
let
inclusion_info = validator.is_previous_epoch_attester.get()
proposer_reward = get_proposer_reward(base_reward)
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
proposer_delta = RewardDelta(rewards: proposer_reward)
fix attestation handling bottlenecks in slot processing and quadratic-time inclusion delay rewards/palties in epoch processing; create infrastructure for dynamically (un)subscribing to/from attestation topics (#1294)
2020-07-10 09:24:04 +00:00
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
let
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
max_attester_reward = base_reward - proposer_reward
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
delta = RewardDelta(rewards: max_attester_reward div inclusion_info.delay)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
proposer_index = inclusion_info.proposer_index;
spec: Option -> Opt (#4488)
2023-01-11 12:29:21 +00:00
return (delta, Opt.some((proposer_index, proposer_delta)))
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
func get_inactivity_penalty_delta*(
validator: RewardStatus,
base_reward: Gwei,
finality_delay: uint64): RewardDelta =
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
## Return inactivity reward/penalty deltas for each validator.
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
var delta: RewardDelta
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if is_in_inactivity_leak(finality_delay):
# If validator is performing optimally this cancels all rewards for a neutral balance
delta.penalties +=
BASE_REWARDS_PER_EPOCH * base_reward - get_proposer_reward(base_reward)
# Additionally, all validators whose FFG target didn't match are penalized extra
# This condition is equivalent to this condition from the spec:
# `index not in get_unslashed_attesting_indices(state, matching_target_attestations)`
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if (validator.flags.contains(RewardFlags.isSlashed)) or
((not validator.flags.contains(RewardFlags.isPreviousEpochTargetAttester))):
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
delta.penalties +=
validator.current_epoch_effective_balance * finality_delay div
INACTIVITY_PENALTY_QUOTIENT
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
delta
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
automated consensus spec URL updating to v1.4.0-beta.6 (#5804)
2024-01-20 11:19:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.6/specs/phase0/beacon-chain.md#get_attestation_deltas
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
func get_attestation_deltas(
state: phase0.BeaconState, info: var phase0.EpochInfo) =
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
## Update rewards with attestation reward/penalty deltas for each validator.
remove v0.11.3 support; add block_sim to CI (#1253) * remove v0.11.3 support; add block_sim to CI * rm stray PERSISTENT_COMMITTEE_PERIOD * remove TopicFilter.InteropAttestations * bump two comment-spec-refs to v0.12.1
2020-06-29 18:08:58 +00:00
let
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
finality_delay = get_finality_delay(state)
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
total_balance = info.balances.current_epoch
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
total_balance_sqrt = integer_squareroot(distinctBase(total_balance))
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
# Filter out ineligible validators. All sub-functions of the spec do this
# except for `get_inclusion_delay_deltas`. It's safe to do so here because
# any validator that is in the unslashed indices of the matching source
# attestations is active, and therefore eligible.
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
for index, validator in info.validators.mpairs():
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
if not is_eligible_validator(validator):
continue
let
base_reward = get_base_reward_sqrt(
state, index.ValidatorIndex, total_balance_sqrt)
improve slot processing speeds (#1670) about 40% better slot processing times (with LTO enabled) - these don't do BLS but are used heavily during replay (state transition = slot + block transition) tests using a recent medalla state and advancing it 1000 slots: ``` ./ncli slots --preState2:state-302271-3c1dbf19-c1f944bf.ssz --slot:1000 --postState2:xx.ssz ``` pre: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 39.236, 0.000, 39.236, 39.236, 1, Load state from file 0.049, 0.002, 0.046, 0.063, 968, Apply slot 256.504, 81.008, 213.471, 591.902, 32, Apply epoch slot 28.597, 0.000, 28.597, 28.597, 1, Save state to file ``` cast: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 37.079, 0.000, 37.079, 37.079, 1, Load state from file 0.042, 0.002, 0.040, 0.090, 968, Apply slot 215.552, 68.763, 180.155, 500.103, 32, Apply epoch slot 25.106, 0.000, 25.106, 25.106, 1, Save state to file ``` cast+rewards: ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 40.049, 0.000, 40.049, 40.049, 1, Load state from file 0.048, 0.001, 0.045, 0.060, 968, Apply slot 164.981, 76.273, 142.099, 477.868, 32, Apply epoch slot 28.498, 0.000, 28.498, 28.498, 1, Save state to file ``` cast+rewards+shr ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 12.898, 0.000, 12.898, 12.898, 1, Load state from file 0.039, 0.002, 0.038, 0.054, 968, Apply slot 139.971, 68.797, 120.088, 428.844, 32, Apply epoch slot 24.761, 0.000, 24.761, 24.761, 1, Save state to file ```
2020-09-16 20:59:33 +00:00
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
let
source_delta = get_source_delta(
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
validator, base_reward, info.balances, finality_delay)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
target_delta = get_target_delta(
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
validator, base_reward, info.balances, finality_delay)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
head_delta = get_head_delta(
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
validator, base_reward, info.balances, finality_delay)
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
(inclusion_delay_delta, proposer_delta) =
get_inclusion_delay_delta(validator, base_reward)
inactivity_delta = get_inactivity_penalty_delta(
validator, base_reward, finality_delay)
validator.delta.add source_delta
validator.delta.add target_delta
validator.delta.add head_delta
validator.delta.add inclusion_delay_delta
validator.delta.add inactivity_delta
if proposer_delta.isSome:
let proposer_index = proposer_delta.get()[0]
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
if proposer_index < info.validators.lenu64:
info.validators[proposer_index].delta.add(
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
proposer_delta.get()[1])
initial 0.12.1 beacon chain spec
2020-06-08 18:41:50 +00:00
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#get_base_reward
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func get_base_reward_increment*(
eip4844 fork and epoch transition tests; some eip4844 gossip (#4393)
2022-12-06 16:43:11 +00:00
state: altair.BeaconState | bellatrix.BeaconState | capella.BeaconState |
add Electra overloads for spec functions; add Electra block processing (#5963)
2024-02-26 02:38:21 +00:00
deneb.BeaconState | electra.BeaconState,
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
index: ValidatorIndex, base_reward_per_increment: Gwei): Gwei =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
## Return the base reward for the validator defined by ``index`` with respect
## to the current ``state``.
let increments =
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
state.validators[index].effective_balance div
EFFECTIVE_BALANCE_INCREMENT.Gwei
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
increments * base_reward_per_increment
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#get_flag_index_deltas
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
func get_flag_index_reward*(
eip4844 fork and epoch transition tests; some eip4844 gossip (#4393)
2022-12-06 16:43:11 +00:00
state: altair.BeaconState | bellatrix.BeaconState | capella.BeaconState |
add Electra overloads for spec functions; add Electra block processing (#5963)
2024-02-26 02:38:21 +00:00
deneb.BeaconState | electra.BeaconState,
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
base_reward: Gwei,
active_increments: uint64,
unslashed_participating_increments: uint64,
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
weight, finality_delay: uint64): Gwei =
[ncli_db] bugfix: take into account finalization delay in reward calc post Altair This fixes a problem affecting Prater's epoch 64444.
2022-01-27 16:37:30 +00:00
if not is_in_inactivity_leak(finality_delay):
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
let reward_numerator =
base_reward * weight * unslashed_participating_increments
reward_numerator div (active_increments * WEIGHT_DENOMINATOR)
else:
0.Gwei
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#get_flag_index_deltas
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func get_unslashed_participating_increment*(
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
info: altair.EpochInfo | bellatrix.BeaconState,
flag_index: TimelyFlag): uint64 =
info.balances.previous_epoch[flag_index] div EFFECTIVE_BALANCE_INCREMENT.Gwei
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
some consensus spec URL updates to v1.5.0-alpha.3 (#6382)
2024-06-22 05:28:19 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.3/specs/altair/beacon-chain.md#get_flag_index_deltas
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
func get_active_increments*(
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
info: altair.EpochInfo | bellatrix.BeaconState): uint64 =
info.balances.current_epoch div EFFECTIVE_BALANCE_INCREMENT.Gwei
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#get_flag_index_deltas
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#modified-get_inactivity_penalty_deltas
automated consensus spec URL updating to v1.5.0-alpha.3 (#6364)
2024-06-16 00:59:25 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.3/specs/bellatrix/beacon-chain.md#modified-get_inactivity_penalty_deltas
optimize epoch transition via get_flag_index_deltas() and get_inactivity_penalty_deltas() (#5404) * optimize epoch transition via get_flag_index_deltas() and get_inactivity_penalty_deltas() * directly mutate Altair info in {get,update}_flag_and_inactivity_deltas
2023-09-07 11:14:52 +00:00
# Combines get_flag_index_deltas() and get_inactivity_penalty_deltas()
refactor get_flag_and_inactivity_deltas() loop body to use per-validator get_flag_and_inactivity_delta() (#6007)
2024-03-01 16:31:25 +00:00
template get_flag_and_inactivity_delta(
state: altair.BeaconState | bellatrix.BeaconState | capella.BeaconState |
deneb.BeaconState | electra.BeaconState,
base_reward_per_increment: Gwei, finality_delay: uint64,
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
previous_epoch: Epoch,
active_increments: uint64,
refactor get_flag_and_inactivity_deltas() loop body to use per-validator get_flag_and_inactivity_delta() (#6007)
2024-03-01 16:31:25 +00:00
penalty_denominator: uint64,
epoch_participation: ptr EpochParticipationFlags,
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
participating_increments: array[3, uint64],
info: var altair.EpochInfo,
vidx: ValidatorIndex
): (ValidatorIndex, Gwei, Gwei, Gwei, Gwei, Gwei, Gwei) =
refactor get_flag_and_inactivity_deltas() loop body to use per-validator get_flag_and_inactivity_delta() (#6007)
2024-03-01 16:31:25 +00:00
let
base_reward = get_base_reward_increment(state, vidx, base_reward_per_increment)
pflags =
if is_active_validator(state.validators[vidx], previous_epoch) and
not state.validators[vidx].slashed:
epoch_participation[].item(vidx)
else:
0
if has_flag(pflags, TIMELY_SOURCE_FLAG_INDEX):
info.validators[vidx].flags.incl ParticipationFlag.timelySourceAttester
if has_flag(pflags, TIMELY_TARGET_FLAG_INDEX):
info.validators[vidx].flags.incl ParticipationFlag.timelyTargetAttester
if has_flag(pflags, TIMELY_HEAD_FLAG_INDEX):
info.validators[vidx].flags.incl ParticipationFlag.timelyHeadAttester
template reward(flag: untyped): untyped =
if has_flag(pflags, flag):
get_flag_index_reward(
state, base_reward, active_increments,
participating_increments[ord(flag)],
PARTICIPATION_FLAG_WEIGHTS[flag], finality_delay)
else:
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
0.Gwei
refactor get_flag_and_inactivity_deltas() loop body to use per-validator get_flag_and_inactivity_delta() (#6007)
2024-03-01 16:31:25 +00:00
template penalty(flag: untyped): untyped =
if not has_flag(pflags, flag):
base_reward * PARTICIPATION_FLAG_WEIGHTS[flag] div WEIGHT_DENOMINATOR
else:
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
0.Gwei
refactor get_flag_and_inactivity_deltas() loop body to use per-validator get_flag_and_inactivity_delta() (#6007)
2024-03-01 16:31:25 +00:00
let inactivity_penalty =
if has_flag(pflags, TIMELY_TARGET_FLAG_INDEX):
0.Gwei
else:
let penalty_numerator =
state.validators[vidx].effective_balance * state.inactivity_scores[vidx]
penalty_numerator div penalty_denominator
(vidx, reward(TIMELY_SOURCE_FLAG_INDEX),
reward(TIMELY_TARGET_FLAG_INDEX), reward(TIMELY_HEAD_FLAG_INDEX),
penalty(TIMELY_SOURCE_FLAG_INDEX), penalty(TIMELY_TARGET_FLAG_INDEX),
inactivity_penalty)
restore full test coverage for process_rewards_and_penalties (#5407) * restore full test coverage for process_rewards_and_penalties * adjust ncli_db to use new iterator
2023-09-08 16:42:18 +00:00
iterator get_flag_and_inactivity_deltas*(
optimize epoch transition via get_flag_index_deltas() and get_inactivity_penalty_deltas() (#5404) * optimize epoch transition via get_flag_index_deltas() and get_inactivity_penalty_deltas() * directly mutate Altair info in {get,update}_flag_and_inactivity_deltas
2023-09-07 11:14:52 +00:00
cfg: RuntimeConfig,
state: altair.BeaconState | bellatrix.BeaconState | capella.BeaconState |
add Electra overloads for spec functions; add Electra block processing (#5963)
2024-02-26 02:38:21 +00:00
deneb.BeaconState | electra.BeaconState,
optimize epoch transition via get_flag_index_deltas() and get_inactivity_penalty_deltas() (#5404) * optimize epoch transition via get_flag_index_deltas() and get_inactivity_penalty_deltas() * directly mutate Altair info in {get,update}_flag_and_inactivity_deltas
2023-09-07 11:14:52 +00:00
base_reward_per_increment: Gwei, info: var altair.EpochInfo,
restore full test coverage for process_rewards_and_penalties (#5407) * restore full test coverage for process_rewards_and_penalties * adjust ncli_db to use new iterator
2023-09-08 16:42:18 +00:00
finality_delay: uint64):
(ValidatorIndex, Gwei, Gwei, Gwei, Gwei, Gwei, Gwei) =
optimize epoch transition via get_flag_index_deltas() and get_inactivity_penalty_deltas() (#5404) * optimize epoch transition via get_flag_index_deltas() and get_inactivity_penalty_deltas() * directly mutate Altair info in {get,update}_flag_and_inactivity_deltas
2023-09-07 11:14:52 +00:00
## Return the deltas for a given ``flag_index`` by scanning through the
## participation flags.
#
# This deviates from spec by processing all flags at once, so does not take a
# flag_index parameter. Fold get_inactivity_penalty_deltas loop into this one
# as well.
const INACTIVITY_PENALTY_QUOTIENT =
when state is altair.BeaconState:
INACTIVITY_PENALTY_QUOTIENT_ALTAIR
else:
INACTIVITY_PENALTY_QUOTIENT_BELLATRIX
static: doAssert ord(high(TimelyFlag)) == 2
let
previous_epoch = get_previous_epoch(state)
active_increments = get_active_increments(info)
penalty_denominator =
cfg.INACTIVITY_SCORE_BIAS * INACTIVITY_PENALTY_QUOTIENT
epoch_participation =
if previous_epoch == get_current_epoch(state):
unsafeAddr state.current_epoch_participation
else:
unsafeAddr state.previous_epoch_participation
participating_increments = [
get_unslashed_participating_increment(info, TIMELY_SOURCE_FLAG_INDEX),
get_unslashed_participating_increment(info, TIMELY_TARGET_FLAG_INDEX),
get_unslashed_participating_increment(info, TIMELY_HEAD_FLAG_INDEX)]
for vidx in state.validators.vindices:
if not is_eligible_validator(info.validators[vidx]):
continue
restore full test coverage for process_rewards_and_penalties (#5407) * restore full test coverage for process_rewards_and_penalties * adjust ncli_db to use new iterator
2023-09-08 16:42:18 +00:00
# Yielding these as a structure with identifiable names rather than
# multiple-return-value style creates spurious nimZeroMem calls.
refactor get_flag_and_inactivity_deltas() loop body to use per-validator get_flag_and_inactivity_delta() (#6007)
2024-03-01 16:31:25 +00:00
yield get_flag_and_inactivity_delta(
state, base_reward_per_increment, finality_delay, previous_epoch,
active_increments, penalty_denominator, epoch_participation,
participating_increments, info, vidx)
use v1.1.4 test vectors
2021-11-05 16:49:03 +00:00
some consensus spec v1.4.0 spec URL updates (#6208)
2024-04-17 03:51:16 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/phase0/beacon-chain.md#rewards-and-penalties-1
add all missing epoch transition tests (#4269)
2022-10-28 08:02:33 +00:00
func process_rewards_and_penalties*(
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
state: var phase0.BeaconState, info: var phase0.EpochInfo) =
update 120+ beacon_chain and validator spec refs from v0.12.2 to v0.12.3 (#1740)
2020-09-24 17:04:10 +00:00
# No rewards are applied at the end of `GENESIS_EPOCH` because rewards are
# for work done in the previous epoch
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
doAssert info.validators.len == state.validators.len
speed up reward/penalty calculation Calculating rewards/penalties is slow due to how we compute sets of attestations validators then use the sets for inclusion checks, to see who attested. The dominant function during validated block processing / epoch processing is hash set building and lookup. This PR inverts the flow by removing the sets and creating a single large validator status list, then applying all relevant state attestations, then updating rewards and penalties. This provides a 10x speedup to epoch processing which in turn speeds up both empty slot and block processing - for example, on startup, we replay all non-finalized blocks to prime fork choice - the same when validating attestations or replaying states on reorg.
2020-10-22 11:08:46 +00:00
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
if get_current_epoch(state) == GENESIS_EPOCH:
return
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
get_attestation_deltas(state, info)
fix attestation selection when calculating rewards and penalties Previously, the we'd use the first source attestation regardless if it matches the filter or not leading to the wrong minimum being used
2019-10-01 14:40:41 +00:00
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
# Here almost all balances are updated (assuming most validators are active) -
# clearing the cache becomes a bottleneck if done item by item because of the
# recursive nature of cache clearing - instead, we clear the whole cache then
# update the raw list directly
state.balances.clearCache()
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
for idx, v in info.validators:
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
var balance = state.balances.item(idx)
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
increase_balance(balance, v.delta.rewards)
decrease_balance(balance, v.delta.penalties)
state.balances.asSeq()[idx] = balance
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
automated v1.4.0-alpha.2 to v1.4.0-alpha.3 consensus spec URL updates (#5065)
2023-06-13 14:03:49 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-alpha.3/specs/altair/beacon-chain.md#rewards-and-penalties
add all missing epoch transition tests (#4269)
2022-10-28 08:02:33 +00:00
func process_rewards_and_penalties*(
eip4844 fork and epoch transition tests; some eip4844 gossip (#4393)
2022-12-06 16:43:11 +00:00
cfg: RuntimeConfig,
state: var (altair.BeaconState | bellatrix.BeaconState |
add Electra overloads for spec functions; add Electra block processing (#5963)
2024-02-26 02:38:21 +00:00
capella.BeaconState | deneb.BeaconState | electra.BeaconState),
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
info: var altair.EpochInfo) =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if get_current_epoch(state) == GENESIS_EPOCH:
return
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
let
total_active_balance = info.balances.current_epoch
base_reward_per_increment = get_base_reward_per_increment(
total_active_balance)
[ncli_db] bugfix: take into account finalization delay in reward calc post Altair This fixes a problem affecting Prater's epoch 64444.
2022-01-27 16:37:30 +00:00
finality_delay = get_finality_delay(state)
avoid repeated `integer_squareroot`s for 20% faster Altair epoch processing (#2797) * avoid repeated integer_squareroots for 20% faster altair epoch processing * the square root of a Gwei isn't a Gwei
2021-08-20 14:41:32 +00:00
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
doAssert state.validators.len() == info.validators.len()
restore full test coverage for process_rewards_and_penalties (#5407) * restore full test coverage for process_rewards_and_penalties * adjust ncli_db to use new iterator
2023-09-08 16:42:18 +00:00
for validator_index, reward0, reward1, reward2, penalty0, penalty1, penalty2 in
get_flag_and_inactivity_deltas(
cfg, state, base_reward_per_increment, info, finality_delay):
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
# templatize this loop? or replicate a couple lines of code?
restore full test coverage for process_rewards_and_penalties (#5407) * restore full test coverage for process_rewards_and_penalties * adjust ncli_db to use new iterator
2023-09-08 16:42:18 +00:00
info.validators[validator_index].delta.rewards += reward0 + reward1 + reward2
info.validators[validator_index].delta.penalties += penalty0 + penalty1 + penalty2
update ChainDAG.effective_balance() to use StateData; rm ChainDAG.getBlockByPreciseSlot() (#2622) * update ChainDAG.effective_balance() to use StateData; rm unused ChainDAG.getBlockByPreciseSlot() * update get_effective_balances to avoid god object; avoid most memory allocation in Altair epoch reward and penalty processing
2021-06-01 12:40:13 +00:00
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
# Here almost all balances are updated (assuming most validators are active) -
# clearing the cache becomes a bottleneck if done item by item because of the
# recursive nature of cache clearing - instead, we clear the whole cache then
# update the raw list directly
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
state.balances.clearCache()
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
for vidx in state.validators.vindices:
var balance = state.balances.item(vidx)
increase_balance(balance, info.validators[vidx].delta.rewards)
decrease_balance(balance, info.validators[vidx].delta.penalties)
state.balances.asSeq()[vidx] = balance
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
optimize epoch registry processing (#5412)
2023-09-11 09:21:50 +00:00
from std/heapqueue import HeapQueue, `[]`, len, push, replace
automated consensus spec URL updating to v1.4.0-beta.6 (#5804)
2024-01-20 11:19:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.6/specs/phase0/beacon-chain.md#registry-updates
test fixture cleanups (#2976) * "official" -> "scenarios", like the submodule * fewer test binaries - various compile hacks have been improved over time, test suite should follow * remove obsolete bls tests - there are better test vectors in nim-blscurve * remove obsolete mentions of `ssz_testing` * remove obsolete comments about proc vs globals, unittest2 already uses proc's
2021-10-12 11:36:52 +00:00
func process_registry_updates*(
add rest of EF consensus spec test Electra epoch transition fixture(s) (#6232)
2024-04-24 12:28:47 +00:00
cfg: RuntimeConfig,
state: var (phase0.BeaconState | altair.BeaconState |
bellatrix.BeaconState | capella.BeaconState |
deneb.BeaconState),
cache: var StateCache): Result[void, cstring] =
test fixture cleanups (#2976) * "official" -> "scenarios", like the submodule * fewer test binaries - various compile hacks have been improved over time, test suite should follow * remove obsolete bls tests - there are better test vectors in nim-blscurve * remove obsolete mentions of `ssz_testing` * remove obsolete comments about proc vs globals, unittest2 already uses proc's
2021-10-12 11:36:52 +00:00
## Process activation eligibility and ejections
# is_active_validator(...) is activation_epoch <= epoch < exit_epoch,
# and changes here to either activation_epoch or exit_epoch only take
# effect with a compute_activation_exit_epoch(...) delay of, based on
# the current epoch, 1 + MAX_SEED_LOOKAHEAD epochs ahead. Thus caches
# remain valid for this epoch through though this function along with
# the rest of the epoch transition.
optimize epoch registry processing (#5412)
2023-09-11 09:21:50 +00:00
#
# This implementation fuses the two loops over all validators in the
# spec code.
## Queue validators eligible for activation and not dequeued for activation
var activation_queue: HeapQueue[(uint64, uint32)]
implement EIP-7514 for Deneb: Add Max Epoch Churn Limit (#5442) * implement EIP-7514 for Deneb: Add Max Epoch Churn Limit Cap activations per epoch according to EIP-7514: - https://eips.ethereum.org/EIPS/eip-7514 - https://github.com/ethereum/consensus-specs/pull/3499 * apply proposer boost to first block in case of equivocation Implement spec changes to fork choice; this only affects equivocation when multiple blocks are signed for the same slot. Regular operation is not changed. - https://github.com/ethereum/consensus-specs/pull/3352 * bump test vectors to v1.4.0-beta.2-hotfix --------- Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-09-20 11:39:56 +00:00
let churn_limit =
convert remaining `toFork` -> `kind` for consistency (#5462) We currently have multiple ways to obtain `ConsensusFork` or `LcDataFork` from a forky type. Rename `toFork` to `kind` for a consistent API naming.
2023-09-27 15:10:28 +00:00
when typeof(state).kind >= ConsensusFork.Deneb:
implement EIP-7514 for Deneb: Add Max Epoch Churn Limit (#5442) * implement EIP-7514 for Deneb: Add Max Epoch Churn Limit Cap activations per epoch according to EIP-7514: - https://eips.ethereum.org/EIPS/eip-7514 - https://github.com/ethereum/consensus-specs/pull/3499 * apply proposer boost to first block in case of equivocation Implement spec changes to fork choice; this only affects equivocation when multiple blocks are signed for the same slot. Regular operation is not changed. - https://github.com/ethereum/consensus-specs/pull/3352 * bump test vectors to v1.4.0-beta.2-hotfix --------- Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-09-20 11:39:56 +00:00
get_validator_activation_churn_limit(cfg, state, cache)
else:
get_validator_churn_limit(cfg, state, cache)
avoid quadratic behavior exiting validators (#6161) * avoid quadratic behavior exiting validators * fix libnfuzz callers
2024-04-02 12:18:40 +00:00
var maybe_exit_queue_info: Opt[ExitQueueInfo]
document and clean up `ValidatorIndex` usage (#3651) * document static vs dynamic range checking requirements * add `vindices` iterator to iterate over valid validator indices in a state * clean up spec comments in general * fixup Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-05-23 23:39:08 +00:00
for vidx in state.validators.vindices:
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
if is_eligible_for_activation_queue(state.validators.item(vidx)):
state.validators.mitem(vidx).activation_eligibility_epoch =
test fixture cleanups (#2976) * "official" -> "scenarios", like the submodule * fewer test binaries - various compile hacks have been improved over time, test suite should follow * remove obsolete bls tests - there are better test vectors in nim-blscurve * remove obsolete mentions of `ssz_testing` * remove obsolete comments about proc vs globals, unittest2 already uses proc's
2021-10-12 11:36:52 +00:00
get_current_epoch(state) + 1
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
if is_active_validator(state.validators.item(vidx), get_current_epoch(state)) and
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
state.validators.item(vidx).effective_balance <= cfg.EJECTION_BALANCE.Gwei:
avoid quadratic behavior exiting validators (#6161) * avoid quadratic behavior exiting validators * fix libnfuzz callers
2024-04-02 12:18:40 +00:00
# Typically, there will be no ejected validators, and even more rarely,
# more than one. Therefore, only calculate the information required for
# initiate_validator_exit if there actually is at least one.
let exit_queue_info = maybe_exit_queue_info.valueOr:
rm unused parameters of get_state_exit_queue_info() (#6167)
2024-04-03 01:45:57 +00:00
let initial_exit_queue_info = get_state_exit_queue_info(state)
avoid quadratic behavior exiting validators (#6161) * avoid quadratic behavior exiting validators * fix libnfuzz callers
2024-04-02 12:18:40 +00:00
maybe_exit_queue_info = Opt.some initial_exit_queue_info
initial_exit_queue_info
maybe_exit_queue_info = Opt.some (? initiate_validator_exit(
cfg, state, vidx, exit_queue_info, cache))
test fixture cleanups (#2976) * "official" -> "scenarios", like the submodule * fewer test binaries - various compile hacks have been improved over time, test suite should follow * remove obsolete bls tests - there are better test vectors in nim-blscurve * remove obsolete mentions of `ssz_testing` * remove obsolete comments about proc vs globals, unittest2 already uses proc's
2021-10-12 11:36:52 +00:00
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
let validator = unsafeAddr state.validators.item(vidx)
test fixture cleanups (#2976) * "official" -> "scenarios", like the submodule * fewer test binaries - various compile hacks have been improved over time, test suite should follow * remove obsolete bls tests - there are better test vectors in nim-blscurve * remove obsolete mentions of `ssz_testing` * remove obsolete comments about proc vs globals, unittest2 already uses proc's
2021-10-12 11:36:52 +00:00
if is_eligible_for_activation(state, validator[]):
optimize epoch registry processing (#5412)
2023-09-11 09:21:50 +00:00
let val_key =
(FAR_FUTURE_EPOCH - validator[].activation_eligibility_epoch,
high(distinctBase(ValidatorIndex)) - distinctBase(vidx))
if activation_queue.len.uint64 < churn_limit:
activation_queue.push val_key
elif val_key > activation_queue[0]:
discard activation_queue.replace val_key
test fixture cleanups (#2976) * "official" -> "scenarios", like the submodule * fewer test binaries - various compile hacks have been improved over time, test suite should follow * remove obsolete bls tests - there are better test vectors in nim-blscurve * remove obsolete mentions of `ssz_testing` * remove obsolete comments about proc vs globals, unittest2 already uses proc's
2021-10-12 11:36:52 +00:00
implement EIP-7514 for Deneb: Add Max Epoch Churn Limit (#5442) * implement EIP-7514 for Deneb: Add Max Epoch Churn Limit Cap activations per epoch according to EIP-7514: - https://eips.ethereum.org/EIPS/eip-7514 - https://github.com/ethereum/consensus-specs/pull/3499 * apply proposer boost to first block in case of equivocation Implement spec changes to fork choice; this only affects equivocation when multiple blocks are signed for the same slot. Regular operation is not changed. - https://github.com/ethereum/consensus-specs/pull/3352 * bump test vectors to v1.4.0-beta.2-hotfix --------- Co-authored-by: tersec <tersec@users.noreply.github.com>
2023-09-20 11:39:56 +00:00
## Dequeued validators for activation up to activation churn limit
## (without resetting activation epoch)
optimize epoch registry processing (#5412)
2023-09-11 09:21:50 +00:00
doAssert activation_queue.len.uint64 <= churn_limit
for i in 0 ..< activation_queue.len:
let (_, vidx_complement) = activation_queue[i]
state.validators.mitem(
high(distinctBase(ValidatorIndex)) - vidx_complement).activation_epoch =
compute_activation_exit_epoch(get_current_epoch(state))
test fixture cleanups (#2976) * "official" -> "scenarios", like the submodule * fewer test binaries - various compile hacks have been improved over time, test suite should follow * remove obsolete bls tests - there are better test vectors in nim-blscurve * remove obsolete mentions of `ssz_testing` * remove obsolete comments about proc vs globals, unittest2 already uses proc's
2021-10-12 11:36:52 +00:00
consensus specs v1.2.0-rc.1 (#3661)
2022-05-25 13:49:29 +00:00
ok()
add rest of EF consensus spec test Electra epoch transition fixture(s) (#6232)
2024-04-24 12:28:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.0/specs/electra/beacon-chain.md#updated--process_registry_updates
func process_registry_updates*(
cfg: RuntimeConfig, state: var electra.BeaconState, cache: var StateCache):
Result[void, cstring] =
# Process activation eligibility and ejections
for index in 0 ..< state.validators.len:
let validator = state.validators.item(index)
if is_eligible_for_activation_queue(validator):
# Usually not too many at once, so do this individually
state.validators.mitem(index).activation_eligibility_epoch =
get_current_epoch(state) + 1
if is_active_validator(validator, get_current_epoch(state)) and
distinctBase(validator.effective_balance) <= cfg.EJECTION_BALANCE:
discard ? initiate_validator_exit(
cfg, state, ValidatorIndex(index), static(default(ExitQueueInfo)), cache)
# Activate all eligible validators
let activation_epoch =
compute_activation_exit_epoch(get_current_epoch(state))
for index in 0 ..< state.validators.len:
if is_eligible_for_activation(state, state.validators.item(index)):
state.validators.mitem(index).activation_epoch = activation_epoch
ok()
some consensus spec v1.4.0-beta.7 spec URL updates (#5945)
2024-02-22 02:42:57 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.7/specs/phase0/beacon-chain.md#slashings
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#slashings
automated consensus spec URL updating to v1.5.0-alpha.3 (#6364)
2024-06-16 00:59:25 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.3/specs/bellatrix/beacon-chain.md#slashings
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func get_adjusted_total_slashing_balance*(
state: ForkyBeaconState, total_balance: Gwei): Gwei =
MEV block proposal (#3883) * MEV validator registration * add nearby canary to detect new beacon chain forks * remove special MEV graffiti * web3signer support * fix trace logging * Nim 1.2 needs raises Defect * use template rather than proc in REST JSON parsing * use --payload-builder-enable and --payload-builder-url * explicitly default MEV to disabled * explicitly empty default value for payload builder URL * revert attestation pool to unstable version
2022-08-01 06:41:47 +00:00
const multiplier =
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
# tradeoff here about interleaving phase0/altair, but for these
# single-constant changes...
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
when state is phase0.BeaconState:
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
PROPORTIONAL_SLASHING_MULTIPLIER
elif state is altair.BeaconState:
PROPORTIONAL_SLASHING_MULTIPLIER_ALTAIR
eip4844 fork and epoch transition tests; some eip4844 gossip (#4393)
2022-12-06 16:43:11 +00:00
elif state is bellatrix.BeaconState or state is capella.BeaconState or
add Electra overloads for spec functions; add Electra block processing (#5963)
2024-02-26 02:38:21 +00:00
state is deneb.BeaconState or state is electra.BeaconState:
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
PROPORTIONAL_SLASHING_MULTIPLIER_BELLATRIX
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
else:
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
{.fatal: "process_slashings: incorrect BeaconState type".}
min(sum(state.slashings.data) * multiplier, total_balance)
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
some consensus spec v1.4.0-beta.7 spec URL updates (#5945)
2024-02-22 02:42:57 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.7/specs/phase0/beacon-chain.md#slashings
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#slashings
automated consensus spec URL updating to v1.5.0-alpha.3 (#6364)
2024-06-16 00:59:25 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.3/specs/bellatrix/beacon-chain.md#slashings
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func slashing_penalty_applies*(validator: Validator, epoch: Epoch): bool =
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
validator.slashed and
epoch + EPOCHS_PER_SLASHINGS_VECTOR div 2 == validator.withdrawable_epoch
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
some consensus spec v1.4.0-beta.7 spec URL updates (#5945)
2024-02-22 02:42:57 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.7/specs/phase0/beacon-chain.md#slashings
some consensus spec URL updates to v1.5.0-alpha.3 (#6382)
2024-06-22 05:28:19 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.3/specs/altair/beacon-chain.md#slashings
automated consensus spec URL updating to v1.5.0-alpha.3 (#6364)
2024-06-16 00:59:25 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.3/specs/bellatrix/beacon-chain.md#slashings
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
func get_slashing_penalty*(validator: Validator,
adjusted_total_slashing_balance,
total_balance: Gwei): Gwei =
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
# Factored out from penalty numerator to avoid uint64 overflow
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
const increment = EFFECTIVE_BALANCE_INCREMENT.Gwei
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
let penalty_numerator = validator.effective_balance div increment *
adjusted_total_slashing_balance
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
penalty_numerator div total_balance * increment
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
some consensus spec v1.4.0-beta.7 spec URL updates (#5945)
2024-02-22 02:42:57 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.7/specs/phase0/beacon-chain.md#slashings
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#slashings
automated consensus spec URL updating to v1.5.0-alpha.3 (#6364)
2024-06-16 00:59:25 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.3/specs/bellatrix/beacon-chain.md#slashings
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
func process_slashings*(state: var ForkyBeaconState, total_balance: Gwei) =
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
let
More 0.8.0 updates (#311) * replace BeaconState.finalized_{epoch,root} with BeaconState.finalized_checkpoint; rename get_delayed_activation_exit_epoch(...) to compute_activation_exit_epoch(...) and mark as 0.8.0; update get_churn_limit(...)/get_validator_churn_limit(...) to 0.8.0; update process_registry_updates(...) to 0.8.0 * update process_crosslinks(...) to 0.8.0; mark compute_start_slot_of_epoch(...) and get_committee_count(...) as 0.8.0 * mark Fork, is_slashable_validator(...), and get_beacon_proposer_index(...) as 0.8.0 * rename LATEST_SLASHED_EXIT_LENGTH to EPOCHS_PER_SLASHINGS_VECTOR; update process_slashings(...) to 0.8.0; remove pointless type conversion warning in get_previous_epoch(...) * convert remaining references to finalized_epoch to finalized_checkpoint.epoch * update slash_validator(...) to 0.8.0; mark inital value, Gwei, and time constants as 0.8.0; mark hash(...) and processBlockHeader(...) as 0.8.0 * rename WHISTLEBLOWING_REWARD_QUOTIENT to WHISTLEBLOWER_REWARD_QUOTIENT; rename LATEST_ACTIVE_INDEX_ROOTS_LENGTH to EPOCHS_PER_HISTORICAL_VECTOR (randao will also get merged into this); remove get_active_index_root(...); mark time parameter, signature domain types, and max operations per block constants as 0.8.0; update rewards and penalties constants to 0.8.0 * update is_valid_indexed_attestation(...) to 0.8.0; mark process_slot(...) as 0.8.0 * replace BeaconState.{current,previous}_justified_{epoch,root} with BeaconState.{current,previous}_justified_checkpoint
2019-07-05 08:30:05 +00:00
epoch = get_current_epoch(state)
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
adjusted_total_slashing_balance = get_adjusted_total_slashing_balance(
state, total_balance)
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
document and clean up `ValidatorIndex` usage (#3651) * document static vs dynamic range checking requirements * add `vindices` iterator to iterate over valid validator indices in a state * clean up spec comments in general * fixup Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-05-23 23:39:08 +00:00
for vidx in state.validators.vindices:
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
let validator = unsafeAddr state.validators.item(vidx)
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
if slashing_penalty_applies(validator[], epoch):
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
let penalty = get_slashing_penalty(
validator[], adjusted_total_slashing_balance, total_balance)
document and clean up `ValidatorIndex` usage (#3651) * document static vs dynamic range checking requirements * add `vindices` iterator to iterate over valid validator indices in a state * clean up spec comments in general * fixup Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-05-23 23:39:08 +00:00
decrease_balance(state, vidx, penalty)
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
automated consensus spec URL updating to v1.4.0-beta.6 (#5804)
2024-01-20 11:19:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.6/specs/phase0/beacon-chain.md#eth1-data-votes-updates
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
func process_eth1_data_reset*(state: var ForkyBeaconState) =
per HF1, split process_final_updates() (#2319)
2021-02-14 19:31:01 +00:00
let next_epoch = get_current_epoch(state) + 1
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Reset eth1 data votes
initial 0.11.0 spec version update
2020-03-14 21:54:45 +00:00
if next_epoch mod EPOCHS_PER_ETH1_VOTING_PERIOD == 0:
SSZ cleanup * be stricter about SSZ length prefix * compute zeroHash list at compile time * remove SSZ schema stuff * move SSZ navigation to ncli * cleanup a few leftover openArray uses
2020-05-27 11:36:02 +00:00
state.eth1_data_votes = default(type state.eth1_data_votes)
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
some consensus spec v1.4.0-beta.7 spec URL updates (#5945)
2024-02-22 02:42:57 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.7/specs/phase0/beacon-chain.md#effective-balances-updates
ShufflingRef approach to next-epoch validator duty calculation/prediction (#5414) * ShufflingRef approach to next-epoch validator duty calculation/prediction * refactor action_tracker.updateActions to take ShufflingRef + beacon_proposers; refactor maybeUpdateActionTrackerNextEpoch to be separate and reused function; add actual fallback logic * document one possible set of conditions * check epoch participation flags and inactivity scores to ensure no penalties and MAX_EFFECTIVE_BALANCE to ensure rewards don't matter * correctly (un)shuffle each proposer index * remove debugging assertion
2023-10-10 00:02:07 +00:00
template effective_balance_might_update*(
balance: Gwei, effective_balance: Gwei): bool =
const
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
HYSTERESIS_INCREMENT =
EFFECTIVE_BALANCE_INCREMENT.Gwei div HYSTERESIS_QUOTIENT
ShufflingRef approach to next-epoch validator duty calculation/prediction (#5414) * ShufflingRef approach to next-epoch validator duty calculation/prediction * refactor action_tracker.updateActions to take ShufflingRef + beacon_proposers; refactor maybeUpdateActionTrackerNextEpoch to be separate and reused function; add actual fallback logic * document one possible set of conditions * check epoch participation flags and inactivity scores to ensure no penalties and MAX_EFFECTIVE_BALANCE to ensure rewards don't matter * correctly (un)shuffle each proposer index * remove debugging assertion
2023-10-10 00:02:07 +00:00
DOWNWARD_THRESHOLD = HYSTERESIS_INCREMENT * HYSTERESIS_DOWNWARD_MULTIPLIER
UPWARD_THRESHOLD = HYSTERESIS_INCREMENT * HYSTERESIS_UPWARD_MULTIPLIER
balance + DOWNWARD_THRESHOLD < effective_balance or
effective_balance + UPWARD_THRESHOLD < balance
some consensus spec v1.4.0 spec URL updates (#6215)
2024-04-18 01:00:04 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/phase0/beacon-chain.md#effective-balances-updates
use EF consensus spec v1.5.0-alpha.1 test vectors (#6249)
2024-04-28 03:33:44 +00:00
func process_effective_balance_updates*(
state: var (phase0.BeaconState | altair.BeaconState |
bellatrix.BeaconState | capella.BeaconState |
deneb.BeaconState)) =
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Update effective balances with hysteresis
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
for vidx in state.validators.vindices:
ShufflingRef approach to next-epoch validator duty calculation/prediction (#5414) * ShufflingRef approach to next-epoch validator duty calculation/prediction * refactor action_tracker.updateActions to take ShufflingRef + beacon_proposers; refactor maybeUpdateActionTrackerNextEpoch to be separate and reused function; add actual fallback logic * document one possible set of conditions * check epoch participation flags and inactivity scores to ensure no penalties and MAX_EFFECTIVE_BALANCE to ensure rewards don't matter * correctly (un)shuffle each proposer index * remove debugging assertion
2023-10-10 00:02:07 +00:00
let
balance = state.balances.item(vidx)
effective_balance = state.validators.item(vidx).effective_balance
if effective_balance_might_update(balance, effective_balance):
speed up epoch processing 6x+ (#3089) * speed up epoch processing 6x+ This change above all helps contain long replay times on epoch change, reorg and deep history inspection via REST/RPC * most effective balances don't actually change due to MAX_EFFECTIVE_BALANCE * ditto for inactivity scores * avoid signature check for trusted sync aggregates pre: ``` ./ncli_db --db:mainnet_0/db bench --start-slot=-3200 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3468.621, 0.000, 3468.621, 3468.621, 1, Initialize DB 0.357, 0.938, 0.171, 52.752, 3155, Load block from database 15691.471, 0.000, 15691.471, 15691.471, 1, Load state from database 6.100, 9.469, 0.033, 526.816, 3101, Advance slot, non-epoch 579.131, 9.523, 566.936, 610.328, 100, Advance slot, epoch 18.551, 16.317, 12.664, 136.668, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` post: ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3488.541, 0.000, 3488.541, 3488.541, 1, Initialize DB 0.369, 1.123, 0.183, 63.208, 3155, Load block from database 13430.642, 0.000, 13430.642, 13430.642, 1, Load state from database 6.522, 1.721, 0.034, 36.708, 3101, Advance slot, non-epoch 89.074, 3.162, 83.573, 101.436, 100, Advance slot, epoch 18.325, 18.346, 13.005, 145.040, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * Update beacon_chain/spec/state_transition_block.nim Co-authored-by: zah <zahary@gmail.com> * avoid copying validator data in accessor ``` 5291.227, 0.000, 5291.227, 5291.227, 1, Initialize DB 0.436, 0.928, 0.138, 51.438, 3155, Load block from database 11962.826, 0.000, 11962.826, 11962.826, 1, Load state from database 6.477, 1.675, 0.037, 34.174, 3101, Advance slot, non-epoch 76.633, 3.705, 71.106, 98.085, 100, Advance slot, epoch 18.301, 18.593, 13.208, 149.153, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * work around compiler bug Co-authored-by: zah <zahary@gmail.com>
2021-11-11 19:24:29 +00:00
let new_effective_balance =
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
min(
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
balance - balance mod EFFECTIVE_BALANCE_INCREMENT.Gwei,
MAX_EFFECTIVE_BALANCE.Gwei)
speed up epoch processing 6x+ (#3089) * speed up epoch processing 6x+ This change above all helps contain long replay times on epoch change, reorg and deep history inspection via REST/RPC * most effective balances don't actually change due to MAX_EFFECTIVE_BALANCE * ditto for inactivity scores * avoid signature check for trusted sync aggregates pre: ``` ./ncli_db --db:mainnet_0/db bench --start-slot=-3200 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3468.621, 0.000, 3468.621, 3468.621, 1, Initialize DB 0.357, 0.938, 0.171, 52.752, 3155, Load block from database 15691.471, 0.000, 15691.471, 15691.471, 1, Load state from database 6.100, 9.469, 0.033, 526.816, 3101, Advance slot, non-epoch 579.131, 9.523, 566.936, 610.328, 100, Advance slot, epoch 18.551, 16.317, 12.664, 136.668, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` post: ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3488.541, 0.000, 3488.541, 3488.541, 1, Initialize DB 0.369, 1.123, 0.183, 63.208, 3155, Load block from database 13430.642, 0.000, 13430.642, 13430.642, 1, Load state from database 6.522, 1.721, 0.034, 36.708, 3101, Advance slot, non-epoch 89.074, 3.162, 83.573, 101.436, 100, Advance slot, epoch 18.325, 18.346, 13.005, 145.040, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * Update beacon_chain/spec/state_transition_block.nim Co-authored-by: zah <zahary@gmail.com> * avoid copying validator data in accessor ``` 5291.227, 0.000, 5291.227, 5291.227, 1, Initialize DB 0.436, 0.928, 0.138, 51.438, 3155, Load block from database 11962.826, 0.000, 11962.826, 11962.826, 1, Load state from database 6.477, 1.675, 0.037, 34.174, 3101, Advance slot, non-epoch 76.633, 3.705, 71.106, 98.085, 100, Advance slot, epoch 18.301, 18.593, 13.208, 149.153, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * work around compiler bug Co-authored-by: zah <zahary@gmail.com>
2021-11-11 19:24:29 +00:00
# Protect against unnecessary cache invalidation
if new_effective_balance != effective_balance:
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
state.validators.mitem(vidx).effective_balance = new_effective_balance
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
use EF consensus spec v1.5.0-alpha.1 test vectors (#6249)
2024-04-28 03:33:44 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.1/specs/electra/beacon-chain.md#updated-process_effective_balance_updates
func process_effective_balance_updates*(state: var electra.BeaconState) =
# Update effective balances with hysteresis
for vidx in state.validators.vindices:
let
balance = state.balances.item(vidx)
effective_balance = state.validators.item(vidx).effective_balance
if effective_balance_might_update(balance, effective_balance):
some debugRaiseAssert to debugComment; unblock some spec tests (#6284)
2024-05-14 09:19:24 +00:00
debugComment "amortize validator read access"
use EF consensus spec v1.5.0-alpha.1 test vectors (#6249)
2024-04-28 03:33:44 +00:00
# Wrapping MAX_EFFECTIVE_BALANCE_ELECTRA.Gwei and
# MIN_ACTIVATION_BALANCE.Gwei in static() results
# in
# beacon_chain/spec/state_transition_epoch.nim(1067, 20) Error: expected: ':', but got: '('
# even though it'd be better to statically verify safety
let
effective_balance_limit =
if has_compounding_withdrawal_credential(
state.validators.item(vidx)):
MAX_EFFECTIVE_BALANCE_ELECTRA.Gwei
else:
MIN_ACTIVATION_BALANCE.Gwei
new_effective_balance =
min(
balance - balance mod EFFECTIVE_BALANCE_INCREMENT.Gwei,
effective_balance_limit)
# Protect against unnecessary cache invalidation
if new_effective_balance != effective_balance:
state.validators.mitem(vidx).effective_balance = new_effective_balance
some consensus spec v1.4.0 spec URL updates (#6215)
2024-04-18 01:00:04 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/phase0/beacon-chain.md#slashings-balances-updates
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
func process_slashings_reset*(state: var ForkyBeaconState) =
per HF1, split process_final_updates() (#2319)
2021-02-14 19:31:01 +00:00
let next_epoch = get_current_epoch(state) + 1
0.8.0, continued (#315) * rename get_genesis_beacon_state(...) to initialize_beacon_state_from_eth1(...); remove unused vestiges get_temporary_block_header(...) and get_empty_block(...); partly align initialize_beacon_state_from_eth1(...) with 0.8.0 version; implement CompactCommittee object type; update process_final_updates(...) to 0.8.0 * mark get_shard_delta(...) as 0.8.0 * mark get_total_active_balance(...) and epoch transition helper functions as 0.8.0 * move FORK_CHOICE_BALANCE_INCREMENT, not in spec anymore, to sole user in fork_choice
2019-07-10 12:23:02 +00:00
# Reset slashings
Small fixes (#1165) * random fixes * create dump dir on startup * don't crash on failure to write dump * fix a few `uint64` instances being used when indexing arrays - this should be a compile error but isn't due to compiler bugs * fix standalone test_block_pool compilation * add signed block processing in ncli * reuse cache entry instead of allocating a new one * allow for small clock disparities when validating blocks
2020-06-12 16:43:20 +00:00
state.slashings[int(next_epoch mod EPOCHS_PER_SLASHINGS_VECTOR)] = 0.Gwei
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
automated consensus spec URL updating to v1.4.0-beta.6 (#5804)
2024-01-20 11:19:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.6/specs/phase0/beacon-chain.md#randao-mixes-updates
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
func process_randao_mixes_reset*(state: var ForkyBeaconState) =
per HF1, split process_final_updates() (#2319)
2021-02-14 19:31:01 +00:00
let
current_epoch = get_current_epoch(state)
next_epoch = current_epoch + 1
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
# Set randao mix
Fix #378, int to bytes32, LATEST_RANDAO_MIXES, get_seed, get_crosslink_committee, get_compact_committee_root (#380)
2019-09-04 13:57:18 +00:00
state.randao_mixes[next_epoch mod EPOCHS_PER_HISTORICAL_VECTOR] =
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
get_randao_mix(state, current_epoch)
era file verification (#3605) * era file verification Implement and document era file verification * era file states now come with block applied for easier verification * clarify conflicting version handling * document verification requirements * remove count from name, use start-era, end-root to discover range * remove obsolete todo * abstract out block root loading
2022-05-10 00:28:46 +00:00
func compute_historical_root*(state: var ForkyBeaconState): Eth2Digest =
# Equivalent to hash_tree_root(foo: HistoricalBatch), but without using
# significant additional stack or heap.
automated consensus spec URL updating to v1.4.0-beta.6 (#5804)
2024-01-20 11:19:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.6/specs/phase0/beacon-chain.md#historicalbatch
era file verification (#3605) * era file verification Implement and document era file verification * era file states now come with block applied for easier verification * clarify conflicting version handling * document verification requirements * remove count from name, use start-era, end-root to discover range * remove obsolete todo * abstract out block root loading
2022-05-10 00:28:46 +00:00
# In response to https://github.com/status-im/nimbus-eth2/issues/921
hash_tree_root([
hash_tree_root(state.block_roots), hash_tree_root(state.state_roots)])
automated consensus spec URL updating to v1.4.0-beta.6 (#5804)
2024-01-20 11:19:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.6/specs/phase0/beacon-chain.md#historical-roots-updates
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
func process_historical_roots_update*(state: var ForkyBeaconState) =
update 62 spec URLs to v1.1.2 (#2979)
2021-10-12 10:17:37 +00:00
## Set historical root accumulator
per HF1, split process_final_updates() (#2319)
2021-02-14 19:31:01 +00:00
let next_epoch = get_current_epoch(state) + 1
Restore compilation with custom presets (#1309) This also assigns precise types to the constants in the minimal and mainnet presets in order to reduce the chance of compilation errors when custom presets are used (previously, only the custom presets have precisely assigned types for the constants).
2020-07-13 14:44:58 +00:00
if next_epoch mod (SLOTS_PER_HISTORICAL_ROOT div SLOTS_PER_EPOCH) == 0:
reduce stack space usage in process_final_updates(...) (#925) * reduce stack space usage in process_final_updates(...) to avoid fuzzed segfault in https://github.com/status-im/nim-beacon-chain/issues/921 * document motivation behind manually constructing hash_tree_root of a HistoricalBatch
2020-04-23 11:39:38 +00:00
# Equivalent to hash_tree_root(foo: HistoricalBatch), but without using
# significant additional stack or heap.
some consensus spec URL updates to v1.5.0-alpha.3 (#6382)
2024-06-22 05:28:19 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.3/specs/phase0/beacon-chain.md#historicalbatch
update nim-beacon-chain to nimbus-eth2 in beacon_chain/, ncli/, tests/, and README.md (#1843)
2020-10-08 19:02:05 +00:00
# In response to https://github.com/status-im/nimbus-eth2/issues/921
era file verification (#3605) * era file verification Implement and document era file verification * era file states now come with block applied for easier verification * clarify conflicting version handling * document verification requirements * remove count from name, use start-era, end-root to discover range * remove obsolete todo * abstract out block root loading
2022-05-10 00:28:46 +00:00
if not state.historical_roots.add state.compute_historical_root():
Reset cached indices when resetting cache on SSZ read (#2480) * Reset cached indices when resetting cache on SSZ read When deserializing into an existing structure, the cache should be cleared - goes for json also. Also improve error messages.
2021-04-08 10:11:04 +00:00
raiseAssert "no more room for historical roots, so long and thanks for the fish!"
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
automated consensus spec URL updating to v1.4.0-beta.6 (#5804)
2024-01-20 11:19:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.6/specs/phase0/beacon-chain.md#participation-records-rotation
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
func process_participation_record_updates*(state: var phase0.BeaconState) =
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
# Rotate current/previous epoch attestations - using swap avoids copying all
# elements using a slow genericSeqAssign
state.previous_epoch_attestations.clear()
swap(state.previous_epoch_attestations, state.current_epoch_attestations)
Split state_transition (#301)
2019-06-28 13:44:44 +00:00
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#participation-flags-updates
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
func process_participation_flag_updates*(
use 'deneb' in 4 more modules (#4658)
2023-02-23 18:06:57 +00:00
state: var (altair.BeaconState | bellatrix.BeaconState |
add Electra overloads for spec functions; add Electra block processing (#5963)
2024-02-26 02:38:21 +00:00
capella.BeaconState | deneb.BeaconState |
electra.BeaconState)) =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
state.previous_epoch_participation = state.current_epoch_participation
block_clearance, ncli, and ncli_db Altair state saving (#2672) * block_clearance, ncli, and ncli_db Altair state saving * avoid invalidating SSZ hash caches with every assignment
2021-06-24 18:34:08 +00:00
const zero = 0.ParticipationFlags
for i in 0 ..< state.current_epoch_participation.len:
60% state replay speedup (#4434) * 60% state replay speedup * don't use HashList for epoch participation - in addition to the code currently clearing the caches several times redundantly, clearing has to be done each block nullifying the benefit (35%) * introduce active balance cache - computing it is slow due to cache unfriendliness in the random access pattern and bounds checking and we do it for every block - this cache follows the same update pattern as the active validator index cache (20%) * avoid recomputing base reward several times per attestation (5%) Applying 1024 blocks goes from 20s to ~8s on my laptop - these kinds of requests happen on historical REST queries but also whenever there's a reorg. * fix test and diffs
2022-12-19 12:01:49 +00:00
asList(state.current_epoch_participation)[i] = zero
block_clearance, ncli, and ncli_db Altair state saving (#2672) * block_clearance, ncli, and ncli_db Altair state saving * avoid invalidating SSZ hash caches with every assignment
2021-06-24 18:34:08 +00:00
# Shouldn't be wasted zeroing, because state.current_epoch_participation only
# grows. New elements are automatically initialized to 0, as required.
60% state replay speedup (#4434) * 60% state replay speedup * don't use HashList for epoch participation - in addition to the code currently clearing the caches several times redundantly, clearing has to be done each block nullifying the benefit (35%) * introduce active balance cache - computing it is slow due to cache unfriendliness in the random access pattern and bounds checking and we do it for every block - this cache follows the same update pattern as the active validator index cache (20%) * avoid recomputing base reward several times per attestation (5%) Applying 1024 blocks goes from 20s to ~8s on my laptop - these kinds of requests happen on historical REST queries but also whenever there's a reorg. * fix test and diffs
2022-12-19 12:01:49 +00:00
doAssert state.current_epoch_participation.asList.setLen(state.validators.len)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#sync-committee-updates
proc -> func, mainly in spec/state transition and adjecent modules (#3405)
2022-02-17 11:53:55 +00:00
func process_sync_committee_updates*(
eip4844 fork and epoch transition tests; some eip4844 gossip (#4393)
2022-12-06 16:43:11 +00:00
state: var (altair.BeaconState | bellatrix.BeaconState |
add Electra overloads for spec functions; add Electra block processing (#5963)
2024-02-26 02:38:21 +00:00
capella.BeaconState | deneb.BeaconState |
electra.BeaconState)) =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
let next_epoch = get_current_epoch(state) + 1
time: spring cleaning (#3262) Time in the beacon chain is expressed relative to the genesis time - this PR creates a `beacon_time` module that collects helpers and utilities for dealing the time units - the new module does not deal with actual wall time (that's remains in `beacon_clock`). Collecting the time related stuff in one place makes it easier to find, avoids some circular imports and allows more easily identifying the code actually needs wall time to operate. * move genesis-time-related functionality into `spec/beacon_time` * avoid using `chronos.Duration` for time differences - it does not support negative values (such as when something happens earlier than it should) * saturate conversions between `FAR_FUTURE_XXX`, so as to avoid overflows * fix delay reporting in validator client so it uses the expected deadline of the slot, not "closest wall slot" * simplify looping over the slots of an epoch * `compute_start_slot_at_epoch` -> `start_slot` * `compute_epoch_at_slot` -> `epoch` A follow-up PR will (likely) introduce saturating arithmetic for the time units - this is merely code moves, renames and fixing of small bugs.
2022-01-11 10:01:54 +00:00
if next_epoch.is_sync_committee_period():
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
state.current_sync_committee = state.next_sync_committee
state.next_sync_committee = get_next_sync_committee(state)
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#inactivity-scores
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
template compute_inactivity_update(
use correct `INACTIVITY_SCORE_RECOVERY_RATE` if overridden from default (#6091) When a config defines a different `INACTIVITY_SCORE_RECOVERY_RATE` than the default, `process_inactivity_updates` uses an incorrect rate ever since #2710 when `INACTIVITY_SCORE_RECOVERY_RATE` became configurable.
2024-03-17 12:32:30 +00:00
cfg: RuntimeConfig,
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
state: altair.BeaconState | bellatrix.BeaconState | capella.BeaconState |
add Electra overloads for spec functions; add Electra block processing (#5963)
2024-02-26 02:38:21 +00:00
deneb.BeaconState | electra.BeaconState,
use correct `INACTIVITY_SCORE_RECOVERY_RATE` if overridden from default (#6091) When a config defines a different `INACTIVITY_SCORE_RECOVERY_RATE` than the default, `process_inactivity_updates` uses an incorrect rate ever since #2710 when `INACTIVITY_SCORE_RECOVERY_RATE` became configurable.
2024-03-17 12:32:30 +00:00
info: altair.EpochInfo,
make `Gwei` `distinct` (#6090) #6087 introduced a subtle change to `nim-web3` resulting in `Gwei` to be serialized differently than before. Using a `distinct` type for `Gwei` improves type safety and avoids such problems in the future.
2024-03-19 13:22:07 +00:00
pre_inactivity_score: uint64): uint64 =
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
let previous_epoch = get_previous_epoch(state) # get_eligible_validator_indices()
# Increase the inactivity score of inactive validators
var inactivity_score = pre_inactivity_score
# TODO activeness already checked; remove redundant checks between
# is_active_validator and is_unslashed_participating_index
if is_unslashed_participating_index(
state, TIMELY_TARGET_FLAG_INDEX, previous_epoch, index.ValidatorIndex):
inactivity_score -= min(1'u64, inactivity_score)
else:
inactivity_score += cfg.INACTIVITY_SCORE_BIAS
# Decrease the inactivity score of all eligible validators during a
# leak-free epoch
if not_in_inactivity_leak:
use correct `INACTIVITY_SCORE_RECOVERY_RATE` if overridden from default (#6091) When a config defines a different `INACTIVITY_SCORE_RECOVERY_RATE` than the default, `process_inactivity_updates` uses an incorrect rate ever since #2710 when `INACTIVITY_SCORE_RECOVERY_RATE` became configurable.
2024-03-17 12:32:30 +00:00
inactivity_score -=
min(cfg.INACTIVITY_SCORE_RECOVERY_RATE, inactivity_score)
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
inactivity_score
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#inactivity-scores
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
func process_inactivity_updates*(
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
cfg: RuntimeConfig,
use 'deneb' in 4 more modules (#4658)
2023-02-23 18:06:57 +00:00
state: var (altair.BeaconState | bellatrix.BeaconState |
add Electra overloads for spec functions; add Electra block processing (#5963)
2024-02-26 02:38:21 +00:00
capella.BeaconState | deneb.BeaconState | electra.BeaconState),
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
info: altair.EpochInfo) =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
# Score updates based on previous epoch participation, skip genesis epoch
if get_current_epoch(state) == GENESIS_EPOCH:
return
remove false OnBlockAdded dependency on phase0 HashedBeaconState (#2661) * remove false OnBlockAdded dependency on phase.HashedBeaconState * introduce altair data types into block_clearance; update some alpha.6 spec refs to alpha.7; add get_active_validator_indices_len ForkedHashedBeaconState wrapper * switch many modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 and remove rm'd G2_POINT_AT_INFINITY * switch more modules from using datatypes (with phase0 states/blocks) to datatypes/base (fork-independent); update spec refs from alpha.6 to alpha.7 * remove unnecessary phase0-only wrapper of get_attesting_indices(); allow signatures_batch to process either fork; remove O(n^2) nested loop in process_inactivity_updates(); add altair support to getAttestationsforTestBlock() * add Altair versions of asSigVerified(), asTrusted(), and makeBeaconBlock() * fix spec URL to be Altair for Altair makeBeaconBlock()
2021-06-21 08:35:24 +00:00
let
[ncli_db] bugfix: take into account finalization delay in reward calc post Altair This fixes a problem affecting Prater's epoch 64444.
2022-01-27 16:37:30 +00:00
finality_delay = get_finality_delay(state)
not_in_inactivity_leak = not is_in_inactivity_leak(finality_delay)
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
for index in 0'u64 ..< state.validators.lenu64:
refactor get_flag_and_inactivity_deltas() loop body to use per-validator get_flag_and_inactivity_delta() (#6007)
2024-03-01 16:31:25 +00:00
if not is_eligible_validator(info.validators[index]):
continue
refactor epoch state transition to facilitate individual validator balance change calculations (#5910)
2024-02-20 05:14:52 +00:00
let
pre_inactivity_score = state.inactivity_scores.asSeq()[index]
inactivity_score =
use correct `INACTIVITY_SCORE_RECOVERY_RATE` if overridden from default (#6091) When a config defines a different `INACTIVITY_SCORE_RECOVERY_RATE` than the default, `process_inactivity_updates` uses an incorrect rate ever since #2710 when `INACTIVITY_SCORE_RECOVERY_RATE` became configurable.
2024-03-17 12:32:30 +00:00
compute_inactivity_update(cfg, state, info, pre_inactivity_score)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
speed up epoch processing 6x+ (#3089) * speed up epoch processing 6x+ This change above all helps contain long replay times on epoch change, reorg and deep history inspection via REST/RPC * most effective balances don't actually change due to MAX_EFFECTIVE_BALANCE * ditto for inactivity scores * avoid signature check for trusted sync aggregates pre: ``` ./ncli_db --db:mainnet_0/db bench --start-slot=-3200 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3468.621, 0.000, 3468.621, 3468.621, 1, Initialize DB 0.357, 0.938, 0.171, 52.752, 3155, Load block from database 15691.471, 0.000, 15691.471, 15691.471, 1, Load state from database 6.100, 9.469, 0.033, 526.816, 3101, Advance slot, non-epoch 579.131, 9.523, 566.936, 610.328, 100, Advance slot, epoch 18.551, 16.317, 12.664, 136.668, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` post: ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3488.541, 0.000, 3488.541, 3488.541, 1, Initialize DB 0.369, 1.123, 0.183, 63.208, 3155, Load block from database 13430.642, 0.000, 13430.642, 13430.642, 1, Load state from database 6.522, 1.721, 0.034, 36.708, 3101, Advance slot, non-epoch 89.074, 3.162, 83.573, 101.436, 100, Advance slot, epoch 18.325, 18.346, 13.005, 145.040, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * Update beacon_chain/spec/state_transition_block.nim Co-authored-by: zah <zahary@gmail.com> * avoid copying validator data in accessor ``` 5291.227, 0.000, 5291.227, 5291.227, 1, Initialize DB 0.436, 0.928, 0.138, 51.438, 3155, Load block from database 11962.826, 0.000, 11962.826, 11962.826, 1, Load state from database 6.477, 1.675, 0.037, 34.174, 3101, Advance slot, non-epoch 76.633, 3.705, 71.106, 98.085, 100, Advance slot, epoch 18.301, 18.593, 13.208, 149.153, 3155, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * work around compiler bug Co-authored-by: zah <zahary@gmail.com>
2021-11-11 19:24:29 +00:00
# Most inactivity scores remain at 0 - avoid invalidating cache
if pre_inactivity_score != inactivity_score:
state.inactivity_scores[index] = inactivity_score
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
some consensus spec URL updates to v1.5.0-alpha.3 (#6382)
2024-06-22 05:28:19 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.3/specs/capella/beacon-chain.md#historical-summaries-updates
use v1.3.0-rc.0 consensus spec test vectors (#4472)
2023-01-09 14:15:43 +00:00
func process_historical_summaries_update*(
add Electra overloads for spec functions; add Electra block processing (#5963)
2024-02-26 02:38:21 +00:00
state: var (capella.BeaconState | deneb.BeaconState | electra.BeaconState)):
use v1.3.0-rc.0 consensus spec test vectors (#4472)
2023-01-09 14:15:43 +00:00
Result[void, cstring] =
# Set historical block root accumulator.
let next_epoch = get_current_epoch(state) + 1
if next_epoch mod (SLOTS_PER_HISTORICAL_ROOT div SLOTS_PER_EPOCH) == 0:
let historical_summary = HistoricalSummary(
block_summary_root: hash_tree_root(state.block_roots),
state_summary_root: hash_tree_root(state.state_roots),
)
if not state.historical_summaries.add(historical_summary):
return err("process_historical_summaries_update: state.historical_summaries full")
ok()
use EF consensus-specs v1.5.0-alpha.3 test vectors (#6362)
2024-06-15 22:15:27 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.3/specs/electra/beacon-chain.md#new-process_pending_balance_deposits
add rest of EF consensus spec test Electra epoch transition fixture(s) (#6232)
2024-04-24 12:28:47 +00:00
func process_pending_balance_deposits*(
cfg: RuntimeConfig, state: var electra.BeaconState,
rm debugRaiseAssert; clean up several debugComments (#6308) * rm debugRaiseAssert; clean up several debugComments * exception linting
2024-05-23 21:51:09 +00:00
cache: var StateCache): Result[void, cstring] =
use EF consensus-specs v1.5.0-alpha.3 test vectors (#6362)
2024-06-15 22:15:27 +00:00
let available_for_processing = state.deposit_balance_to_consume +
get_activation_exit_churn_limit(cfg, state, cache)
add rest of EF consensus spec test Electra epoch transition fixture(s) (#6232)
2024-04-24 12:28:47 +00:00
var
processed_amount = 0.Gwei
use EF consensus-specs v1.5.0-alpha.3 test vectors (#6362)
2024-06-15 22:15:27 +00:00
next_deposit_index = 0
deposits_to_postpone: seq[PendingBalanceDeposit]
add rest of EF consensus spec test Electra epoch transition fixture(s) (#6232)
2024-04-24 12:28:47 +00:00
for deposit in state.pending_balance_deposits:
use EF consensus-specs v1.5.0-alpha.3 test vectors (#6362)
2024-06-15 22:15:27 +00:00
let validator = state.validators.item(deposit.index)
rm debugRaiseAssert; clean up several debugComments (#6308) * rm debugRaiseAssert; clean up several debugComments * exception linting
2024-05-23 21:51:09 +00:00
let deposit_validator_index = ValidatorIndex.init(deposit.index).valueOr:
use EF consensus-specs v1.5.0-alpha.3 test vectors (#6362)
2024-06-15 22:15:27 +00:00
# TODO this function in spec doesn't really have error returns as such
rm debugRaiseAssert; clean up several debugComments (#6308) * rm debugRaiseAssert; clean up several debugComments * exception linting
2024-05-23 21:51:09 +00:00
return err("process_pending_balance_deposits: deposit index out of range")
use EF consensus-specs v1.5.0-alpha.3 test vectors (#6362)
2024-06-15 22:15:27 +00:00
# Validator is exiting, postpone the deposit until after withdrawable epoch
if validator.exit_epoch < FAR_FUTURE_EPOCH:
if get_current_epoch(state) <= validator.withdrawable_epoch:
deposits_to_postpone.add(deposit)
# Deposited balance will never become active. Increase balance but do not
# consume churn
else:
increase_balance(state, deposit_validator_index, deposit.amount)
# Validator is not exiting, attempt to process deposit
else:
# Deposit does not fit in the churn, no more deposit processing in this
# epoch.
if processed_amount + deposit.amount > available_for_processing:
break
# Deposit fits in the churn, process it. Increase balance and consume churn.
else:
increase_balance(state, deposit_validator_index, deposit.amount)
processed_amount += deposit.amount
# Regardless of how the deposit was handled, we move on in the queue.
next_deposit_index += 1
add rest of EF consensus spec test Electra epoch transition fixture(s) (#6232)
2024-04-24 12:28:47 +00:00
state.pending_balance_deposits =
HashList[PendingBalanceDeposit, Limit PENDING_BALANCE_DEPOSITS_LIMIT].init(
state.pending_balance_deposits.asSeq[next_deposit_index..^1])
if len(state.pending_balance_deposits) == 0:
state.deposit_balance_to_consume = Gwei(0)
else:
state.deposit_balance_to_consume =
available_for_processing - processed_amount
use EF consensus-specs v1.5.0-alpha.3 test vectors (#6362)
2024-06-15 22:15:27 +00:00
debugComment "yet another in-theory-might-overflow-maybe things, look at these more carefully"
if len(deposits_to_postpone) > 0:
discard state.pending_balance_deposits.add deposits_to_postpone
rm debugRaiseAssert; clean up several debugComments (#6308) * rm debugRaiseAssert; clean up several debugComments * exception linting
2024-05-23 21:51:09 +00:00
ok()
add rest of EF consensus spec test Electra epoch transition fixture(s) (#6232)
2024-04-24 12:28:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.0/specs/electra/beacon-chain.md#new-process_pending_consolidations
rm debugRaiseAssert; clean up several debugComments (#6308) * rm debugRaiseAssert; clean up several debugComments * exception linting
2024-05-23 21:51:09 +00:00
func process_pending_consolidations*(
cfg: RuntimeConfig, state: var electra.BeaconState):
Result[void, cstring] =
add rest of EF consensus spec test Electra epoch transition fixture(s) (#6232)
2024-04-24 12:28:47 +00:00
var next_pending_consolidation = 0
for pending_consolidation in state.pending_consolidations:
let source_validator =
state.validators.item(pending_consolidation.source_index)
if source_validator.slashed:
next_pending_consolidation += 1
continue
if source_validator.withdrawable_epoch > get_current_epoch(state):
break
rm debugRaiseAssert; clean up several debugComments (#6308) * rm debugRaiseAssert; clean up several debugComments * exception linting
2024-05-23 21:51:09 +00:00
let
source_validator_index = ValidatorIndex.init(
pending_consolidation.source_index).valueOr:
return err("process_pending_consolidations: source index out of range")
target_validator_index = ValidatorIndex.init(
pending_consolidation.target_index).valueOr:
return err("process_pending_consolidations: target index out of range")
add rest of EF consensus spec test Electra epoch transition fixture(s) (#6232)
2024-04-24 12:28:47 +00:00
# Churn any target excess active balance of target and raise its max
rm debugRaiseAssert; clean up several debugComments (#6308) * rm debugRaiseAssert; clean up several debugComments * exception linting
2024-05-23 21:51:09 +00:00
switch_to_compounding_validator(state, target_validator_index)
add rest of EF consensus spec test Electra epoch transition fixture(s) (#6232)
2024-04-24 12:28:47 +00:00
# Move active balance to target. Excess balance is withdrawable.
rm debugRaiseAssert; clean up several debugComments (#6308) * rm debugRaiseAssert; clean up several debugComments * exception linting
2024-05-23 21:51:09 +00:00
let active_balance = get_active_balance(state, source_validator_index)
decrease_balance(state, source_validator_index, active_balance)
increase_balance(state, target_validator_index, active_balance)
add rest of EF consensus spec test Electra epoch transition fixture(s) (#6232)
2024-04-24 12:28:47 +00:00
inc next_pending_consolidation
state.pending_consolidations =
HashList[PendingConsolidation, Limit PENDING_CONSOLIDATIONS_LIMIT].init(
state.pending_consolidations.asSeq[next_pending_consolidation..^1])
rm debugRaiseAssert; clean up several debugComments (#6308) * rm debugRaiseAssert; clean up several debugComments * exception linting
2024-05-23 21:51:09 +00:00
ok()
some consensus spec v1.4.0 spec URL updates (#6208)
2024-04-17 03:51:16 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/phase0/beacon-chain.md#epoch-processing
avoid extraneous hash root calculation (#2537) When applying a block, we'll currently compute a state root for the state after slot processing but before block processing - this is unnecessary when a block is being applied because the intermediate state root is never observed.
2021-05-05 06:54:21 +00:00
proc process_epoch*(
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
cfg: RuntimeConfig, state: var phase0.BeaconState, flags: UpdateFlags,
consensus specs v1.2.0-rc.1 (#3661)
2022-05-25 13:49:29 +00:00
cache: var StateCache, info: var phase0.EpochInfo): Result[void, cstring] =
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
let epoch = get_current_epoch(state)
trace "process_epoch", epoch
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
info.init(state)
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
info.process_attestations(state, cache)
Justification/finalization amelioration using testnet1 shard count and epoch length (#331) * working justification, 1 node, 64 validators * closer to tn1 params: 256 validators * more debugging output; switch to minimum test case * working justification and finalization in local network simulation * fix currently incorrect state transition/attestation test assumption
2019-08-14 08:56:32 +00:00
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
process_justification_and_finalization(state, info.balances, flags)
Increase finalization and finalization checking robustness (#990) * fix some warnings related to beacon_node splitting; reimplement finalization verification more robustly; improve attestation pool block selection logic * re-add missing import * whitelist allowed state transition flags and make rollback/restore naming more consistent * restore usage of update flags passed into skipAndUpdateState(...) in addition to the potential verifyFinalization flag * switch rest of rollback -> restore
2020-05-09 12:43:15 +00:00
# state.slot hasn't been incremented yet.
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
if strictVerification in flags and epoch >= 2:
doAssert state.current_justified_checkpoint.epoch + 2 >= epoch
verifyFinalization checks for justification being consistent with later finalization (#1386)
2020-07-28 17:35:32 +00:00
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
if strictVerification in flags and epoch >= 3:
Increase finalization and finalization checking robustness (#990) * fix some warnings related to beacon_node splitting; reimplement finalization verification more robustly; improve attestation pool block selection logic * re-add missing import * whitelist allowed state transition flags and make rollback/restore naming more consistent * restore usage of update flags passed into skipAndUpdateState(...) in addition to the potential verifyFinalization flag * switch rest of rollback -> restore
2020-05-09 12:43:15 +00:00
# Rule 2/3/4 finalization results in the most pessimal case. The other
# three finalization rules finalize more quickly as long as the any of
# the finalization rules triggered.
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
doAssert state.finalized_checkpoint.epoch + 3 >= epoch
Justification/finalization amelioration using testnet1 shard count and epoch length (#331) * working justification, 1 node, 64 validators * closer to tn1 params: 256 validators * more debugging output; switch to minimum test case * working justification and finalization in local network simulation * fix currently incorrect state transition/attestation test assumption
2019-08-14 08:56:32 +00:00
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
process_rewards_and_penalties(state, info)
consensus specs v1.2.0-rc.1 (#3661)
2022-05-25 13:49:29 +00:00
? process_registry_updates(cfg, state, cache)
Cleanups (#3123) Renames and cleanups split out from the validator monitoring branch, so as to reduce conflict area vs other PR:s * add constants for expected message timing * name validators after the messages they validate, mostly, to make grepping easier * unify field naming of EpochInfo across forks to make cross-fork code easier
2021-11-25 12:20:36 +00:00
process_slashings(state, info.balances.current_epoch)
split state_transition() into slots/block parts and use only block where appropriate (#2630)
2021-06-03 09:42:25 +00:00
process_eth1_data_reset(state)
process_effective_balance_updates(state)
process_slashings_reset(state)
process_randao_mixes_reset(state)
process_historical_roots_update(state)
process_participation_record_updates(state)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
consensus specs v1.2.0-rc.1 (#3661)
2022-05-25 13:49:29 +00:00
ok()
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
func init*(
info: var altair.EpochInfo,
eip4844 fork and epoch transition tests; some eip4844 gossip (#4393)
2022-12-06 16:43:11 +00:00
state: altair.BeaconState | bellatrix.BeaconState | capella.BeaconState |
add Electra overloads for spec functions; add Electra block processing (#5963)
2024-02-26 02:38:21 +00:00
deneb.BeaconState | electra.BeaconState) =
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
# init participation, overwriting the full structure
info.balances = get_unslashed_participating_balances(state)
info.validators.setLen(state.validators.len())
let previous_epoch = get_previous_epoch(state)
for index in 0..<state.validators.len():
var flags: set[ParticipationFlag]
if is_eligible_validator(state.validators[index], previous_epoch):
flags.incl ParticipationFlag.eligible
info.validators[index] = ParticipationInfo(
flags: flags
)
func init*(
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
T: type altair.EpochInfo,
eip4844 fork and epoch transition tests; some eip4844 gossip (#4393)
2022-12-06 16:43:11 +00:00
state: altair.BeaconState | bellatrix.BeaconState | capella.BeaconState |
add 10 sub-test-fixtures for electra epoch transitions; use v2.0.4 for Nim 2.x (#6229)
2024-04-23 01:15:55 +00:00
deneb.BeaconState | electra.BeaconState): T =
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
init(result, state)
some consensus spec URL updates to v1.5.0-alpha.3 (#6382)
2024-06-22 05:28:19 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.3/specs/altair/beacon-chain.md#epoch-processing
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
proc process_epoch*(
use v1.3.0-alpha.1 consensus spec test vectors (#4338)
2022-11-21 07:44:49 +00:00
cfg: RuntimeConfig,
use v1.3.0-rc.0 consensus spec test vectors (#4472)
2023-01-09 14:15:43 +00:00
state: var (altair.BeaconState | bellatrix.BeaconState),
consensus specs v1.2.0-rc.1 (#3661)
2022-05-25 13:49:29 +00:00
flags: UpdateFlags, cache: var StateCache, info: var altair.EpochInfo):
Result[void, cstring] =
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
let epoch = get_current_epoch(state)
trace "process_epoch", epoch
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
info.init(state)
Fix Altair epoch processing and add/enable remaining epoch processing tests (#2619) * fix Altair epoch state transitions * re-enable and re-add near-complete Altair epoch processing test vectors: inactivity, justification & finalization, participation flag, slashings, and sync committee updates
2021-05-30 20:23:10 +00:00
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#justification-and-finalization
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
# [Modified in Altair]
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
process_justification_and_finalization(state, info.balances, flags)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
# state.slot hasn't been incremented yet.
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
if strictVerification in flags and epoch >= 2:
doAssert state.current_justified_checkpoint.epoch + 2 >= epoch
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
if strictVerification in flags and epoch >= 3:
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
# Rule 2/3/4 finalization results in the most pessimal case. The other
# three finalization rules finalize more quickly as long as the any of
# the finalization rules triggered.
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
doAssert state.finalized_checkpoint.epoch + 3 >= epoch
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
process_inactivity_updates(cfg, state, info) # [New in Altair]
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#rewards-and-penalties
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
process_rewards_and_penalties(cfg, state, info) # [Modified in Altair]
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
some consensus spec v1.4.0 spec URL updates (#6215)
2024-04-18 01:00:04 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/phase0/beacon-chain.md#registry-updates
consensus specs v1.2.0-rc.1 (#3661)
2022-05-25 13:49:29 +00:00
? process_registry_updates(cfg, state, cache)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#slashings
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
process_slashings(state, info.balances.current_epoch) # [Modified in Altair]
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
process_eth1_data_reset(state)
process_effective_balance_updates(state)
process_slashings_reset(state)
process_randao_mixes_reset(state)
process_historical_roots_update(state)
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
process_participation_flag_updates(state) # [New in Altair]
process_sync_committee_updates(state) # [New in Altair]
consensus specs v1.2.0-rc.1 (#3661)
2022-05-25 13:49:29 +00:00
ok()
use v1.3.0-rc.0 consensus spec test vectors (#4472)
2023-01-09 14:15:43 +00:00
automated consensus spec URL updating to v1.4.0-beta.5 (#5647)
2023-12-05 02:34:45 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.5/specs/capella/beacon-chain.md#epoch-processing
use v1.3.0-rc.0 consensus spec test vectors (#4472)
2023-01-09 14:15:43 +00:00
proc process_epoch*(
cfg: RuntimeConfig,
add rest of EF consensus spec test Electra epoch transition fixture(s) (#6232)
2024-04-24 12:28:47 +00:00
state: var (capella.BeaconState | deneb.BeaconState),
flags: UpdateFlags, cache: var StateCache, info: var altair.EpochInfo):
Result[void, cstring] =
let epoch = get_current_epoch(state)
trace "process_epoch", epoch
info.init(state)
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#justification-and-finalization
process_justification_and_finalization(state, info.balances, flags)
# state.slot hasn't been incremented yet.
if strictVerification in flags:
# Rule 2/3/4 finalization results in the most pessimal case. The other
# three finalization rules finalize more quickly as long as the any of
# the finalization rules triggered.
if (epoch >= 2 and state.current_justified_checkpoint.epoch + 2 < epoch) or
(epoch >= 3 and state.finalized_checkpoint.epoch + 3 < epoch):
fatal "The network did not finalize",
epoch, finalizedEpoch = state.finalized_checkpoint.epoch
quit 1
process_inactivity_updates(cfg, state, info)
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#rewards-and-penalties
process_rewards_and_penalties(cfg, state, info)
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.6/specs/phase0/beacon-chain.md#registry-updates
? process_registry_updates(cfg, state, cache)
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#slashings
process_slashings(state, info.balances.current_epoch)
process_eth1_data_reset(state)
process_effective_balance_updates(state)
process_slashings_reset(state)
process_randao_mixes_reset(state)
? process_historical_summaries_update(state) # [Modified in Capella]
process_participation_flag_updates(state)
process_sync_committee_updates(state)
ok()
add EF consensus spec test Electra block sanity fixtures (#6250) * add EF consensus spec test Electra block sanity fixtures * move process_consolidation/process_deposit_receipt before usage
2024-04-28 09:15:03 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.5.0-alpha.1/specs/electra/beacon-chain.md#epoch-processing
add rest of EF consensus spec test Electra epoch transition fixture(s) (#6232)
2024-04-24 12:28:47 +00:00
proc process_epoch*(
cfg: RuntimeConfig, state: var electra.BeaconState,
use v1.3.0-rc.0 consensus spec test vectors (#4472)
2023-01-09 14:15:43 +00:00
flags: UpdateFlags, cache: var StateCache, info: var altair.EpochInfo):
Result[void, cstring] =
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
let epoch = get_current_epoch(state)
trace "process_epoch", epoch
use v1.3.0-rc.0 consensus spec test vectors (#4472)
2023-01-09 14:15:43 +00:00
info.init(state)
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#justification-and-finalization
use v1.3.0-rc.0 consensus spec test vectors (#4472)
2023-01-09 14:15:43 +00:00
process_justification_and_finalization(state, info.balances, flags)
# state.slot hasn't been incremented yet.
Local sim impovements (#4551) * Local sim impovements * Added support for running Capella and EIP-4844 simulations by downloading the correct version of Geth. * Added support for using Nimbus remote signer and Web3Signer. Use 2 out of 3 threshold signing configuration in the mainnet configuration and regular remote signing in the minimal one. * The local testnet simulation can now use a payload builder. This is currently not activated in CI due to lack of automated procedures for installing third-party relays or builders. You are adviced to use mergemock for now, but for most realistic results, we can create a simple builder based on the nimbus-eth1 codebase that will be able to propose transactions from the regular network mempool. * Start the simulation from a merged state. This would allow us to start removing pre-merge functionality such as the gossip subsciption logic. The commit also removes the merge-forcing hack installed after the TTD removal. * Consolidate all the tools used in the local simulation into a single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
if strictVerification in flags:
use v1.3.0-rc.0 consensus spec test vectors (#4472)
2023-01-09 14:15:43 +00:00
# Rule 2/3/4 finalization results in the most pessimal case. The other
# three finalization rules finalize more quickly as long as the any of
# the finalization rules triggered.
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
if (epoch >= 2 and state.current_justified_checkpoint.epoch + 2 < epoch) or
(epoch >= 3 and state.finalized_checkpoint.epoch + 3 < epoch):
Local sim impovements (#4551) * Local sim impovements * Added support for running Capella and EIP-4844 simulations by downloading the correct version of Geth. * Added support for using Nimbus remote signer and Web3Signer. Use 2 out of 3 threshold signing configuration in the mainnet configuration and regular remote signing in the minimal one. * The local testnet simulation can now use a payload builder. This is currently not activated in CI due to lack of automated procedures for installing third-party relays or builders. You are adviced to use mergemock for now, but for most realistic results, we can create a simple builder based on the nimbus-eth1 codebase that will be able to propose transactions from the regular network mempool. * Start the simulation from a merged state. This would allow us to start removing pre-merge functionality such as the gossip subsciption logic. The commit also removes the merge-forcing hack installed after the TTD removal. * Consolidate all the tools used in the local simulation into a single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
fatal "The network did not finalize",
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
epoch, finalizedEpoch = state.finalized_checkpoint.epoch
Local sim impovements (#4551) * Local sim impovements * Added support for running Capella and EIP-4844 simulations by downloading the correct version of Geth. * Added support for using Nimbus remote signer and Web3Signer. Use 2 out of 3 threshold signing configuration in the mainnet configuration and regular remote signing in the minimal one. * The local testnet simulation can now use a payload builder. This is currently not activated in CI due to lack of automated procedures for installing third-party relays or builders. You are adviced to use mergemock for now, but for most realistic results, we can create a simple builder based on the nimbus-eth1 codebase that will be able to propose transactions from the regular network mempool. * Start the simulation from a merged state. This would allow us to start removing pre-merge functionality such as the gossip subsciption logic. The commit also removes the merge-forcing hack installed after the TTD removal. * Consolidate all the tools used in the local simulation into a single `ncli_testnet` binary.
2023-02-23 02:10:07 +00:00
quit 1
use v1.3.0-rc.0 consensus spec test vectors (#4472)
2023-01-09 14:15:43 +00:00
process_inactivity_updates(cfg, state, info)
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#rewards-and-penalties
use v1.3.0-rc.0 consensus spec test vectors (#4472)
2023-01-09 14:15:43 +00:00
process_rewards_and_penalties(cfg, state, info)
automated consensus spec URL updating to v1.4.0-beta.6 (#5804)
2024-01-20 11:19:47 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0-beta.6/specs/phase0/beacon-chain.md#registry-updates
add EF consensus spec test Electra block sanity fixtures (#6250) * add EF consensus spec test Electra block sanity fixtures * move process_consolidation/process_deposit_receipt before usage
2024-04-28 09:15:03 +00:00
? process_registry_updates(cfg, state, cache) # [Modified in Electra:EIP7251]
use v1.3.0-rc.0 consensus spec test vectors (#4472)
2023-01-09 14:15:43 +00:00
automated consensus spec URL updating to v1.4.0 (#6074)
2024-03-14 06:26:36 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.4.0/specs/altair/beacon-chain.md#slashings
use v1.3.0-rc.0 consensus spec test vectors (#4472)
2023-01-09 14:15:43 +00:00
process_slashings(state, info.balances.current_epoch)
process_eth1_data_reset(state)
rm debugRaiseAssert; clean up several debugComments (#6308) * rm debugRaiseAssert; clean up several debugComments * exception linting
2024-05-23 21:51:09 +00:00
? process_pending_balance_deposits(cfg, state, cache) # [New in Electra:EIP7251]
? process_pending_consolidations(cfg, state) # [New in Electra:EIP7251]
add EF consensus spec test Electra block sanity fixtures (#6250) * add EF consensus spec test Electra block sanity fixtures * move process_consolidation/process_deposit_receipt before usage
2024-04-28 09:15:03 +00:00
process_effective_balance_updates(state) # [Modified in Electra:EIP7251]
use v1.3.0-rc.0 consensus spec test vectors (#4472)
2023-01-09 14:15:43 +00:00
process_slashings_reset(state)
process_randao_mixes_reset(state)
cleanup `state_transition_epoch` and bump to v1.3.0 (#4922) * cleanup `state_transition_epoch` and bump to v1.3.0 More v1.3.0 consensus-specs bumps, focused on `state_transition_epoch`. Also fixed `current_epoch` spurious style check warning, and cleanup. * Update beacon_chain/spec/state_transition_epoch.nim
2023-05-10 02:31:23 +00:00
? process_historical_summaries_update(state) # [Modified in Capella]
use v1.3.0-rc.0 consensus spec test vectors (#4472)
2023-01-09 14:15:43 +00:00
process_participation_flag_updates(state)
process_sync_committee_updates(state)
ok()